Hey, everyone!

It's almost time for the Wheel Reinvention Jam! It's starting on the 27th, which is this coming Monday. Be sure to explore or post project ideas, register for the jam, and get all of your coffee ready. I'm so excited to see what you all cook up!

During the jam, we encourage sharing progress (and result) either on the Discord (we'll have a special channel set up for jam showcase content) or on the forums. If you end up participating on Discord, be sure to connect your Handmade Network account to your Discord account!

To get things kicked off before the jam, the Handmade Network staff (myself, Ben, and Asaf) will be joining the Zig Showtime show tomorrow. We'll be chatting with Loris Cro, who runs the show, about reinventing wheels, the jam, and all things Handmade. Be sure to tune in.

Shortly after, we'll be having a coffee chat in the Handmade Network Discord, as usual for Saturdays.

That's all for now. Looking forward to seeing you all during the jam!

-Ryan

I recently removed untyped literals, and that complexity is now leaving me free to actually add code where there's a tangible benefit to it.

It might not be immediately obvious that having untyped literals adds complexity to a language. For C3, with its C-like implicit conversions and untyped macros, it adds more complexity than in languages like Go, where explicit conversions are enforced.

The sheer amount of complexity I had added to just do this surprised even me though. For example, every expression needed to pass down a "target type" just in case the analysis encountered an untyped literal to give a type to. In some cases this was far from trivial, and analysis had to be done in a certain order to prevent unnecessary error situations, where the incorrect order would only be obvious when the expressions where combined in some special way.

As I removed the code I also discovered I could change how the "failables" (error unions) were handled, which further reduced complexity. This in turn allowed me to do simplifications in tracking constant and "pure" expressions.

Lots of code that I had left half unfinished – with special, problematic, cases to solve another day – would after refactoring easily cover all cases and be smaller.

The code now seems so much easier to grasp, and it seems crazy I didn't removed this feature - that at the most saved a little typing here and there - earlier. But the problem was that it was a gentle creep. I did the untyped literals early, so when I added more complex features I didn't have a comparison with how it would look with untyped literals removed.

When I look at the code now, I see something that more easily can accommodate added features and behaviours. The semantic analysis had before by necessity been much more coupled due to type flow going both bottom up and top down.

There's a lesson here – which is that a seemingly simple and "nice to have" feature might by accident end up making a code base more than twice as complex to read and reason about. And that complexity cost takes away the resources the compiler (and the language) has for other features – features that may actually have a cost that is equal to its benefits.

In removing this small feature with little practical benefit, I am now free to actually add a little complexity where it really helps the users.

This post is mirrored on my website.

It is common advice that it isn't worth automating something unless the time saved doing the task is greater than the time required to automate it.

Randall Munroe has two relevant xkcd comics (license):

This comic is the most relevant to this article:

When I talk about automation, I include things like code generation and macros, which automate code writing.

I am going to argue that doing work to automate things may still be worth your time, even if it takes longer than it would have without the automation.

Ethics of unergonomic interfaces

Recently, I have been thinking about user interface ergonomics, or how well an interface, be it textual code, physical hardware, or application UI, is designed around humans.

The goal of automation is typically to increase the ergonomic quality of some interface. Rather than doing a mindless and repetitive task (unergonomic), it is done automatically so one can spend their time doing more valuable work (ergonomic).

Jonathan Blow originally made me see interfaces in this way in an On Doubt interview:

If by spending 10 hours of your time, you can save a typical person using software 1 minute of bad experience, then at some level of usage that number just dwarfs your hours of time, and it becomes an ethical failing to not do that work.

Level editing

I heard an especially good example of this. Some game designers I know spent entire weeks hand-writing 2,000-line XML files. These files determined the basic setup of a level in the game.

They would open the game in a specific world, then hand-copy hundreds of floating point coordinates provided by their mouse pointer. These coordinates would be input (by hand!) to this file to precisely place items.

The file would look something like this, only there would be thousands of unique entries:

<level>
  <building x=1234.23 y=928.2233 z=977.22>Hut</building>
  <building x=928.23 y=110.2233 z=5638.22>Shack</building>
</level>

If an artist wanted a building rotated, they would send the designer a message to have them tweak the value. These are creative people with Master's degrees, typing numbers into a file for days on end. These files could have been generated by the computer---there was no creativity involved in their copying to these files.

This is an example of an unethical interface. One week of programmer time would have easily saved multiple weeks of designer time. Not only that, but the programmer would likely get some amount of enjoyment and satisfaction from creating a good tool.

When developing things which have a human interacting with it, you should consider how much you value your life time, and have empathy for your users' life time.

Effects of automation on creations

Automation not only changes how you do something, but what you will be willing to do.

In the level editing example, designers would be much more willing to create levels if they didn't have to suffer every time they did it. They would spend less time copying numbers to index cards and more time being creative and iterating.

I have spent a large amount of time on a new programming language, Cakelisp. I could have implemented all the software I have written with Cakelisp with existing languages. Thanks to Turing completeness, it follows that for any program I can write in Cakelisp, I could have saved all that up-front time making Cakelisp by writing the program in C instead.

This is too naive. I work on both C and C++ programs professionally. I am very familiar with how to get real things done with those languages. I am also intimately familiar with their limitations. When I worked on projects at home, I became more frustrated with those limitations.

Cakelisp was a revelation because it was my chance to escape from these limitations. I had to pay my time in order to make something I consider better, but it was absolutely worth it. I was now in charge of the interface. I shaped the language, the language didn't shape me.

This resulted in much higher motivation to do my own projects. If my goal is to eventually become self-sufficient by selling my software, doing more projects in a more enjoyable way is valuable.

Macro magic

Modern computer processors are very parallel. A good processor nowadays has sixteen or more cores. However, writing multi-threaded software to exploit these processors remains difficult as ever.

In my File Helper project, I needed to keep the user interface responsive while performing a scan of the entire filesystem. My first implementation used SDL's thread API to create a thread dedicated to file system scanning. I used a heavy-handed mutex to ensure the main thread and the scanning thread never stomped on each other's shared memory.

This system was complicated, fragile, and difficult to change.

Task systems assist the programmer by abstracting work that needs to be done into tasks or jobs. In this example, scanning the filesystem would be considered a task.

The problem is, C doesn't have a great interface to these systems.

Here's an example of the C I would have to write to create and run a task in EnkiTS:

taskScheduler= enkiNewTaskScheduler();
enkiInitTaskScheduler(taskScheduler);

// myLongTask is a function which must match a special signature
enkiTaskSet* myTask = enkiCreateTaskSet(taskScheduler, myLongTask);

enkiParamsTaskSet taskParams = enkiGetParamsTaskSet(myTask);
// How many things need processing
taskParams.setSize= 100;
// How many things should be processed in each task bucket
taskParams.minRange= 10;
enkiSetParamsTaskSet(myTask, taskParams);

// Set task on-complete
enkiCompletionAction* completionAction =
    enkiCreateCompletionAction(taskScheduler,
                               onMyLongTaskComplete);
enkiParamsCompletionAction completionArgs =
    enkiGetParamsCompletionAction(completionAction);
completionArgs.pDependency= enkiGetCompletableFromTaskSet(myTask);
enkiSetParamsCompletionAction(completionAction, completionArgs);

// Start the tasks
enkiAddTaskSet(taskScheduler, myTask);
enkiWaitforAllAndShutdown(taskScheduler);

enkiDeleteTaskSet(taskScheduler, myTask);
enkiDeleteCompletionAction(taskScheduler, completionAction);
enkiDeleteTaskScheduler(taskScheduler);

As you can see, this is a lot worse than a simple function call:

myLongTask();

This friction in the interface changes how often I am willing to use the task system. Additionally, it imposes a maintenance burden simply by virtue of being much more lines of code, which encourages bad habits like copy-pasting.

Cakelisp provides me the opportunity to eliminate this friction via full-power macros, which allow me to do arbitrary compile-time code generation.

This means I get to define the interface. The task system becomes a domain-specific language where I only type what I must:

;; I define a task just like a function, with regular arguments
;; These arguments are automatically structured and de-structured
(def-task treemap-update-state-task (state (* treemap-state))
 (do-the-long-scanning state))

;; Start the task, passing the arguments from the current context (state)
(task-system-execute
 (treemap-update-state-task state)
 ;; This will run after the previous task, on any thread
 (classify-treemap-paths state)
 (treemap-update-state-done :pin-to-main-thread state))

This task-system-execute macro took about one week to implement (it's here). That is a large time investment, but it will continue to pay dividends the more I use it. In fact, I have already used it three times in File Helper, with great success.

This macro not only eliminated the fragile mess that was the hand-rolled file system thread code, it made me move even more things off the main thread.

Another example, this time with tasks that run in parallel:

(task-system-execute
 (export-category-paths copied-entries copied-categories)
 (parallel ;; These tasks can run at the same time
  (open-userdata-system-file-explorer
   open-file-explorer-on-complete)
  (on-export-complete :pin-to-main-thread
                      copied-entries copied-categories)))

There are some drawbacks, of course. This macro adds a small amount of time to the compile phase, and produces code that is likely slower than hand-rolled threading code. However, the amount of time recovered by being more willing to make things threaded should easily make up for its drawbacks.

This tool changes how willing I am to write things which require multi-threading. It changes what kinds of things I even consider making.

Conclusion

Automation not only saves you time, it changes what you do. The ergonomics and interface friction influence how you approach tasks. More ergonomic interfaces give you more time to do more interesting work.