This article is mirrored on my blog.

I recently released File Helper, a file organization application I wrote using Cakelisp.

This application had only two external files that were necessary for it to fully function:

• A font
• An application icon

I packaged File Helper in a .zip or .tar.gz for Windows or Linux respectively. These archives contain the platform executable as well as a license file and the two necessary font and icon files.

However, wouldn't it be nice if instead I shipped a single executable, thereby eliminating the extract step?

It might sound trivial, but eliminating that extra step has many benefits:

• Less technical users won't get confused. Double-clicking an archive usually opens it in a browser rather than extracting it, which might confuse them and cause them to not use my product.
• The application has no risk of breaking if the executable is moved.
• The user doesn't have to delete or move the archive after they extract it.

Bundling files into executables

An executable is just a file format which your operating system understands. It is essentially a header and a whole bunch of sections filled with binary data.

Typically, a linker converts a collection of object files into a single executable. Because executables are containers which can hold various kinds of data, we can package data only our application understands in the same container as the application code.

The operating system is fine with this because it only needs to map the executable into memory and start executing code at a designated entry point. It is then up to the program to decide how to interpret the various executable sections.

Platform differences

There are many different file formats for executables. Usually, an operating system only supports one executable file format. On Windows, it's the Win32 Portable Executable format, typically with extension .exe. On Linux, it's usually ELF.

I am only targeting those two platforms, so I can add code to specifically support those formats when building Cakelisp programs.

On Windows, data is added to executables via Resource Files. I wrote a tutorial on how to do this.

On Linux, data can be added via dumping the data to an object file which defines a couple symbols. This is a great tutorial on how to do that.

Good and bad ways

Like everything in programming, you'll hear different advice on how to bundle data.

The most common alternative method is to convert your data to a C-style array definition. This has many limitations, and in my opinion should be avoided:

• Some compilers (MSVC included) limit the number of elements in an array, which therefore limits the size of the bundled data.
• Your compiler has to do extra processing (tokenization, parsing, etc.) to that data which it should actually just treat as a giant binary blob. Extra unnecessary processing means longer build times.
• An extra stage has to be created and compiled as part of your build system, which adds complexity.

We are going to proceed with the platform dependent but much more robust approach, which is to convert our data to object files without using a C/C++ compiler.

Integrated build system

Whether we are on Windows or Linux, we need to process our data file into some other form in order for the linker to properly understand the data package. This means adding a step to our build to process the data, because we want it to automatically stay up-to-date when linked in the executable.

Cakelisp includes a simple C/C++ build system as well as compile-time code execution. We need to create a new build step to process our binary data into object files. In order to do that, we use a compile-time build hook to execute a function which performs the conversion.

The full code is here.

The end-user interface is simply:

(import "DataBundle.cake")
(bundle-file data-start data-end (const char)
             "../data/MyFont.ttf")

We declare data-start and data-end to represent pointers to the symbols associated with our data.

That bundle-file invocation is a macro that adds the data file to a list. It also generates the variables we can use to refer to the data.

Finally, a compile-time function convert-all-bundle-files calls the necessary objcopy (or Resource Compiler on Windows[^1]) to generate the actual object file for each bundle-file. It only does this if the data files are changed or the object files don't already exist in the cache.

We can then link the generated objects into the executable alongside our code object files. It also adds that object file to the linker command line.

This function is integrated into the Cakelisp build sequence like so:

(add-compile-time-hook-module pre-build convert-all-bundle-files)

Conclusion

This is pretty great: we extended our build system to support bundling arbitrary data files, all without touching Cakelisp's internals itself.

Not only that, we extended the system in the same language we write our application code, and within the same invocation---we didn't need to create some other phase. We were also able to provide the user with an extremely simple interface to bundling files.

[^1]: On Windows, we need to generate a .rc file with a list of all the resources that should be compiled into a single object file. Because Cakelisp allows arbitrary compile-time code execution, we can easily do this by writing the filenames out to the generated rc, then invoking the Resource Compiler on that file. This platform-specific step can be completely automated!

Hi,

We have been working on many different variants of AOgmaNeo that support Unsupervised Behavioral Learning (UBL).

For those that don't know, UBL is a sort of alternative to classic reinforcement learning (RL). It's a bit different of a paradigm, but instead of optimizing a reward function, it learns the dynamics of the environment and provides a kind if programmable interface to it. The main motivation behind UBL is that we want an agent that is easier to use with real-world robotics which may require a lot of hand-crafting. It also is able to handle instantaneously changing objectives, which regular RL cannot really (even with goal conditioning).

Currently, the best performing UBL branches are able to reproduce some of the results from the original RL version, but not yet all. There is still work to do!

As a result, it will still take a bit before UBL has a chance at making it into the master branch. If you wish to try it anyways, the latest experimental branches are "ubl" and "ubl_cart". There are several others, but those two are the most interesting at this time. Both of those two branches use recurrence instead of exponential memory to handle short-term memory, as we found that having a fast-moving top layer helps performance a lot.

We also started thinking of UBL as a machine that executes programs rather than chasing a goal state. This new interpretation required some changes internally, but it seems to be working much better as a result. The idea is that UBL receives a sequence of Columnar Sparse Distributed Representations (CSDRs) that act as a program, with the UBL hierarchy being a sort of interface to the environment the agent is operating in. The hierarchy executes the instructions in that CSDR program while abstracting away many of the details of the environment that is learned from.

Finally, we have been playing around with Odin some more, working on some tools that use it first - before hopefully re-writing all of AOgmaNeo in Odin!