Handmade Network»Forums
Aurita
4 posts
Anyone interested in an easy-to-use build system?

Hi all. Although I've just registered an account, I have been reading posts here for a long time, and appreciate the handmade philosophy to construct software.

I'm aware that some people like Casey don't use a build system, and just use a plain .bat script to build everything sequentially. Also some new programming languages like Jai, Odin and Zig tries to eliminate the need of a build system, and handle the building process with language tooling.

I however have a different opinion, I think build systems are still useful in the long run, and it's just that existing build systems are too difficult to use. To optimize development process of a software, multiple different tools are required in many cases. For example my entire game project is now using a total of 6 programming languages, and it's not really practical to replace one of them with another. And to optimize for the shipping product of the software, lengthy compile times would always be useful. Theoretically, it's always possible for the compiler to take more time to produce binaries a little better. Therefore I think a dedicated build system that handles a complex build process in a parallel fashion would always be useful.

Struggling to find a good build system for my own game project, I made new one that is sufficiently easy to use. Slightly inaccurately, with my new build system, user build scripts are written in Lua, and the user only needs to specify what shell commands to execute, and for each shell command, which shell commands it depends on. And then, everything is taken care of by the build system automatically. This is in contrary to the traditional build system designs, where shell commands depend on input files, but not directly depend on other shell commands.

The build system is also powerful enough so that it can efficiently subsume other build systems, like make, ninja, CMake, or any of the new programming languages with a builtin build tooling. Nothing special is required to make this happen, but it's best to attach a stateful storage to the shell commands that calls out to those build systems, so that they can reuse build objects from previous build runs. Stateful storage is a notion introduced in my build system, they are explicitly managed, and in practice are the only places that things could go wrong. When that happens, the user can request to clean the storage, and the build system takes care of the rest. Otherwise cleaning is only necessary to save disk space.

To achieve the mentioned functionality, my build system uses FUSE under Linux to automatically detect file operations, and to present a virtual build-time intermediate filesystem to the shell commands. It also uses inotify to monitor file changes, and to support the feature of continuous rebuilds. It may sound tricky to implement correctly and robustly, but I'm an experienced developer, and this stuff is not difficult for me. I've been using the build system for my game project for several months, and it has been a pleasure to use.

I intend to open source this build system soon, after things like documentation is ready. In the meanwhile, I'd like hear your thoughts, both on build systems in general, and this specific new build system. There are more features of it I haven't mentioned, but it's already a long post. Thank you for reading!

Simon Anciaux
1259 posts
Anyone interested in an easy-to-use build system?
Edited by Simon Anciaux on

Welcome.

In my opinion it looks like there are two different use cases for build systems:

  • You're actively developing a project;
  • You're a "user" of the project and just want to compile it a few time and maybe do some modifications.

In the first case I think any build system can be used, it's up to the developer to choose what they are OK to deal with. But if you want to share the project with other, I think it would be nice to generate simple batch/bash file.

If it's a small project I prefer to just have a batch or bash file (or just the command line in a comment in the main source file) because I don't like to have to learn a new "language/build description thing" or install software just to compile the project.

For bigger project I'm OK to install a build system, but if something goes wrong I sometimes don't have a clue of what's going wrong and how to fix it. It also often not clear if the build system will handle dependency, verify version of dependency and tools and so I end up lost and not compiling the thing. So in the end I still prefer a batch/bash file even if the compile process is slower or doesn't reuse previous object files, isn't parallel... It also might be that it's not the build system that is the issue but the project "architecture".

I also think that it might be a good idea to consider that everybody works in a different way and so I believe that most people have some "high level" build system to handle their projects. I use a build system I have written in C that is mostly what I was doing with batch files, but in C because it's a lot easier for me to work with C than batch/bash languages. But I can pass a parameter to output the generated command so when I need to share some code, it's easy to create a batch/bash file. It's also tailored to how I like to organize my projects, which wouldn't transfer to anybody else. And I don't work on big projects so I don't have a complete view of the "problem".

The build system is also powerful enough so that it can efficiently subsume other build systems, like make, ninja, CMake, or any of the new programming languages with a builtin build tooling.

I think that in Jai (and maybe other languages) one of the point of having the build system part of the language is that during compilation you can handle messages and generate/modify code on the fly. That has probably more to do with the compiler implementation than the language, but I'm assuming that your build system can't do that king of things (I never used Jai so this might not be accurate).

I would be interested to look at your system, if only to see how it works.

65 posts
Anyone interested in an easy-to-use build system?
Edited by Shastic on
Replying to mrmixer (#26859)

"I use a build system I have written in C that is mostly what I was doing with batch files, but in C because it's a lot easier for me to work with C than batch/bash languages. "

I've come around to the idea of using C as a build system, instead of bash/bat files. I haven't tried it yet on my project, so I'll have to see how it goes. As long as the C project is a unity build, a C build system should be simple.

If you use C as a build system, that's two less languages you have to learn to get a C project to build. If you use any other build system, that is yet another language you have to learn before you can build C.

I also think build systems where they stuff every operating system combo into one file are terrible. It might look tidy in the directory having one build file, but searching through a rats nest of if elses, to find the places to make the mods to make it work on an OS the authors didn't know about is annoying, when all you want to do is run it to try it out.

The simplest one I've used, was where they had a clearly marked build file for each OS. That way I copied a file that was close to what I needed, renamed it, and changed a couple of lines. There were only a few lines in each file. This C code was last maintained more than 20 years ago, and apart from a few memory errors, which the compiler picked up, it built and ran fine. I found that amazing.

Aurita
4 posts
Anyone interested in an easy-to-use build system?
Edited by Aurita on
Replying to mrmixer (#26859)

Welcome.

Thank you, and thank you for your input!

In my opinion it looks like there are two different use cases for build systems:

  • You're actively developing a project;
  • You're a "user" of the project and just want to compile it a few time and maybe do some modifications.

Yeah, you are right. I was focusing on the first use case.

For the second use case, I think it would be possible for my build system to export a simple shell script, so that a software project maintainer can include it when shipping, and the users of the project can build it by simply running the script without installing my build system.

Exporting a Makefile or Ninja file is also possible, with the added benefit of parallel building. Incremental rebuild is also possible with the exported build file, but it won't be reliable. Although not perfect, I think this exporting feature is already sufficient for many use cases.

I think that in Jai (and maybe other languages) one of the point of having the build system part of the language is that during compilation you can handle messages and generate/modify code on the fly. That has probably more to do with the compiler implementation than the language, but I'm assuming that your build system can't do that king of things (I never used Jai so this might not be accurate).

I need to clarify that by "subsume", I meant that my build system can incorporate other build systems as a part of larger build process, and this integration would work well. For example, a developer won't need to manually invoke each build system that third-party software uses, and an edit to the source of third-party software would be picked up by my build system, and the whole project would be correctly incrementally rebuilt.

Having features like this, I think my build system is well-suited for "high level" build systems. And I think it would work well with arbitrary ways to organize a project.

In my opinion, there are two layers of build systems that we see today:

  • One layer is like Makefile and Ninja. They are general purpose, and don't impose any restrictions on how a project is organized. But their build files use DSLs with weak expressiveness. They are also arguably half-broken in the sense that incremental rebuild is not guaranteed to be correct, requiring the user to carefully specify all dependent input files.

  • Another layer is like CMake and Premake. They work on top of the half-broken first layer, and provide a more convenient way to build a typical project. But they would impose opinions on how a project should be organized.

The core of my build system is on the first layer except it's not broken, but can also be extended to the second layer with Lua libraries. The core primitives are very simple. I want to leave a proper introduction to the documentation, but here's a simple example to show how it could work:

local a = task("gcc -c -o a.o a.c")
local b = task("gcc -c -o b.o b.c")
return task("gcc -o final a.o b.o", {deps={a,b}})

And it's Lua script, you can do arbitrary things in it. I suppose it won't be too different from how you would do in C, but I wonder what you think?

I would be interested to look at your system, if only to see how it works.

Thank you! I will keep the thread updated when it's released :)

Aurita
4 posts
Anyone interested in an easy-to-use build system?
Edited by Aurita on
Replying to Shastic (#26864)

Thank you for your reply as well!

As long as the C project is a unity build, a C build system should be simple.

I would agree with you. A unity build only needs a single shell command to invoke the compiler, and a dedicated build system won't help much here.

The simplest one I've used, was where they had a clearly marked build file for each OS.

I don't know whether these marked build files include in turn some common build files. If that is the case, I think it's a common pattern to use Makefile. Otherwise, things could get replicated among those files, and the maintainer needs to update each of them when changing the build process, and that would be a tradeoff.

Anyway, my build system would allow both ways. :)

This C code was last maintained more than 20 years ago, and apart from a few memory errors, which the compiler picked up, it built and ran fine. I found that amazing.

I appreciate this as well. I am using a pinned NixOS environment in conjunction to my build system, and I maintain local copies or forks of third-party code. In this way, my build would literally never break, unless I upgrade something explicitly.

Simon Anciaux
1259 posts
Anyone interested in an easy-to-use build system?
Replying to aurita (#26865)

And it's Lua script, you can do arbitrary things in it. I suppose it won't be too different from how you would do in C, but I wonder what you think?

The example is so simple that a build system doesn't make much sens (I understand it was just an illustration). I don't often work on medium/big projects that require build system, so I don't have much feedback to give. Maybe you could try to convert some open source project's build system to use your system to see if it actually works and help ?

Aurita
4 posts
Anyone interested in an easy-to-use build system?
Edited by Aurita on
Replying to mrmixer (#26867)

The example is so simple that a build system doesn't make much sens (I understand it was just an illustration).

Despite being very simple, the code would actually work for my build system, and this is what I meant by "easy-to-use", and I probably should have shown the example earlier. It would also be possible to do something like this:

local a = task("cmake -G ninja ..", {mounts={["."] = {"storage", "cmake-state"}}})
return task("ninja", {deps={a}, mounts={["."] = {"storage", "cmake-state"}}})

This is using the stateful storage feature I mentioned earlier. I have other features to make the build scripts feel more convenient and more conventional, but those are auxiliary, while the most important core primitive is just the task Lua function to build up an arbitrarily complex dependency graph.

As I've said, the core of my build system works on the first layer, like Makefile or Ninja, and their core mechanism is similarly simple. Most of the complexity of Makefile comes from its DSL to provide more expressiveness, but with my build system the user can simply use Lua.

For a real world project, the build script could look like a longer version of my example, or it could also be that those task function calls are buried inside abstractions built with Lua code, or it could be anything in between. It's up to the project maintainer how much abstraction is used.

Maybe you could try to convert some open source project's build system to use your system to see if it actually works and help ?

Thank you anyway. Instead of converting a particular project, I'm more interested in supporting existing build systems' build files, like parsing Ninja build files or supporting Premake's Lua API. This would enable a large number of open source projects to be built with my build system. It's very likely this can be done, but I need to start working on it to be sure.