Hi all. Although I've just registered an account, I have been reading posts here for a long time, and appreciate the handmade philosophy to construct software.
I'm aware that some people like Casey don't use a build system, and just use a plain .bat script to build everything sequentially. Also some new programming languages like Jai, Odin and Zig tries to eliminate the need of a build system, and handle the building process with language tooling.
I however have a different opinion, I think build systems are still useful in the long run, and it's just that existing build systems are too difficult to use. To optimize development process of a software, multiple different tools are required in many cases. For example my entire game project is now using a total of 6 programming languages, and it's not really practical to replace one of them with another. And to optimize for the shipping product of the software, lengthy compile times would always be useful. Theoretically, it's always possible for the compiler to take more time to produce binaries a little better. Therefore I think a dedicated build system that handles a complex build process in a parallel fashion would always be useful.
Struggling to find a good build system for my own game project, I made new one that is sufficiently easy to use. Slightly inaccurately, with my new build system, user build scripts are written in Lua, and the user only needs to specify what shell commands to execute, and for each shell command, which shell commands it depends on. And then, everything is taken care of by the build system automatically. This is in contrary to the traditional build system designs, where shell commands depend on input files, but not directly depend on other shell commands.
The build system is also powerful enough so that it can efficiently subsume other build systems, like make, ninja, CMake, or any of the new programming languages with a builtin build tooling. Nothing special is required to make this happen, but it's best to attach a stateful storage to the shell commands that calls out to those build systems, so that they can reuse build objects from previous build runs. Stateful storage is a notion introduced in my build system, they are explicitly managed, and in practice are the only places that things could go wrong. When that happens, the user can request to clean the storage, and the build system takes care of the rest. Otherwise cleaning is only necessary to save disk space.
To achieve the mentioned functionality, my build system uses FUSE under Linux to automatically detect file operations, and to present a virtual build-time intermediate filesystem to the shell commands. It also uses inotify to monitor file changes, and to support the feature of continuous rebuilds. It may sound tricky to implement correctly and robustly, but I'm an experienced developer, and this stuff is not difficult for me. I've been using the build system for my game project for several months, and it has been a pleasure to use.
I intend to open source this build system soon, after things like documentation is ready. In the meanwhile, I'd like hear your thoughts, both on build systems in general, and this specific new build system. There are more features of it I haven't mentioned, but it's already a long post. Thank you for reading!