I spent the last two days building a plugin-driven tool for static code analysis.

• I evaluated whether building something like this is even still a good idea. (Spoiler: it is) Maybe I should turn that into a blog post?
• I used a three-part plugin architecture that I've been thinking about for quite some time. Turns out go vet is built the same way, that might be where I got the idea from. It held up to use nicely enough and now I've got a clear example to point to.
• I proved to myself that this can be built within a weekend.

https://github.com/unnamedtiger/check

As promised: editing. Now the algorithm correctly classifies the situation when you rename a node somewhere in the tree. &diffest2

I haven't shown you how moved content looks like when it's more than a leaf node in the last post. That's because there is a technically-cheaper solution that instead inserts a new parent node and removes the old parent node to avoid duplication (testcase 410). While a correct solution, it is not intuitive and the way I'm displaying it does it no favors.
However I just realized that if I put enough stuff between the old and new place the subtree attaches to, the cheapest solution will end up being a move again. And so I decided to show you all how that looks (testcase 411). Borders following the text lines aren't pretty either, sadly.
Anyway, back to what I was doing before I came up with that which was adding functionality that allows you to edit a node in the tree. &diffest2

I've improved on the testcases from two posts ago by highlighting both places the moved node shows up as one. You might remember that I tracked moved content the same way in last year's exploration. &diffest2

You might have noticed that I conspicuously left out a few test cases in the screenshots I've shared up to now. That was because the results didn't look right. The core part of my comparison algorithm works itself in a depth-first way through the nodes and is blind to the rest of the tree structure. When the last child-node of a given parent changes depth that becomes a problem. I've spent yesterday afternoon adding a bunch of code for dealing with these cases. At the point in time where I'm turning the result back into a tree I add additional nodes that represent the new or old parents. This doesn't feel like a good solution though, since now we have nodes duplicated in the merged tree. There is a lot more visualization work that needs to be done for this project than I realized. &diffest2

I enhanced my cost function by adding penalties for undesirable solutions. This is to ensure that all these test cases look as obvious as possible. &diffest2

My previous screenshot could have been generated by simply passing the two inputs to a naive text-based diff tool. They weren't, but they could have. That's why I said "nothing exciting".
This new set of tests now works with actual trees. I'm adding or removing both leaf nodes as well as nodes inside the tree (branch nodes). As you can see, the appropriate areas are marked as insertions or deletions respectively. &diffest2

Dear guests,

it is my honor to welcome you all to this year's Visibility Jam, right here in #project-showcase or wherever you're reading this around the web.

My fellow jammers and I will be taking this long weekend to bring light into darkness and explore the invisible. You'll excuse us if our posts are less polished than usual, we take longer to engage in discussions, or resort to only reacting with emoji. We'll have all the time in the world to talk about our projects, after we've caught up on sleep.
-# this is a joke. please sleep enough.

To my fellow jammers: I wish you all success with your projects, few bugs, and of course good luck and have fun!

I'll be spending this weekend looking at trees. The digital kind. Your filesystem is (mostly) a tree, but the content of individual files is also often structured as a tree. Specifically I want to explore how to compare two versions of the same tree and highlight differences. This morning I've built a test setup and started producing results for some of the simplest trees there are. In the screenshot I've got one test per row, and in each row the two trees that are input into the algorithm, and on the right the resulting output. So far, nothing exciting. Next up, I want to look at trees with more than one level. &diffest2

Imagine this: the Big Boss has some important changes to make and pushes them to Gitea. Gitea notifies Aura about the push and Aura queues jobs to be handled as soon as possible. These jobs then run, quickly succeed, and the result shows back up in Gitea. All this is now working in Aura. (Big Boss' satisfaction is not guaranteed.) &aura

With the release of Darke Files 0.5.0, I included a webhook feature that lets you configure Darke Files server to post some info to a specified URL whenever a new commit is pushed. I wanted to integrate Darke Files into a CI system but I didn't find any that fit my needs, so the webhook was the only feature you got. I went into more detail back when I released that version here https://discord.com/channels/239737791225790464/1053016918023487549/1053018234628087828

Times have changed and now I have a build server that can do the things I want it to do. Aura now contains an endpoint specific to the webhook payloads Darke Files server sends out. You just have to map between the two systems in Aura's config and provide the missing pieces of information. You can read more about the integration in the expanded Aura README and the submit-darke docs.

The screenshot montage contains (from top to bottom):

• the repository on Darke Files server
• the project in Aura
• the config for the integration in Aura

Yes, this is a crossover showcase-post.
https://darke.handmade.network/ | https://darkedev.itch.io/darke-files | &darke
https://handmade.network/p/431/aura/ | https://github.com/unnamedtiger/aura | &aura

Just successfully leaked a secret in an Aura build job. Y'all just have to be careful. &aura

Aura now lets you create a new project from the web interface. I think that vital feature nicely rounds out what I've got time for this jam. Of course there are things I didn't get to, most notably authentication, artifacts, and integration into several major and minor version control servers, but I'll save these for later.

With that Aura 0.1.0 is released! Try it out and let me know what you think.
https://handmade.network/p/431/aura/ | https://github.com/unnamedtiger/aura | &aura

By popular demand (it's me, i'm popular demand) timestamps are now relative. They also update, just in case you're looking at a job that just completed. And if you don't want to do the math, the entire thingy has a tooltip with the full date and time. Also visible here: the job log containing whatever the job wrote to stdout/stderr. &aura

Holy ship, it's alive. I've actually got a runner that just sits there and regularly checks in with the controller, asking for work to do. A tiny, productive robot. &aura

If things don't seem to be moving fast enough, you can check the job page in Aura to see exactly what a job is still waiting for. In the first case, the "test:macos" job is waiting for the "build:macos" job to complete, but the latter also hasn't started yet. In the second case the "deploy" job was created with a built-in delay. &aura

The new Runner Status page in Aura lets you check if all the robots who should be running build jobs for you are productive. It also lists the tags that are used for picking the correct build runner for a given job. &aura

Aura now has collections, so if you want to group a bunch of commits under for example a merge request, you can. &aura

Put all the job data into the database in Aura. With that the page listing all jobs for a commit and the page for a single job are also done. Then I added a page listing all queued jobs. I'll get the timestamps fixed before the week is up, hopefully. &aura

A bunch of things are clickable in the Aura web interface now. All of the data comes directly from the SQLite database. If you just want to poke around the program, you can run the executable in demo mode (as seen here) which will already contain some sample data. &aura

The main page of Aura features a list of all projects on the server. It won't stay quite as empty, there are still a few more links coming, once I've created the pages behind them and figured out what goes where. I don't expect this to be used much, most people probably open a page with a deeplink posted where ever the job was started from, e.g. a bot commenting in your MR.
Tech wise the entire thing now sits on top of an SQLite database which is the source for the list of projects you can see here. &aura

If we zoom out to look at an entire project, there might be different places your jobs could live. Attach your jobs to commits, days, or anything else that makes sense to you. Then group those entities in collections, for example representing branches, merge requests, or more. &aura

I've fought with CSS, and while I'm not sure if I won, I'm coming back with loot. If you've got a bunch of build jobs running for your project you might want to inspect one of these more closely. The job page has the info you need, with metadata in the left column, and more importantly the job log to the right. &aura

This Wheel Reinvention Jam I'm building a build server I've named Aura. The goal I've set myself for Monday is to create the major pages in the Web UI. I've started with the most complex one and you can see the first revision below. This page lists out all build jobs that exist for a single commit. &aura

I've added support for diffing multiple files at once to Diffest. While this example is not of a huge code base, the code-aware diff correctly tracks code that was moved between files. &diffest

I'm now grouping and highlighting related changes with surrounding context if possible. &diffest

You might be wondering why I picked Python for the first few test cases. The answer is this optional comma featured in test cases 04 and 05. &diffest

If someone is trying to sneak a change into the middle of a big reformatting, a code-aware diff can tell you exactly what's going on.
(In this jam, I'm exploring what a code-aware diff tool could do for you. Past updates and more info at https://handmade.network/p/366/diffest/, &diffest)

And with that the fourth box, "Code-Aware Diff", is filled. What's that, it doesn't look very spectacular? Well, there haven't been any spectacular changes here, have there? &diffest

I haven't quite filled that fourth box yet but I've made a lot of progress. I'm parsing both files into a syntax tree which I'll then use to calculate a diff between those files. This syntax tree is a combined abstract and concrete syntax tree: The black nodes form the AST and the gray nodes come from the CST and fill in the gaps between the nodes of the AST. The content in quotes comes directly from the source code while anything else is tree-types and structure. The grey nodes are actually two different types: necessary punctuation and optional whitespace; they are not differentiated in this visualization. Because of its structure and because it's neither an AST nor a CST I'm calling it an "expandable syntax tree" or EST (which then brings us to "diffEST" 😛 ). Up next: Fill that fourth box. &diffest

I spent the morning finalizing ThePlan™ and creating the first dozen test cases. Then I built a way to visualize the diffs I'm about to produce and hooked up a regular text diff (as available in Darke) for comparison. Up next: Fill that fourth box.
(In this jam, I'm exploring what a code-aware diff tool could do for you. Past updates and more info at https://handmade.network/p/366/diffest/, &diffest)

I'm in.
Despite our best efforts, the vast majority of source code is still in plain text. Lots of tools operating on source code only work with that plain text. Advanced tools however, like the Language Server in your favorite IDE, use this plain text as a starting point to build up rich, complex, and ultimately incredibly useful data structures in which your code lives. These tools understand your code, in many cases better than you do yourself. (And way-too-often they need to be restarted because they manage to run themselves into some infinite loop or whatever but I digress.)
A tool that is at least slightly aware of the code you have written could compare two states of the same code base and:
1️⃣ Look past changes in formatting to focus on what's actually relevant.
2️⃣ Group related changes so renaming a constant is one change instead of several dozen.
3️⃣ Examine the entire code base instead of handling every file individually.
4️⃣ Track where code came from and where it was moved to.
5️⃣ Present the entirety of the changes in a sensible and ordered way.
6️⃣ And finally, figure out what changes weren't even made.
And that's what I'll be exploring during the Visibility Jam.
diff, differ, &diffest

(originally posted on 2023-03-04 in #jam)

I've released version 0.5.0 of Darke Files, my version control system.
It features

• completely rewritten Git import feature, new Git export feature, and the option to do delta / incremental imports and exports
• a Visual Studio Code Extension for a quick overview on the state of the files in your repository
• a webhook feature so you can trigger other systems after a commit
  Hop in the thread for a lot more info.

https://darke.handmade.network | &darke

I've worked my way through TypeScript and especially the Visual Studio Code Extension API the past week to build such an extension for Darke Files.
It populates the Source Control menu with the status of the files in your repository and clicking on them brings up the diff view you're probably familiar with. Additionally the extension colors filenames in the editor tabs according to the file's status and adds colored gutters in a text editor to show you which lines have changed.
Replacing the CLI is explicitly a non-goal for the extension, so you'll still be using the CLI to commit changes but the extension will help you stay on top of the status of your repository and let you review changes easily before committing.
All this - and more - in the next release. &darke

Finished up work on improving Git interoperability in Darke Files today. I've rewritten the Git importer from the ground up which fixed a few bugs and uncool edge cases that were still hiding in the code. I've added a command to export a Darke Files repository to Git so you're not stuck with this version control system and can mirror your code to public software forges. And additionally both commands now support delta operations, ie. the option to only im-/export the data that changed since the last im-/export. All this - and more - in the next release. &darke

I've just released version 0.4.0 of Darke Files. The big new feature is the Scalable Repository. It lets you select exactly which older files should be saved on your computer and which should only be available on the server. Learn more in the thread.

The screenshot shows how you would configure your repository to exclude all files older than 30 days and all assets older than 7 days from your computer.

https://darke.handmade.network/ | &darke

In the next version of Darke Files you'll be able to decide exactly what files and commits to keep on your local machine. When trying to access something you don't have downloaded the CLI will let you know and helpfully spit out the command you can use to get it. In this example I've decided I only want to keep one commit in my local repository and so daf log ends up pretty short. &darke

I've just released version 0.3.2 of Darke Files. It comes with two improvements to ignoring files:
First, an all-new daf ignores command for easy viewing and editing of all file patterns you ignored (see screenshot).
Second, a rewritten-from-scratch pattern matching algorithm which now also supports patterns with ** wildcards.

https://darke.handmade.network | &darke

I added support for two-factor authentication with TOTP to Darke Files (0.3.1 just released) so if that's something you want in your security model the new release is for you. Simply get started with darke account register-mfa.

https://darke.handmade.network/ | &darke

I've released version 0.3.0 of Darke Files, my version control system. Key new features in 0.3.0 are registering accounts on a server, login and permission management for repos. Find the full changelog on the project page. Download: https://darkedev.itch.io/darke-files | Project page: https://darke.handmade.network/ | &darke

I'm currently working on repository permissions for Darke. With the next release you - as the owner of a repository - will be able to choose who may write to your repository. When write access is enforced you need to be logged-in when writing to the repo. &darke

I've released version 0.2.1 of Darke, my version control system. It's a bit of an incremental update containing fixes and additions suggested by y'all. I've also made changes to make the inner workings clearer, for example with the new /admin page on the server, pictured below. Find more info, the complete changelog, and download link over at https://darke.handmade.network/ &darke

I've just released the new version of my version control system Darke Files.

Darke Files' goal - if you haven't heard - is to scale seamlessly between a full-featured version control system and an easy-to-use file synchronization tool in a single repository. You can find more info about the programs features on the project page.

Since the first release I've done a bunch of clean-up and bug fixes, but more importantly major improvements. I've completely rewritten daf sync which gets your changes to the server easily in a single command. I've also upgraded the algorithm for merging changes to a three-way merge which allows it to deal with a few more situations automatically. The big new feature is daf info which I've already teased a week ago (see snippets on project page); with it you can process a template file (written in Go template syntax) with access to the info contained in a repository to extract whatever output you want from the repository. A list of all changes is available in the changelog file in the download package.

Download: https://darkedev.itch.io/darke-files

More info: https://handmade.network/p/213/darke/

&darke

This is probably the single most important / powerful feature in Darke. Even with everything else I have planned. &darke

With the amount of tasks in the 0.2.0 milestone for Darke actually going down I've spent the last few days unifying and improving the design system. Therefore I have finally produced something visual to show again. Below are "covers" so that the Darke products can look good on the virtual shelf. &darke

I love my beta testers for being able to put a whole bug report into a single screenshot. I'll have to make a pass over errors, not just this one but all of them. &darke

I've got something for you to play around with... https://handmade.network/forums/jam/t/8117-git_importer_for_darke_files

And that's a wrap. The Darke CLI is officially self-version-controlling. (if that's a word.) Bye, Git 👋

and fixed!

renaming a ref is really easy to do

Well, seems like this'll be a tiny rename everyone will have to make...

Importing a Git repository into Darke is done!

As already hinted at in the introduction, the differences in user identity handling between Git and Darke is a key point in this integration.

Git uses a user name and an email address to uniquely identity the person who committed a change. Darke replaces that with a Darke identity which specifies your username and Darke server in the same way an email address does. (Darke in general tries to stay away from using email for anything.)

I have implemented the options to generate such a mapping file and use it during the import process. By default darke.invalid is used as the Darke server. (That host will be handled specially in the UI.)

And that's a dry-run of the entire fast-export file done. I'm not doing anything with the data yet, just reading enough that I can skip over almost everything. Next up: Actually using some of this data, starting with the author and committer fields.

Every line read before this thing crashes is a win in my book. Let's get going!

I've been working on a version control system for the last few months. The goal is to have a system that can handle small text files as well as huge binary files while being as full-featured as Git if you want it to or as easy to use as Dropbox if you don't.

Recently I've added a few web pages to the server so I finally have something pretty to show off that isn't just the screenshot of a terminal. The image shows the history of one of my testing repositories. &darke