jam day 7/7
I finished mere minutes before the deadline.
2025-06-16 04:58:54
X-Ray Jam
June 9 - 15, 2025
I've been silently working on my jam project on exploring the NVidia Ampere GPU ISA binary encoding and just finished wrapping up with a GitHub repo https://github.com/NoxNode/AmpItUp and some blog posts: https://handmade.network/p/691/ampitup/blog/ &-it-up
I wrote a short blog post with more detail about &mDNS, DNS-SD, and how my tool works (including some limitations): https://handmade.network/p/688/buongiorno/blog/p/9033-detailed_writeup
project wrap up:
if the screen captures i posted here didn't make much sense to you, believe it or not, that was actually the whole point of my project. and by that i mean that the goal was to build something tailor made for me to better understand an algorithm i'm working on. and without knowledge of that algorithm, the data just doesn't make a whole lot of sense.
i wanted to figure out what it would take to create a custom visualization and how that process could be streamlined. because i think custom algorithm visualizations are a huge thing we're missing out on, not only for debugging, but also for internal developer documentation. some things are just much easier to understand with an interactive visual.
but while it's certainly nice to have visualizations, there's a question of whether they make "economic sense" to create. for example, if you spend 6 hours to build a visualization, which then allows you to fix a bug in 1 hour, and without the visualization it would have taken you 5 hours to fix the issue, then you saved -2 hours. in other words, you lost time. though there is a potential argument to be made that the tool may pay more dividends in the future, when you or some other dev comes back to the algorithm. then again, since the tool is tied closely to the implementation, there are also additional future costs for maintenance to be considered.
in practice, it took me around 10 hours to build the tool (excluding project setup time), and i spent around 7.5 hours using the tool to debug issues. if we exclude potential future time savings or costs (and points for "i just like doing this sort of thing"), then the tool would have had to save me at least 10 hours of debugging time. while it did certainly accelerate debugging significantly, i think that the savings are well below 10 hours, maybe around 2-3 hours. if we then also factor project setup time back in, i think we may have some anecdotal evidence for why "building custom visualizations" is just not a thing we do, unless we really need it.
my goal is now to bring down the time it takes to create such visualizations to under 2 hours and ideally closer to 30 minutes.
jam day 6/7
I was distracted by a message that caused me to work on another project for the whole day. Oops.
2025-06-15 15:56:44
Now my &mDNS tool Buongiorno can also map out which devices have requested which services. After letting this run for a while you get a pretty nice little matrix - both of my laptops need my printer, and my phone can AirPlay to both my laptops, but my phone will only do Handoff with my personal laptop. Also, my printer apparently serves a website.
it feels so cool to open Spotify on my phone and immediately see all of the following appear in my tool (&mdns)
Finally some meaningful project on my &mDNS project for the jam! I am assembling a list of advertised local services (e.g. AirPlay, Spotify Connect, printers) by performing a packet capture of all mDNS traffic on my system. Now that I have some raw data, I can collect it into a more meaningful UI.
jam day 5/7
I took a rest day to ponder on the alternation issue a bit more. There are two full days left before the jam ends, so there should be enough time to reach a minimum viable product.
2025-06-14 15:02:59
Final screenshot before bed to shows off the new compact layout and coloring based on executable name. The image shows the build of a dependency-crazed rust project. Tool is starting to feel useful because with the new layout you can see places where cmake built things serially that could've been built concurrently. &wtf
made it so it automatically starts recording when it detects a build happening. supports swift, make, cmake, xcodebuild, cargo, etc &wtf
day 5: added interactive navigation.
been using the tool (alongside graphviz) and have identified and fixed 5 bugs so far that would have been pretty hard to diagnose otherwise. &dataflowdebugger
jam day 4/7
The lexer now supports numbered repetition bounds, escape sequences for tab and newline, and lazy repetition. Its code has also been cleaned up to reduce copy pasting.
I'm still unsure of how to handle unicode. I'm currently favoring the byte level parsing route, which would mean that "." will be replaced with "[\x00-\x09\x0b-\xff]", to prevent regex engines that operate on unicode codepoints from using a model different from the verifier's. This would be necessary even if I choose the codepoint level parsing route as apparently some regex engines have additional characters they exclude, e.g. carriage return.
Alternation is a bit trickier to handle than I initially thought. The problem comes from wanting to support regexes that are natural to write, e.g. "^(?:foo|foobar|bar)", while having consistent behavior across different regex engines. For an input of "foobarbaz", apparently some engines will return "foo" as the match, while others will return "foobar". A related issue is that the ordering of the cases may affect which branch is matched.
I think the path forward for alternation, when one case may be the prefix of another, is to decompose it to an equivalent form and check that the cases are disjoint. This may reject more regexes than I'd like, but I'm currently out of ideas on how to resolve this.
python3 handmade_jam_r5.py
b'foo'
[(<Token_t.raw_string: 0>, b'foo')]
b'fo+'
[(<Token_t.raw_string: 0>, b'fo'), (<Token_t.repeat_plus: 11>, False)]
b'(?:abc|def)+([abc]*)?'
[(<Token_t.group_non_capture_start: 4>, 4), (<Token_t.raw_string: 0>, b'abc'), (<Token_t.alternation: 3>,), (<Token_t.raw_string: 0>, b'def'), (<Token_t.group_end: 6>, 0, None), (<Token_t.repeat_plus: 11>, False), (<Token_t.group_capture_start: 5>, 9), (<Token_t.char_set: 1>, [b'a', b'b', b'c']), (<Token_t.repeat_star: 10>, False), (<Token_t.group_end: 6>, 6, 0), (<Token_t.repeat_question: 9>, False)]
b'a{2}b{3,}c{5,7}'
[(<Token_t.raw_string: 0>, b'a'), (<Token_t.repeat_count: 12>, 2, 2, False), (<Token_t.raw_string: 0>, b'b'), (<Token_t.repeat_count: 12>, 3, None, False), (<Token_t.raw_string: 0>, b'c'), (<Token_t.repeat_count: 12>, 5, 7, False)]
b'^$|^foo...|bar$'
[(<Token_t.string_start: 7>,), (<Token_t.string_end: 8>,), (<Token_t.alternation: 3>,), (<Token_t.string_start: 7>,), (<Token_t.raw_string: 0>, b'foo'), (<Token_t.any_char: 2>,), (<Token_t.any_char: 2>,), (<Token_t.any_char: 2>,), (<Token_t.alternation: 3>,), (<Token_t.raw_string: 0>, b'bar'), (<Token_t.string_end: 8>,)]
b'[\\x11-\\x14][-.][.-<][a-z0-9.-]'
[(<Token_t.char_set: 1>, [(b'\x11', b'\x14')]), (<Token_t.char_set: 1>, [b'-', b'.']), (<Token_t.char_set: 1>, [(b'.', b'<')]), (<Token_t.char_set: 1>, [(b'a', b'z'), (b'0', b'9'), b'.', b'-'])]
b'a??b*?c+?\\td{1}?e{4,}?f{9,16}?\\n'
[(<Token_t.raw_string: 0>, b'a'), (<Token_t.repeat_question: 9>, True), (<Token_t.raw_string: 0>, b'b'), (<Token_t.repeat_star: 10>, True), (<Token_t.raw_string: 0>, b'c'), (<Token_t.repeat_plus: 11>, True), (<Token_t.raw_string: 0>, b'\td'), (<Token_t.repeat_count: 12>, 1, 1, True), (<Token_t.raw_string: 0>, b'e'), (<Token_t.repeat_count: 12>, 4, None, True), (<Token_t.raw_string: 0>, b'f'), (<Token_t.repeat_count: 12>, 9, 16, True), (<Token_t.raw_string: 0>, b'\n')]
2025-06-13 14:09:06
the timeline now draws a curve from the start of each process to its parent process. I added some logic to detect when one process reads a file created by another process, and draws a green line from the last write to the first read to visualize that dependency. Clicking on a process reveals more information about it, including the full command that launched the process, and a list of files it read from and wrote to. I have a filter list to hide common system paths, because theres so many that they drown out the relevant paths. Still trying to figure out what other useful information I can extract, and how best to display it. I've seen that even simple projects tend to have a lot of temporary files that are created and destroyed within one build process, so I'm wondering if I should have a timeline for files too. Not sure if that would give any actionable information though. &wtf
day 4: now showing the diff between outputs when processing the work list forwards vs backwards. if my implementation was correct, this view would be empty. but then i also wouldn't need a dataflow debugger 🌚
added tabs to explore the history of dataflow nodes in each of the logs. &dataflowdebugger
jam day 3/7
I wasn't able to finish parsing, but I'm almost done with lexing. Currently, the lexer can't handle numbered repetition bounds, non-hex escapes, negative character sets, and lazy repetition. The code is also in dire need of a cleanup.
I'm not sure on how I want to handle unicode yet. My two ideas are to either parse on the byte level or parse on the codepoint level. The first option provides full functionality at the cost of some verbosity. The second option simplifies usage, but complicates the implementation with regard to collisions with regexes meant to be used on byte strings. I can transparently translate codepoint-specified regexes into the byte-specified equivalent, but that might be very confusing when that's mixed with byte level parsing and the verifier rejects the regex.
Another decision I need to make is which escapes I'll support. The escaping of meta characters and bytes (via the hex code) is already done. I'll likely add support for tab and newline and likely won't add support for the digit/space/word escapes, but I don't know about the rest yet.
$ python3 handmade_jam_r5.py
b'foo'
[(<Token_t.raw_string: 0>, b'foo')]
b'fo+'
[(<Token_t.raw_string: 0>, b'fo'), (<Token_t.repeat_plus: 11>,)]
b'(?:abc|def)+([abc]*)?'
[(<Token_t.group_non_capture_start: 4>, 4), (<Token_t.raw_string: 0>, b'abc'), (<Token_t.alternation: 3>,), (<Token_t.raw_string: 0>, b'def'), (<Token_t.group_end: 6>, 0, None), (<Token_t.repeat_plus: 11>,), (<Token_t.group_capture_start: 5>, 9), (<Token_t.char_set: 1>, [b'a', b'b', b'c']), (<Token_t.repeat_star: 10>,), (<Token_t.group_end: 6>, 6, 0), (<Token_t.repeat_question: 9>,)]
b'a{2}b{3,}c{5,7}'
Exception
b'^$|^foo...|bar$'
[(<Token_t.string_start: 7>,), (<Token_t.string_end: 8>,), (<Token_t.alternation: 3>,), (<Token_t.string_start: 7>,), (<Token_t.raw_string: 0>, b'foo'), (<Token_t.any_char: 2>,), (<Token_t.any_char: 2>,), (<Token_t.any_char: 2>,), (<Token_t.alternation: 3>,), (<Token_t.raw_string: 0>, b'bar'), (<Token_t.string_end: 8>,)]
b'[\\x11-\\x14][-.][.-<][a-z0-9.-]'
[(<Token_t.char_set: 1>, [(b'\x11', b'\x14')]), (<Token_t.char_set: 1>, [b'-', b'.']), (<Token_t.char_set: 1>, [(b'.', b'<')]), (<Token_t.char_set: 1>, [(b'a', b'z'), (b'0', b'9'), b'.', b'-'])]
2025-06-12 13:38:07
&loctree Got a basic web UI up, with an interactive collapsible tree view and filtering by language. So far it's all a completely statically built page, with javascript taking the raw LOC data and rendering the tree from scratch on every filter. This ran reasonably on the orca codebase, but on chromium I'm getting absolutely destroyed. Going to see how much performance I can get out of the static rendering, if that'll work or if I'll have to set up a local server to stream the data and/or do some server-side html generation.
My project can now draw a timeline for make and all the child processes it launches. In this video it's profiling the build of an open source rust project. My program collects syscall/process/timing data in realtime, so in the video you see the timeline scaling to make room for more samples. Above the timeline, I have a list of all the executables used over the course of the build. &wtf
success: i just found the first bug in the dataflow analysis :weewah:
in this step, r3 becomes ? in vals_out, but for some reason it remains unmodified in restored_vals_out.
right now, i'm just manually changing which log file to view and which dataflow node to display in the lua script. would be nice if i didn't have to restart the program all the time, but it's already kinda usable, so that's nice. &dataflowdebugger
&loctree day 2ish - I'm creating some data visualization for lines-of-code-count data - I've often found myself wanting to drill down into lines of code for subdirectories of a project instead of just a whole-project summary. A tool called tokei exposes raw data per file, and you can use it as a library, but so far the only visualization I've seen is a somewhat inflexible pie chart. So I'm hoping to use that to export a static, interactive html page, maybe something like windirstat. Admittedly less x-ray-ish than some of the project I've seen so far, but I figured better to participate and try to scratch an itch than not.
Yesterday and this morning was getting data into a format I can use and starting to generate HTML. I've got some rust code that uses the tokei library to export the data, then using python to transform it since I'm too dumb to write tree building logic and fight the rust compiler at the same time, but I hope to have it all in native code either by the end of the jam or shortly after.
jam day 2/7
I didn't work on the project at all, which is setting me up for a time crunch in the next few days. To make up for today, I want to finish parsing and start on the control flow graph analysis by the end of day 3.
2025-06-11 10:56:41
day 2:
created a structured log of "things happening in the dataflow analysis".
then experimented with a generic viewer for that log, where specific log messages could be replaced with a custom ui builder.
ended up not really liking that. so tomorrow, i'll pre-process the log to build a more useful data structure. and then just build custom uis for different debugging use cases. &dataflowdebugger
After the first day I have a program that logs all the syscalls of a process and its children. Unlike some other projects here, I'm trying to understand the aggregate behavior of a system (processes communicating and launching each other) rather than the atomic parts (syscalls). So now that I have this stream of juicy data, I want to start extracting some useful insights: what files does each process read and write? What are the dependencies between processes? how long does each process take and what's the bottleneck? The screenshot shows some syscalls from make running ls, with the syscalls of ls indented. &wtf
jam day 1/7
I didn't get much done except for the basic skeleton for parsing.
$ python3 handmade_jam_r5.py bytearray(b'f') bytearray(b'o') bytearray(b'o')
2025-06-10 11:28:42
day 1: just some project setup.
plan is to do some structured logging, then figure out how to visualize that in a way that's useful.
decided to use lua both for the log format and the ui implementation, hoping that that will give me a fast iteration cycle.
didn't have much time today, and most of that went into creating lua bindings for my ui library.
