I recently unified the codepaths for recording data, evaluating expressions "currently", and evaluating them historically. This was primarily so that I could finalize conditional breakpoints with arbitrary expressions.
Because they're unified, it made it straightforward to allow conditions to mix and match current and historical data, e.g. in this video val > val @ 0 is true when the current value is greater than the value at sample 0.
Previously, the assumption was that recording would always use be really simple expressions (e.g. variables, pointer derefs, struct member access etc), and that all the full expression work would be historical.

The new conditions play nicely with other major recent additions to &WhiteBox: breakpoint actions to enable/disable other breakpoints/inspections, and (auto-layouting) hardware breakpoints.

These solve the main things I find frustrating with data breakpoints in other debuggers that I've tried:

• Data breakpoint disable between runs (presumably due to ASLR & not knowing where to replace them).
  • Solution: auto-enable them at specific points in the code, reevaluating the expressions for addresses then.
• Data breakpoints on local vars trigger in some completely different callstack that you don't care about after the function finishes.
  • Solution: auto-disable them either at particular points in the code or when the function returns.

In the video:

• I place a sampling breakpoint both as a place to enable/disable the data breakpoint, and as a fixed point of reference for comparing val > val @ 0
• The data breakpoint is set to sample data and then continue execution
• I rerun multiple times with different conditions (including historical self-reference) and get a fresh timeline showing different all the times val was written to when the condition was true.
• (You can see all the changes for local vars on multiple levels of the stack, as well as watch expressions currently evaluating at the top level.)

I did my first &WhiteBox dev stream in a very long time and it went really well!

I added basic, but fully-functional support for hardware breakpoints, including exec/read/write triggers for all valid sizes.
It also has expression evaluation with immediate inline feedback on the address.

It already works with WhiteBox's sampling & timeline to let you see every change to some data, not just stop at the next one 😁
(You can use it to break at the next change as well...)
Ignoring the WIP/dev breakpoint UI, below you can see on the timeline every time val gets written to when parsing "236":

Major new upgrade to &WhiteBox's timeline incoming: showing data at multiple levels of the callstack at once!
(Here you can see n in [Stack 3] is changing at the same time as *out in [Stack 4].)

WIP shot of EXE callstacks across time with data from a selected function in &WhiteBox. You can switch between functions to show recorded data for each.
(Showing data for multiple functions at once is coming, but requires a rework of the UX model because it breaks the current assumption that there's only one expression per row)

&WhiteBox https://fxtwitter.com/whitebox_sys/status/1734239475176296811

&WhiteBox https://fxtwitter.com/whitebox_sys/status/1734210818294657078

&WhiteBox v0.122.0

New Features

• Add simple watch expressions for the timeline, with feedback messages and underlines for parse errors
• Add call stacks to timeline (including arguments & return values where captured)
• Add text scaling (Ctrl + =/Ctrl + -)
• Speed up data inspection, which bounds total running speed for user code
• Add right-click options to timeline values to copy to clipboard
• Visualize jumps in disassembly
• Add AT&T/Intel disassembly syntax options
• Add support for non-trivial copy constructors

Improvements & Fixes

• Better handling for anonymous records
• Improve font rendering with DirectWrite, FreeType, and TrueType backends
• Improve C++ constructor handling
• Show min/max values for timeline expressions
• Improve data tree value graph
• No longer break on Linux signals that are recommended to be ignored by default (e.g. SIGWINCH - change window size)
• Animate the status bar colour when new messages are added to the console
• Fix emacs plugin port issue
• Fix misc issues; add resiliency to more edge-cases; reduce jank; ...

Known Issues

• Globals that are mutated and not reset in user code are not always consistently reset between runs (e.g. changing functions back and forth with a locked caller). Saving your code or using Compiler > Reset (Ctrl+Shift+R) forces them to be reset.

Get access to the latest version here: https://azmr.itch.io/whitebox

N.B. If you were on the WhiteBox discord (https://chat.whitebox.systems) you'd have known about this last week 😁

New &WhiteBox feature: call stacks!

Each level shows the arguments that function was called with, and the return value if it was recorded.

They're currently aligned to the inspected function, but this is likely to change as we move toward inspecting multiple functions at once.

On request from a non-HMN user, I've just added a right-click option to &WhiteBox for copying values from the timeline.
This can then be fed back into the function caller to rerun, starting with the given value (for simple data, not pointer nests)...
(N.B. it copies the value with the current formatting, so to make it a valid C initializer Struct member names needs to be disabled)

@philliptrudeau has done some great work adding a DirectWrite backend for the text rendering in &WhiteBox: small text now looks much better!
(ideally viewed at 100%)

I spent some extra time on this at the end of last week getting &WhiteBox's disassembly jump lines to merge when they have the same destination.

I'm quite pleased with how much it improves the overall legibility of the potential jumps.

Can you spot the codegen oddity in the disassembly?

Thanks to some prompting from @megawolf, the jumps in &WhiteBox's disassembly view are now also coloured by direction

I just added the option to use Intel syntax (as well as AT&T) in &WhiteBox's disassembly, as well as arrows showing the destination of each jump (unconditional jumps are brighter than conditional jumps)

A user got &WhiteBox working with @raysan5's raylib, and I realised I hadn't tried it yet myself. This was something that needed correcting immediately!
It became possible with the concurrency features @philliptrudeau worked hard to get into v0.116.0

There are still things to hammer out from the previous interaction model (when to rerun the code, LOD, recording selection...)

But take a look:
live, graphed data & an instant edit-run loop for games!

raylib made this very easy:

• download raylib-4.5.0_win64_msvc16.zip from https://github.com/raysan5/raylib/releases/latest
• add -I[...]/raylib/include to the compiler flags
• add [...]/raylib/lib/raylib.dll to the DLLs list
• connect to WhiteBox from an example raylib file
• start playing

We accidentally turned &WhiteBox into a graphing calculator 😄

&WhiteBox now has a pretty good heuristic for determining uninitialized values, which means we can show them as distinct and not include them in min/max stats: 🥳
e.g. k is uninit until line 8 finishes

&WhiteBox now has a pretty good heuristic for determining uninitialized values, which means we can show them as distinct and not include them in min/max stats: 🥳

With the jam finished, here's a little demo video that highlights most of the project's features :)

https://handmade.network/p/392/expertmemer-experimental-memory-access-visualizer/ &memer

Here is my project for the HMN visibility jam, introspect. It lets you browse a C file as a nonlinear web of declarations and usage sites. There are a lot of bugs in how I handle libclang parsing due to time constraints unfortunately (I only had a couple days), but if anybody wants to fix it up/make improvements the source, as well as executable to try it out on your own code, is here: https://github.com/creikey/introspect . The code being visualized is the source for introspect itself &introspect

A bit late for the jam 😅 but I wanted to share anyway.
https://gitlab.com/tanoxyz/object-disaster

Updated description to include YouTube video. I was using Safari last night and, now, I'm using Chrome.

Didn't have much time during the day to jam, but managed to whack in a couple UX features before 12am tonight: colored allocation blocks based on a string tag, and frame markers. &memview

I did a lot of work today on the memory viewer.
I implemented scrolling the viewer with mouse wheel, as well as a "goto address" command window that lets you jump to any address.
The first video shows the navigation features and the second video demonstrates load and store instructions moving data to and from memory.
&bitwitch-nes

Day 2

My tasks for today were:

• get the disassembler working
• implement simple text rendering of ttf fonts
• start the memory viewer window

This is my first jam - I decided to try and visualize how the string allocator that I am trying to write works (or doesn't). I spent most of this weekend trying to fix the allocator after I successfully visualized some of it's many problems...so I'd say this was a success!

For my &standoff visualizer, added the ability to show underline styles and save the text and annotations to URL's query string. Now you can share a specific set of annotations and query strings via URL.

JSON viewer final jam screenshot - scrolling and colors &mjson
Download @ https://handmade.network/p/383/mark-s-json-viewer/

End of jam report: I added a system for mixing all input tracks into a master track to perform additional analysis & visualization on the combined track. I added a pretty typical visualization for the mixed track that also serves to visually "frame" the video a little bit. I tuned up the effects to make them just right for the final videos. Finally I added command line parameterization so I can reuse the program without having to edit the parameters and generate noisy git commits.

Speaking of git you can now check out source over here: https://mr4th.com/link/podcastoscope

&rede Better NFA superstate diffing for stepping backwards. Node hit-testing. Red flash for entering null-state. Strict/Loose matching with contextual underline.

Show markup as Markdown is now supported in the &standoff visualizer.

&WhiteBox timeline data visualisation part 3: using de-emphasis for the "non-value" part of the bar instead of emphasis for "value" part itself. To me this reduces visual noise, but I'd like to hear thoughts from others. The value colour is what was the normal bar colour... it might be worth increasing the contrast somewhat.
Importantly, it also obscures the text less.
(Ignore the fact that [line] is coloured as it was previously, synthetic values need to be handled slightly differently)

&WhiteBox Noticed a bug in the previous image as I was writing about unit size: there's an off-by-"1" in max value for floats, which visually halves the 0.0-0.9 range shown.
This is now fixed:

&WhiteBox Noticed a bug in the previous image as I was writing about unit size: there's an off-by-"1" in max value, which visually halves the 0-1 range shown.
This is now fixed:

&WhiteBox timeline data visualisation part 2: lines for integral values have a height based on the span/range of values for that expression. This is particularly useful for expressions with small ranges like booleans!
This is meaningful because integers have a "unit size" of 1, but floats don't really have that. I could use "smallest non-zero exponent" or "smallest difference between values" or even "ordinal rank of all seen values (sorted)". The last of these would work for integers as well with large ranges but few values. It's better for distinguishing between values but not so good at making the actual value perceptually apparent.

I took some output from &uloc and put it into Google Sheets, and made some charts. Shown is the number of source lines compared to number of unique lines over the course of some git commit history. The overlayed line shows the rate at which no. of source lines is increasing relative to unique lines, ranging from 0x to 4x. In the first pic you can see the rate is somewhat tame compared to the second pic, where there's some spikes going up to 3x - 4x, hinting that the code added at those commits was largely duplicate code.

Here's a bug you may not have considered, that &yno could help you catch.

&elf Next step is writing the loader simulator, it shouldn't be hard considering my scope but it should be able to display when and where things are allocated and loaded.

Press CTRL-ALT-J to view JSON data from your clipboard &mjson

&WhiteBox timeline data visualisation, part 1: horizontal lines indicating value for primitive types

I've added support for diffing multiple files at once to Diffest. While this example is not of a huge code base, the code-aware diff correctly tracks code that was moved between files. &diffest

Did some polish yesterday: embed a nerd font, clean up help and error messages.

I'm happy overall with the approach I'm trying for UI:

• Spin up a web server with Axum (Rust) for static content and a websocket
• Intentionally simple front-end (Tailwind+Preact). No build step, no business logic

Added JSON output support to &uloc. The * syntax is handled by the shell though and using jq for formatting.

Well I think basically this is it :) I don't really need a ton of other pieces of information on my website.

The statistics page is accessible through the link on the footer of my blog. https://www.wellobserve.com/

i got the very beginnings of a memory viewer working. its kind of cool to watch data fly around in memory with a game running.
&bitwitch-nes

Instrumented DOOM and got its Z_Malloc()/Z_Free() allocations streaming live to &memview. Added a couple UX improvements. Unfortunately due to a bug in the Windows version of Zig, callstacks are currently only available for zig modules. The linux version should work fine, though I need to finish the port tomorrow.

A few more &yno scenarios, all trying to verify if plain old HTTP requests are working.

1. Caddy is trying to force all traffic to HTTPS, which doesn't work. yno shows you the redirect packets.
2. Caddy is now only handling HTTPS, rejecting my initial request.
3. I have now forced Caddy to use HTTP, and things work!

This was real debugging; I didn't mean to configure Caddy this way, and yno actually helped me find the issue 😁

Day 2 progress report: I've definitely got an audio visualizer now. Today my goal was to get it good enough that I would be willing to use to actually render a video for YouTube. The biggest steps forward are a system to analyze when to highlight speakers that does not change too often, or ignore overlapping speakers, and sensitive enough frequency analysis that effects rendered from it really do match what you're hearing. It's still very bare bones, but I think it's looking pretty nice!

Finally have something to show! &yno is performing a battery of checks to figure out why HTTP connections aren't working, including sniffing packets to figure out which programs are actually handling the traffic despite connections not working.

i got my disassembler working and you can step through instructions.
the current instruction that the program counter is pointing to is highlighted.
you can also step by frame.
&bitwitch-nes

I'm now grouping and highlighting related changes with surrounding context if possible. &diffest

Today I added directory support to &uloc as well as csv/tsv output, so you can tranform or view the data with external tools. Also fixed some bugs and made a github repo :)

Ok, my little &standoff visualizer can now show the HTML so you can compare between a hierarchical format like HTML and standoff markup. Try out overlapping multiple italic and bold annotations to see how quirky the HTML can get!

You might be wondering why I picked Python for the first few test cases. The answer is this optional comma featured in test cases 04 and 05. &diffest

If someone is trying to sneak a change into the middle of a big reformatting, a code-aware diff can tell you exactly what's going on.
(In this jam, I'm exploring what a code-aware diff tool could do for you. Past updates and more info at https://handmade.network/p/366/diffest/, &diffest)

Scrolling through reddit: 23 seconds of stalls out of 46 seconds of usage &theblazingweb
(I still need to work on a real score calculation. Obviously this should get a zero)

Collecting information about frame stalls on websites through a browser extension &theblazingweb

Day 1

I had to work today so I only got a few hours of jammin in. I was mainly focused on trying to figure out how to write a disassembler. I want to be able to display a games' assembly and let the user step through it as the game runs. I also want to be able to display a certain number of instructions before the current instruction that the cpu's program counter is pointing to. The 6502 has variable length instructions so this is not straight forward to seek backwards from the current program counter to determine previous instructions.

I also cannot disassemble the whole memory up front, because there are chips called "mappers" inside NES cartridges that dynamically remap addresses. The part of the address space visible by the cpu that is occupied by cartridge memory can be remapped by the mapper to any arbitrary memory within the cartridge.

I think a viable approach will be to just cache a certain number of instruction addresses. The program counter can be cached before each instruction that the cpu executes, replacing the oldest address in cache. This should make it possible to disassemble code from a few instructions prior to a few instructions following the current instruction.

I had to do some refactoring work just to get the code into a place where this is possible. It has been awhile since I touched this codebase as well. I wasn't able to get it working tonight, so I will go at it again tomorrow.

A few days leading up to the jam I did some prep work. I created a second window where all the visibility stuff will go, and displayed pattern table memory in it. I posted some screenshots of pattern tables for various games in #project-showcase.

I am playing around with my NES emulator for the Visibility Jam (&bitwitch-nes).
Here are some examples of pattern table memory dumped from some NES cartridges and visualized.
Tomorrow I hope to get a disassembler working, so you can step through code as a game is running.

I don't have a lot of free time each day, so to get this project to a reasonable state of completion I needed to do some pregaming. At the beginning of the jam, Memview could read allocation messages over the network, and visualize them within a scrubbable timeline and zoomable memory view.

After a couple evenings, callstack support has been added, so you can select blocks to see their size and the allocating callstack.

Put together a help page

Added NFA superstate visualization and particles my jam project &rede (https://handmade.network/p/369/rede/)
Planning on making it more clear what path the individual states go through as well as be better at stepping backwards

And with that the fourth box, "Code-Aware Diff", is filled. What's that, it doesn't look very spectacular? Well, there haven't been any spectacular changes here, have there? &diffest

I haven't quite filled that fourth box yet but I've made a lot of progress. I'm parsing both files into a syntax tree which I'll then use to calculate a diff between those files. This syntax tree is a combined abstract and concrete syntax tree: The black nodes form the AST and the gray nodes come from the CST and fill in the gaps between the nodes of the AST. The content in quotes comes directly from the source code while anything else is tree-types and structure. The grey nodes are actually two different types: necessary punctuation and optional whitespace; they are not differentiated in this visualization. Because of its structure and because it's neither an AST nor a CST I'm calling it an "expandable syntax tree" or EST (which then brings us to "diffEST" 😛 ). Up next: Fill that fourth box. &diffest

plotting target scroll and smoothly animated scroll against time

I spent the morning finalizing ThePlan™ and creating the first dozen test cases. Then I built a way to visualize the diffs I'm about to produce and hooked up a regular text diff (as available in Darke) for comparison. Up next: Fill that fourth box.
(In this jam, I'm exploring what a code-aware diff tool could do for you. Past updates and more info at https://handmade.network/p/366/diffest/, &diffest)

I'm in.
Despite our best efforts, the vast majority of source code is still in plain text. Lots of tools operating on source code only work with that plain text. Advanced tools however, like the Language Server in your favorite IDE, use this plain text as a starting point to build up rich, complex, and ultimately incredibly useful data structures in which your code lives. These tools understand your code, in many cases better than you do yourself. (And way-too-often they need to be restarted because they manage to run themselves into some infinite loop or whatever but I digress.)
A tool that is at least slightly aware of the code you have written could compare two states of the same code base and:
1️⃣ Look past changes in formatting to focus on what's actually relevant.
2️⃣ Group related changes so renaming a constant is one change instead of several dozen.
3️⃣ Examine the entire code base instead of handling every file individually.
4️⃣ Track where code came from and where it was moved to.
5️⃣ Present the entirety of the changes in a sensible and ordered way.
6️⃣ And finally, figure out what changes weren't even made.
And that's what I'll be exploring during the Visibility Jam.
diff, differ, &diffest

(originally posted on 2023-03-04 in #jam)

Got a basic statistic graph working on my blog 😄 &lamdwiki https://www.wellobserve.com/index.php?&statistics=all

A simple graph written in php/svg directly 😄 &lamdwiki

&WhiteBox is prepping to go native, reading PDBs for arbitrary executables in place of the DWARF it generates internally.
Unsurprisingly, @NeGate has been doing a great job on this! 😄
Here's WhiteBox inspecting its own PDB functions, showing line info & local variables

&WhiteBox Handmade Seattle demo is now public
https://twitter.com/whitebox_sys/status/1598824487084068864
https://www.youtube.com/watch?v=mNFDr33NbyQ

&WhiteBox
https://twitter.com/whitebox_sys/status/1577053893150019584

I've been working my way through the ANSI control codes standard (+ extras) for WhiteBox's new in-built console.

We're capturing stdout & stderr from the user's process, reading ~all controls & handling more graphical ones than many terminals!

Can yours do curly red underlines?

&WhiteBox https://twitter.com/whitebox_sys/status/1573418093531271169

Just playing around with a WIP build of WhiteBox after @phillip_trudeau has made some upgrades.

I'm really enjoying taking advantage of this breakout, full-width timeline 😀

(thanks to @ocornut for Dear ImGui + docking + viewports!)

I added a red line to the end of the &WhiteBox timeline when a run had an error that prevented the function from returning.
I also fixed the status message telling you what line the error occurred on.
(There's a new release coming very, very soon!)

&WhiteBox https://twitter.com/whitebox_sys/status/1534480349820071936

&WhiteBox now has a welcome screen with instructions on how to install the plugin for each editor, thanks to @BaremetalBaron

&whitebox https://twitter.com/whitebox_sys/status/1528796270797651968

&whitebox https://twitter.com/whitebox_sys/status/1519811544766369792

I'm really excited about this update to &WhiteBox - it adds the ability to call into other functions in your code (not just the function that's being recorded).
This means you can call main() (or something else that calls your function) and see how your function behaves with all of the parameters & extra context from runtime without having to input them manually.

It's now at a point where this seems to be working, it just needs to do a little tidying up after itself...

Reworked &WhiteBox's recording of the running process' memory to be faster, simpler, and to better capture pages changing protection/accessibility

I haven't shown much for &WhiteBox recently as it's all been pretty non-visual. Here's a little expression builder I made on stream for debugging expression evaluations.
It shows the value of that expression at a given sample of the run as well as a list of all the times the value changed during the run.

Having it as a builder means that I can enforce the validity of the expression, and it also means I don't have to handle shadowed variables with the same identifier as you would for text parsing. It's a little less clear how identities should be handled when taking a large span of time into account instead of just a single instant, so this lets me defer decisions about that.