!til &learningjam2024
Just finished processing this paper. I found it very informative but also hard to follow since the math notation made the equations look the same. My notes are basically a condensed version of the whole text, including all the 131 equations of which I'll review and color code the variables.
https://jcgt.org/published/0003/02/03/paper.pdf

&learningjam2024
Recently finished reading through the paper [PM] I mentioned in my last post. It certainly clarified some things and my summary has some inaccuracies in it.
Until I get around to address that I'd like to mention that equation (3), which I got from the Disney paper [DS], also has to be multiplied with a "normalization" term which is shown at the top of page 25 (eq. 3, the left fraction on the right side). This wasn't made clear to me until it was mentioned in [PM].
This is my usual experience with reading papers of this kind. Often I need multiple sources in order to fill in the gaps of my understanding.

Testing in my Shadertoy demo, the specular highlight (D-term) now behaves more or less as I would expect. Still got aliasing at the penumbra which appears to be due to the G-term (which handles micro facet shadowing), so that's what I'll be working on next.

!til &reveng-ann As a sign-off, for the teaching portion decided to majorly revise the first part of my writings (on solving Caesar ciphers using a transformer) to ensure overall quality. Gave a project recap and had a fun time!

!til &learningjam2024 Linking to the paper I mentioned in the recap stream. The best resource on the subject I've found so far.
https://blog.selfshadow.com/publications/s2013-shading-course/hoffman/s2013_pbs_physics_math_notes.pdf

!til late addition to my learning jam project on &colors: how to derive the HSL/HSV color models from an RGB cube: https://youtu.be/1zwXnf0G2II

!til the sRGB curve is slightly different from a simple gamma curve, and the difference is actually noticeable on darker &colors. So here's a follow-up video on the proper way to do sRGB conversions: https://youtu.be/T54SX-QwFpc

!til &modular-forms I've written up my understanding of what modular forms are here: https://graemephi.github.io/posts/modular-forms/ I've tried pretty hard to keep the maths simple and have pictures for the stuff that needs visual intuition. Also, I've come across some pretty good resources for learning this stuff so if nothing else my post will point you at good places to learn further if you, like me, want to know wtf modular forms are

!til here's part two of my learning jam project &colors where I explain gamma correction and sRGB encoding, and update the vector graphics renderer of &orca to be gamma correct
https://youtu.be/cFoi1OLHFQ0

Now I've published the source code and changed the description, this is looking like a real project!

!til &learningjam2024 Got started on implementing a material demo in Shadertoy but had to realize that it's going to take its time, so a brief summary will have to suffice. I've put it up for download on my site.
https://unlitart.com/download/HMN_LearningJam2024_BRDF_Summary.pdf
Big thanks to the HMN team for arranging this event. To be honest the concept didn't excite me too much when it was announced but it turned out to be a great opportunity to check off something from the mental bucket list.

!til &learning-jam-2024 Here is my write up for my deepdives in the inner workings of containers
https://www.annwan.me/computers/what-why-how-containers/

!til a bit about &colors, so here are some ramblings about the XYZ color space and xy chromaticity diagram: https://youtu.be/IOJe1ugkc9Y

!til &virtual-machine I had planned to do a stream today continuing from yesterday, adding more instructions and explaining IO. But, I had a lot to do today so it didn't work out. overall though, I think yesterday's stream was pretty good, even if no one watched 🥲. That concludes my contribution to this year's learning jam and the initial creation of Gepetto. I'll publish the code to my source repo tomorrow and continue development. This was fun!

!til &learning-jam my post is done! now i may need to translate it

Made a demo of timing attacks as part of my &spectre writeup...hopefully I still have time to actually explain the Spectre part 😅

!til &junkworks And stretch goal has been reached! I've got my UDP client/server prototype controlling two characters. Clients send position packets to a server, the server forwards those packets to the clients, and the clients update the other character's position in space.

Next steps are to make state server-authoritative, implement client-side prediction, rollback, and interpolation. This has been a great jam. I learned a ton and have prototype code to build off of (cough cough refactor cough). Thank you @bvisness and all of the admin team for setting everything up!

!til &junkworks Awwww yeah, we've got client-server-client UDP action happening.

These two clients are running on the same Windows machine, but they're exchanging packets with a server in VA. The server forwards complementary packets to each client (client A gets client B's data, client B gets client A's data).

I can also throw in a third Linux client (A gets B and C data, B gets A and C data, C gets A and B data).

Next step is to hook this up to the renderer I've been working on and let some players run around.

!til &junkworks There are subtle and obnoxious API differences between Windows and Linux's implementations of recvfrom and sendto. Linux sends and copies buffers of void *, but Windows sends and copies buffers of char *. Linux uses size_t for the sizes of incoming and outgoing buffers; Windows uses int. Separate UDP socket implementations for both OS's might be worthwhile. That way you could include one or the other based on preprocessor directives rather than having preprocessor conditionals sprinkled over one implementation.

!til working on my &spectre article this morning, plz tell me if my handwavey pipeline diagram is too handwavey

Memory Management in Windows - Part 4: Conclusion

Well, that's it, 15,000 characters worth of me trying to summarize how memory management works in Windows! There was a ton that I didn't cover. Copy-on-write, hardware permissions on physical pages, translation buffers and caches, and much more. My hope with the below is that it has piqued your interest and made you want to learn more about this fascinating OS. I know there are some very knowledgeable folks around here, so if there's something I've got wrong or misunderstood, please don't hesitate to let me know.

Thank you so much to Ben and Asaf for putting the jam together and helping with some technical difficulties I had when making these posts. And thank you to the entire HMN community, for being such welcoming and encouraging folks. If anyone finds this useful, please let me know here or on Discord. Cheers!

Memory Management in Windows – Part 3: Physical Memory

Just like virtual memory, processes also have a limited amount of physical memory, referred to as the process’ Working Set. There are also System and Session Working Sets, which contain subsets of pageable code and data scoped to those spaces. While it is possible for a process to specify a minimum and maximum working set size, these limits are almost always ignored in practice. The default minimum and maximum per process is 50 and 345 pages, respectively. You can self-impose a hard cap on maximum working set if you’d like to do that for some reason. Without these self-imposed restrictions, working sets will grow or shrink beyond these limits as demand requires and resources allow.

When memory is getting low, working sets are trimmed by the working set manager, taking into account a number of factors including minimum working set sizes and the last time pages were accessed. Page faults under these conditions that require another page from the working set are handled by replacing pages on the working set and paging out old ones, rather than allocating new pages. It is possible to manually trigger the trimming of a process’ working set at anytime by setting the maximum working set size while the program is running. When memory is plentiful, the working set manager instead spends its time calculating how many pages could be removed from working sets should the need ever arise.

On a system-wide scale, the total memory available for allocation is referred to as the “system commit limit.” This comprises the sum of all physical memory available to the system plus the size of any page files. It is possible to run Windows with no page file at all (though this is almost universally discouraged), in which case the commit limit is simply the amount of physical memory available to Windows. Note that it is likely that this number does not coincide exactly with the amount of physical memory installed, as certain hardware reserves memory for itself independent of the operating system.

Any memory allocated against the system commit limit is considered the “system commit charge” and represents everything that must be kept either in RAM, or in the page file (so basically, anything that isn’t a file on disk other than the page file). When the system commit limit is reached, the system will attempt to increase the size of the page file. If this does not succeed (or, seemingly, if it can’t be carried out fast enough), memory allocations will fail. Each process also has a process page file quota, which tracks its contributions to the system commit charge. It’s worth noting that the commit charge and process page file quotas reflect the maximum theoretical, rather than actual, usage. Windows will not allocate any memory that it could not actually provide if necessary, even if many of those allocations have not and may never take place.

As you can imagine, pages added to a process’ working set are not chosen at random (well, they kind of are, but ASLR is another topic). Windows keeps track of every physical page of memory in the Page Frame Number database. These pages can be in one of nine states: Free, Zeroed, Modified, Modified No-Write, Standby, Transition, Active, Rom, or Bad. Active, or Valid, pages are either part of a working set or in use by some other means and typically have a valid PTE pointing to them. Transition pages are currently undergoing I/O, not part of a working set, and not on a page list. Modified no-write is a special case of the modified state where the page won’t be written to disk. This state is only used by file system drivers in specific scenarios. The other six states’ pages are each tracked in their own linked list for quick access by the memory manager.

Every page in the system starts out on the free page list, and returns there when it is no longer in use. These free pages are zeroed by the zero page thread and placed on the zero page list. Memory is (typically) pulled from the zero page list into a working set. When memory is trimmed from a working set, it either goes onto the modified or standby list. Modified pages have been changed in some way since they were last written to disk, and therefore must have their contents saved by the modified page writer before becoming standby pages. Standby pages can be reused immediately by whatever was using them previously, because their content has not changed since the last time it was written to disk. Rom pages are read-only, and bad pages have failed a consistency check should not be used. Most new allocations happen from the zero page list. Kernel mode processes are permitted to pull directly from the free page list in some cases, so long as the memory’s content has been overwritten before it makes it to user mode.

!til I'm writing a tutorial on my blog in portuguese but I plan to translate it to english later

Memory Management in Windows - Part 2: Address Translation

Note: We’re going to have to break this into three parts, not two, because I wrote so much on the address translation piece I’ve hit the character limit for HMN posts. And I didn't even get to talk about TLBs or anything!

While virtual memory address spaces can be as large as the implementation supports, 128 terabytes on modern Windows [https://learn.microsoft.com/en-us/windows/win32/memory/memory-limits-for-windows-releases#memory-and-address-space-limits], physical memory is limited by how much we’ve put in the computer. Windows Server 2022 supports up to 48 terabytes of RAM [https://www.microsoft.com/en-us/windows-server] (oh the things I would do for access to a Windows server like that!), but most of us will likely be working with much less. So, how is 128 terabytes of virtual address space per process (and that’s just the user mode portion, there’s another 128 terabytes for kernel mode) squeezed into a comparatively minuscule amount of physical memory?

Let’s start by getting a feel for what the journey from virtual to physical memory looks like. This process is called translation and it is so performance-critical that special hardware has been invented to assist in carrying it out [https://en.wikipedia.org/wiki/Memory_management_unit]. A page of virtual memory’s location in physical memory is noted within a page table, in a page table entry or PTE. These page tables and PTEs must be in a specific format to work with the available hardware. Page tables themselves can take up a fair chunk of physical memory, and this cost can add up quickly. Since a page table must always be resident in physical memory for the translation to work (it would be hard to find where a page is in physical memory if the thing that tells you where the page is isn’t there!), we would end up taking up a large amount of physical memory just for the page tables themselves. To get around this, many systems (Windows included) create page tables of page tables. That way, only the top-level page table need stay in memory.

The top-level page table in Windows is the Page Directory Pointer Table (PDPT). The PDPT is always guaranteed to be resident in physical memory, and its physical address is stored within the process’ data structures and loaded into a processor register whenever one of the process’ threads is executing there. From the PDPT, one can find a PDPE (E standing for Entry), which will lead to the physical address of a Page Directory. From the Page Directory, one can find a PDE pointing to a Page Table. From the Page Table, one can find a PTE and finally the physical page of memory where the information resides. Once we have found the physical page, we can, at long last, get the content at the physical address using the offset of that address from the start of the process’ virtual address space.

But how do we account for this massive four levels of indirection efficiently? How do we go from table spanning 128 TB of virtual address space to a single address in physical memory? The answer is absolutely brilliant: We need nothing more than the virtual memory address itself! By assigning specific portions of the address to specific indices in each stage, we can go down this entire chain with just one number. I can’t help but mention that this was (in my opinion) the single coolest thing I learned in the entire jam. I’ve attached an image from Windows Internals showing this breakdown for x86 PAE virtual addresses (hopefully this won’t cause any DMCA notices to be sent to HMN). There is also a fantastic overview of virtual memory, which includes discussing this translation in particular (in a more general sense) linked in a previous post in this project.

So, we can now translate a virtual address to a physical address, assuming the virtual address indeed resides in physical memory. But we already know it may very well not. It might be a file on disk, or it might be a page that has been sent out to the page file. When this happens, it is called a page fault. When a page fault occurs, the memory manager is summoned to deal with the problem. These faults can be as easy as giving the process another page of memory or expanding a thread stack, to so catastrophic it will crash the system. I’ve attached a table of the possible faults and their consequences to this post. Since these faults can occur for many reasons, the “page fault” counts in tools like Process Explorer can get quite high (Discord was sitting over one million after around 40 minutes of uptime on a system with ample physical memory). This is not necessarily something to be alarmed about, and does not indicate resource starvation. Most of these faults are harmless and occurring by design.

Now that we know how and when this translation is carried out, let’s look at how Windows manages physical memory.

!til &junkworks You have to explicitly initialize sockets with a special function call on Windows before you can start binding to ports or sending bytes around. See "Initializing the socket layer" from here for reference.

Memory Management in Windows - Part 1: Virtual Memory

Windows implements virtual memory via pages. A page is (typically) a 4 kilobyte [https://devblogs.microsoft.com/oldnewthing/20210510-00/?p=105200] chunk of memory, though on some architectures it is larger [https://devblogs.microsoft.com/oldnewthing/20040908-00/?p=37923]. Pages in a virtual address space can be in one of four states at any given time: free, reserved, committed, or shareable.

Free, as the name suggests, means that the page is not currently in use and is available for reservation and committal. Reserved pages have been earmarked for use, but are not yet backed by physical memory or the page file (together referred to as the “commit charge”, more on that when we discuss physical memory). Committed, often called private, pages are private to their process and allocated in physical memory or the page file. Finally, shareable pages are also allocated to physical memory or the page file, but these may be shared between processes. It is important to note that they are are not so by default, however. This is often noted by the distinction between shareable and shared.

Windows works on a demand-paged basis, which effectively means it isn’t going to give anything (even itself) any committed or shareable pages until it absolutely must do so. There are a few ways it finds out when this must be done, such as special guard pages present in a thread’s stack. Initially, a thread stack is just one page, with the next page being reserved and marked as a guard page. The number of pages allocated to the stack grows as this guard page is accessed. Likewise, when reserved memory is first accessed, a demand-zero fault is triggered, signaling that a reserved page is now in use and must be committed. When a page is committed, it is always in a zeroed state for anything in user mode. Occasionally, a non-zeroed page is given to something in kernel mode so as not to reduce the number of zeroed physical pages (more on this later).

While free, reserved, and committed pages are somewhat self-explanatory once one is familiar with the terminology, shareable pages are more complex. While it’s fairly easy to envision a programmer designating some portion of their virtual address space as a shared scratch space for multiple processes, much of shared memory is devoted to section objects. Section objects, or file mapping objects as they are called in Win32, are shareable portions of memory that either reside in physical memory, the page file, or in a file on disk. It’s important to note that a section object is not necessarily shared, as it may only be mapped into one process’ address space. Section objects backed by files on disk are called mapped files, and those backed by memory are called page-file-backed sections (even if there is no page file enabled). Pieces of files can be mapped selectively using a view, specified via a range.

This mapped file mechanism is obviously powerful, and quite heavily used. It allows many applications to take advantage of shared resources (DLL files for example), in addition to accessing their own image. However, it has an interesting side effect when it comes to getting a sense for the “true” memory usage of a process. Measuring private memory is often sufficient, but it may not tell the full story. If a process is the only process using a DLL, is that not part of the “memory usage” of that process? That shareable memory will go unaccounted for in the general “memory usage” section of most tools (task manager, for example). No doubt the easily obtainable measurement of memory private to a process is sufficient in most cases.

“Large” pages are also available to processes which support them. A large page is 2 megabytes on x64. Large pages have a few restrictions that normal pages do not: They must be reserved and committed in one operation, are always present in physical memory (there’s no file system support for paging large pages to disk), and are not considered part of the working set. They are, however, accounted for as part of the process’ private bytes. There are also huge pages, 1 gigabyte in size, available with newer versions of Windows 10. Per Windows Internals, huge pages are given automatically when more than 1 gigabyte of large pages is requested. It also appears to be possible to request huge pages explicitly via VirtualAlloc2.

Sources:
Mark Russinovich’s talk on virtual memory: https://www.youtube.com/watch?v=TrFEgHr72Yg.
Windows Internals 7th Edition, Part 1

!til &virtual-machine the 3 categories of instructions needed for a useful system seems to be arithmetic (bitwise, addition, multiplication, etc), control flow (i.e. jmp) and data flow (load, store, mov, etc).

!til &window-manager In windows applications there is an invisible border that shows resizing pointer when you hover over it. So if you want to draw a custom colored border around any window you must offset its size. The problem is some apps have found a way to not draw this border (vscode, slack, maybe other electron apps), so if you offset your custom border it wont look good for all apps.

!til &reveng-ann I was coding along with a popular article on ChatGPT by Stephen Wolfram. It suggests that tiny neural nets can't reproduce a certain function (using automated training). By reverse-engineering how some larger nets solved it, I discovered that actually the tiny nets can reproduce the function, and arbitrarily well. This discovery will be the basis of my jam entry.

!til &learningjam2024 Currently reading through this paper.
https://media.disneyanimation.com/uploads/production/publication_asset/48/asset/s2012_pbs_disney_brdf_notes_v3.pdf
The scope has shifted from understanding BRDF's to learn about material models based on the micro facet model, which are nowadays commonly employed in both games and films, what is usualy referred to as "physically based".
As mentioned before, BRDF simply denotes a function that returns reflectance based on view, and light direction, so to learn anything of substance we need to look at the models themselves. The time has come where I finally study lighting. I'm thankful for this jam giving me the incentive.

!til &compilers To improve the ease of GPU programming, this dissertation presents a system forfully-automatic parallelization for C and C++ codes for GPUs. The system consistsof a compiler and a run-time system. The compiler generates pipeline parallelizationsfor GPUs and the run-time system provides software-only shared memory. The maincontributions are: the first automatic data management and communication opti-mization framework for GPUs and the first automatic pipeline parallelization systemfor GPUs.

Reference: https://liberty.princeton.edu/Publications/phdthesis_tjablin.pdf

!til &compilers
I maintain my Compilers research and notes https://nixpienotes.notion.site/Everything-About-Compilers-8561aeae66384a60902961e422035f81
But will be sharing the facts and tidbits here too

!til &compilers Hey everyone,
I am very excited for this learning jam but due to my university exams i am not able to give much time on it but still I wanna give it a try and continue researching the compiler space as a part of this learning jam.

!til that &spectre is in principle possible for any kind of speculative execution, simply because if the CPU does the wrong thing, the behavior can be observed, whether by cache timing, or contention for internal CPU resources, or by monitoring power consumption...or basically anything else

!til &virtual-machine on the topic of IO, there's 3 schemes of communication with external devices: Interrupts, Ports and Memory Mapped IO. One of these is most superior for my use case; I'll reveal which one along with details about all 3 schemes in my livestream

!til &virtual-machine In order for my generic VM to be useful, it needs to be able to handle any computation, meaning Turing Completeness. Only once I have achieved that can I start to worry about MMIO and other devices. So far, the requirements for turing completeness appear to be (a) minimal arithmetic ops like ADD, NOT and XOR, and (b) a loop construct. With those, you can theoretically compute any arithmetic operation. There are some One Instructions Set Computers that achieve Turing Completeness (I'll go on a tangent about them on stream prob), but I want my VM to be tolerable to write ASM for, and enjoyable to program in general, so I think I'll need a few more

!til &virtual-machine Threaded code is a commonly used technique when implementing Forths primarily. It's an interesting strategy that I'll talk about Next Saturday 🙂

!til &virtual-machine In addition to register based machines, which provide memory and dedicated registers, there exists stack based machines which only provide memory and a stack pointer (typically at the end of memory) that is decremented when new values are pushed onto it. Register machines are arguably harder to construct, but produce faster results that can be optimized, whereas stack machines require fewer instructions and are simpler to implement, but are a lot slower. An example of a good stack VM that's so simple it's commonly programmed in it's assembly language is https://wiki.xxiivv.com/site/uxn.html

uxn has the design closest to what I'm looking for, but I'll talk more about that next week

!til &learningjam2024 The concept of BRDF's turned out simple enough. It's a function that returns the amount of light being reflected based on a given direction pointing towards the viewer and a direction pointing towards the light source. Whether they are vectors or angles based on surface normal seems to vary between sources.
A BRDF is meant to be a self contained thing in the rendering algorithm that can be swapped at will. Basically it's an abstraction.
In practice a BRDF implements some material model. Apparently it's common to combine one for diffuse and one for specular. Any old material model you might have heard of (Lambert, Phong etc) counts as a BRDF. Nowadays BRDF's are usually based on the micro facet model, both in games and film.
This is currently my primary source, which I'm still reading through.
https://boksajak.github.io/files/CrashCourseBRDF.pdf

!til &learning-jam-2024 on Linux cgroups and user namespaces weren't originally created with containers in mind

cgroups were created to stop programs fighting for control over the hardware,
namespaces were created to allow for setting quotas on ressource usage

they just happen to allow for containers when used together

!til Reducing the precision of performance.now() was a waste of time and only done to annoy me. &spectre

!til &junkworks Clients behind NAT that want to receive data via UDP from a server outside of the NAT must first send a dummy packet to the server they want to hear back from. Also the server has to send its response to the same port as the client's dummy packet port. And the client still might not even receive the server's packet because NAT can choose to send the client's packet out on a different port that the port that was specified by the client. 😩 #necessarybutnotsufficient

!til The origin of &modular-forms, even though today they're just a number theory thing, was through the physics of... elastic ribbons!?

!til &learning-jam I wrote a simple 6507 emulator. It only initializes and does not execute any opcode. The idea is to make a progressive emulator while making the markdown explanation.

!til &windowsmm Page tables are a big deal. Such a big deal, in fact, that there are special caches and hardware devoted entirely to working with them. Also, you can have page table entries that point to other page tables, so you can page out page tables. This video series has been tremendously helpful and I'd highly recommend it to anyone trying to grok virtual memory. https://www.youtube.com/watch?v=qcBIvnQt0Bw&list=PLiwt1iVUib9s2Uo5BeYmwkDFUh70fJPxX

!til There is a conceptually simpler way of figuring out which value was read out of bounds in a &spectre attack - just index using an in-bounds value immediately afterward and see if the access is fast or slow. If it's fast, and you know the value used for the index, then you know that the secret out-of-bounds value == your index, and you've extracted the information.

!til &learning-jam there is no difference between opcodes and data in the 6507, the only thing that makes a byte an opcode is the initialization. When the CPU initializes it already expects an opcode, so the first byte will be an opcode, that makes the code valid. there is always an origin point

!til &virtual-machine I made a simple lc3 vm in order to test my VM making skills. Unfortunately, I can't get the thing to actually work due to console input nonsense. But, I've learnt what I needed to from it (mainly the general architecture of a VM and their implementations) and I've already spent around 4hours on it. Next up is looking at differing VM designs. I might even program another VM for fun.

!til about the BTB and RSB...so many ways for CPUs to speculate &spectre

!til &virtual-machine Two's complement is the leading way to represent signed numbers in binary, and it's pretty simple. For example, the number 1110 unsigned is 14, but when treated as a signed number with two's complement 1110 means -6. The way we determine that number is pretty interesting. I'll explain on the livestream insha'Allah.

!til &virtual-machine CISC computers lead to simpler assembly but RISC computers are easier to design and implement. For this jam, I'll keep looking into RISC architectures.

!til &learning-jam-2024 contrary to what I thought there is no one "Container" thing at the kernel level that user level systems are interfaces to, every container system has it's own implementation using similar kernel mechanisms in similar but subtly different ways, and there is like 5 different major ones on linux alone

!til There are multiple variants of &spectre described in the original paper, not just the out-of-bounds read that I've seen discussed in many places. (Also the original paper is very clear and well-written.)

!til &learning-jam-2024 actually chroot and the concept of "I want to run it but not with the rest of my system" is basically as old as Unix

!til people were extracting data via cache timing attacks back in like 2005, and yet somehow &spectre was not recognized until 2017?

!til &learning-jam-2024 Linux Containers are actually related to FreeBSD Jails https://docs.freebsd.org/en/books/handbook/jails/

!til &learning-jam Today I bought a MOS 6507, so I'll make it my study subject for this jam!