When do libraries go sour?
The Handmade community is often opposed to using libraries. But let's get more specific about why that can be, and whether that's reasonable. What do we look for in a library? When do libraries go sour? How do we evaluate libraries before using them? How can the libraries we make avoid these problems?
One example I can give is pgx: https://pkg.go.dev/github.com/jackc/pgx/v4
It's a postgres library for Go that we use for the Handmade Network website. The main thing that it solves for us is a knowledge-acquisition problem. It talks to postgres over its binary protocol, which to the best of my knowledge is undocumented. So doing it ourselves would require reverse engineering libpq (the official C library).
Interesting connection here, one important aspect of code is the fact that it captures knowledge. It's not explicit, direct knowledge - it's baked down quite a bit - but the knowledge is really the most useful part. So for stb_image, the useful part is not quite the API (you could imagine tweaking it), it's rather that decoding a PNG in a fairly good way is difficult enough that unpacking all the knowledge is fairly high-effort. Same with meow_hash or for that matter any other hash - producing a hash that has fairly good results for a given problem is a difficult process that requires expertise and care, and it's often quite orthogonal to the problem in question
For me personally, I've had the most success with "data-in, data-out" libraries - a few examples of which are well-known in the community. stb_image, meow_hash, stb_truetype
You also refer to them as "leaf libraries", right?
You also refer to them as "leaf libraries", right?
Yeah. I don't think libraries are only possibly useful as "leafs", but I think they have a higher chance of being useful when they are, if that makes sense.
a related concept I use, similar to "leaf", is whether or not using the library forces me into using their type system
4
To me leaf just means that it's replaceable in a bounded way without affecting the rest of the tree.
so a replacement would need to be compatible only in terms of API? it doesn't necessarily imply anything about the size correct?
if the library just accepts arrays of primitives (as an example) then any library could do that
For sure. I think that this also becomes much easier as an API designer when it's a leaf-like scenario
a related concept I use, similar to "leaf", is whether or not using the library forces me into using their type system
this is weird spot in C because the type system is limited enough that everyone and their moms must redefine a slice or other common structures
this is weird spot in C because the type system is limited enough that everyone and their moms must redefine a slice or other common structures
personally I rarely have a problem designing library interfaces that only rely on primitives, or on extremely shallow structs
2
Anyways I suppose this is going down an API design rabbit hole, which is maybe a bit off-topic
yeah I agree we shouldn't dig too deep on library design as such, but I think it highlights some of the aspects that contribute to code reuse decisions
1
so Ryan, wrt "Could we replace them easily if we needed to?", i think we've reached a spot where most of our foundational libraries probably can't (sometimes won't) be replaced by us but there's also a lot to be said that we don't necessarily need all of stb_image to get work done
Hmmm so what do you mean by "foundational libraries"?
but rarely does anyone replace stb_truetype with something that isn't a bigger and weirder library
like for all the text editor memeing we've had around here, it's rare for someone to make an stb_truetype replacement, they just use FreeType or something
like for all the text editor memeing we've had around here, it's rare for someone to make an stb_truetype replacement, they just use FreeType or something
but FreeType2 is a very big library...
Yes. And that diff tends to grow larger as your project grows.
1
One point I wanted to mention that's related is the idea that code is always written given a set of assumptions (or constraints). Properly analyzing a library requires understanding those assumptions. So for example, stb_image assumes that you trust your input data. (Same with stb_truetype). They assume a few other things (which are related to our earlier discussion on API design), and basically libraries become more problematic when the diff between the library's assumptions and your assumptions grows large.
well, I'm afraid that's the price you pay for the library
well, I'm afraid that's the price you pay for the library
sometimes you'll either compromise or pull a handmade or something
it's ok to have assumptions, it's just that it's also ok not to pick a library because yours dont line up with theirs
Yeah it's not only okay, it's required - you cannot write code given no constraints
1
Furthermore I think this gets at one aspect of the modern code reuse culture - if you pull in a gigantic DAG of libraries just by trying to grab one, you've introduced a gigantic bundle of assumptions that you have not vetted, and basically cannot vet
yea this is a big problem when it comes to threading especially in the types of languages we work in, like in our lovely imperative languages you need to understand data flow and mutability to be able to actually schedule things to happen in parallel, but once you lug around 20 libraries it only takes one of them being nasty about data flow for you to lose a bunch of potential performance gains
yea this is a big problem when it comes to threading especially in the types of languages we work in, like in our lovely imperative languages you need to understand data flow and mutability to be able to actually schedule things to happen in parallel, but once you lug around 20 libraries it only takes one of them being nasty about data flow for you to lose a bunch of potential performance gains
For sure
On this subject & connecting with Asaf's original prompt, I think clearly documenting the assumptions you make for a given API is probably a good place to start, w.r.t. making a library easily responsibly used. (And flipping it, when you're a user, trying to fish out the assumptions, and clearly documenting your project's)
i like the idea of making certain things the default:
* functions are pure (given the same input, it produces the same output)
* you must pass in a non-NULL pointer
* functions which take in isolated inputs are therefore thread-safe.
these are some simple ones but you can probably imagine more, and whenever things don't match this
we must document it.
1
It's something that people should pay attention to when vetting a library, but it's not something that's often talked about.
1
Oh, this touches on yet another thing - sorry, I'm sort of throwing all of my thoughts in and hoping one will stick - "code reuse" =/= "library that you cannot edit". There is another form of code reuse that arises, which is "I wrote a renderer in my last project, but I want to have a renderer in my new project - instead of trusting my old assumptions, or throwing the old renderer away, I can just duplicate it and mutate it as needed". This can be an invaluable time-saver, without the common drawbacks of overly-assumptive libraries.
2
Oh, this touches on yet another thing - sorry, I'm sort of throwing all of my thoughts in and hoping one will stick - "code reuse" =/= "library that you cannot edit". There is another form of code reuse that arises, which is "I wrote a renderer in my last project, but I want to have a renderer in my new project - instead of trusting my old assumptions, or throwing the old renderer away, I can just duplicate it and mutate it as needed". This can be an invaluable time-saver, without the common drawbacks of overly-assumptive libraries.
the problem I see with external libraries that you need to edit for your project is maintenance
the problem I see with external libraries that you need to edit for your project is maintenance
for this I think it's worth somehow distinguishing evolving vs stable libraries
for this I think it's worth somehow distinguishing evolving vs stable libraries
libraries can always change or need to change for specific projects needs
You can't know what will be added/fixed in the future, so you'd generally expect the library to be good enough as is, no?
1
You can't know what will be added/fixed in the future, so you'd generally expect the library to be good enough as is, no?
this is not my experience with how most people use libraries
Or more accurately, there's explicit anti-adaptability.
1
"it keeps getting better, and I don't have to do anything"
I imagine it depends on the type of library and the functionality provided, for example stb_image functionality is very focused, chances to have problems with it are low (with trusted input data)
libraries can always change or need to change for specific projects needs
sometimes you end up where you make changes into the internals of a library and since it's not in the trunk then you're kinda fucked even if just small changes comes by just because modifying inside of a library doesn't have the same stability guarentees as the API, this is where i'd just say "lmao dont update your tools" or "you might wanna consider just branching off into your own thing if the library is already doing what it needs to do"
1
sometimes you end up where you make changes into the internals of a library and since it's not in the trunk then you're kinda fucked even if just small changes comes by just because modifying inside of a library doesn't have the same stability guarentees as the API, this is where i'd just say "lmao dont update your tools" or "you might wanna consider just branching off into your own thing if the library is already doing what it needs to do"
you dont need to completely drop the library at this point, it's sour enough for you not to continue in their updates but still keep your modifications and the library... mostly
I think most people use libraries with the specific intent to benefit from rolling updates
this is true but in practice that promise isn't always kept, sometimes things get deprecated and after a while they just get removed
1
Just for concrete reference, @bvisness and I had to modify a markdown library because of an issue when compiling it to WASM. It was literally not workable without the change.
1
I agree with @demetrispanos, though, that this is how most people use libraries, and I think that is one place where the "Handmade narrative" explicitly disagrees with the general programming culture, at least in my mind. If code inside of your project is swept out from under your feet, and you have not tested with it, then in my mind it's quite unethical to ship that to users. Even if it's ostensibly "an improvement", that is really just a prediction, and that prediction is often wrong.
1
yeah this is my point really, obviously I'm aware of the ways it can go wrong but this is by far the dominant mode of use
1
because it's not that big it's probably something a person could in theory learn and maybe you dont need to drop it and we can approach with my based strat of forking stuff 
Yeah, I think sometimes this is necessary, especially when the "knowledge acquisition" aspect is important, but the offered API or other characteristics of a library fail to meet your constraints. I mean, obviously, always check the license on a library, but if it's MIT or something like that, then a very effective strategy is reusing & readapting the code that the library-author wrote for a given difficult/opaque data transform that deals with a difficult subject (reverse-engineering, undocumented formats, a subtle mathematical transform, etc.)
1
But then you'd make your own?
a new standard! 
this is spot where they got their stuff out of the way
but it's not a library most people might be willing to accept, i'd be willing to call it sour
I mean your own implementation for parsing DWARF for instance.
Right
Would you rather have that knowledge in the form of code that can be run/debugged, or in the form of a spec document?
It's hard to say. I guess it depends on what kind of code 
Because while the knowledge from e.g. Clang is useful, it's baked in with a bunch of other assumptions/constraints that are incorrect - e.g. assumptions that make things very slow, heavyweight, or unreliable
Clang can also be called sour but depending on how much you use it doesn't matter
and if it's an open source project then the documentation of the file format should also be open
1
I think if the code is reasonable and if it's possible to run/debug, that is indeed often preferable - documentation writing is a skill, and your documentation doesn't have a typechecker and you can't run/test it. That being said, I think the line between the two is more artificial than it could be - but that is perhaps a subject for another day
3
My question is should people push for more knowledge sharing in the form of documentation?
I think some documentation is important and useful but it could be redundant and verbose, a simple API cheatsheet and some commented code examples could work better
I think if the code is reasonable and if it's possible to run/debug, that is indeed often preferable - documentation writing is a skill, and your documentation doesn't have a typechecker and you can't run/test it. That being said, I think the line between the two is more artificial than it could be - but that is perhaps a subject for another day
Well, there's an emotional component to it. Like, if I really hate using it.
1
Well, there's an emotional component to it. Like, if I really hate using it.
then there's that
But primarily it's a question of how many bugs or other trouble is this going to introduce to my codebase over time.
Agree, I had that experience with one raylib external library, many related issues/missing features all the time
Agree, I had that experience with one raylib external library, many related issues/missing features all the time
that's the brings up a fun one, as much as i'd hate to admit it, rewriting massive mature projects is an easy way to introduce bugs they got resolved a long time ago in the older product, also means you dont have the ecosystem to engage with testing and potentially not even be able to share some testing suites... brings up the evil side of the question "when is a rewrite too sour to be done?"
A corollary question: How do you determine the limits of a library ahead of time?
you simply dont have the same resources usually which is a good type of limitation to a degree, ideally it means you focus on your use case because once you're rewriting it it doesn't need to be a library anymore, at least it doesn't if you dont want it to be one
A corollary question: How do you determine the limits of a library ahead of time?
It depends, I usually check for some "elements" that I consider useful, like no-memory allocations or memory allocation replacement option, same for external file accessors, option for memory accessors...
Looking at the docs is one thing, but do you also read much of the code?
I check the code, the exposed API and the internal API, personally I also check the coding conventions
physically recoiling at the idea of the code probably means you shouldn't rewrite, physically recoiling at the code is probably a reason to rewrite it, just out of being a moral person trying to make the world not suck as much
physically recoiling at the idea of the code probably means you shouldn't rewrite, physically recoiling at the code is probably a reason to rewrite it, just out of being a moral person trying to make the world not suck as much
There's also the idea of contributing to the library to make it better, but we can touch on that later.
On thing that I find is often missing from library docs is the mental model behind the library. But as @ryanfleury said, documentation is a skill and writing up the mental model is probably one of the harder aspects of it.
1
And on the subject of vetting, what can you do to make your library more easily vettable?
Well, no allocators customization or no memory data access is something I don't like
i say this like i haven't done this... im the problem sometimes
when they force you to do file I/O via their stuff ::((((
that's horrible... but it depends on the type of library...
i say this like i haven't done this... im the problem sometimes
me too
When you buy into a big library, you sell some personal responsibility away which can be good... until things go wrong
1
I think there's a point in every library where it stops making things easier and starts making things harder.
From my experience, that happens when you try to give too much control to the user
as much as we might meme about weird dependency stacks i feel like this might be a "justified stack", the low level users have one library and the high level people can wrap over it... sometimes
or in the reuse case, you factor out the BS to just handle your case
that's the brings up a fun one, as much as i'd hate to admit it, rewriting massive mature projects is an easy way to introduce bugs they got resolved a long time ago in the older product, also means you dont have the ecosystem to engage with testing and potentially not even be able to share some testing suites... brings up the evil side of the question "when is a rewrite too sour to be done?"
i brought up the side of "oh the rewrite might introduce bugs" but now i think i'll try to bring over the "a rewrite can be a chance to fix bugs" side
A lot of people are also worried about introducing bugs.
It's a legit worry, because most contributions come with no maintenance
It's a legit worry, because most contributions come with no maintenance
I was thinking more in terms of using "battle tested" vs writing your own from scratch.
1
And one thing that we haven't explicitly discussed is the aspect of saving time by using libraries.
you usually save time... if everything goes as expected...
I think people have the impression that any decently-used library had it's bugs worked out.
this is true in our perfect world but a lot of projects will grow in stars and popularity far faster than the project can actually """scale to enterprise"""
1
I think people have the impression that any decently-used library had it's bugs worked out.
it's not that they're all worked out, it's that many of them have been encountered
I've had "popular" libraries break just a few days after I started using them.
1
I think people have the impression that any decently-used library had it's bugs worked out.
Completely agree, far from reality
yea that's one of the weird spots which is that a lot of the people who might contribute to the popularity of a project aren't necessarily contributing or seriously engaging with that project
1
but you might be willing to deal with 10k lines you factored out of said project
I agree but I think this is kind of changing the goalpost ... if you can replace a 1M library with 10kloc you control, obviously that's good
I agree but I think this is kind of changing the goalpost ... if you can replace a 1M library with 10kloc you control, obviously that's good
not quite, im not saying rewrite 1M into 1kloc, im saying take out the relevant details and just have that in your rewrite
So another point for library evaluation: "how debuggable does this look?"
i'd just poke it with a debugger and if i cry then it's bad, it's really abstract for me
So another point for library evaluation: "how debuggable does this look?"
That's a good reason to take a look to code formatting and conventions before using it, just in case... following some code bases could be really complex
I want to touch on the time saving aspect a bit more, as this is a primary reason people go for libraries in the first place.
I think it's worth making the strongest possible argument for preferring existing libraries, so we can frame our criticisms or reactions
- save time coding it yourself, possibly a huge amount of time if it's a lot of code or if you'd need to learn a lot to do it
- get something higher quality than you would make yourself, because it is used in many different situations and has been subjected to selection pressure; also, if you are not an expert in the problem domain and the library author is
- get a built-in knowledge community you can ask for help with problems, since other people are using the library
- increase transferability of skills across teams, since the same external library can be used in many teams
4
We're at the two hours mark, so I'd like to switch gears and talk about the community's stance on library usage.
2
I think, at the very least, Handmade explicitly rejects the idea that you should rely on a library without having a fairly strong understanding of what it is doing, and being capable of at least learning how to replace it if necessary. Handmade promotes the idea of self-reliance and responsibility over what you ship to users, so if you use a library, you are responsible for what the library does
2
I think, at the very least, Handmade explicitly rejects the idea that you should rely on a library without having a fairly strong understanding of what it is doing, and being capable of at least learning how to replace it if necessary. Handmade promotes the idea of self-reliance and responsibility over what you ship to users, so if you use a library, you are responsible for what the library does
I'd agree with this but I think it's a process not really an end-state
It's not that you have to go study the PNG spec before using stb_image. It's that, if stb_image had some critical flaw for your purposes, you'd be capable of ditching it, when push comes to shove
1
It's not that you have to go study the PNG spec before using stb_image. It's that, if stb_image had some critical flaw for your purposes, you'd be capable of ditching it, when push comes to shove
well, I created my own single-file header-only PNG loader... 
well, I created my own single-file header-only PNG loader...
Yeah, and that may have made sense for your purposes - and you could do it!
I think for example if you get off the ground with stb_image and then someday decide you need to really be robust against possibly-hostile inputs, at that point you should be able to drop down and replace it. Or, I mean, hire someone who does, for that matter
2
(So even if you aren't willing to invest the time to go figure out PNG yourself, you're familiar with the nuances of the issue and what you'd do to avoid certain flaws in a library)
3
Because I agree that fundamentally, you can't know everything - and certainly every time you use code you haven't fully vetted, you could be relying on someone else's bad decisions. That's sort of unavoidable. But I guess Handmade's "take" - if there is one - would be just pointing out this fact, and so minimizing the drawbacks wherever possible. Not just shipping random software that you naively trust to users.
3
Computers should be programmable, and they should be as programmable as possible to as many people as possible.
2
2
2
Computers should be programmable, and they should be as programmable as possible to as many people as possible.
the problem that comes up is that if you make certain things easier it doesn't mean people will do them less, it means people can do them faster and will do them more often
Well I think it's not necessarily about changing everyone's minds by persuasion. Personally, I am sort of more in favor of forcing people to change their minds by competition. If Handmade ideas cannot reasonably outcompete everyone in an area, then hey, maybe people have just decided that the crappy drawbacks they're subject to are worth it, and the benefits from higher quality are not yet big enough.
yea this is why i really like seeing new projects pop by and why i think that we should invest some time into incremental change, there's a lot of things that exist in the culture and if you start to show people that you're right, they can actually move towards you
"we're some Assembly, C and C++ nerds who like rewrites :P" we don't look that good in a lot of ways but if you show someone remedybg you might turn some heads
1
I think this is why it's also important to advertise when doing things differently. Like how raylib mentions that it uses no external dependencies, and how it clearly improves the build process.
1
"we're some Assembly, C and C++ nerds who like rewrites :P" we don't look that good in a lot of ways but if you show someone remedybg you might turn some heads
NPM looks cool because you can do anything with it if you type the right install command, same with python and some imports... we dont have that sort of appeal of package management and we really dont need it... the solution to package management is just not to centralize it in my opinion, there doesn't need to be only one way to grab packages, another thing we run into is that we've separated stuff like cloning a repo from install a package and now it's more promoted to just use what you're being fed and not just fork and make your own changes or even really look into the codebase
I think expertise in using a library, or libraries, is its own skill
being good at a library, especially a big and popular one is really just the first step to understanding the status quo and from the status quo is where you learn the problem and how we can move forward... it's rare that a big project has such deep complaints someone with a surface level understanding can levy
Well, we've gone for nearly 3 hours, so I think it's a good time to call it.
2
Thanks for organizing @AsafGartner, was a fun conversation!
1