There's no pressure encouraging software engineering culture to converge on values that actually make sense and are true.

-Jonathan Blow https://youtu.be/dS6rCaDSwW8?t=956


In this programming stream excerpt, JB calls out a phenomenon I keep wondering about. Why do practices and values that result in inefficiently creating slow and unmaintainable software dominate the industry? Wouldn't you think they would get out-competed by more effective practices and values?

I'm curious what you think about why this is. A few possible answers:

1. This can only be a partial explanation, but (a) computers are fast enough to still sort of work even with all the bloated code we throw at them, and (b) users have been taught that software in general is low-quality. But this still doesn't explain why (c) we don't have new companies writing higher-quality software and out-competing the incumbents overnight, and (d) so many companies remain competitive even when standard practices make their software extremely expensive to maintain.
2. For upstream reasons relating to economic inequality, we're in a New Tech Startup Bubble. In short, there's just so much VC funding available that companies are not incentivized to spend it efficiently on evidence-based software engineering. Instead they use technologies that will generate the most hype among both investors and potential employees.
3. More effective practices will eventually dominate; evolution is just a slow process.
4. ?...

I feel like it's a combination of the above and other factors. Any other thoughts? I think about this all the time for some reason. :P

Competition is a complex topic, but here's why I think most companies don't compete on technical ability:

1. Most software is not functionally equivalent to most other software. Blender does not totally compete with Maya (yet?), Photoshop alternatives can't 100% replace Photoshop, Vim vs Emacs, etc..
2. Most of the competition happens in the UX space (in terms of features and looks). You would choose between alternatives based on the workflows they enable, not based on performance or stability (usually).
3. You only discover technical issues after an initial investment of time and/or money, and you don't really know if an alternative product would fare any better.
4. The baseline quality of software in the industry is very low, and most products cluster around that baseline, so you don't really get to see exceptional software very often.
5. Due to that common baseline, most programmers don't know that it's possible to produce higher-quality software with the same amount of effort.
6. Most companies (especially web companies) can afford inefficiencies in production and maintenance.

I don't think we'll see progress as a result of competition between companies.
You do see some competition between programmers. Every company is looking for good programmers and is willing to pay them a lot of money. But it's also pretty difficult to tell what makes a good programmer, or how to become one (I believe this is what Jon means when he talks about values).

So far most of the progress I've seen came through the hype machine. A few years ago Google taught webdevs that maintaining a 60fps framerate is important*. Nowadays a lot of people are switching to static typing.

We need to raise the quality baseline, and that push doesn't need to come from upper management. We need to redefine what it means to be proud of your work. This can be done by setting a good example, showing people that certain things are possible and within reach, and occasionally using the hype machine. Unfortunately this is a fairly slow process (unless you are a big name in your part of the industry) and I don't know how to accelerate it.

* This is a rare example where it can actually be attributed to Google's competition with Apple. Sadly Android is still full of inefficiencies in spite of that.

AsafG

I don't think we'll see progress as a result of competition between companies.
You do see some competition between programmers. Every company is looking for good programmers and is willing to pay them a lot of money. But it's also pretty difficult to tell what makes a good programmer, or how to become one (I believe this is what Jon means when he talks about values).

Plus it's really really easy to *seem* like a good programmer by simply preaching some ideology and flexing your pedantry during code reviews. Pragmatic people have a harder time selling their ideas because their ideas aren't flashy or interesting. They simply work.

> Why do practices and values that result in inefficiently creating slow and unmaintainable software dominate the industry?

Because slow and unmaintainable software is easier to replicate enmass than fast, maintainable software.

> Wouldn't you think they would get out-competed by more effective practices and values?

They are in landscapes where more effective practices and values are more viable.


Poor quality software is a short-term bet to cast a wide net to cover as much territory with the smallest resource investment possible. There is constant threat of being replaced by a superior approach, but only in small isolated pockets where the value trade off really pays off.

AsafG

2. Most of the competition happens in the UX space (in terms of features and looks). You would choose between alternatives based on the workflows they enable, not based on performance or stability (usually).
3. You only discover technical issues after an initial investment of time and/or money, and you don't really know if an alternative product would fare any better.
5. ... most programmers don't know that it's possible to produce higher-quality software with the same amount of effort.

Thanks for your thoughts, I think the above were especially important points.

You also note that "Most companies (especially web companies) can afford inefficiencies in production and maintenance," but how do they afford the inefficiencies? Is it due to the VC bubble / concentration of wealth that I mentioned?

Also a good point from tedbendixson. Maybe there still isn't as much crossover as there could be between people in charge of business/hiring and people who know what they're doing technically.

JesseMeyer

Poor quality software is a short-term bet to cast a wide net to cover as much territory with the smallest resource investment possible.

So IIUC, basically the profit motives are not set up to produce quality software for the most part. Get a new app to market, capture market share, and by the time your scale is causing unmaintainability to really bite you, your users are locked in. So this kind of relates to Asaf's #3 point.

(BTW, great avatar, I'm a big Alan Watts fan...)

I was also remiss in forgetting to mention Casey's 30-Million Line Problem talk, which is amazing for going way upstream on the technical side of this problem. Basically, computers could still be fundamentally simple to program, if they were fully spec'd at the hardware level (i.e. had instruction set architectures covering all functions, not just CPU operations). Because we rely instead on drivers to interface with hardware, operating systems become overly complex, but necessary in order to do things users care about, and then application developers are beholden to the complexity of operating systems which is out of their control, just as operating systems are beholden to the lack of hardware specs, which is out of their control...

According to Casey, the root problem is only solvable by chip manufacturers, and in order for them to make the necessary investments in a new line of fully spec'd hardware (i.e. a high-powered system on a chip with a stable ISA, and aimed at consumer PCs), they need to be confident in market demand.

But another talk Casey did on HMH about CPU architectures and how chips are actually fabbed gave me the impression that if you have a good chip design, it doesn't actually take that much capital (I mean, more than I have personally, but less than Intel has) to get a company in China or Taiwan to just make a bunch of it for you. That talk was really out of my area of expertise (though I was fascinated), so there's probably something I don't understand, but I came away from it thinking ... why has no one started their own line of fully spec'd hardware as called for in the 30-Million Line Problem talk?

winnythefoo
You also note that "Most companies (especially web companies) can afford inefficiencies in production and maintenance," but how do they afford the inefficiencies? Is it due to the VC bubble / concentration of wealth that I mentioned?

It's different for different companies. I suspect some of them are just VC backed and are trying to take over a segment of the market, or just waiting for a big IPO. I'm sure some of them also make a lot of money by selling data, but the richer companies probably just over-charge for their services, which isn't difficult to do since software scales so easily and there's not much apples-to-apples competition in services.

winnythefoo
So IIUC, basically the profit motives are not set up to produce quality software for the most part.

If we had a higher quality baseline we would get higher quality software at no additional cost to companies and without requiring any profit motives. Programmers tend to look at each other's work to determine what is "good enough". Companies/management are not the barrier here.

winnythefoo
According to Casey, the root problem is only solvable by chip manufacturers

I think Casey and Jon look at this problem from a bottom-up viewpoint. If you create a streamlined platform it would enable higher-quality software to be built on top of it, exceeding what is possible on today's platforms and making everything generally easier. However, this only really affects programmers who are currently fighting against the limitations of their platform. Personally I take the top-down approach (though I think the bottom-up approach will become vital as the industry improves, and improving that sooner would be better than later), where it's apparent that programmers are not even taking full advantage of the existing platforms. When you see a janky website it's not because it's impossible to implement that website in today's browsers, it's because of sloppy programming. I would like to see the industry get to a point where even programmers who work on the highest-level platforms (web/mobile/Unity/whatever) try to make the most out of their platform, even if that platform itself is inefficient. At that point they would be fighting against the limitations of their platform and I imagine we'll start seeing layers getting knocked out of the platform stack.

winnythefoo
That talk was really out of my area of expertise (though I was fascinated)

I think you'll enjoy this talk: https://www.youtube.com/watch?v=OqM2mSkSyFY

AsafG
When you see a janky website it's not because it's impossible to implement that website in today's browsers, it's because of sloppy programming. I would like to see the industry get to a point where even programmers who work on the highest-level platforms (web/mobile/Unity/whatever) try to make the most out of their platform, even if that platform itself is inefficient.

Working in the mobile telephone software application space, I can agree that it is mostly sloppy programming but it is also this crazy mixture of businesses making weird decisions and generally being unable to get their web api shit together.

Example: a previous client couldn't simply build a web service so we could pull all of a video asset's data with a single request to the server. Instead, we had to make like three separate requests, each to get one little piece (one thing to display it this way, another thing for analytics, etc.)

Most apps I work on have no web caching mechanism. We just straight up grab all of the stuff from the server every time the app launches.

So we aren't even making optimizations at the most *expensive* level, talking to a server. To get people to start thinking about whether such-and-such call into a native SDK is efficient or whether we're missing the cache is like a galaxy away from where we are now. You can barely get these business managers to acknowledge that things like React Native are slow and bug prone, just to provide some context.

In my experience, fellow mobile developers have been less lazy and more dogmatic/unthinking. They go full blast on some "pattern" they read about in a Medium article, and suddenly we're doing the MVVM "architecture" and there are all of these absurd layers of indirection for what should in principle be a simple app.

I've tried to sell people on Casey's idea of Compression Oriented Programming, but it's either blank stares of vehement rejection of the idea. I figured the speed and quality of my work would speak for itself, but people just gotta buy into whatever fad thing.

Hello,

winnythefoo
Why do practices and values that result in inefficiently creating slow and unmaintainable software dominate the industry? Wouldn't you think they would get out-competed by more effective practices and values?

I'm not sure I can find a single software product that dominates its field because of its efficiency and performance. All the dominating products I can think of are dominating because they arrived the first and competitors struggle to outperform them, and often used this opportunity to use aggressive business tactics (see Windows. Also works with Autodesk or Adobe products, which are taught in school, even when teaching Blender alongside would also make a lot of sense, and wouldn't lock students into a single technology. I've seen a professional artist REFUSE to switch from Maya to 3DS Max, even though that's what the company was most used too. I also saw a lot of freshly-out-of-school artists spit on Blender without being really able to explain why).
Some others, with some powerful marketing, manage to become number one (see Chrome, which everyone agreed was the fastest web browser when it came out, but now, even though it's become an unusable bloat, a lot of people are convinced there's no better alternative. And you can't expect users to constantly benchmark all competing browsers to find the best one).

So in my opinion, part of the state of dominant software products is caused by users being reluctant to change, and powerful companies convincing their users they don't have to change, and also that there's no proof I can think of that better programming practices can result in a more successful product or company.
(Maybe one whose products are directly targeted at competent programmers like RAD?)

Even among programmers, who are usually more computer-literate, most of the ones I know (and I consider a lot of them better than me) use the tools that are provided to them by default (Visual Studio's editor and debugger...) except when they need a lot of efficiency for builds. Benchmarking takes a lot of time that few companies are willing to allocate, and benchmarking so many tools before you realize you should make your own takes even more time. And it is hard to judge when to decide that all the small inefficiencies in the tools you use every day make it worth to look for something else.

https://thedailywtf.com/ is full of stories of companies that get away with using sloppily-developed slow tools, and no one seems to care when a concerned programmer makes it 10 times faster.

I could give one example of working on a, both mobile and desktop, 3D architecture design app, where people were more concerned that in Ted Bendixson's example (it sometimes also needed to download large stuff from a server, but at least we cached it and we stored a checksum both on the server and in the cache to decide if the cache was already up to date :)), which is probably because people didn't have the usual profile of mobile developers (most of them worked in games for years prior to this project, and the team was small enough that ones who worked on the heaviest parts of the app (3D engine, mesh-generation algorithms...) also worked on the most mundane ones (build systems, UIs, managing in-app purchases or ads...)).

There's no institution to do it yet.

Since global warming is really trendy among the kids right now, and it has strong elite backing all the way to the bankster level, I'd suggest someone set up a racket where devs pay your institution a fee to look at their code to see how Green Friendly it is, and then give it a grade they can use to virtue signal how Green they are on their advertising materials. They'd lose points for using horribly slow languages like Java or Python, memory managed languages and so on. The less resources their app uses, the better grade they'd get. If you get the woke crowd interested in this idea, they'd agitate to get laws put in place to tax the Brogrammers out of the business.

I've always resented having to replace perfectly good computers, so I can run software that does the same thing much slower, that faster software did better 10 years ago, on much slower computers. Its like watching property developers tear down perfectly good modest houses in my neighborhood to send to the garbage dump, and then they replace them with ugly low quality McMansions that cover every inch of the block, and will be falling down in 20 years.

Mór
There's no institution to do it yet.

Since global warming is really trendy among the kids right now, and it has strong elite backing all the way to the bankster level, I'd suggest someone set up a racket where devs pay your institution a fee to look at their code to see how Green Friendly it is, and then give it a grade they can use to virtue signal how Green they are on their advertising materials. They'd lose points for using horribly slow languages like Java or Python, memory managed languages and so on. The less resources their app uses, the better grade they'd get. If you get the woke crowd interested in this idea, they'd agitate to get laws put in place to tax the Brogrammers out of the business.

I've always resented having to replace perfectly good computers, so I can run software that does the same thing much slower, that faster software did better 10 years ago, on much slower computers. Its like watching property developers tear down perfectly good modest houses in my neighborhood to send to the garbage dump, and then they replace them with ugly low quality McMansions that cover every inch of the block, and will be falling down in 20 years.

Yes but we need to come up with some kind "meme" or "hashtag" for this thing, as the kids would have it.

#allocationshaming
#objectorientedplundering
#abstractasshole

???

I'm open to suggestions

winnythefoo
Get a new app to market, capture market share, and by the time your scale is causing unmaintainability to really bite you, your users are locked in. So this kind of relates to Asaf's #3 point.

This was basically the explicit strategy of my last employer. It wasn't about being the best, it was about being the first, even at the cost of being mediocre.

Amusingly, in this case, they were laughably bad at that too. When I was hired, they were "a few weeks out" from finishing and launching a key project of theirs. A year later they were still "a few weeks out". All their attempts to rush the process so they could ship yesterday just created more delays cleaning up messes and putting out fires. And they never seemed to learn anything from the experience.

Guntha
I'm not sure I can find a single software product that dominates its field because of its efficiency and performance. All the dominating products I can think of are dominating because they arrived the first and competitors struggle to outperform them, and often used this opportunity to use aggressive business tactics (see Windows. Also works with Autodesk or Adobe products

Yup. But let us remember the first rule of investment: never follow the crowd. When everyone's doing something because everyone's doing it, it becomes unsustainable, and huge opportunities open up for those who pivot in a different direction.

I think the same thing applies here. It may be that performance doesn't sell...right now. It may also be that efficiency will be the big selling point in the next few years.

Vinyl records were dead 10 years ago. Now vinyl is a mainstream music technology again. Trends reverse, and they're hard to predict.

Guntha
I also saw a lot of freshly-out-of-school artists spit on Blender without being really able to explain why

As someone who started out doing 3D art before becoming a programmer, I can tell you why. It's because up until recently, Blender's workflow and UX have sucked. There's been a lot of hype about Blender 2.8, and for good reason. It's a huge step up in those respects.

I was just thinking about this & i think its pretty much been covered but I'll mention it. I recently got a go pro & downloaded their quick editing software, but once I closed it I couldnt reopen it & it looking on the website its a well known bug, and they posted some things to try (which I couldn't get to work) but doesn't seem very urgent & then when I plugged the camera in & it doesn't detect it, even in their app (which I reinstalled to open again), so to get the footage off I've got to Bluetooth it via the iPad. I think its the first time when I thought not just could this be faster or stabler but how did this get out the door? But the reason I mention it is I bought go pro stocks a few years ago, & know they haven't been doing to well, but coming to the point now 😜, I have been reading 'the lean startup' by Eric reis & the advice in the book is to ship stuff before its ready, like the opposite of drug trials, to not lock in design choices & to stay relevant etc. (Sorry if its a bit oversimplified) in a super evolving industry. But when you look at it through this lens, I thought no wonder companies ship early because the whole company could go down, silicon valley advice of lean startup & 'hyper growth', keeping investors interested, & this is all at odds of creating reliable, stable software, its more of a good enough to get to the next round of funding.