Hey guys,

I do web programming in my day job in Clojure (a LISP that runs on the JVM). I also have a little background in low level programming, mostly from messing around with microcontrollers, and a course in C at university. Web programming has a bad reputation, and for good reason - it's generally an inefficient, overly complex mess.

I was wondering how you approach web development for the Handmade site. Part of my interest in the Handmade community is how to take these ideas and apply them to web development / business applications; essentially how to do a better job, despite the horribleness.

So my questions are:

1) What is the site written in on the backend? What database are you using?

2) Do you need / want help? I'm happy to fix bugs, and add features, when I have time.

Hey Oliver,

I'm not the webmaster here but I just checked BuiltWith and Netcraftwhere you can see that it's pretty minimalistic and they've done a fantastic job with security as well since nearly nothing is exposed there! I'm sure Abner or one of the other guys can give you the full scoop.

I'm in the same boat as you, I'm currently working with ASP.NET MVC 5 / Entity Framework / Web API 2 right now and the whole me learning C thing / Handmade stuff is kind of a private endeavor, but I'm sad to see that it seems a lot of other native programmers sort of see web development as the armpit of programming. I don't think it needs to be that way and if you would be interested in working together on a cool project to try and improve the web dev world, I would be interested as well.

One thing I've really not enjoyed about web development, is it seems that most time is spent fighting with tools and frameworks rather than programming, whereas when I use C and write my own libraries, I tend to work on problems that I feel make me a better actual programmer rather than learning some other peoples' frameworks if that makes sense.

Hey Todd,

Thanks for the reply. Nginx is always a good choice. :) I'm hoping that Django is a mistake though. I checked my companies website, and it said we are using Rails, completely untrue, so I guess their introspection isn't that accurate always.

I'm definitely interested in cool web projects to improve the web dev world. There's so much we can learn from native programmers for sure.

I agree with the frameworks/tools issues - that really annoys me too. These days I've been trying to minimize my use of them in personal projects. I think the real reason for the success of frameworks like Rails etc is simply consistent naming, and knowing where everything is. It's definitely possible to get that without a framework too though. The Javascript world is particularly insane when it comes to frameworks/build tools/you name it. I'd like to experiment with doing plain JS single page apps, with basically no deps, and no fancy build tools.

Oliver,

I agree! And yes I found this site quite handy for that: https://plainjs.com/

They have code counterparts to everything that jQuery and several other popular libraries/frameworks does but in just plain JavaScript.

The only thing I think JavaScript really needs to improve on is the AJAX requests... They seem way outdated and totally not user-friendly which is why I think most people bring jQuery in. But IMO it's quite insane to drop the entire jQuery lib in just to make a few AJAX requests.

You know the ecosystem is a nightmare when you have to refer to a language as "vanilla" by default or else people assume you're in a framework, LOL!

That's an awesome resource, thanks for sharing. Bookmarked. :) Just what I've been looking for actually.

Yeah I agree, the AJAX isn't great, though it's probably not hard to wrap it yourself in a more pleasant way, without, as you said, dragging in the entire JQuery.

Hahah so true. Another thing I had a laugh about the other day was Facebook's new build tool "yarn". How do you install it?

"npm install yarn"

Reminds me of the Xzibit meme. Yo Dawg, I heard you like JS build tools. So I put a build tool, in your build tool... ;)

greywolve
That's an awesome resource, thanks for sharing. Bookmarked. :) Just what I've been looking for actually.

Yeah I agree, the AJAX isn't great, though it's probably not hard to wrap it yourself in a more pleasant way, without, as you said, dragging in the entire JQuery.

Hahah so true. Another thing I had a laugh about the other day was Facebook's new build tool "yarn". How do you install it?

"npm install yarn"

Reminds me of the Xzibit meme. Yo Dawg, I heard you like JS build tools. So I put a build tool, in your build tool... ;)

Hahaha oh you're gonna love this one: https://www.npmjs.com/package/is-positive

I don't know what's scarier... The fact that people need a package to determine if a number is positive, or the fact that there have been 1,625 downloads in the last month...

And 4 versions... Apparently it took until version 2-3 for them to realize 0 wasn't positive.

That's just crazy, I had a good laugh now. That's actually worse than leftpad, haha.

Things like that make me not want to be a web programmer, sigh, but I know we can do better.

There's even an issue on the github repo for that. "We need to improve the performance." And people discussing parts of the code. Crazy.

greywolve

There's even an issue on the github repo for that. "We need to improve the performance."

Hey man, determining if a number is positive is some serious business. We might have to get the interlock compare exchange action goin on.

greywolve
Hey guys,

I do web programming in my day job in Clojure (a LISP that runs on the JVM). I also have a little background in low level programming, mostly from messing around with microcontrollers, and a course in C at university. Web programming has a bad reputation, and for good reason - it's generally an inefficient, overly complex mess.

I was wondering how you approach web development for the Handmade site. Part of my interest in the Handmade community is how to take these ideas and apply them to web development / business applications; essentially how to do a better job, despite the horribleness.

So my questions are:

1) What is the site written in on the backend? What database are you using?

2) Do you need / want help? I'm happy to fix bugs, and add features, when I have time.

Hi Oliver,

The site's built in Python using a tiny bit from Django, backed by PostgreSQL. It's not a Django site in the sense that most Django sites are. Anyway, long story. After we hit v1, I'll probably end up writing a long form blog post series about the site backend to finish what I started with the discussion on how the 'unread markers' work.

Next year Miblo will take over maintenance from me, adding some more features we have planned post-v1. At that point I'll start on the rewrite to C and turn it into a codebase we can comfortably build on for the next two decades. That'll be built from the ground up with a certain technology in mind that'll help improve security further, but which also has a beneficial side effect on speed as a side effect. That's normally an either/or consideration, but we're doing a complete rethink of the way sites are built and coming at this from a rather unusual angle.

It's not the kind of 'new' that's the same as what came before with new window dressing as is generally the case in this field, but something built from the ground up with security in mind as the first pillar, making it hard for the programmer to introduce a problem by accident. Such measures would normally slow things down, but by leaning into it, we can do things more efficiently as well. I'm being purposefully vague about this until we have a working prototype of this technology, but the general idea behind it works out on paper.

Once work on v2 is underway I might very well start a new blog series on the underpinnings of the C version, because we will end up opening that source (under MIT or BSD) once it's feature complete. In case anyone questions whether it's a good idea to write a website in C, I'd ask them to consider what their safe language of choice is implemented in. ;-)

In any event, as to your other question, adding more people at this point would slow us down. We have a whole lot of volunteers to help out should we need it. I'll happily add your name to the list.

Best regards,
Jeroen

Kelimion

greywolve
We have a whole lot of volunteers to help out should we need it.

Sounds like a high-class problem. Must be nice.


On a more serious note, Kelimion, that sounds absolutely amazing and exciting and I can't wait to see what you guys come up with.

I'm sure you guys will run into all kinds of interesting flak along the way but that's what makes it fun and I read a quote yesterday:

The person who says it cannot be done should not interrupt the person doing it.

The fact is, there's a ton of room for improvement in web development, so have fun!

PS: This website is already done very well and in fact it is the fastest website I visit. Kudo's for the minimal overhead, it shows.

Todd
PS: This website is already done very well and in fact it is the fastest website I visit. Kudo's for the minimal overhead, it shows.

Thanks :)
There's still a number of rough edges to sand off, but we're getting there.

One of the reasons it's relatively speedy is that I spent a lot of time getting the data model just right. The Handmade Hero forums originally ran on Kunena (a Joomla addon), and while importing them I restructured how things were stored along the way. Leaving aside for a moment the impact of their code on performance, just structuring the data more sanely made for a marked improvement.

When I get around to that blog post series about the backend design, I'll dedicate a post to the data model and how splitting things up a certain way makes more sense than doing it another way, based on what data you want where (on which types of pages). Fairly straightforward stuff, but it may be beneficial to go into a bit of detail there.

how do you even program a site in C? where would one look to learn that skill?

Keyword to look up is "CGI".
Here's a few links with simple introduction on this:
https://www.cs.tut.fi/~jkorpela/forms/cgic.html
http://www6.uniovi.es/cscene/topics/web/cs2-12.xml.html

mmozeiko
Keyword to look up is "CGI".
Here's a few links with simple introduction on this:
https://www.cs.tut.fi/~jkorpela/forms/cgic.html
http://www6.uniovi.es/cscene/topics/web/cs2-12.xml.html

Or if you don't want to depend on the CGI server,

learn how server sockets and accepting incoming TCP connections works. Most tutorials on sockets will have a short section on that.

Then learning all the intricacies of the http protocol. It's big it's complex, ... It's web.

Then adding encryption (though you really shouldn't code encryption yourself, there's too much that you can do wrong that would break the encryption)

ratchetfreak

mmozeiko
Keyword to look up is "CGI".
Here's a few links with simple introduction on this:
https://www.cs.tut.fi/~jkorpela/forms/cgic.html
http://www6.uniovi.es/cscene/topics/web/cs2-12.xml.html

(though you really shouldn't code encryption yourself, there's too much that you can do wrong that would break the encryption)

I swear I hear this so much, I almost want to spend like 10 years studying encryption and mathematics just specifically so I can roll my own for the challenge of it, dammit. :P


On a more serious note though, anyone have any idea why web has become so complicated? When I'm programming for web, I honestly rarely ever enjoy myself because it just seems overly-complicated. I feel like it's more battling with tools/frameworks than actual solving programming problems.