When you want to send a picture of a cat to your buddy, get the latest weather, or watch a video, your browser sends a request to the website using a protocol called HTTP. HTTP messages are chunked up and sent over a protocol called TCP, which are routed to the destination by a protocol called IP.

This article, and others in our networking series, will walk you through every part of the web stack.

Heads up! If you're an absolute beginner to web programming, you may wish to start with this article.

Increasingly accurate depiction of the internet.

HTTP: Where your message lives

Browsers use HTTP (HyperText Transfer Protocol) to communicate with web servers. The browser uses HTTP to request and receive web pages, and to perform basic transactions, like updating your account, posting images, or getting the latest weather forecast.

Basically, HTTP defines a format for the messages that are sent over the internet. It describes the action being performed (like GET, PUT, or DELETE), the URL to perform the action on, and the data format being used, such as images, text, or JSON.

Read the following sections of Mozilla's HTTP overview:

  • Components of HTTP-based systems (excluding Proxies). This will teach you basic terminology.
  • HTTP flow and HTTP messages: This will teach you what HTTP requests and responses look like.

If you use a command-line program called curl, you can see the raw HTTP messages for yourself. Open Command Prompt on Windows or Terminal on macOS, then run the following:

curl -v example.org

You should get a result like the following:

> GET / HTTP/1.1
> Host: example.org
> [request info]
>
< HTTP/1.1 200 OK
< [response info]
<
[a bunch of HTML]

The lines prefixed by > are the HTTP request; the lines prefixed by < are the HTTP response. Try other websites and see what you get!

Now read Mozilla's article about HTTP messages to learn more about the HTTP format. (You can skip the section on HTTP/2 Frames.)

This is a good time to learn about several common HTTP headers that are frequently used in web requests: Host, User-Agent, Content-Type, and Content-Length. Don't worry if some terminology is still unfamiliar. Feel free to look up the definitions of any other headers you see in your test requests.

You can easily run an HTTP server on your own computer. Python is the easiest way to get started. Once you have Python installed, run the following on your command line:

python -m http.server

Then visit localhost:8000 in your browser. You should get a directory listing of the folder in which you are running Python. Python here is running an HTTP server on your local computer, and your browser is making HTTP requests to fetch data from it. You can try running curl -v localhost:8000 as well to see the details of the response.

Check your knowledge:

  • What is the structure of an HTTP request or response?
  • Which part of an HTTP request contains the domain name?
  • Which part of an HTTP request contains the rest of the URL?

TCP: Retry, retry, retry

HTTP messages are sent via a protocol called TCP (Transmission Control Protocol). TCP's job is to ensure that the message is delivered reliably—that all the data arrives in one piece.

TCP chunks messages into pieces called packets. These packets are individually sent across the internet, and each may take a different route to the destination. There are many reasons for this, but the primary one is simply to allow the network to be shared by many devices—by chunking messages into pieces, many messages can be interleaved and sent across the same network.

However, packets are not guaranteed to reach their destination. Packets can get lost, arrive out of order, or get corrupted along the way. Tom Scott covers some of this with his overview of the Two Generals problem:

Packets can get dropped along the way, or arrive out of order.

TCP is designed to deal with these problems, and therefore make it easier for programmers to write network code. Packets are assigned numbers, and the receiving end will ensure that all packets are passed to the application in order. The receiving end acknowledges each packet, and the sender will re-send any packets that are not acknowledged. TCP also automatically scales how fast it sends messages, so as to not overwhelm the network.

TCP packets also have a port, which tells the destination which program the packet is for. For example, when sending an HTTP request to Handmade Network, your computer will send a TCP message with port 443 to the Handmade Network server. On our end, our server will route the incoming TCP packets to our website, which is listening on port 443.

Watch the following videos to learn more about TCP:

Ben Eater explains at a high level the reason TCP exists and what problems it solves for programmers.

Eater delves deeper to explain the TCP connection and disconnection process.

Exercise time: make your own TCP server! Follow Bharat Chauhan's guide. In step 1, you will create a TCP server that echoes the data it receives. Then, you will expand it to handle HTTP messages.

IP: What's my address?

Underlying TCP is IP, the Internet Protocol. IP contains the essentials to deliver data from A to B, primarily the source IP address and destination IP address. Internet routers use IP addresses to route the packet across the internet.

Watch the following video to learn more about IP:

Ben Eater explains the fundamentals of internet routing.

Exercise time

So, how do you actually send the cat though? How do you send a real packet yourself? It's time to make that theory stick.

If you didn't do it above, go through Bharat Chauhan's article Writing an HTTP server from scratch. It will give you a good high-level view of what a web server actually does—critical knowledge for a variety of web applications.

Then, it's time to dive deep. Go through Beej's Guide to Sockets. You will learn how to create a server from the ground up (well, from the operating system up).

DNS: Wait, how do I get an IP?

The last really important bit for day-to-day network programming is DNS. The job of DNS is to help you find IP addresses for domain names, like "handmade.network".

For a high-level introduction to DNS, there's no one better than Julia Evans. Here are a few very short comics that give you the basics:

If these interest you, her full zine on DNS is well worth the money.

Now, learn how DNS works in practice by going through her "mess with dns" tool: https://messwithdns.net/

Finally, here are a couple other resources to round things out:

Ethernet and PHY: Rock bottom

Ok, so you've got some of the basics down, and you're ready for some serious spelunking? Let's talk bits and bytes.

Watch this series of videos by Ben Eater, stopping after video 8 (The Internet Protocol).


Written by Colin Davidson. Edited by Ben Visness. Art by Jacob Bell.