handmade.network » Forums » Work-in-Progress » Thinking about starting a Handmade Bootloader project.
stevetranby
Steve Tranby
11 posts

Code Monkey

#11236 Thinking about starting a Handmade Bootloader project.
4 months, 2 weeks ago

Questions:
1. I was wondering if there's any interest in a written or video series on writing a real-mode bootloader?
2. Is anyone else already doing a project on Bootloaders or Kernel?
3. Do y'all prefer the stream of consciousness or edited videos? What about the upcoming educational written format instead?

I'd lean toward writing a series of blog posts or a short book (maybe even taking IntermezzOS book and writing the rest of it while adding a C/Rust toggle as well??), but I've never done video streaming/producing so that might be more interesting. Though both could also be a possibility.

Be brutally honest if you think this would be a waste of my and everyone else's time.

------------------------------------------------------------------------------------------------------------

I am definitely no expert, and haven't worked on OS kernels or drivers professionally, but I can pretend ;-p

Initial topics would include: x86, BIOS, assembly (NASM/registers/etc), real-mode memory models (segmentation), simple code loading BIN/COM/COFF, QEMU/Bochs, debugging, FAT12, Segment:Offset Memory, simple shell. (MacOS dev env.)

Future direction ideas: build/run within linux/win dev env, C/C++/Jai/Odin/Zig/Rust/Swift, UEFI, drivers, ELF linking/loading, protected/long modes, paging, multitasking, ring protection, user-mode, sys-calls, memory manager, BASIC/Lisp interpreter, VGA, RaspberryPi game launcher OS, PCI, sandboxing, networking, etc. Many of which would be more of overview showing code pulled from elsewhere - dumb to re-write everything, amirite.

Maybe cross reference with other low-level handmade projects or series.

------------------------------------------------------------------------------------------------------------

I've been mostly a passive participant here watching various streams and have visited this site/forum every so often for anything of interest, but I've been meaning to dive into something to add to the network of "knowledge" and the new education push also inspired me to try and move forward by posting this topic.

The instigator of this project is that recently, while taking breaks from Game Development work, I've have been at it again this time writing a real-mode boot loader. It involved re-learning "useless" stuff like 16-bit real mode and it's <sarcasm>glorious</sarcasm> segmentation memory model and calling BIOS interrupts among other things. This got me thinking maybe I should document the process more formally.

I did also previously worked on a simple c/c++ kernel TranbyOS (narcissistic much, eh), if one could even call it that, and it pulled from a lot of other sources (OSDev wiki, tutorials, manuals) so I didn't write it from scratch by any means, but it did involve learning, experiments, and trail+error.

I thought about posting along these lines when this website first launched, but noticed Pseudonym's (Andrew's) Handmade OS project. It appears that was short-lived, however, though I should probably look at his one saved stream as reference.

It'd be fun to see a full OS, or maybe something more along the lines of SteamOS, developed by the community, but that's for another time.

Future idea could be discussing and walking-through more mature projects:
Redox, IncludeOS, BareMetal, TempleOS, Minix, Haiku, ReactOS, etc.

===============================================================================

So, any thoughts?

Feel free to share any ideas, comments, warnings, criticism, or whatever that you have.

I see 0xdeadc0de everywhere
mmozeiko
Mārtiņš Možeiko
1404 posts
1 project
#11237 Thinking about starting a Handmade Bootloader project.
4 months, 2 weeks ago Edited by Mārtiņš Možeiko on March 9, 2017, 6:18 a.m.

I think MBR bootloaders and BIOS should just die completely. They are mostly dead already. UEFI is the way to go.

UEFI is so much more nicer to use. No more problems with active partitions. No more problems about conflicting bootloaders and which partition to install it, or Windows fighting with Linux. If anything I would prefer to see articles on UEFI loaders, capabilities, building then and debugging it. There are a ton of guides on BIOS bootloaders already out there, but not a lot on UEFI.

I run triple-boot Windows, OSX and Linux on my work laptop, and I don't even want to try to remember how to do that with MBR and bios. With UEFI it is trivial - just an EFI partition (which is FAT) and some config files for your EFI loader.
stevetranby
Steve Tranby
11 posts

Code Monkey

#11238 Thinking about starting a Handmade Bootloader project.
4 months, 2 weeks ago Edited by Steve Tranby on March 10, 2017, 12:16 p.m. Reason: typo

Thanks for the input; I definitely agree.

However, my goal is not to release a product, nor is it to explain how to triple-boot or use a bootloader to run your favorite operating system, sorry if my intent wasn't clear. My intent here is to produce educational content and I do like your idea of explaining how to write and debug a UEFI bootloader and if that's too small then start building it into an actual kernel targeting a few hardware platforms.

How about the process of writing low-level drivers or writing code to interface with hardware? I'm just trying to gauge how much of my goal is irrelevant and what if anything you think would be interesting or useful to future generations.

I guess my thought was that most of the Handmade Education projects presented in Abner's blog post, if not all of them, are technically re-writing the wheel and already well covered in tutorials, videos, and other mediums found through a simple search. Though the stated goal is to produce material that's more polished and thoughtful than everything else, which would be my goal.

All that said, I will always have concern of wasting other people's time, as Jonathan Blow is always quick to point out.

p.s. if I pursue anything maybe I'll try using your profiler on part or over the entire boot process ;)

I see 0xdeadc0de everywhere
mmozeiko
Mārtiņš Možeiko
1404 posts
1 project
#11239 Thinking about starting a Handmade Bootloader project.
4 months, 2 weeks ago Edited by Mārtiņš Možeiko on March 9, 2017, 8:11 a.m.

Yeah, I'm not suggesting making product. I actually mean explanation and guide how UEFI works and how you implement stuff there + a little bit how it boots real kernel (maybe show simple hello world). Because there's not a lot information out there on this topic. On MBR + regular kernels there is a ton.
In UEFI you can do a ton of more interesting stuff, than simple MBR bootloader. You can use graphics mode and draw fancy menu, use a mouse, usb devices & much more. There's a ton of stuff to show and tell about UEFI environment.
stevetranby
Steve Tranby
11 posts

Code Monkey

#11240 Thinking about starting a Handmade Bootloader project.
4 months, 2 weeks ago

Gotcha, yeah I definitely wanted to get into and learn UEFI as part of this a little later on, but I think you're right that it makes sense to start using it immediately as the core of the project. Thanks.

I see 0xdeadc0de everywhere
CaptainKraft
Jeremiah
143 posts
2 projects

Father, husband, C programmer, and Linux apologist. Think before you code.

#11243 Thinking about starting a Handmade Bootloader project.
4 months, 2 weeks ago

This is a really interesting project idea. It's not something I've even considered looking into, but you have piqued my interest to know more.

I say go for it.

mmozeiko:
I think MBR bootloaders and BIOS should just die completely. They are mostly dead already. UEFI is the way to go.

UEFI is so much more nicer to use. No more problems with active partitions. No more problems about conflicting bootloaders and which partition to install it, or Windows fighting with Linux. If anything I would prefer to see articles on UEFI loaders, capabilities, building then and debugging it. There are a ton of guides on BIOS bootloaders already out there, but not a lot on UEFI.

I run triple-boot Windows, OSX and Linux on my work laptop, and I don't even want to try to remember how to do that with MBR and bios. With UEFI it is trivial - just an EFI partition (which is FAT) and some config files for your EFI loader.


This is concerning because I've had issues with UEFI for a while. It started with trying to dual boot a Macbook and now it causes headaches on one of my desktop machines. MBR + BIOS just worked for me. I use Grub and didn't really have any issues.

I've since ditched the Macbook, but on my desktop Grub doesn't even detect Windows anymore. I have to mash the Delete key to get to the boot menu and manually select Windows from there (or Linux if Windows is the default).

I must be doing something seriously wrong.

Build a man a fire, he'll be warm for a day.
Set a man on fire, he'll be warm for the rest of his life.
mmozeiko
Mārtiņš Možeiko
1404 posts
1 project
#11247 Thinking about starting a Handmade Bootloader project.
4 months, 2 weeks ago Edited by Mārtiņš Možeiko on March 9, 2017, 6:58 p.m.

Well it probably depends on what kind of UEFI loader you used. I know its possible to use grub for that, but I think grub is too overcomplicated and in most cases nobody needs its complexity or features. I use systemd-boot (former gummiboot) everywhere. It's really as as simple as putting .efi files on EFI system partition and adding tiny config files. No need to edit one file and generate another like grub does.

The complexity is obvious if you look up documentation on both of them. For example, Arch wiki pages:
GRUB - https://wiki.archlinux.org/index.php/GRUB
systemd-boot - https://wiki.archlinux.org/index.php/systemd-boot

This is how my linux config file looks like (approximately):
1
2
3
4
title          Arch Linux
linux          /vmlinuz-linux
initrd         /initramfs-linux.img
options        root=PARTUUID=14420948-2cea-4de7-b042-40f67c618660 rw

And this is how Windows looks like:
1
2
title Windows
loader EFI/MICROSOFT/BOOT/BOOTMGFW.EFI

Very readable and easy to understand. Not like MBR where you need to care about which partition is active one, or which partition you need to specify in grub or whatever. With good UEFI loader all the config is readable as simple file paths in simple config files - whether you use systemd-boot, or something more fancy like rEFInd.

You can actually avoid using any UEFI loader, if you care only about booting Linux on your PC. Because kernel itself is a valid UEFI loader. So just configure UEFI bios to use vmlinuz-linux file as bootlaoder and everything will boot!

With MBR there is so many disadvantages. Sure you can configure it and make it work, I'm not saying it doesnt work. But it has more complex setup. Yes, grub hides some of complexity, but if something breaks or doesn't work properly, then it is more effort to debug problem (ask Kknewkles on IRC about his mbr grub setup experience :) For example, worrying about 4 primary and extended partitions. What happens when you reinstall Windows? It potentially corrupts grub bootloader if you have complex/non-standard setup, so you need to boot into recovery and fix everything. With UEFI it is as simple as simply choosing different loader during system startup, nothing else is corrupted. And so on... There's also secure boot option with UEFI if you are paranoid about security.
CaptainKraft
Jeremiah
143 posts
2 projects

Father, husband, C programmer, and Linux apologist. Think before you code.

#11249 Thinking about starting a Handmade Bootloader project.
4 months, 2 weeks ago

I like the simplicity of those config files. I'll be looking into changing over from Grub soon. Thanks!

Build a man a fire, he'll be warm for a day.
Set a man on fire, he'll be warm for the rest of his life.
godratio
Ray Garner
39 posts
#11253 Thinking about starting a Handmade Bootloader project.
4 months, 2 weeks ago

Im interested in seeing this move forward.
Mr4thDimention
Allen Webster
253 posts
2 projects

Heyo

#11255 Thinking about starting a Handmade Bootloader project.
4 months, 2 weeks ago Edited by Allen Webster on March 10, 2017, 2:57 p.m.

All I can say is that my understanding of "booting" is almost non-existent. Even after messing with boot processes for dual booting, I never really understood them. So I can't weigh in on what topics you should do, because I don't even know what the topics are. I vote for a beginner's guide to what happens during the boot process and a high level overview of what these different approaches even mean, and then moving on to your idea of writing your own "booty thingy". (I apologise for not being sure what term to put there.)
stevetranby
Steve Tranby
11 posts

Code Monkey

#11263 Thinking about starting a Handmade Bootloader project.
4 months, 2 weeks ago Edited by Steve Tranby on March 11, 2017, 2:14 a.m. Reason: finish thoughts

Haha, enjoyed this reply. What are your thoughts on the video vs stream vs written approaches? Since you've done (or will be doing) both. I lean toward written so I can continually edit and improve, and will likely start with it ... but maybe streaming a few research/coding sessions could be fun addendum. I like watching video even when there's code involved, but I do skip around, watch at 2x, and usually find some written copy or git repo when actually applying the knowledge from the video.

Oh, as far as your suggestions, I like the idea of doing both what happens (and why) as well as the how (UEFI, code, etc). I will refer to other material that already exists where possible and bring in summary info when I do. Thanks.

I see 0xdeadc0de everywhere
Mr4thDimention
Allen Webster
253 posts
2 projects

Heyo

#11265 Thinking about starting a Handmade Bootloader project.
4 months, 2 weeks ago

Hmm I think the very basic intro stuff should definitely be written. If you need to show an animation you can always add that as a part of the text, but a video/stream of explaining things just never matches the written thing. That way it's easy to search, read at your own pace, edit over time etc.

After that I don't really know enough about the topic to choose a format, they're all good, you should feel it out. For instance you can always do a couple streams and if it's not feeling right switch to another format.

Hope that helps somewhat!
Todd
93 posts

I use x86-64 assembly, Python, C, and SQL to fight malware.

#11268 Thinking about starting a Handmade Bootloader project.
4 months, 2 weeks ago

mmozeiko:
I run triple-boot Windows, OSX and Linux on my work laptop


Oh Martins... Our resident Handmade Chuck Norris.

I'd be interested in helping with the project as long as you're okay with a total noob helping and could at least explain what needs to get done, because I've not done any of what you said but I do know C at a moderate level and x86 assembly enough to get around, though with x86, I've spent more time reversing than forwarding :)

I'm the smartest man in Athens because I know that I know nothing. ~ Socrates