The Arduino uses an off-the-shelf
ATmega chip. You can buy these for a few dollars each without the rest of the Arduino board. They may come with or without the Arduino bootloader already flashed.
In any case, you can flash them yourself with your custom software. You need to build a fairly simple circuit, hook it up to your computer's serial port (or USB adapter), and flash it using Atmel's software.
They have some competitors. For example, you could get a
PIC microcontroller. I never used one, but I assume the process is mostly the same: build the flashing circuit and use the manufacturer's compiler and flashing software.
To be honest, for the most part it's just more busy work compared to using an Arduino, with very little benefit. You'll be able to flash slightly larger programs (since you don't use the Arduino's bootloader), but it's not really going to teach you much about the hardware.
You can go more low-level than that, but you'll need to have a project in mind before you go and buy components.
You may be interested in Ben Eater's
work, who was mentioned in
Abner's education update.