If you're running regular Linux and want to just customize kernel/userspace, then easiest way is to do NFS boot over network. Newer Pi's don't need sdcard at all for this. All you do is connect to network, and provide NFS setup on other machine in same network where you have Pi's root filesystem unpacked, and it'll use files from there. See:
This is very common how embedded devices are developed - as you can easily modify files on your host/development machine and just trigger reboot on device to use them next time.
If you want to write custom OS, then simplest way would be to use serial port. Write small bootloader than you flash to sdcard and use it permanently. And this small bootloader will request actual boot kernel/files sent over serial port. Then you connect serial port with USB to TTL cable to your PC where you run server that provides these files over serial port. For examples see:
As for debugging, the simplest thing would be to write code that can do gdbserver compatible protocol. Then you'll be able to use regular gdb to debug remotely from your host PC. Here's an example of such implementation (no idea how good or full-featured it is): https://github.com/dwelch67/raspberrypi/tree/master/armjtag
I don't know much about JTAG, never used it, but here's some information on it: https://github.com/dwelch67/raspberrypi/tree/master/armjtag
Also for debugging you can leverage qemu - as it has support for rpi emulation. Probably not all the hardware, but for basic stuff like bootloader/serial port setup you can easily debug that without real hardware, just with emulation.