I took a step back and started working on a 1 dimensional circular convolution. I got it working, but still don't have the adjoint up and running.

You can see the stream here: https://www.youtube.com/watch?v=HHLcrmeuwhs