As an optimization I incorporated some ray tracing into the ray marching routine.
First we trace a bounding box covering the objects and also trace the ground plane. If we miss the box then there's no need to ray march anything and the results of the plane trace is returned.
In the case of a box hit, the ray march will commence from the entry point until minimum distance or the exit point has been reached. In the case of the latter, the plane trace is returned as well.

Unsure about the performance gain. In standard Shadertoy editor mode it seems to be over 10% (according to Psensor readout) but less than that in full screen. At the very least I got a clean trace of the ground plane now.