I figured out some more things that are starting to make me realize the power of FluidSynth as a synth engine. To start, I purchased one of the newly-released Pi 3's and set up my code on it. I was able to get audio.period-size down to 64 and audio.periods to 3 with no audio stutters/clicks, which gets the latency below 10ms - now playing feels "right".
I had been trying to figure out a way to modify FluidSynth to allow me to change synth parameters like the filter cutoff frequency in real-time using control changes. This was looking like a difficult task, but after a more careful reading of the soundfont spec I realized it's already in there. Modulators can be added to soundfonts to allow CCs to change things like filter cutoff, envelope attack time, LFO, etc. I used the polyphone soundfont editor to create a simple font with saw waves and modulators tying filter resonance and cutoff to CCs 71 and 74. After some trial-and-error with the parameter ranges, it was working exactly the way I wanted.
This shows me that the features that feel like they should exist often already do if I just know where to look for them. I don't have to write a new/customized synth engine - the real power of the SquishBox is the ability to describe performance setups - i.e. how my keyboard should control those synths.