The panzoom library was being an absolute pain to work with. If building Pith has taught me anything it's that working with prebuilt packages is often not worth it if they're poorly documented, maintained, and would not be that difficult to put together on your own. Just spend the extra couple hours and build it yourself. Despite the best efforts of panzoom's nasty interface and thanks to @sydney's engineering prowess we got a much better system sorted out that zooms in perfectly to the center at nice clean intervals. Likely will be returning to this at a later date and building our own panning library so we don't have to keep dealing with panzoom's annoyances.
I also got rid of double clicking to zoom in, and we added an option to show a higher level view of the full board.