I’ve been surfing through the website for the German FREMO club America-N a lot lately. They manage a terrific website and within it a page outlining their method of manually controlling turnouts. They too are relying on a DPDT switch to both control the polarity at the frog and also to lock the blades in either position.
Here’s a link to their page:
At the heart of their design is a brass rod from the fascia. This rod is slid in and out to move the blades. What I liked was how they have separated out the switch and the wire to the turnout. This would make it a lot easier to install both and also build in some spring action to compensate for the broad throw of the switch and the much more limited travel an N scale turnout needs. Their method is pretty simple and makes terrific use of readily available parts. Check it out.