For model railroad applications at the hobbyist level where it's only going to be used a few hours a day or what not I would just go with button cells, you can get them cheap in bulk on Ebay... Run a bank of them and you can get enough power to run a decent amount of LEDs for a decent length of time...
Personally I would skip the on/off switch and just go with installing/uninstalling the LEDs if I went this way...
In the end it's really about what your end goal is, as Harald said there are commercially available options that will really simplify things... But, if you got into electronics you could do more...