Ken Mills
Hello & Welcome to the Forum!

The modern manual switch/turnout blues.
This seems to be a problem across the board not just with Lionel Fast Track, but other scales too.
The problem of the plastic frog and not fully adjusted slide rails.
Here is where a set of jewelers file's come into play.  the frog needs to be filed down, it raises the wheel as  the train passes over it and if the slide rail isn't sitting tight, right off she goes.  With judicious filing & a little bit of bending  you'll be able roll through your switches without derailing.
FYI... I only use antique/vintage O gauge all metal manual switches. Other scales/gauges also operated with manual switches.
A bicycle brake handle & cable can be used to operate a manual switch that is difficult to reach.

Happy Rails.
dave