The solenoid when receiving 12v allows the fuel into the system. When no 12volt, the spring on the solenoid itself should push the plunger down, cutting off fuel. If the plunger pushing down is not shutting off the fuel, somethings wrong in the pump. If the plunger is hanging up from rust/ setting, (I've seen that) it can hang up inside the solenoid, allowing the engine to still run. Turn it upside down (solenoid), dismounted from the pump, and spray some penetrating oil on it, and work it back and forth till it pops back down easily.
You can manually run the button that the plunger pushes against with the handle end of a screwdriver, just to make sure somethings not wrong inside the pump, maybe working that lever back and forth, but I bet your plunger is just hanging up.
The shut off, that the plunger controls could be gummed up/ crudded up from sitting also, and it may need a good cleaning out.
If its not that (pump/ fuel), then you're leading to all kinds of other possibilities, like that its not running on fuel, but engine oil, but you don't want to go down that path.