As others have mentioned, try to get yourself a decent base to set the pipe on. As you backfill make sure that you are using good quality material that you can get compaction on. Bring up lifts on each side evenly and compact as you go. You're already into this project for the excavation and pipe, don't cheap out and ruin the outcome by not backfilling with good material. This means hauling it in if the material on site isn't very good. I don't know how much water you're dealing with, but if the existing material around the culvert is soft and wet then you should try to remove any mud and replace with good fill.
I'm not sure what you mean with your 5' number, but if you're saying you have 5' cover on top of the pipe just make sure you compact as you go up to eliminate settling issues. If you only have 1' cover on top of the culvert, I'm not sure what the manufacturer recommends for a minimum coverage. You may have to add some fill on top to gain more coverage.