So it sounds like there's an almost broken connection someplace along the line. Hows the voltage at the battery with the key on? start there and work your way up along the power bus. Should be a larger gauge wire feeding the panel, perhaps a fuse is cracked. Also, with the wiring diagram, find all your grounds and give them a cleaning if they look/feel dodgy