The fact that those allen bolt heads are rounded off sounds alarm bells to me about the condition of the shaft and bushing even before you get the assembly disassembled .......
So if I understand you correctly the inside of the idler where the bush fits is worn down, correct..? I wouldn't expect the shaft or bush to wear so long as the idler still had some traces of lubricant in it.The shaft and bushing seem pretty solid, it had a bunch of really hard almost cement like crud inside the hub that wore it down