When I got my skid loader there were a few side panels that were vibrating all over the place. Nothing mechanical, but just a lot of vibrations. I straightened out metal tabs and added rubber cushion to certain areas which stopped the noise and made a huge improvement. Case sure could have invested just a little more in any number of off-the-shelf motor mounts with 'real' cushion which would have made these machines a lot more pleasurable to use. I think just about all skid loaders have a certain amount of hydraulic 'whine' from the pump and valves.
I wonder if your machine is supposed to use HTO additive and if it's possible the previous owner didn't use it (due to the cost or just wasn't aware of it) and that might be possibly contributing to the noise?? You might poke your head into the service bay of your local Case dealer and ask one of the better mechanics if NOT using HTO additive to the hydraulic fluid (which is an anti-foaming agent and might contain other agents) could be contributing to the excessive hydraulic noise. Or if there is a specific fluid type it should use. Or, if the noise could be pointing to any type of wear or be a precursor to a larger (and more expensive) coming problem? You probably should know about it sooner than later so if that were the case, it may be an inexpensive adjustment, seal or part to deal with now vs a more expensive repair after something goes 'boing'!!.
If there were something simple contributing to a hydraulic motor wear issue and you didn't deal with it, fixing those can be extremely expensive -- like $2000+ ea! You might want to run it past the mechanic just to see. Maybe record the noise with your phone and see if he can detect something from that. Have you checked the fluid levels of the hydraulic motors (I lot of guys don't know they're supposed to check/change the fluid)? How about the hydraulic fluid ...have you changed it, and is the level good? Chain case fluid?? Adjustments at the wheels? Lube each wheel bearing (but don't pump the seal out!)?? Just throwing some things out there. Good luck!!