I have a very strange issue where the view in some games quickly wobbles left and right. It seems completely random because the same section will sometimes run smooth as butter and then go back to wobbling and I don't think it's performance because it happens in games like Adventure Climb VR that shouldn't impact my 3070 and 5600X much at all. I can play Into the Radius and heavily populated worlds in VRChat without it happening at all but I can't get rid of it in a few I'm really keen on trying.
I've noticed it in Adventure Climb VR, Half Life Alyx and Ruins Magus, do they have anything in common maybe? 

It's kinda hard to see in the clip but imagine that tenfold and in VR. Completely unplayable. Anyone have any ideas? I haven't been able to find anything while googling.
Thanks!
https://streamable.com/ua2q82