>>414095825
>100mbit ports on a modern'ish router
the shit manufacturers skimp out on

It really shouldn't make much of a difference if your bitrate is low enough.
Your LAN ports are 100mbit: that's 100Megabits per second, not to be confused with 100MegaBytes per second, which is what 1gbit would be.
Quest1 can do up to 100mbit, quest2 150mbit, so lower your bitrate in VD and see if your latency improves. 
Keep in mind that there is a "automatically adjust bitrate" option in the streamer app that will override the setting your use in the quest app. 

Also enable the performance overlay in VD and see what it says; if network is taking ages you may want to swap the router. I'd be incredibly suspicious of anything with 100mbit ports in this day and age either way.