>>401009552
>>401009678
I forgot to actually make my point because I was horny; Real time conversion is not far off. Processing power requirements are still pretty high, but it is very feasible that some dedicated research work (maybe someone's PhD) could bring processing cost down with a good model, and GPUs get fast enough to put it on some RTX cores for sub 100ms conersion times. 
The main problem with this is the problem of 'continuous feed' - You're going to have to buffer some audio to work on it; and the longer the sample the better the overall conversion will be; if you're converting 10ms chunks of audio it's going to be a mess but that's how you're gonna get 10ms latency on it. I think you could do something clever with a shifting window of recording though; like you take the latest 40ms of audio, combine it with the last 500ms of audio, and convert that entire block for a better conversion.
Or take part of the model's conversion from last time and apply it to the new block.