I'm going to try your first approach, which seems to be what I want.
The second one looks tempting, but the first one seems to be more secure, I think.
If I use the same server to host games and apps, how am I going to redirect URLs to the correct reverse proxy ? I think I need a little more research, but I agree that putting the reverse proxy in the same place as what I'm hosting is a better idea.
On the other hand, if I just use tailscale, how can people connect to the game servers without having to install anything ? Or I just don't understand how it works.
I'm going to try the first approach of @Limonene.
I hope the amount of lag is not that high, and thank you for reminding me.