Yes, you can easily do that. Set the container name and make them on the same network. Used caddy and whole bunch of Selfhostable services with it and I reverse proxy as
One of the main reason I switched to podman was its compatibility with firewalld. Haven't used rootless docker, but podman and podman-compose gets the job done for me.
I am not self hosting an LLM, but running on my laptop with Alpaca. Google's Gemma 2B. On my hardware its pretty slow, but kind of gets the work done. My hardware is getting old, need to upgrade soon.
Left thing is the right thing