Portainer using remote nfs volume with docker fails to startup unless port 922 is also open in firewall

[bmguyii]

Ask a Question!

I cannot get Portainer using a remote nfs docker volume to complete "boot" unless I also open up port 922 in the firewall (see details below on the setup).

docker logs -f --until=2s portainer
2025/01/17 07:16PM INF github.com/portainer/portainer/api/cmd/portainer/main.go:370 > encryption key file not present | filename=portainer
2025/01/17 07:16PM INF github.com/portainer/portainer/api/cmd/portainer/main.go:393 > proceeding without encryption key |
2025/01/17 07:16PM INF github.com/portainer/portainer/api/database/boltdb/db.go:125 > loading PortainerDB | filename=portainer.db

never gets past this unless I open up port 922

If I do open port 922 (which I cannot find documented anywhere or related to nfs or docker anything), the "boot" completes and the interface comes up:

docker logs -f --until=2s portainer
2025/01/17 07:48PM INF github.com/portainer/portainer/api/cmd/portainer/main.go:370 > encryption key file not present | filename=portainer
2025/01/17 07:48PM INF github.com/portainer/portainer/api/cmd/portainer/main.go:393 > proceeding without encryption key |
2025/01/17 07:48PM INF github.com/portainer/portainer/api/database/boltdb/db.go:125 > loading PortainerDB | filename=portainer.db
2025/01/17 07:49PM INF github.com/portainer/portainer/api/chisel/service.go:200 > found Chisel private key file on disk | private-key=/data/chisel/private-key.pem
2025/01/17 19:49:03 server: Reverse tunnelling enabled
2025/01/17 19:49:03 server: Fingerprint {removed}=
2025/01/17 19:49:03 server: Listening on http://0.0.0.0:8000
2025/01/17 07:49PM INF github.com/portainer/portainer/api/datastore/postinit/migrate_post_init.go:107 > Executing post init migration for environment 2 |
2025/01/17 07:49PM INF github.com/portainer/portainer/api/cmd/portainer/main.go:655 > starting Portainer | build_number=137 go_version=go1.21.13 image_tag=linux-amd64-2.21.5 nodejs_version=18.20.5 version=2.21.5 webpack_version=5.88.2 yarn_version=1.22.22
2025/01/17 07:49PM INF github.com/portainer/portainer/api/http/server.go:370 > starting HTTPS server | bind_address=:9443
2025/01/17 07:49PM INF github.com/portainer/portainer/api/http/server.go:354 > starting HTTP server | bind_address=:9000

There's not a lot of details but could be stalled on chisel service not starting? Some attempt to verify no MITM condition?

Setup/recreation details:
truenas core server running on 192.168.0.40
proxmox 8.3.2 running a ubuntu server vm isolated in a vlan tag=1000 at 10.10.10.82
docker installed, nfs-common installed
firewall rule to allow traffic from 10.10.10.82 to 192.168.0.40 (ports 111, 2049, 712 allowed, all other inter-vlan traffic dropped, internet accessible)
(successfully) created a volume for portainer on a remote (192.168.0.40) truenas core nfs share

docker volume create --driver local --opt type=nfs --opt o=addr=192.168.0.40,rw --opt device=:/mnt/docker_vols/portainer_data portainer_data
docker run -d -p 8000:8000 -p 9443:9443 --name=portainer --restart=always -v /var/run/docker.sock:/var/run/docker.sock -v portainer_data:/data portainer/portainer-ce:2.21.5

(I can mount any nfs share from that truenas server to the filesystem through the firewall without issue - so it doesn't seem to be nfs port related).
Inspection of the truenas nfs share shows portainer created files etc on the volume.
I tshark captured with "drop inter-vlan traffic" enabled (meaning only ports 111,2049,712 allowed) where the boot stall occurs and noticed a request for port 922.
Adding 922 to the allowable list enabled a successful portainer "boot"completion with the interface coming up.
However, analysis of a tshark capture when "922 enabled" did not show any instances of port 922 actually being used.
I suspect that the issue has to do with Chisel checking for a MITM. But that's just a guess.

As a side experiment I repeated the build on the same subnet as the truenas server (i.e. doesn't have to go through the firewall) and things worked as expected.

I also tried a version where the docker volume was "local" and that did not have the issue (which would be expected since the firewall is not in play and all ports would be accessible to localhost).

Seems the issue only presents itself when using a docker volume nfs mounted across a firewall and the undocumented port 922 is blocked.