Unable to log into Portainer with initial password

[benjii]

Bug description
The password that we had been using for Portainer suddenly stopped working (I have been unable to determine why). However, in scouting around and discovering that there is no way to reset the password via the UI, I followed various threads to remedy the issue.

One of those was to remove the data directory / volume. I didn't do that, but I did remove portainer.db. This caused Portainer to re-prompt me for a password. I can enter it and create a new user. I have verified that a record is created for the admin user and the hashed password is present.

When I try to log in immediately after, however, I get Invalid Credentials.

Expected behavior
I should be able to log in (I think).

Steps to reproduce the issue:

1. Remove the portainer.db file;
2. Enter new password;
3. Try to log in with new password.

Technical details:

• Portainer version: (Unknown - I can't log in to find out).
• Docker version (managed by Portainer): 18.05.0-ce, build f150324
• Platform (windows/linux): Ubuntu 16.04
• Command used to start Portainer (docker run -p 9000:9000 portainer/portainer): Unknown (if I had this my first step would be to remove the existing service and re-create it).
• Browser: Chrome

Additional context
N/A

[ncresswell]

Are you using our RBAC extension? Please make sure to try to login using incognito/private mode to ensure no cached cookies remain.

[tr1et]

Hello,

We meet this issue also:

• 2 days ago, it works OK.
• Today, we can not login, it show error: "Invalid credentials".
• All 5 servers of us encounter this issue, browser cached or not, incognito or not.
• We don't use any extension.
• We have:
  • Removed the stack
  • Removed all Portainer containers
  • Removed the volume
  • Re-added the stack
  • Can not create Admin account, it somehow added it into the DB but failed, subsequent creations show error "There is already an admin account", and still can not login.
• This is the logs it it help:

portainer_portainer.1.lb6irooidv2w@ip-10-0-0-169    | 2019/08/19 07:32:01 server: Reverse tunnelling enabled
portainer_portainer.1.lb6irooidv2w@ip-10-0-0-169    | 2019/08/19 07:32:01 server: Fingerprint 97:8d:2d:ab:4b:2f:3c:2c:6d:2c:82:25:32:ae:f0:04
portainer_portainer.1.lb6irooidv2w@ip-10-0-0-169    | 2019/08/19 07:32:01 server: Listening on 0.0.0.0:8000...
portainer_portainer.1.lb6irooidv2w@ip-10-0-0-169    | 2019/08/19 07:32:01 Templates already registered inside the database. Skipping template import.
portainer_portainer.1.lb6irooidv2w@ip-10-0-0-169    | 2019/08/19 07:32:01 server: Reverse tunnelling enabled
portainer_portainer.1.lb6irooidv2w@ip-10-0-0-169    | 2019/08/19 07:32:01 server: Fingerprint 97:8d:2d:ab:4b:2f:3c:2c:6d:2c:82:25:32:ae:f0:04
portainer_portainer.1.lb6irooidv2w@ip-10-0-0-169    | 2019/08/19 07:32:01 server: Listening on 0.0.0.0:8000...
portainer_portainer.1.lb6irooidv2w@ip-10-0-0-169    | 2019/08/19 07:32:01 Starting Portainer 1.22.0 on :9000
portainer_portainer.1.lb6irooidv2w@ip-10-0-0-169    | 2019/08/19 07:32:01 [DEBUG] [chisel, monitoring] [check_interval_seconds: 10.000000] [message: starting tunnel management process]
portainer_portainer.1.lb6irooidv2w@ip-10-0-0-169    | 2019/08/19 07:32:02 http error: Invalid credentials (err=Unauthorized) (code=422)
portainer_portainer.1.lb6irooidv2w@ip-10-0-0-169    | 2019/08/19 07:32:03 http error: Invalid credentials (err=Unauthorized) (code=422)
portainer_portainer.1.lb6irooidv2w@ip-10-0-0-169    | 2019/08/19 07:32:04 http error: Invalid credentials (err=Unauthorized) (code=422)
portainer_portainer.1.lb6irooidv2w@ip-10-0-0-169    | 2019/08/19 07:32:04 http error: Invalid credentials (err=Unauthorized) (code=422)
portainer_portainer.1.lb6irooidv2w@ip-10-0-0-169    | 2019/08/19 07:32:04 http error: Invalid credentials (err=Unauthorized) (code=422)
portainer_portainer.1.lb6irooidv2w@ip-10-0-0-169    | 2019/08/19 07:32:05 http error: Invalid credentials (err=Unauthorized) (code=422)
portainer_portainer.1.lb6irooidv2w@ip-10-0-0-169    | 2019/08/19 07:32:05 http error: Invalid credentials (err=Unauthorized) (code=422)
portainer_portainer.1.lb6irooidv2w@ip-10-0-0-169    | 2019/08/19 07:32:05 http error: Invalid credentials (err=Unauthorized) (code=422)
portainer_portainer.1.lb6irooidv2w@ip-10-0-0-169    | 2019/08/19 07:32:05 http error: Invalid credentials (err=Unauthorized) (code=422)
portainer_portainer.1.lb6irooidv2w@ip-10-0-0-169    | 2019/08/19 07:32:07 http error: Invalid credentials (err=Unauthorized) (code=422)
portainer_portainer.1.lb6irooidv2w@ip-10-0-0-169    | 2019/08/19 07:32:08 http error: Invalid credentials (err=Unauthorized) (code=422)
portainer_portainer.1.lb6irooidv2w@ip-10-0-0-169    | 2019/08/19 07:32:09 http error: Invalid credentials (err=Unauthorized) (code=422)
portainer_portainer.1.lb6irooidv2w@ip-10-0-0-169    | 2019/08/19 07:32:10 http error: Invalid credentials (err=Unauthorized) (code=422)
portainer_portainer.1.lb6irooidv2w@ip-10-0-0-169    | 2019/08/19 07:32:10 http error: Invalid credentials (err=Unauthorized) (code=422)
portainer_portainer.1.lb6irooidv2w@ip-10-0-0-169    | 2019/08/19 07:32:10 http error: Invalid credentials (err=Unauthorized) (code=422)
portainer_portainer.1.lb6irooidv2w@ip-10-0-0-169    | 2019/08/19 07:32:11 http error: Invalid credentials (err=Unauthorized) (code=422)
portainer_portainer.1.lb6irooidv2w@ip-10-0-0-169    | 2019/08/19 07:32:11 http error: Invalid credentials (err=Unauthorized) (code=422)
portainer_portainer.1.lb6irooidv2w@ip-10-0-0-169    | 2019/08/19 07:32:11 http error: Invalid credentials (err=Unauthorized) (code=422)
portainer_portainer.1.lb6irooidv2w@ip-10-0-0-169    | 2019/08/19 07:32:13 http error: Invalid credentials (err=Unauthorized) (code=422)
portainer_portainer.1.lb6irooidv2w@ip-10-0-0-169    | 2019/08/19 07:32:13 http error: Invalid credentials (err=Unauthorized) (code=422)
portainer_portainer.1.lb6irooidv2w@ip-10-0-0-169    | 2019/08/19 07:32:14 http error: Invalid credentials (err=Unauthorized) (code=422)
portainer_portainer.1.lb6irooidv2w@ip-10-0-0-169    | 2019/08/19 07:32:15 http error: Invalid credentials (err=Unauthorized) (code=422)
portainer_portainer.1.lb6irooidv2w@ip-10-0-0-169    | 2019/08/19 07:32:16 http error: Invalid credentials (err=Unauthorized) (code=422)
portainer_portainer.1.lb6irooidv2w@ip-10-0-0-169    | 2019/08/19 07:32:16 http error: Access denied (err=Access denied to resource) (code=403)
portainer_portainer.1.lb6irooidv2w@ip-10-0-0-169    | 2019/08/19 07:32:16 http error: Access denied (err=Access denied to resource) (code=403)
portainer_portainer.1.lb6irooidv2w@ip-10-0-0-169    | 2019/08/19 07:32:17 http error: Access denied (err=Access denied to resource) (code=403)
portainer_portainer.1.lb6irooidv2w@ip-10-0-0-169    | 2019/08/19 07:32:17 http error: Access denied (err=Access denied to resource) (code=403)
portainer_portainer.1.lb6irooidv2w@ip-10-0-0-169    | 2019/08/19 07:32:19 http error: Access denied (err=Access denied to resource) (code=403)
portainer_portainer.1.lb6irooidv2w@ip-10-0-0-169    | 2019/08/19 07:32:19 http error: Access denied (err=Access denied to resource) (code=403)
portainer_portainer.1.lb6irooidv2w@ip-10-0-0-169    | 2019/08/19 07:32:19 http error: Access denied (err=Access denied to resource) (code=403)
portainer_portainer.1.lb6irooidv2w@ip-10-0-0-169    | 2019/08/19 07:32:20 http error: Access denied (err=Access denied to resource) (code=403)

[benjii]

Yes, similar experience and I do not use any extension. I have not gone so far as to remove and re-create the service, but it's next on my list. Here are my logs:

{"log":"2019/08/19 05:23:10 Instance already has defined endpoints. Skipping the endpoint defined via CLI.\n","stream":"stderr","time":"2019-08-19T05:23:10.443397742Z"}
{"log":"2019/08/19 05:23:10 Starting Portainer 1.17.1 on :9000\n","stream":"stderr","time":"2019-08-19T05:23:10.443426242Z"}
{"log":"2019/08/19 05:23:11 http error: Invalid credentials (code=422)\n","stream":"stderr","time":"2019-08-19T05:23:11.372841271Z"}
{"log":"2019/08/19 05:23:11 http error: Invalid credentials (code=422)\n","stream":"stderr","time":"2019-08-19T05:23:11.67974224Z"}
{"log":"2019/08/19 05:23:13 http error: Invalid credentials (code=422)\n","stream":"stderr","time":"2019-08-19T05:23:13.04716957Z"}
{"log":"2019/08/19 05:23:13 http error: Invalid credentials (code=422)\n","stream":"stderr","time":"2019-08-19T05:23:13.187370622Z"}
{"log":"2019/08/19 05:23:13 http error: Invalid credentials (code=422)\n","stream":"stderr","time":"2019-08-19T05:23:13.214197789Z"}
{"log":"2019/08/19 05:23:14 http error: Invalid credentials (code=422)\n","stream":"stderr","time":"2019-08-19T05:23:14.208023784Z"}
{"log":"2019/08/19 05:23:14 http error: Invalid credentials (code=422)\n","stream":"stderr","time":"2019-08-19T05:23:14.770695498Z"}
{"log":"2019/08/19 05:23:15 http error: Invalid credentials (code=422)\n","stream":"stderr","time":"2019-08-19T05:23:15.698891332Z"}
{"log":"2019/08/19 05:23:16 http error: Invalid credentials (code=422)\n","stream":"stderr","time":"2019-08-19T05:23:16.347750964Z"}
{"log":"2019/08/19 05:23:17 http error: Invalid credentials (code=422)\n","stream":"stderr","time":"2019-08-19T05:23:17.437348607Z"}
{"log":"2019/08/19 05:23:17 http error: Invalid credentials (code=422)\n","stream":"stderr","time":"2019-08-19T05:23:17.445793828Z"}
{"log":"2019/08/19 05:23:17 http error: Invalid credentials (code=422)\n","stream":"stderr","time":"2019-08-19T05:23:17.892114952Z"}
{"log":"2019/08/19 05:23:17 http error: Invalid credentials (code=422)\n","stream":"stderr","time":"2019-08-19T05:23:17.919705422Z"}
{"log":"2019/08/19 05:23:18 http error: Invalid credentials (code=422)\n","stream":"stderr","time":"2019-08-19T05:23:18.466256499Z"}
{"log":"2019/08/19 05:23:18 http error: Invalid credentials (code=422)\n","stream":"stderr","time":"2019-08-19T05:23:18.624945499Z"}
{"log":"2019/08/19 05:23:19 http error: Invalid credentials (code=422)\n","stream":"stderr","time":"2019-08-19T05:23:19.58887353Z"}
{"log":"2019/08/19 05:23:20 http error: Invalid credentials (code=422)\n","stream":"stderr","time":"2019-08-19T05:23:20.235247361Z"}
{"log":"2019/08/19 05:23:20 http error: Invalid credentials (code=422)\n","stream":"stderr","time":"2019-08-19T05:23:20.616137823Z"}
{"log":"2019/08/19 05:23:21 http error: Invalid credentials (code=422)\n","stream":"stderr","time":"2019-08-19T05:23:21.320939103Z"}
{"log":"2019/08/19 05:23:21 http error: Invalid credentials (code=422)\n","stream":"stderr","time":"2019-08-19T05:23:21.408390924Z"}
{"log":"2019/08/19 06:23:22 http error: Invalid credentials (code=422)\n","stream":"stderr","time":"2019-08-19T06:23:22.434644199Z"}
{"log":"2019/08/19 06:23:23 http error: Invalid credentials (code=422)\n","stream":"stderr","time":"2019-08-19T06:23:23.054537029Z"}
{"log":"2019/08/19 06:23:23 http error: Invalid credentials (code=422)\n","stream":"stderr","time":"2019-08-19T06:23:23.91635863Z"}
{"log":"2019/08/19 06:23:24 http error: Invalid credentials (code=422)\n","stream":"stderr","time":"2019-08-19T06:23:24.361425531Z"}
{"log":"2019/08/19 06:23:24 http error: Invalid credentials (code=422)\n","stream":"stderr","time":"2019-08-19T06:23:24.510555667Z"}
{"log":"2019/08/19 06:23:24 http error: Invalid credentials (code=422)\n","stream":"stderr","time":"2019-08-19T06:23:24.674391857Z"}
{"log":"2019/08/19 06:23:24 http error: Invalid credentials (code=422)\n","stream":"stderr","time":"2019-08-19T06:23:24.719887821Z"}
{"log":"2019/08/19 06:23:24 http error: Invalid credentials (code=422)\n","stream":"stderr","time":"2019-08-19T06:23:24.904282584Z"}
{"log":"2019/08/19 06:23:25 http error: Invalid credentials (code=422)\n","stream":"stderr","time":"2019-08-19T06:23:25.139330429Z"}
{"log":"2019/08/19 06:23:25 http error: Invalid credentials (code=422)\n","stream":"stderr","time":"2019-08-19T06:23:25.445523731Z"}
{"log":"2019/08/19 06:23:27 http error: Invalid credentials (code=422)\n","stream":"stderr","time":"2019-08-19T06:23:27.868985449Z"}
{"log":"2019/08/19 06:23:28 http error: Invalid credentials (code=422)\n","stream":"stderr","time":"2019-08-19T06:23:28.301130504Z"}
{"log":"2019/08/19 06:23:28 http error: Invalid credentials (code=422)\n","stream":"stderr","time":"2019-08-19T06:23:28.452370548Z"}
{"log":"2019/08/19 06:23:29 http error: Invalid credentials (code=422)\n","stream":"stderr","time":"2019-08-19T06:23:29.390418923Z"}
{"log":"2019/08/19 06:23:29 http error: Invalid credentials (code=422)\n","stream":"stderr","time":"2019-08-19T06:23:29.399156954Z"}
{"log":"2019/08/19 06:23:29 http error: Invalid credentials (code=422)\n","stream":"stderr","time":"2019-08-19T06:23:29.539396859Z"}
{"log":"2019/08/19 06:23:30 http error: Invalid credentials (code=422)\n","stream":"stderr","time":"2019-08-19T06:23:30.844305453Z"}
{"log":"2019/08/19 06:23:31 http error: Invalid credentials (code=422)\n","stream":"stderr","time":"2019-08-19T06:23:31.415278407Z"}
{"log":"2019/08/19 06:23:31 http error: Invalid credentials (code=422)\n","stream":"stderr","time":"2019-08-19T06:23:31.807825219Z"}
{"log":"2019/08/19 07:23:32 http error: Invalid credentials (code=422)\n","stream":"stderr","time":"2019-08-19T07:23:32.315005043Z"}
{"log":"2019/08/19 07:23:32 http error: Invalid credentials (code=422)\n","stream":"stderr","time":"2019-08-19T07:23:32.679308554Z"}
{"log":"2019/08/19 07:23:34 http error: Invalid credentials (code=422)\n","stream":"stderr","time":"2019-08-19T07:23:34.497780193Z"}
{"log":"2019/08/19 07:23:34 http error: Invalid credentials (code=422)\n","stream":"stderr","time":"2019-08-19T07:23:34.631475011Z"}
{"log":"2019/08/19 07:23:34 http error: Invalid credentials (code=422)\n","stream":"stderr","time":"2019-08-19T07:23:34.858071688Z"}
{"log":"2019/08/19 07:23:34 http error: Invalid credentials (code=422)\n","stream":"stderr","time":"2019-08-19T07:23:34.863498409Z"}
{"log":"2019/08/19 07:23:35 http error: Invalid credentials (code=422)\n","stream":"stderr","time":"2019-08-19T07:23:35.275695104Z"}
{"log":"2019/08/19 07:23:36 http error: Invalid credentials (code=422)\n","stream":"stderr","time":"2019-08-19T07:23:36.767514478Z"}
{"log":"2019/08/19 07:23:37 http error: Invalid credentials (code=422)\n","stream":"stderr","time":"2019-08-19T07:23:37.007494207Z"}
{"log":"2019/08/19 07:23:38 http error: Invalid credentials (code=422)\n","stream":"stderr","time":"2019-08-19T07:23:38.008948182Z"}
{"log":"2019/08/19 07:23:38 http error: Invalid credentials (code=422)\n","stream":"stderr","time":"2019-08-19T07:23:38.409140631Z"}
{"log":"2019/08/19 07:23:38 http error: Invalid credentials (code=422)\n","stream":"stderr","time":"2019-08-19T07:23:38.444295367Z"}
{"log":"2019/08/19 07:23:38 http error: Invalid credentials (code=422)\n","stream":"stderr","time":"2019-08-19T07:23:38.537400727Z"}
{"log":"2019/08/19 07:23:40 http error: Invalid credentials (code=422)\n","stream":"stderr","time":"2019-08-19T07:23:40.16890634Z"}
{"log":"2019/08/19 07:23:40 http error: Invalid credentials (code=422)\n","stream":"stderr","time":"2019-08-19T07:23:40.974040055Z"}
{"log":"2019/08/19 07:23:41 http error: Invalid credentials (code=422)\n","stream":"stderr","time":"2019-08-19T07:23:41.793170024Z"}
{"log":"2019/08/19 07:23:42 http error: Invalid credentials (code=422)\n","stream":"stderr","time":"2019-08-19T07:23:42.608454777Z"}
{"log":"2019/08/19 07:23:42 http error: Invalid credentials (code=422)\n","stream":"stderr","time":"2019-08-19T07:23:42.625010841Z"}
{"log":"2019/08/19 07:23:43 http error: Invalid credentials (code=422)\n","stream":"stderr","time":"2019-08-19T07:23:43.215951627Z"}
{"log":"2019/08/19 07:23:43 http error: Invalid credentials (code=422)\n","stream":"stderr","time":"2019-08-19T07:23:43.92249296Z"}
{"log":"2019/08/19 07:23:43 http error: Invalid credentials (code=422)\n","stream":"stderr","time":"2019-08-19T07:23:43.969532142Z"}
{"log":"2019/08/19 07:23:46 http error: Invalid credentials (code=422)\n","stream":"stderr","time":"2019-08-19T07:23:46.22873418Z"}
{"log":"2019/08/19 07:23:46 http error: Invalid credentials (code=422)\n","stream":"stderr","time":"2019-08-19T07:23:46.838754238Z"}
{"log":"2019/08/19 07:23:46 http error: Invalid credentials (code=422)\n","stream":"stderr","time":"2019-08-19T07:23:46.999787061Z"}

I used boltbrowser to manually set the (hashed) password in portainer.db. After doing so I manually checked that the hash that I put in place matched the password that I thought that it did (and it did).

And still no dice logging in...

Edit: and I did try incognito mode & different browsers... but again, no good.

[deviantony]

@benjii can you share the logs of the Portainer container with us? It will help us identify the portainer version as well.

@tr1et how did you deploy Portainer? Can you share the entire logs of the Portainer service?

[mattchatterley]

Hello,

We meet this issue also:

• 2 days ago, it works OK.
• Today, we can not login, it show error: "Invalid credentials".
• All 5 servers of us encounter this issue, browser cached or not, incognito or not.
• We don't use any extension.

Same here - it worked perfectly on friday but does not work today - we have rebooted our swarm but this hasn't helped. No server updates have been installed, nor have any containers/images been updated, so I'm baffled as to what has changed to cause this!

[i2egular]

I managed to fix this issue on my portainers by using 'sudo' with docker,
~~I've used both normal and stack deploy version. ~~

In normal version, just delete container (keep the volume) and start new one with "sudo" command

In stack version, remove stack (keep the volume) and download new stack yml file and start stack with "sudo" command.
docker stack rm portainer
curl -L https://downloads.portainer.io/portainer-agent-stack.yml -o portainer-agent-stack.yml
sudo docker stack deploy --compose-file=portainer-agent-stack.yml portainer

Now i can login with my user/pass again...

Updated - Now i am unable to login again

[leyk0001]

Exactly the same issue as described by @tr1et to the letter. I have tried using "sudo" as suggested by @i2egular but it did not help.

[deviantony]

Have you guys been able to restart the Portainer container/service to see if this fixes the issue?

@benjii have you restarted the portainer container after removing the portainer database?

[balta3]

A recreation of the portainer service does not fix anything, interestingly a restart of the portainer service did help for one out of our four instances

[balta3]

I have to correct myself: After restarting the portainer instance, logging in is working exactly once, this is reproducible on all systems

[deviantony]

@balta3 what version of Portainer are you using?

[balta3]

I was using 1.21.0 (Image from 2 months age, ID: da2759008147). Because of the problems of today I pulled the latest tag which is from 3 weeks ago, ID: 2b4ddf654e1c.

The UI is showing 1.21.0 still but in docker hub this seems to be 1.22.0

[benjii]

@deviantony I have pasted the trimmed logs below (trimmed because it's just the same message all the way through).

And to answer your question my procedure for resetting Portainer is as follow (my only Portainer instance runs on my master node);

1. docker service scale portainer=0
2. Remove the portainer.db file
3. docker service scale portainer=1
4. Enter new password when prompted by Portainer within the web UI.

And I have a similar experience as some others above; I can enter the password and it is accepted (I can even see it when I view the portainer.db file). But, as before, I can not log in with the new password.

And the logs...

2019/08/19 05:23:10 Instance already has defined endpoints. Skipping the endpoint defined via CLI.
2019/08/19 05:23:10 Starting Portainer 1.17.1 on :9000
2019/08/19 05:23:11 http error: Invalid credentials (code=422)
2019/08/19 05:23:11 http error: Invalid credentials (code=422)
2019/08/19 05:23:13 http error: Invalid credentials (code=422)
2019/08/19 05:23:13 http error: Invalid credentials (code=422)
2019/08/19 05:23:13 http error: Invalid credentials (code=422)
2019/08/19 05:23:14 http error: Invalid credentials (code=422)
2019/08/19 05:23:14 http error: Invalid credentials (code=422)
2019/08/19 05:23:15 http error: Invalid credentials (code=422)
2019/08/19 05:23:16 http error: Invalid credentials (code=422)
2019/08/19 05:23:17 http error: Invalid credentials (code=422)
2019/08/19 05:23:17 http error: Invalid credentials (code=422)
2019/08/19 05:23:17 http error: Invalid credentials (code=422)
2019/08/19 05:23:17 http error: Invalid credentials (code=422)
2019/08/19 05:23:18 http error: Invalid credentials (code=422)
2019/08/19 05:23:18 http error: Invalid credentials (code=422)
2019/08/19 05:23:19 http error: Invalid credentials (code=422)
2019/08/19 05:23:20 http error: Invalid credentials (code=422)
2019/08/19 05:23:20 http error: Invalid credentials (code=422)
2019/08/19 05:23:21 http error: Invalid credentials (code=422)
2019/08/19 05:23:21 http error: Invalid credentials (code=422)
2019/08/19 06:23:22 http error: Invalid credentials (code=422)
2019/08/19 06:23:23 http error: Invalid credentials (code=422)
2019/08/19 06:23:23 http error: Invalid credentials (code=422)
2019/08/19 06:23:24 http error: Invalid credentials (code=422)

Edit: I even tried a more "direct" way of updating the password. I manually created a hashed password and used sed to do a direct substitution in the portainer.db file. Even then, no good (and I even confirmed that the hash matched what I expected the password to be).

[jellingsen]

Experiencing the exact same issue on a 6 month old portainer container.

2-5 "Access denied (err=Access denied to resource) (code=403)" per second and sporadic "Invalid credentials (err=Unauthorized) (code=422)"

Is there a "lock-down" feature for possible brute-force attacks than could have been triggered?

[deviantony]

Regarding these http error: Access denied (err=Access denied to resource) (code=403) logs.

The only occasion that can return a 403 on login (via the /api/auth operation) is through the rate limiter: https://github.com/portainer/portainer/blob/develop/api/http/security/rate_limiter.go#L29-L38

It seems that the rate limiter reports the current user (IP) as banned and returns a 403.

We use the request RemoteAddr field to retrieve the IP that we check for ban and it seems that this could be problematic in a reverse proxy / load balancer environment: https://husobee.github.io/golang/ip-address/2015/12/17/remote-ip-go.html

I'm still trying to figure out why some users do not have this error message though and only have the invalid credentials message.

[RobertBarachini]

The issue is still present in the latest release. I've set up the Portainer docker instance in a new (Hyper-V) VM which I use exclusively for development and basic monitoring of running containers. The initial setup was successful and I was able to create a new user/password which worked fine the first few times.

OS: Kubuntu 22.04 LTS
Docker version: 20.10.17, build 100c701
portainer/portainer-ce:latest: DIGEST:sha256:9cd7e0bcea468af297c37ede624b9aa2e6168973a977045ad6236d5d8622d3a6

Set up script:

# From https://docs.portainer.io/start/install/server/docker/linux
sudo docker volume create portainer_data
sudo docker run -d -p 8000:8000 -p 9443:9443 --name portainer --restart=always -v /var/run/docker.sock:/var/run/docker.sock -v portainer_data:/data portainer/portainer-ce:latest

POST request to https://localhost:9443/api/auth returns status code 422 and the message containing "Invalid credentials". The user/password combo seemed to work fine the first few times + it is auto-filled from browser store / password manager.

My current "solution": removing the volume and reinstalling Portainer

[samdulam]

Share portainer logs when this issue is happening to help troubleshoot.

[RobertBarachini]

Unfortunately I've already wiped the entire setup before reinstalling but I'll provide logs if I can reproduce the bug.

[zx900930]

This can be temporarily fixed by docker restart portainer if you really want to regain access to portainer.
Sometimes the restart didn't work so you need to delete the portainer_data data volume and recreate it (This will reset portainer data, make a backup before doing this!).

For heimdall user, it's better to create a non admin user and give enviroment access to it and use that separate account to display the portainer status.
In this way your main account won't be locked out because of rate limits.

[slimshizn]

Just ran into this issue today. Here's the issue posted on Heimdall. Had to remove the login creds from Heimdall, then rebooted Heimdall, Portainer, and my Custom OAuth provider.

linuxserver/Heimdall#921

Heimdall absolutely hammers Portainer log in. Prob best to not even use that function.

As stated in the issue above, I'm not sure if it's a user related error, I was able to see the stats Heimdall puts out through logging into Portainer, even with OAuth, and hidden login.

[atsen-dev]

I've also the same issue, i'm using Portainer on a Docker container on a raspbery pi 4.
When i log for the first time, I put a password.

If just after in logout, i'm unable to login with previous password (and without any password too)

[naidinp]

You have this page : https://docs.portainer.io/advanced/reset-admin
This works for me Method 1 : Resseting the admin password if Portainer run as container
My config : Proxmox/Portainer/docker
Hope this helps.

[kamichal]

Yes, so there is May 2023 and it's still happening. Clearing web browser cookies for localhost helps each time, but at the next day - the problem goes back.

[moninformateur]

Complete UI redesign? Sure!

Fixing an actual 4 year old issue? Nah!

[Pablo-Aldana]

Same issue here, at first I thought something went wrong and reset my password, after the It is happening again. After four year it is something to be considered, as for a production environment, one cannot depend on portainer's will to access. ie: there is a P1 you cannot just hope that portioner won't ban you and you'll have to recreate the password.

docker restart didn't work
clear safari cache didn't work
go to chrome and use same password worked.

Safari logs:

api/endpoints?limit=1&sort=&order=asc
Status: 401 Unauthorized
Source: Network
Address: MYIP:443
Initiator: 
xhr.js:210

Request
GET /portainer/api/endpoints HTTP/1.1
Accept: application/json, text/plain, */*
Accept-Encoding: gzip, deflate, br
Accept-Language: en-GB,en;q=0.9
Connection: keep-alive
Cookie: _pk_id.1.1f37=dc8367e3c550d0f9.1697441083.; _pk_ses.1.1f37=1
Host: MYHOST
Referer: https://MYHOST/portainer/
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-origin
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.6 Safari/605.1.15

Response
HTTP/1.1 401 Unauthorized
Connection: keep-alive
Content-Length: 78
Content-Type: application/json
Date: Mon, 16 Oct 2023 07:26:28 GMT
Server: nginx/1.22.1
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block

Query String Parameters
limit: 1
sort
order: asc

[rit001]

As people are still seeing this issue and posting to this thread - depending on your environment

Does your infrastructure frontend your deployed Portainer environment with any session management/traffic routing tools? Cloudflare Zero Trust and Teleport are good examples of these types of tools.

Portainer is a thick JS client that runs in the Browser and a Server backend that communicates by API calls. So if the communications path is not fully open the client can show things like the login page, but can not complete the login process as the API calls can not be made. This is made more complicated by the fact that the client is very bad at reporting API layer issues.

For our environment which is front-ended by Cloudflare this showed up due to 'panic' config changes made due to Cloudflare's problems last weekend. Some of our session time limits had been cut back. Portainer on one system then started to fail logins and at times would not refresh content. All the menus were shown, but no data was being retrieved from the server, just large numbers of 'fail' messages.

I have raised a ticket saying that having the client show better error messages could help in this situation.

[kevinbull]

I've been dealing with an issue that's been very time-consuming to diagnose today. I've been getting sporadic 403 'Access Denied' errors, and restarting the Portainer container temporarily solves the problem. However, the issue returns after some time.

Upon further investigation, I've found that the 403 response is specific to the /api/auth endpoint, which is affected by an internal rate limiter. Would love something in the docs calling this out!! Legitimate requests exceeding the predetermined limit result in the banning of the requesting client's IP.

The apparent solution is to use access tokens when using the Portainer API. However, certain APIs, specifically the websocket APIs, require the JWT token. Getting the JWT token involves making calls to the /api/auth endpoint, which means being subjected to rate limits determined by Portainer. At the moment, with the current release 2.20.x, I don't see any way around this issue.

[nikz]

Hi @kevinbull - thanks for adding to the discussion.

I believe the rate limit on the /api/auth endpoint is set to 10 requests per second by default - are you able to let us know how many approximately you're making? Perhaps there's something not quite right there.

Additionally, the auth tokens themselves have a configurable duration (default to ~8 hours) and you can continue to use a single token for that time. So unless I'm misunderstanding, there should be no need to hit the endpoint every time you're making a request.

Let me know!

[kevinbull]

the problem lies in the fact that we have approx 100 services hitting a portainer instance and I don't control all of them. Hence being able rate limit failed requests would be nice. Or if everything is always internal the ability to turn off the rate limited completely would be nice.

[nikz]

Sure, and we can certainly queue up a task to make this configurable. My only suggestion here is that you may end up with further downstream problems by removing this safeguard.

I understand you don't control the services, but would it be possible to recommend to downstream users that they cache the generated JWT if they are hitting 429s?

[kevinbull]

Sure I can suggest/reccommend that downstream services do something but will they is another matter. Since the auth endpoint is the only endpoint being rate limited, could the websocket endpoints be made to accept an access token? We use access tokens for other requests but these endpoints seem to only allow the JWT token.
As an aside, it would be nice to allow JWT tokens that have been handed out to persist until they reach their expiration. I have noticed that as soon as the portainer container is restarted all JWT tokens are now invalid. This causes problems too and I suspect why some downstream clients have opted for getting a fresh token when one is required because even though the token they are holding purports to be valid it may not be if portainer has been restarted.

[nikz]

Yes - the JWTs expiring on restart is something we're aware of and tracking, so I will add your vote to that :)

[rit001]

Sadly if you dig into the sourcecode you will find that the rate limiter is hardcoded with the following values

 max requests 10
 duration time 1 second  (time over which the checks are executed)
 ban time 1 hour

I came across it while working for a small company, so it was not more than a future security issue as we had few staff and portainer was not published to the outside world.

Such values will never match any company's internal security policies and will cause problems if a gateway is put in front that causes all requests to come from the gateway's IP address.

All 3 values should be adjustable so that portainer can be configured to meet its deployer's requirements, with an additional option to just disable the feature.

[nikz]

Whereabouts are you seeing this? This line would indicate 10 per second - and has existed for about 6 years now 😂

[rit001]

Whereabouts are you seeing this? This line would indicate 10 per second - and has existed for about 6 years now 😂

Sorry that was a typo as I used the ban value twice, I've amended my post.

[Graimalkin]

Ran into this one today running Portainer CE locally on Docker. Password is stored in a pw manager, so it's not an inability to type thing.

Portainer | build_number=163 go_version=1.23.5 image_tag=2.27.0-linux-amd64 nodejs_version=18.20.6 version=2.27.0 webpack_version=5.88.2 yarn_version=1.22.22

I could not log into Portainer via a normal browser - cleared cache history, deleted cookies, nothing. I could log in with Chrome incognito mode of all things.

Ended up blowing away my instance to create new (re-used my old password). Still couldn't log into the non-incognito instance. Tried to reset my pw with the incognito instance where I was logged in - password does not match existing password error.

I also did some console tracing while I was at it. saw this being passed to the api/auth endpoint and found it pretty interesting. Something seems to be escaping the backslash in the password there. I didn't think to test this escaped pw, sorry. 🤷

Anyways, I've since reset my pw with the pw reset tool, so it's not in a bad state anymore and I can't test if that changed pw would work or not. But I did find the "password does not match existing password" error when I tried to reset it while logged in, and the password mismatch in the API call interesting.

[Pablo-Aldana]

The way I had to stop having this issue was to stop using Safari and using Portainer only in Chrome, even with the same password on Safari I was having the issue however on Chrome I could access without problem. I didn't have the time to look deeper, but at least is a "workaround".