Internal server error after upgrade to 2.11

[dejantep]

Just upgraded from 2.10.6 to 2.11 end get internal server error

Netbox-rq is not starting

apr 16 20:30:24 netbox systemd[1]: netbox-rq.service: Main process exited, code=exited, status=1/FAILURE
apr 16 20:30:24 netbox systemd[1]: netbox-rq.service: Failed with result 'exit-code'.
root@netbox:/opt/netbox# systemctl status netbox-rq
â—� netbox-rq.service - NetBox Request Queue Worker
Loaded: loaded (/etc/systemd/system/netbox-rq.service; enabled; vendor preset: enabled)
Active: activating (auto-restart) (Result: exit-code) since Fri 2021-04-16 20:30:56 CEST; 1s ago
Docs: https://netbox.readthedocs.io/en/stable/
Process: 28865 ExecStart=/usr/bin/python3 /opt/netbox/netbox/manage.py rqworker (code=exited, status=1/FAILURE)
Main PID: 28865 (code=exited, status=1/FAILURE)

Cannot see any other errors in the upgrade procedure

[dejantep]

After checking journalctl it looked like tablib package was missing so i ran pip3 install -r requierments.txt

now i see stuff like this

apr 16 20:53:47 netbox gunicorn[19981]: File "", line 219, in _call_with_frames_removed
apr 16 20:53:47 netbox gunicorn[19981]: File "/opt/netbox/netbox/netbox/wsgi.py", line 7, in
apr 16 20:53:47 netbox gunicorn[19981]: application = get_wsgi_application()
apr 16 20:53:47 netbox gunicorn[19981]: File "/usr/local/lib/python3.6/dist-packages/django/core/wsgi.py", line 13, in get_wsgi_application
apr 16 20:53:47 netbox gunicorn[19981]: return WSGIHandler()
apr 16 20:53:47 netbox gunicorn[19981]: File "/usr/local/lib/python3.6/dist-packages/django/core/handlers/wsgi.py", line 127, in init
apr 16 20:53:47 netbox gunicorn[19981]: self.load_middleware()
apr 16 20:53:47 netbox gunicorn[19981]: File "/usr/local/lib/python3.6/dist-packages/django/core/handlers/base.py", line 40, in load_middleware
apr 16 20:53:47 netbox gunicorn[19981]: middleware = import_string(middleware_path)
apr 16 20:53:47 netbox gunicorn[19981]: File "/usr/local/lib/python3.6/dist-packages/django/utils/module_loading.py", line 17, in import_string
apr 16 20:53:47 netbox gunicorn[19981]: module = import_module(module_path)
apr 16 20:53:47 netbox gunicorn[19981]: File "/usr/lib/python3.6/importlib/init.py", line 126, in import_module
apr 16 20:53:47 netbox gunicorn[19981]: return _bootstrap._gcd_import(name[level:], package, level)
apr 16 20:53:47 netbox gunicorn[19981]: File "", line 994, in _gcd_import
apr 16 20:53:47 netbox gunicorn[19981]: File "", line 971, in _find_and_load
apr 16 20:53:47 netbox gunicorn[19981]: File "", line 955, in _find_and_load_unlocked
apr 16 20:53:47 netbox gunicorn[19981]: File "", line 665, in _load_unlocked
apr 16 20:53:47 netbox gunicorn[19981]: File "", line 678, in exec_module
apr 16 20:53:47 netbox gunicorn[19981]: File "", line 219, in _call_with_frames_removed
apr 16 20:53:47 netbox gunicorn[19981]: File "/opt/netbox/netbox/netbox/middleware.py", line 11, in
apr 16 20:53:47 netbox gunicorn[19981]: from netbox.views import server_error
apr 16 20:53:47 netbox gunicorn[19981]: File "/opt/netbox/netbox/netbox/views/init.py", line 26, in
apr 16 20:53:47 netbox gunicorn[19981]: from netbox.releases import get_latest_release
apr 16 20:53:47 netbox gunicorn[19981]: File "/opt/netbox/netbox/netbox/releases.py", line 7, in
apr 16 20:53:47 netbox gunicorn[19981]: from utilities.background_tasks import get_releases
apr 16 20:53:47 netbox gunicorn[19981]: File "/opt/netbox/netbox/utilities/background_tasks.py", line 13, in
apr 16 20:53:47 netbox gunicorn[19981]: @job('check_releases')
apr 16 20:53:47 netbox gunicorn[19981]: File "/usr/local/lib/python3.6/dist-packages/django_rq/decorators.py", line 28, in job
apr 16 20:53:47 netbox gunicorn[19981]: queue = get_queue(queue)
apr 16 20:53:47 netbox gunicorn[19981]: File "/usr/local/lib/python3.6/dist-packages/django_rq/queues.py", line 146, in get_queue
apr 16 20:53:47 netbox gunicorn[19981]: connection = get_connection(name)
apr 16 20:53:47 netbox gunicorn[19981]: File "/usr/local/lib/python3.6/dist-packages/django_rq/queues.py", line 123, in get_connection
apr 16 20:53:47 netbox gunicorn[19981]: return get_redis_connection(QUEUES[name], use_strict_redis)
apr 16 20:53:47 netbox gunicorn[19981]: File "/usr/local/lib/python3.6/dist-packages/django_rq/queues.py", line 115, in get_redis_connection
apr 16 20:53:47 netbox gunicorn[19981]: return redis_cls(host=config['HOST'], port=config['PORT'], db=config.get('DB', 0), username=config.get('USERNAME', None), password=config.get('PASSWORD'), ssl=config.get('SSL', False), ssl
apr 16 20:53:47 netbox gunicorn[19981]: TypeError: init() got an unexpected keyword argument 'username'
apr 16 20:53:47 netbox gunicorn[19981]: [2021-04-16 20:53:47 +0200] [20005] [INFO] Worker exiting (pid: 20005)
apr 16 20:53:47 netbox systemd[1]: netbox-rq.service: Main process exited, code=exited, status=1/FAILURE
apr 16 20:53:47 netbox systemd[1]: netbox-rq.service: Failed with result 'exit-code'.
apr 16 20:53:47 netbox gunicorn[19981]: [2021-04-16 20:53:47 +0200] [19981] [WARNING] Worker with pid 20005 was terminated due to signal 15
apr 16 20:53:47 netbox gunicorn[19981]: [2021-04-16 20:53:47 +0200] [19981] [WARNING] Worker with pid 19998 was terminated due to signal 15
apr 16 20:53:47 netbox gunicorn[19981]: [2021-04-16 20:53:47 +0200] [19981] [WARNING] Worker with pid 20001 was terminated due to signal 15
apr 16 20:53:47 netbox gunicorn[19981]: [2021-04-16 20:53:47 +0200] [19981] [WARNING] Worker with pid 19999 was terminated due to signal 15
apr 16 20:53:48 netbox gunicorn[19981]: [2021-04-16 20:53:48 +0200] [19981] [INFO] Shutting down: Master
apr 16 20:53:48 netbox gunicorn[19981]: [2021-04-16 20:53:48 +0200] [19981] [INFO] Reason: Worker failed to boot.
apr 16 20:53:48 netbox systemd[1]: netbox.service: Main process exited, code=exited, status=3/NOTIMPLEMENTED
apr 16 20:53:48 netbox systemd[1]: netbox.service: Failed with result 'exit-code'.

[dejantep]

yes, i believe i did but i might have done it first with wrong user and something didnt install properly

i see this as well >> AttributeError: 'Redis' object has no attribute 'connection'

[dejantep]

i use option with git repository

[candlerb]

Is this a git checkout of master branch which you upgraded using "git pull"? Or is this a tarball installation? Or something else?

Once you've answered this, we can suggest how to start again.

AttributeError: 'Redis' object has no attribute 'connection'

You didn't show the full error (including the backtrace which shows the source file line that this error occurred in), and you didn't show the exact commands you typed, so it's hard to provide any help.

Maybe you've got the wrong version of libraries installed in the virtualenv, or you untarred one version of Netbox on top of another version (which you can't do, as it would leave behind files which shouldn't be there), or maybe something else is wrong.

[candlerb]

OK, so show the full output of running

cd /opt/netbox
git pull
./upgrade.sh

[dejantep]

Here is the output as attached file
netbox_upgrade_error.txt

[markkuleinio]

Process: 28865 ExecStart=/usr/bin/python3 /opt/netbox/netbox/manage.py rqworker (code=exited, status=1/FAILURE)

Also ensure that your /etc/systemd/system/netbox-rq.service is correct: the python path does not point to the virtual environment.

There is the sample: https://github.com/netbox-community/netbox/blob/develop/contrib/netbox-rq.service

[dejantep]

hmm..it doesnt look like that example (e.g no venv in the path) but i have never touched that file

[candlerb]

Someone must have installed it in the first place :-)

Netbox 2.7 changed to using a virtualenv for all python dependencies. If you originally installed pre-2.7, and then upgraded, then you should have updated the system unit files.

[dejantep]

of course :)
but it worked fine as far as 2.10.6
Should i change as example above?

[candlerb]

Change both netbox.service and netbox-rq.service to modern versions. If it worked before, it was pure chance; probably only worked if you ended up installing packages globally (e.g. using pip3 install -r requirements.txt), which there was no need to do.

[dejantep]

guys it did work. 👍

Its up and running again. Thanks alot. I learned thing or two as well. Have a great weekend