How to modify an existing device using a custom script

[WhaleJ84]

Hey all,

I'd like to start by saying I'd rather be taught how to teach myself the answer to these things but to keep to the point the situation is as so:

I'm developing a custom script that connects to LibreNMS (an SNMP monitoring system) and will update a given device with the SNMP information found.

My script at the moment is as so:

from django.utils.text import slugify
from librenms_handler.devices import Devices

from dcim.models import Device, DeviceType, Manufacturer, Platform
from extras.scripts import *


class TestScript(Script):
    class Meta:
        name = "Get device SNMP details"
        description = "Fetches relevant available SNMP information from LibreNMS"
        field_order = ['device_type', 'librenms_host', 'librenms_api_key']

    manufacturer = ObjectVar(
        model=Manufacturer,
        required=False
    )
    device_type = ObjectVar(
        required=False,
        description="Device model",
        model=DeviceType,
        display_field="model",
        query_params={
            "manufacturer_id": "$manufacturer"
        }
    )
    device_name = ObjectVar(
        description="Device name",
        model=Device,
        display_field="name",
        query_params={
            "manufacturer_id": "$manufacturer"
        }
    )
    librenms_host = StringVar(
        description="FQDN/IP address of LibreNMS host (e.g. https://librenms.example.com)"
    )
    # NOTE: The API key is sensitive data - obscuring the input like a password is advisable
    librenms_api_key = StringVar(
        description="API key to allow access to the LibreNMS instance"
    )

    def run(self, data, commit):
        # get LibreNMS data for device
        librenms_devices = Devices(
            data["librenms_host"],
            data["librenms_api_key"],
            verify=False
        )
        device_data = librenms_devices.get_device(data["device_name"])

        # Try to create the platform if it doesn't exist
        platform = Platform(
            name=device_data.json()["devices"][0]["version"],
            slug=slugify(device_data.json()["devices"][0]["version"])
        )
        try:
            platform.save()
            self.log_success(f"Created new platform: {platform}")
        except Exception:
            self.log_warning('The platform is already exists')

        # update the device
        updated_device = Device.objects.get(name=data["device_name"])
        updated_device.platform = device_data.json()["devices"][0]["version"]
        try:
            updated_device.save()
            self.log_success('wow it worked!')
        except Exception:
            self.log_warning(':(')

        if device_data.json()["status"] == "ok":
            return device_data.json()["devices"][0]
        return device_data.status_code

But what happens when I run the script is I get the following exception:

An exception occurred: TransactionManagementError: An error occurred in the current transaction. You can't execute queries until the end of the 'atomic' block.

Traceback (most recent call last):
  File "/opt/netbox-2.10.5/netbox/extras/scripts.py", line 438, in _run_script
    script.output = script.run(data=data, commit=commit)
  File "/opt/netbox-2.10.5/netbox/scripts/test.py", line 64, in run
    updated_device = Device.objects.get(name=data["device_name"])
  File "/opt/netbox/venv/lib/python3.8/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/opt/netbox/venv/lib/python3.8/site-packages/cacheops/query.py", line 353, in get
    return qs._no_monkey.get(qs, *args, **kwargs)
  File "/opt/netbox/venv/lib/python3.8/site-packages/django/db/models/query.py", line 425, in get
    num = len(clone)
  File "/opt/netbox/venv/lib/python3.8/site-packages/django/db/models/query.py", line 269, in __len__
    self._fetch_all()
  File "/opt/netbox/venv/lib/python3.8/site-packages/cacheops/query.py", line 283, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/opt/netbox/venv/lib/python3.8/site-packages/django/db/models/query.py", line 53, in __iter__
    results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
  File "/opt/netbox/venv/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1156, in execute_sql
    cursor.execute(sql, params)
  File "/opt/netbox/venv/lib/python3.8/site-packages/cacheops/transaction.py", line 93, in execute
    result = self._no_monkey.execute(self, sql, params)
  File "/opt/netbox/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/opt/netbox/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/opt/netbox/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 78, in _execute
    self.db.validate_no_broken_transaction()
  File "/opt/netbox/venv/lib/python3.8/site-packages/django/db/backends/base/base.py", line 447, in validate_no_broken_transaction
    raise TransactionManagementError(
django.db.transaction.TransactionManagementError: An error occurred in the current transaction. You can't execute queries until the end of the 'atomic' block.

The error on my end occurs with the line: updated_device = Device.objects.get(name=data["device_name"]), which suggests I can't try to manipulate an existing device using that method.
I've tried my best to look around and figure this out but to no avail.

Like said at the beginning, I'd rather be taught how to teach myself but as far as I can tell, the documentation doesn't explain the inner workings of how the Netbox Python code handles certain transactions, so I have no easy reference to follow.

Thanks in advance!

[candlerb]

See https://docs.djangoproject.com/en/3.2/topics/db/transactions/ at the section headed "Avoid catching exceptions inside atomic!"

You are intentionally causing an IntegrityError with your failure to create a duplicate object, but this breaks the transaction in the database. Even though you catch the exception, the database layer has left the transaction in a broken state.

The solution suggested is to wrap that bit of code in transaction.atomic():

from django.db import transaction
...
        try:
            with transaction.atomic():
                platform.save()
                self.log_success(f"Created new platform: {platform}")
        except Exception:
            self.log_warning('The platform is already exists')

[candlerb]

I think you misunderstood. What I'm saying is:

1. You perform a database operation (which fails) - a database integrity error when you try to insert a duplicate Platform
2. You catch the exception
3. The failed operation, in the middle of an open transaction, leaves the database connection in a bad state
4. The next database operation which you try - fetching a device - then fails with "An error occurred in the current transaction" because the transaction is broken

Solution: if you wrap the operation (1) inside an atomic block (which I believe uses a nested transaction or savepoint), it allows the ORM to rollback just that part of transaction on exception, and put it back into a good state.

[WhaleJ84]

I gave your suggestion another try, this time modifying the code a little more to better demonstrate:

from django.db import transaction
from django.utils.text import slugify
from librenms_handler.devices import Devices

from dcim.models import Device, DeviceType, Manufacturer, Platform
from extras.scripts import *


class TestScript(Script):
    class Meta:
        name = "Get device SNMP details"
        description = "Fetches relevant available SNMP information from LibreNMS"
        field_order = ['device_type', 'librenms_host', 'librenms_api_key']

    manufacturer = ObjectVar(
        model=Manufacturer,
        required=False
    )
    device_type = ObjectVar(
        required=False,
        description="Device model",
        model=DeviceType,
        display_field="model",
        query_params={
            "manufacturer_id": "$manufacturer"
        }
    )
    device_name = ObjectVar(
        description="Device name",
        model=Device,
        display_field="name",
        query_params={
            "manufacturer_id": "$manufacturer"
        }
    )
    librenms_host = StringVar(
        description="FQDN/IP address of LibreNMS host (e.g. https://librenms.example.com)"
    )
    # NOTE: The API key is sensitive data - obscuring the input like a password is advisable
    librenms_api_key = StringVar(
        description="API key to allow access to the LibreNMS instance"
    )

    def run(self, data, commit):
        # get LibreNMS data for device
        librenms_devices = Devices(
            data["librenms_host"],
            data["librenms_api_key"],
            verify=False
        )
        device_data = librenms_devices.get_device(data["device_name"])

        # Try to create the platform if it doesn't exist
        platform = Platform(
            name=device_data.json()["devices"][0]["version"],
            slug=slugify(device_data.json()["devices"][0]["version"])
        )
        try:
            platform.save()
            self.log_success(f"Created new platform: {platform}")
        except Exception:
            self.log_warning('The platform is already exists')

        # update the device
        try:
            with transaction.atomic():
                updated_device = Device.objects.get(name=data["device_name"])
        except Exception:
            self.log_warning('I failed trying to access the device')

        try:
            with transaction.atomic():
                updated_device.platform = device_data.json()["devices"][0]["version"]
                updated_device.save()
            self.log_success('wow it worked!')
        except Exception:
            self.log_warning('I failed trying to modify the device')

        if device_data.json()["status"] == "ok":
            return device_data.json()["devices"][0]
        return device_data.status_code

So I split down the erroneous code into two segments:

        try:
            with transaction.atomic():
                updated_device = Device.objects.get(name=data["device_name"])
        except Exception:
            self.log_warning('I failed trying to access the device')

and

        try:
            with transaction.atomic():
                updated_device.platform = device_data.json()["devices"][0]["version"]
                updated_device.save()
            self.log_success('wow it worked!')
        except Exception:
            self.log_warning('I failed trying to modify the device')

The output of the code is as follows:

Line	Level	Message
1	Warning	The platform is already exists
2	Warning	I failed trying to access the device
3	Warning	I failed trying to modify the device

So even with the wrapper, I still cannot grab the device to modify in the first place. Without being able to get the existing device, the next section will fail also.

[candlerb]

Sorry, but you really aren't listening are you?

The database access which needs to be wrapped is the one where the first database error occurs, in other words, this code:

        platform = Platform(
            name=device_data.json()["devices"][0]["version"],
            slug=slugify(device_data.json()["devices"][0]["version"])
        )
        try:
            platform.save()
            self.log_success(f"Created new platform: {platform}")
        except Exception:
            self.log_warning('The platform is already exists')

After this code runs, if a platform with the same name already exists, the database is in an invalid state. This is because you allowed a database integrity exception to occur outside of a transaction.atomic block, which means that the Django ORM can't reset the transaction back to a valid state.

Once the database is in this invalid state, then any subsequent database access WILL fail - whether it's wrapped or not makes no difference.

By putting the above code into a transaction.atomic block, you allow the Django ORM to revert the transaction into a valid state, before continuing to the subsequent code.

[WhaleJ84]

I was trying my best to follow and understand, but I have never worked with Django before, nor have much experience with databases so I apologize for the frustration.

You have indeed managed to get it through to me (third time's the charm).
Now that I've finally understood that, I can make some progress, so thank you very much.

[candlerb]

Glad it's working, and sorry if I wasn't explaining clearly enough before.

[WhaleJ84]

To answer my own question, I have discovered a few ways to go about it.

As Candlerb shows in great detail above, using transaction.atomic to wrap around the saving transactions can help prevent failures, but my testing found it cannot be used as an absolute solution; In my case it only allowed me to carry on one more save before having no effect (but as you can see from above, I'm probably still not understanding all it properly, so that's on me not working with Netbox scripts before :) ).

Next I started looking around for people's existing scripts and found one by Barbarossa. Inside you can see they write methods to both check if an attribute already exists within Netbox, and methods to update a given item with said attribute.
Using this method, I'm able to prevent the errors that occurred in the example above and working very nicely (although still leaves you scouring through the Netbox code to try figure out how things work).

The method I've decided to go down is to instead use the very well documented API that Netbox provides. Although it seems a bit redundant making a HTTP request to yourself, creating an authorization token for it, it means you don't have to try and figure out how all the code works as the API handles it for you.
I'm not against learning about the inner workings of Netbox, but code changes - with Netbox very frequently too. It makes sense to utilise the API they very graciously provide.

I hope the information shown here helps anybody else in the future.

[jeremystretch]

Thanks for following up with your solution!