NetworkManager inappropriately configures the SR-IOV interface on Azure

[freedge]

Describe the bug

deploying a VM using accelerated networking on Azure,
Azure exposes a second interface, that is supposed to stay unmanaged, but NetworkManager still configures it.

Reproduction steps

1. deploy a VM with accelerated networking
2. reboot it
3. ip a

Expected behavior

eth0 should be the only managed and configured interface.

When explicitly telling the NetworkManager to not manage this interface as doc, the second interface should not receive an IP

Actual behavior

both eth0 and enP36480s1 are configured and given an IP.

When creating a 99-azure-unmanaged-devices.conf, enP36480s1 is not managed, but still has an IP configured on it. Maybe CoreOS is starting the NetworkManager a first time in the initrd and it does not see the configuration.

System details

$ rpm-ostree status -b
State: idle
AutomaticUpdatesDriver: Zincati
  DriverState: active; periodically polling for updates (last checked Sat 2023-01-14 23:31:17 UTC)
BootedDeployment:
● fedora:fedora/aarch64/coreos/next
                  Version: 37.20230110.1.0 (2023-01-11T23:24:37Z)
                   Commit: 28729e10c569ae46560de6fb251d45be9ea7efa6666b98790b09f7ad4af8c8bc
             GPGSignature: Valid signature by ACB5EE4E831C74BB7C168D27F55AD3FB5323552A

• Azure with accelerated networking
• Arm64 in my case, but should be reproducible with x64 as well

Ignition config

variant: fcos
version: 1.4.0

storage:

  directories:

    - path: /etc/NetworkManager/conf.d

  files:

    
    # https://learn.microsoft.com/en-us/azure/virtual-network/accelerated-networking-overview
    - path: /etc/NetworkManager/conf.d/99-azure-unmanaged-devices.conf
      contents:
        inline: |
          # Ignore SR-IOV interface on Azure, since it'll be transparently bonded
          # to the synthetic interface
          [keyfile]
          unmanaged-devices=driver:mlx4_core;driver:mlx5_core

Additional information

reported on RHCOS through a case to the vendor, but since it is reproduced on fcos, I figured I'll send an issue here too.

[dustymabe]

Thanks @freedge for the report. Also thanks for running aarch64 Azure instances. They are relatively new to our offering, but glad to know you have at least had some success there.

When creating a 99-azure-unmanaged-devices.conf, enP36480s1 is not managed, but still has an IP configured on it. Maybe CoreOS is starting the NetworkManager a first time in the initrd and it does not see the configuration.

Correct. I suspect it is the case that this configuration you've laid down in /etc/NetworkManager/conf.d/99-azure-unmanaged-devices.conf doesn't make it into the initramfs and thus isn't applied there.

I think we could take a step back here and ask if this configuration should be delivered another way by us and not by the user directly (i.e. the WALinuxAgent-udev package (source, which we do ship). I guess we could/should ask ourselves if this is safe to ship everywhere or not. i.e. the mlx4_core and mlx5_core drivers probably are used outside of Azure too.

[bgilbert]

We already ship a udev rule for this, but presumably it isn't matching on aarch64.

[dustymabe]

We already ship a udev rule for this, but presumably it isn't matching on aarch64.

Thanks @bgilbert for that link. I see DRIVERS=="hv_pci" there but the NetworkManager config snippet from the azure docs seems to have unmanaged-devices=driver:mlx4_core;driver:mlx5_core, which implies to me maybe we need to update our snippet to mention these other drivers?

[freedge]

there is an issue with NetworkManager still deciding to manage this interface despite the udev rule actually matching.

(this issue is not specific to fcos or aarch64). There were similar issues with NetworkManager like https://bugzilla.redhat.com/show_bug.cgi?id=1896866 and https://bugzilla.redhat.com/show_bug.cgi?id=2149012

however the IP being added through DHCP when in the ramdisk seems to be a race condition that triggers more often with aarch64. Adding the 99-azure-unmanaged-devices.conf file does not workaround this one, so this is really something unique to CoreOS.

[dustymabe]

there is an issue with NetworkManager still deciding to manage this interface despite the udev rule actually matching.

but does the udev rule match? See #1383 (comment)

Adding the 99-azure-unmanaged-devices.conf file does not workaround this one, so this is really something unique to CoreOS.

Right. When you create /etc/NetworkManager/conf.d/99-azure-unmanaged-devices.conf it's not having the desired effect for you because that's not in the initramfs AND currently we always bring up networking in the initramfs on Azure (see the code).

I think bringing up the networking in the initramfs on every boot was unintended. The intention I beleive was to have it be activated unconditionally on first boot.

So.. Since the network is activated in the initramfs on Azure always, your applied /etc/NetworkManager/conf.d/99-azure-unmanaged-devices.conf isn't having the desired effect.

[dustymabe]

I think bringing up the networking in the initramfs on every boot was unintended. The intention I beleive was to have it be activated unconditionally on first boot.

Let's test that theory in coreos/fedora-coreos-config#2176

[dustymabe]

I also found Azure/WALinuxAgent#1622, which is an effort to upstream this udev rule in question here.

[freedge]

but does the udev rule match?

yes

[core@fcosa ~]$ nmcli dev
DEVICE      TYPE      STATE      CONNECTION       
eth0        ethernet  connected  Wired Connection 
enP35237s1  ethernet  connected  Wired Connection 
lo          loopback  unmanaged  --     
          
[core@fcosa ~]$ udevadm info /sys/class/net/enP35237s1 
P: /devices/LNXSYSTM:00/LNXSYBUS:00/ACPI0004:00/VMBUS:00/2be94203-89a5-4c73-b9a9-06ee28f2ba7a/pci89a5:00/89a5:00:02.0/net/enP35237s1
M: enP35237s1
R: 1
U: net
I: 3
E: DEVPATH=/devices/LNXSYSTM:00/LNXSYBUS:00/ACPI0004:00/VMBUS:00/2be94203-89a5-4c73-b9a9-06ee28f2ba7a/pci89a5:00/89a5:00:02.0/net/enP35237s1
E: SUBSYSTEM=net
E: INTERFACE=enP35237s1
E: IFINDEX=3
E: USEC_INITIALIZED=9531286
E: NM_UNMANAGED=1  <------------
E: ID_NET_NAMING_SCHEME=v251
E: ID_NET_NAME_MAC=enx6045bd9dea9c
E: ID_OUI_FROM_DATABASE=Microsoft
E: ID_NET_NAME_PATH=enP35237p0s2
E: ID_NET_NAME_SLOT=enP35237s1
E: ID_BUS=pci
E: ID_VENDOR_ID=0x15b3
E: ID_MODEL_ID=0x1018
E: ID_PCI_CLASS_FROM_DATABASE=Network controller
E: ID_PCI_SUBCLASS_FROM_DATABASE=Ethernet controller
E: ID_VENDOR_FROM_DATABASE=Mellanox Technologies
E: ID_MODEL_FROM_DATABASE=MT27800 Family [ConnectX-5 Virtual Function]
E: ID_PATH=acpi-VMBUS:00-pci-89a5:00:02.0
E: ID_PATH_TAG=acpi-VMBUS_00-pci-89a5_00_02_0
E: ID_NET_DRIVER=mlx5_core
E: ID_NET_LINK_FILE=/usr/lib/systemd/network/99-default.link
E: ID_NET_NAME=enP35237s1
E: SYSTEMD_ALIAS=/sys/subsystem/net/devices/enP35237s1
E: TAGS=:systemd:
E: CURRENT_TAGS=:systemd:

and logs

[core@fcosa ~]$ journalctl -b 0 -g enP35237s1
Jan 16 21:02:09 localhost kernel: mlx5_core 89a5:00:02.0 enP35237s1: renamed from eth1
Jan 16 21:02:10 localhost.localdomain kernel: mlx5_core 89a5:00:02.0 enP35237s1: Link up
Jan 16 21:02:10 localhost.localdomain kernel: hv_netvsc 6045bd9d-ea9c-6045-bd9d-ea9c6045bd9d eth0: Data path switched to VF: enP35237s1
Jan 16 21:02:10 localhost.localdomain NetworkManager[571]: <info>  [1673902930.1280] device (enP35237s1): carrier: link connected
Jan 16 21:02:10 localhost.localdomain NetworkManager[571]: <info>  [1673902930.1288] manager: (enP35237s1): new Ethernet device (/org/freedesktop/NetworkManager/Devices/3)
Jan 16 21:02:10 localhost.localdomain NetworkManager[571]: <info>  [1673902930.1291] device (enP35237s1): state change: unmanaged -> unavailable (reason 'managed', sys-iface-state: 'external')
Jan 16 21:02:10 localhost.localdomain NetworkManager[571]: <info>  [1673902930.1324] device (enP35237s1): state change: unavailable -> disconnected (reason 'none', sys-iface-state: 'managed')
Jan 16 21:02:10 localhost.localdomain NetworkManager[571]: <info>  [1673902930.1341] device (enP35237s1): Activation: starting connection 'Wired Connection' (d214b555-e097-4d39-81d2-6ce459a1f3af)
Jan 16 21:02:10 localhost.localdomain NetworkManager[571]: <info>  [1673902930.1350] device (enP35237s1): state change: disconnected -> prepare (reason 'none', sys-iface-state: 'managed')
Jan 16 21:02:10 localhost.localdomain NetworkManager[571]: <info>  [1673902930.1352] device (enP35237s1): state change: prepare -> config (reason 'none', sys-iface-state: 'managed')
Jan 16 21:02:10 localhost.localdomain NetworkManager[571]: <info>  [1673902930.1397] device (enP35237s1): state change: config -> ip-config (reason 'none', sys-iface-state: 'managed')
Jan 16 21:02:10 localhost.localdomain NetworkManager[571]: <info>  [1673902930.1399] dhcp4 (enP35237s1): activation: beginning transaction (timeout in 90 seconds)
Jan 16 21:02:10 localhost.localdomain NetworkManager[571]: <info>  [1673902930.1605] dhcp4 (enP35237s1): state changed new lease, address=10.56.142.170  <-------------- what scares me
Jan 16 21:02:10 localhost.localdomain NetworkManager[571]: <info>  [1673902930.1718] device (enP35237s1): state change: ip-config -> ip-check (reason 'none', sys-iface-state: 'managed')
Jan 16 21:02:10 localhost.localdomain NetworkManager[571]: <info>  [1673902930.1741] device (enP35237s1): state change: ip-check -> secondaries (reason 'none', sys-iface-state: 'managed')
Jan 16 21:02:10 localhost.localdomain NetworkManager[571]: <info>  [1673902930.1742] device (enP35237s1): state change: secondaries -> activated (reason 'none', sys-iface-state: 'managed')
Jan 16 21:02:10 localhost.localdomain NetworkManager[571]: <info>  [1673902930.1746] device (enP35237s1): Activation: successful, device activated.
Jan 16 21:02:11 localhost.localdomain NetworkManager[571]: <info>  [1673902931.8333] dhcp4 (enP35237s1): canceled DHCP transaction
Jan 16 21:02:11 localhost.localdomain NetworkManager[571]: <info>  [1673902931.8334] dhcp4 (enP35237s1): activation: beginning transaction (timeout in 90 seconds)
Jan 16 21:02:11 localhost.localdomain NetworkManager[571]: <info>  [1673902931.8334] dhcp4 (enP35237s1): state changed no lease
Jan 16 21:02:14 fcosa NetworkManager[571]: <info>  [1673902931.8333] dhcp4 (enP35237s1): canceled DHCP transaction
Jan 16 21:02:14 fcosa NetworkManager[571]: <info>  [1673902931.8334] dhcp4 (enP35237s1): activation: beginning transaction (timeout in 90 seconds)
Jan 16 21:02:14 fcosa NetworkManager[571]: <info>  [1673902931.8334] dhcp4 (enP35237s1): state changed no lease
Jan 16 21:02:16 fcosa NetworkManager[957]: <info>  [1673902936.5786] device (enP35237s1): carrier: link connected
Jan 16 21:02:16 fcosa NetworkManager[957]: <info>  [1673902936.5794] manager: (enP35237s1): new Ethernet device (/org/freedesktop/NetworkManager/Devices/3)
Jan 16 21:02:16 fcosa NetworkManager[957]: <info>  [1673902936.5798] device (enP35237s1): state change: unmanaged -> unavailable (reason 'managed', sys-iface-state: 'external')
Jan 16 21:02:16 fcosa NetworkManager[957]: <info>  [1673902936.5833] device (enP35237s1): state change: unavailable -> disconnected (reason 'none', sys-iface-state: 'managed')
Jan 16 21:02:16 fcosa NetworkManager[957]: <info>  [1673902936.5861] device (enP35237s1): Activation: starting connection 'Wired Connection' (d214b555-e097-4d39-81d2-6ce459a1f3af)
Jan 16 21:02:16 fcosa NetworkManager[957]: <info>  [1673902936.5864] device (enP35237s1): state change: disconnected -> prepare (reason 'none', sys-iface-state: 'managed')
Jan 16 21:02:16 fcosa NetworkManager[957]: <info>  [1673902936.5866] device (enP35237s1): state change: prepare -> config (reason 'none', sys-iface-state: 'managed')
Jan 16 21:02:16 fcosa NetworkManager[957]: <info>  [1673902936.5886] device (enP35237s1): state change: config -> ip-config (reason 'none', sys-iface-state: 'managed')
Jan 16 21:02:16 fcosa NetworkManager[957]: <info>  [1673902936.5889] dhcp4 (enP35237s1): activation: beginning transaction (timeout in 90 seconds)
Jan 16 21:02:16 fcosa NetworkManager[957]: <info>  [1673902936.5912] dhcp4 (enP35237s1): state changed new lease, address=10.56.142.170
Jan 16 21:02:16 fcosa systemd-resolved[948]: enP35237s1: Bus client set search domain list to: reddog.microsoft.com
Jan 16 21:02:16 fcosa systemd-resolved[948]: enP35237s1: Bus client set default route setting: yes
Jan 16 21:02:16 fcosa systemd-resolved[948]: enP35237s1: Bus client set DNS server list to: 10.56.128.190
Jan 16 21:02:16 fcosa NetworkManager[957]: <info>  [1673902936.5943] device (enP35237s1): state change: ip-config -> ip-check (reason 'none', sys-iface-state: 'managed')
Jan 16 21:02:16 fcosa NetworkManager[957]: <info>  [1673902936.6558] device (enP35237s1): state change: ip-check -> secondaries (reason 'none', sys-iface-state: 'managed')
Jan 16 21:02:16 fcosa NetworkManager[957]: <info>  [1673902936.6560] device (enP35237s1): state change: secondaries -> activated (reason 'none', sys-iface-state: 'managed')
Jan 16 21:02:16 fcosa NetworkManager[957]: <info>  [1673902936.6563] device (enP35237s1): Activation: successful, device activated.
[core@fcosa ~]$ 

Your pull request, not bringing the network in the ramdisk, should directly win us 90s of boot time on our x64 machines, where the DHCP request does not succeed, and so it waits for 90s (I just noticed that!)

[dustymabe]

This should be fixed (in two ways) by coreos/fedora-coreos-config#2176

[freedge]

Thanks a lot! Can I just rebase to the next next release to try it?
(note that following this change, I still expect NetworkManager-wait-online to fail after the first reboot, unless I deploy 99-azure-unmanaged-devices.conf)

[dustymabe]

Thanks a lot! Can I just rebase to the next next release to try it?

We'll put out a new testing and next release next week that will have this change. However, I'll link you to a development pipeline build here soon that you can try before then.

(note that following this change, I still expect NetworkManager-wait-online to fail after the first reboot, unless I deploy 99-azure-unmanaged-devices.conf)

You shouldn't need to deploy 99-azure-unmanaged-devices.conf after these changes. I fixed the problem in two ways in coreos/fedora-coreos-config#2176:

• the network is no longer brought up on subsequent boots
• the udev rule applies in the initramfs
  • so even if the network did come up on subsequent boots the enP* interface would be ignored anyway.

[dustymabe]

You can test out this change with the following testing-devel build:

• fedora-coreos-37.20230114.20.3-azure.x86_64.vhd.xz

[dustymabe]

Just to add more context to this issue. The existing udev rule that we had in Fedora CoreOS already made it so that this SR-IOV interface was unmanaged:

SUBSYSTEM=="net", DRIVERS=="hv_pci", ACTION=="add", ENV{NM_UNMANAGED}="1"

However there was a case where the enP* device would end up managed anyway:

• the rule wasn't in the initramfs so any networking brought up there would bring up the enP* device.
• on first boot this wouldn't matter because all networking gets torn down before the switch to the real root.
• azure FCOS instances were configured in such a way to bring up initramfs networking on every boot
• on a subequent boot initramfs networking would bring up the enP* device (managed by NM) and it wouldn't get torn down on the switch to the real root so it would persist even though there was a udev rule in the real root that told the device to be unmanaged

So we made two fixes here:

1. don't bring up Azure networking on every boot. It wasn't necessary.
2. apply the udev rule to the initramfs. So even if networking does get brought up it still won't bring up that device.
   • In testing this I found the udev rule we had needed a tweak in order to properly apply in the initramfs.

[freedge]

also verified with fedora-coreos-37.20230122.20.0-azure.aarch64.vhd, thanks!
If you have testing builds of rhos I'd be happy to test those as well.
Many thanks for your work!

[dustymabe]

The fix for this went into next stream release 37.20230122.1.1. Please try out the new release and report issues.

[dustymabe]

The fix for this went into testing stream release 37.20230122.2.0. Please try out the new release and report issues.

[dustymabe]

The fix for this went into stable stream release 37.20230122.3.0.