phase1: remove kmods in target packages if archive is enabled

[dangowrt]

OPKG gets confused if kmod packages are present in both, target packages as well as kernel version specific folder.
Remove them from target packages to make opkg pick the kmods from kmod archive folder only.

I didn't test this and I'm not very familiar with buildbot. Just noticing that currently, older snapshots fail to install packages because newer-version kmods from target packages are selected instead of the ones matching the kernel version, so this should be fixed somehow...

[ynezz]

OPKG gets confused if kmod packages are present in both, target packages as well as kernel version specific folder.

Do you've more details about the issue, ideally reproducer? I just tried following and don't see any issue.

root@OpenWrt:/# ubus call system board
{
        "kernel": "6.6.32",
        "hostname": "OpenWrt",
        "system": "ARMv8 Processor rev 4",
        "model": "Linksys E8450",
        "board_name": "linksys,e8450",
        "rootfs_type": "initramfs",
        "release": {
                "distribution": "OpenWrt",
                "version": "SNAPSHOT",
                "revision": "r26463-a1a9572f43",
                "target": "mediatek/mt7622",
                "description": "OpenWrt SNAPSHOT r26463-a1a9572f43"
        }
}

root@OpenWrt:/# opkg update 
Downloading https://downloads.openwrt.org/snapshots/targets/mediatek/mt7622/packages/Packages.gz
Updated list of available packages in /var/opkg-lists/openwrt_core
Downloading https://downloads.openwrt.org/snapshots/targets/mediatek/mt7622/packages/Packages.sig
Signature check passed.
Downloading https://downloads.openwrt.org/snapshots/packages/aarch64_cortex-a53/base/Packages.gz
Updated list of available packages in /var/opkg-lists/openwrt_base
Downloading https://downloads.openwrt.org/snapshots/packages/aarch64_cortex-a53/base/Packages.sig
Signature check passed.
Downloading https://downloads.openwrt.org/snapshots/targets/mediatek/mt7622/kmods/6.6.32-1-5ca3cc895cfde6fa5bcd4cac1657aceb/Packages.gz
Updated list of available packages in /var/opkg-lists/openwrt_kmods
Downloading https://downloads.openwrt.org/snapshots/targets/mediatek/mt7622/kmods/6.6.32-1-5ca3cc895cfde6fa5bcd4cac1657aceb/Packages.sig
Signature check passed.
Downloading https://downloads.openwrt.org/snapshots/packages/aarch64_cortex-a53/luci/Packages.gz
Updated list of available packages in /var/opkg-lists/openwrt_luci
Downloading https://downloads.openwrt.org/snapshots/packages/aarch64_cortex-a53/luci/Packages.sig
Signature check passed.
Downloading https://downloads.openwrt.org/snapshots/packages/aarch64_cortex-a53/packages/Packages.gz
Updated list of available packages in /var/opkg-lists/openwrt_packages
Downloading https://downloads.openwrt.org/snapshots/packages/aarch64_cortex-a53/packages/Packages.sig
Signature check passed.
Downloading https://downloads.openwrt.org/snapshots/packages/aarch64_cortex-a53/routing/Packages.gz
Updated list of available packages in /var/opkg-lists/openwrt_routing
Downloading https://downloads.openwrt.org/snapshots/packages/aarch64_cortex-a53/routing/Packages.sig
Signature check passed.
Downloading https://downloads.openwrt.org/snapshots/packages/aarch64_cortex-a53/telephony/Packages.gz
Updated list of available packages in /var/opkg-lists/openwrt_telephony
Downloading https://downloads.openwrt.org/snapshots/packages/aarch64_cortex-a53/telephony/Packages.sig
Signature check passed.

root@OpenWrt:/# opkg install kmod-usb-serial
Installing kmod-usb-serial (6.6.32-r1) to root...
Downloading https://downloads.openwrt.org/snapshots/targets/mediatek/mt7622/kmods/6.6.32-1-5ca3cc895cfde6fa5bcd4cac1657aceb/kmod-usb-serial_6.6.32-r1_aarch64_cortex-a53.ipk
Installing kmod-nls-base (6.6.32-r1) to root...
Downloading https://downloads.openwrt.org/snapshots/targets/mediatek/mt7622/kmods/6.6.32-1-5ca3cc895cfde6fa5bcd4cac1657aceb/kmod-nls-base_6.6.32-r1_aarch64_cortex-a53.ipk
Installing kmod-usb-core (6.6.32-r1) to root...
Downloading https://downloads.openwrt.org/snapshots/targets/mediatek/mt7622/kmods/6.6.32-1-5ca3cc895cfde6fa5bcd4cac1657aceb/kmod-usb-core_6.6.32-r1_aarch64_cortex-a53.ipk
Configuring kmod-nls-base.
Configuring kmod-usb-core.
Configuring kmod-usb-serial.

root@OpenWrt:/# dmesg | grep usbserial
[  144.192904] usbcore: registered new interface driver usbserial_generic
[  144.192952] usbserial: USB Serial support registered for generic

[dangowrt]

@ynezz Reproducing is easy. Install a snapshot today and try installing a kmod package or anything depending on a kmod package in a few weeks, after a kernel version bump. Despite the kmod archive feed being present, opkg well prefer the no longer matching newer kmods instead of choosing the ones from the kmod archive.

[jow-]

I can confirm this wrong opkg behavior and purging the kmods from the core repo will fix it. Basically, our opkg fork is broken when it has to pick a dependency with version constraints from multiple providers. Likely one of our opkg memory usage reduction patches is breaking this.

[Ansuel]

(this problem with kmods and dependency was one of the main reason we wants to move to apk)

[aparcar]

I think we tried to tackle this issue back in 2020 and even made it working. I wonder what broke it again...

[ynezz]

I've just deployed this change "manually" to https://buildbot.staging.openwrt.org

[aparcar]

Long live the OPKG

[dangowrt]

@Ansuel Thank you! You are too fast 😜

[Ansuel]

@dangowrt ehehehe problem is that i think this needs to be manually picked for staging buildbot :(

[ynezz]

Thanks, fix deployed.

[Ansuel]

Lovely find command that loves order of args...

find: paths must precede expression: `bin/targets/malta/be/packages/'
find: possible unquoted pattern after predicate `-delete'?

@ynezz can you redeploy one last time :( ?

[ynezz]

@ynezz can you redeploy one last time :( ?

Done

[ynezz]

Just added apk compat and printing of deleted files, deployed to staging.

[Ansuel]

@ynezz in case you miss IRC i think an intermediate commit was deployed to staging
https://buildbot.staging.openwrt.org/images/#/builders/5/builds/5

[ynezz]

@ynezz in case you miss IRC i think an intermediate commit was deployed to staging https://buildbot.staging.openwrt.org/images/#/builders/5/builds/5

Yep, should be fixed. I've as well (hopefully) fixed the wrong find expression.

[Ansuel]

@ynezz I think i understood why this fail

https://buildbot.staging.openwrt.org/images/#/builders/1/builds/8/steps/59/logs/stdio

the file list is generated by the sha256sums file that is generated before the kmodclean is called (when the kmods are still in the target directory) hence rsync complain that the file can't be found.

Solution is to move the step before the checksums call so that file gets correctly deleted and are not included in the final file.

Can you check if this theory is correct and redeploy?

If it does look ugly we may have to add an additional step to remove the missing entry in the sha256sums (if we want to keep everything in order)

[Ansuel]

Well it seems at the end we really need to ""fix"" the sha256sums... Alternative solution is to use a tmp directory to move the kmods.

Honestly they are both bad solution but one should be faster than moving lots of files around one extra time...

[Ansuel]

@ynezz any idea how to handle the lint problem with this?

[Ansuel]

Just to assert the situation.

We really need to fix this:

• kmods are placed on target packages dir
• APK will install package from here instead of the kmods directory

TODO:

• create sha256sums of the file in kmods (and sign it i guess)
• correctly create a target packages index AFTER the kmods are moved to the dedicated directory