Inserting seccomp notify filter in WSL kernel 5.15.83.1 fails with EBUSY

[ctalledo]

Version

Microsoft Windows [Version 10.0.22621.1245]

WSL Version

• WSL 2
• WSL 1

Kernel Version

5.15.83.1-microsoft-standard-WSL2

Distro Version

Ubuntu 22.04

Other Software

No response

Repro Steps

Starting with WSL kernel 5.15.83.1, there is an issue inserting seccomp filters that have the "notify" action (see SECCOMP_FILTER_FLAG_NEW_LISTENER in the Linux seccomp man page). Inserting such filters returns EBUSY from the kernel.

Possible cause is that PID 1 in the WSL distro appears to have a seccomp filter attached to it by default:

wsl -d Ubuntu-22.04

$ grep -i seccomp /proc/1/status
Seccomp:        2
Seccomp_filters:        1        <<< HERE

I believe the presence of this filter on PID 1 is causing the problem: the filter is inherited by all child processes in the distro, and therefore a child process that tries to insert a seccomp notify filter (SECCOMP_FILTER_FLAG_NEW_LISTENER) gets an EBUSY response from the kernel since only one such filter per process is allowed.

The problem does NOT occur in WSL kernel 5.15.79.1. In that kernel, PID 1 has no seccomp filter on it (as expected):

$ grep -i seccomp /proc/1/status
Seccomp:        0
Seccomp_filters:        0      

Expected Behavior

A process running in a WSL Linux distro should be able to insert a seccomp notify filter (SECCOMP_FILTER_FLAG_NEW_LISTENER) without problem.

In addition, PID 1 in the WSL Linux distro should not have a seccomp filter attached to it by default.

Actual Behavior

Starting with WSL kernel 5.15.83.1, inserting seccomp filters that have the "notify" action (SECCOMP_FILTER_FLAG_NEW_LISTENER) returns EBUSY from the kernel.

PID 1 in the WSL distro has an unexpected seccomp filter attached to it by default:

wsl -d Ubuntu-22.04

$ grep -i seccomp /proc/1/status
Seccomp:        2
Seccomp_filters:        1        <<< HERE

Diagnostic Logs

No response

[ctalledo]

Another data point to help debug: In WSL kernel 5.15.83.1, I tried to dump the seccomp filter for PID 1 using the seccomp-tool, but the dump fails as follows:

$ sudo seccomp-tools dump -p 1
[ERROR] Permission denied - ptrace seccomp_get_filter failed
        PTRACE_SECCOMP_GET_FILTER requires CAP_SYS_ADMIN

Though it reports a permission error, the process doing the call has CAP_SYS_ADMIN (via sudo). The same occurs when trying to dump seccomp filters for any other process in the WSL distro (i.e., can't dump the seccomp filters).

In constrast, in WSL kernel 5.15.79.1, there is no error (and no filters are reported since there are none on PID 1):

$ sudo seccomp-tools dump -p 1
$ echo $?
0

[benhillis]

@kelsey-steele - do you have any idea what might be going on here?

[kelsey-steele]

I don't at first glance, though I'll do some investigating.

[ctalledo]

Thanks @kelsey-steele for the help. For context, I work at Docker and came across this while doing some development work on a container runtime that uses seccomp-notify, and which we hope to support on WSL. Things were working well with WSL kernel 5.15.79.1 but broke in 5.15.83.1.

[kelsey-steele]

Thank you for all the information and I appreciate your patience @ctalledo! I didn't have luck reproducing this on my current machine running the 5.15.83 WSL kernel, though I am on Win11 and may have some other differences on my environment. I have on my queue to test with a Win10 instance tomorrow.

Some questions that may help me reproduce:
Are you running on a x86 or arm64 machine?
What version of WSL are you running?
When you went back to 5.15.79, did you also change WSL versions or only the kernel?
Have you created a new WSL Ubuntu instance to see if the issue also appears there?

I did get a new WSL kernel release out on Friday (5.15.90.1). If you have the ability and time to test that kernel, let me know if the issue persists. Can always be hopeful this was something caused upstream and already fixed. :)

[ctalledo]

Hi @kelsey-steele, thanks for taking a closer look, much appreciated.

Are you running on a x86 or arm64 machine?

x86.

What version of WSL are you running?

C:\> wsl --version
WSL version: 1.1.2.0
Kernel version: 5.15.83.1
WSLg version: 1.0.49
MSRDC version: 1.2.3770
Direct3D version: 1.608.2-61064218
DXCore version: 10.0.25131.1002-220531-1700.rs-onecore-base2-hyp
Windows version: 10.0.22621.1250

NOTE: I used the Windows Insider program to download the latest WSL kernel (5.15.83.1) a couple of weeks ago.

When you went back to 5.15.79, did you also change WSL versions or only the kernel?

I didn't go back to 5.15.79; rather, my Windows dev machine had 5.15.79.1 and things were working perfectly. I then upgraded to 5.15.83.1 and noticed the problem. I then tried on another Windows machine with 5.15.79.1 and things worked there too.

Have you created a new WSL Ubuntu instance to see if the issue also appears there?

The issue is present in all distros on the machine where the WSL kernel is 5.15.83.1 (Ubuntu distro, docker-desktop's custom distro, etc.) I believe that's because it's a kernel level problem and all distros are sharing the same kernel.

I did get a new WSL kernel release out on Friday (5.15.90.1). If you have the ability and time to test that kernel, let me know if the issue persists. Can always be hopeful this was something caused upstream and already fixed. :)

Sure, let me give that a try and report.

Thanks again!

[ctalledo]

I did get a new WSL kernel release out on Friday (5.15.90.1). If you have the ability and time to test that kernel, let me know if the issue persists.

@kelsey-steele: how do I update the WSL kernel to that release? I tried wsl.exe --update but it says I am already using the latest kernel (5.15.83.1). Thanks.

[kelsey-steele]

@ctalledo thank you for the additional info!

The WSL2 kernel gets released on the WSL2 kernel repo first and then will be included in the next WSL release. Currently we're in that stage where the new kernel is so far only released on the WSL2 kernel repo. Here's the steps you can follow to boot with the 5.15.90 WSL kernel:

1. Clone the WSL2 Kernel Repo

2. Make sure you're on the 'linux-msft-wsl-5.15.y' branch or checkout the 'linux-msft-wsl-5.15.90.1' tag

3. Follow the build instructions in the WSL2 kernel README

4. After building, you will find your new kernel image inside the WSL2 tree at 'arch/x86/boot/bzImage'

5. copy this to your windows files for easy access. i.e. $ cp arch/x86/boot/bzImage /mnt/c/Users/$USERNAME/5.15.90.1-wsl

6. Add the new kernel to your .wslconfig file:

    [wsl2]
    kernel=C:\\Users\\$USERNAME\\5.15.90.1-wsl
   

7. Run 'wsl --shutdown' inside powershell

8. Restart your WSL instance and verify new kernel is being used: $ uname -r

These are the same steps you'd use to boot with a custom or upstream kernel on x86 machines. Just make sure to build any kernel you want to use with the config file from WSL2 (can be found in the WSL2 tree under Microsoft/config-wsl).

Let me know if you have any questions or unable to take these steps. Hoping I'll be able to reproduce on my side today so I can poke at things.

[ctalledo]

Hi @kelsey-steele, thanks for the detailed instructions to upgrade the WSL kernel, super clear and easy to follow.

I updated my WSL kernel to 5.15.90.1, but unfortunately still hit the same issue:

C:\Users\Cesar> wsl -d Ubuntu-22.04

cesar@xps15:/mnt/c/Users/Cesar$ uname -a
Linux xps15 5.15.90.1-microsoft-standard-WSL2+ #1 SMP Tue Feb 7 10:58:54 PST 2023 x86_64 x86_64 x86_64 GNU/Linux

cesar@xps15:/mnt/c/Users/Cesar$ grep -i seccomp /proc/1/status
Seccomp:     2
Seccomp_filters: 1        <<< HERE: why does the init process have a seccomp filter on it?

As a result, when I try to run a container with Docker Desktop on WSL, using this new runtime that uses seccomp-notify, it fails with:

$ docker run -it --rm alpine
docker: Error response from daemon: failed to create shim task: OCI runtime create failed: container_linux.go:429: starting container process caused: init_linux.go:631: loading seccomp notification rules caused: error loading seccomp filter into kernel: device or resource busy: unknown.

Could you double check on your side if /proc/1/status on your WSL kernel 5.15.90.1 shows a seccomp filter on it?

Thanks!

[ctalledo]

Thinking out loud, I wonder if the init process in the WSL distro (PID 1) is inserting that seccomp filter. I assume it's a custom init process for WSL, is the code for it available so I can take a look? I believe also that the init process is the same for all WSL distros correct?

[kelsey-steele]

@ctalledo I was able to reproduce this. This requires WSL 1.1.0 or later to reproduce.

I tested rolling the kernel back to 5.15.79.1 while on WSL 1.1.0 and 1.1.2 which did not resolve the problem. However, once I rolled the WSL version back to 1.0.3, the seccomp filter on PID 1 reports as expected (with 0) with each kernel (5.15.79.1 - 5.15.90.1)

I believe your thinking is on the right path. I also believe the portion of code that needs to be examined is closed and needs to be checked on our side, though I very much appreciate all your help to debug! For the question if the init process is the same for all WSL distros, that escapes a bit out of my current knowledge. @benhillis do you have an answer for this?

@benhillis I see in the WSL 1.1.0 release notes there is a change "Use seccomp in localhost relay". Can I get help investigating that patch to see if it's linked here?

@ctalledo for a temporary fix, I'd advise rolling WSL back to 1.0.3:

• uninstall WSL (not the distros, just "Windows Subsystem for Linux").
• Download the msixbundle from the 1.0.3 release page.
• Install by running the downloaded file or in powershell run $ Add-AppxPackage -Path "Path to msixbundle"

edit: your distro installs should be ok throughout this process, though it's always smart to backup first just in case. You can use "wsl --export <Distro> <FileName>" to accomplish that. :)

@benhillis if there's a more appropriate route for downgrading WSL, please correct me. :)

[ctalledo]

Hi @kelsey-steele, thanks for the update, glad you were able to reproduce it.

This requires WSL 1.1.0 or later to reproduce.

Got it; I can confirm that my machine where it fails has WSL 1.1.2.0, while the machine where it passes has WSL 1.0.3.0.

I believe your thinking is on the right path. I also believe the portion of code that needs to be examined is closed and needs to be checked on our side

Got it, thanks for letting me know.

To help a bit more:

The problem is that when installing a seccomp-notify filter into the WSL kernel we are getting back EBUSY. Per seccomp(2), this occurs when there is prior seccomp-notify filter already installed for the process:

       EBUSY  While installing a new filter, the
              SECCOMP_FILTER_FLAG_NEW_LISTENER flag was specified, but a
              previous filter had already been installed with that flag.

I suspect that PID 1 in the WSL distros is already under such a filter (as evidenced by the output of grep -i seccomp /proc/1/status above) and since a process inherits seccomp filters from its parent, it means all processes in the WSL distro have the filter applied. Thus when our container runtime inside WSL tries to insert another seccomp-notify filter for the container, it gets an EBUSY.

I'd advise rolling WSL back to 1.0.3.

Thanks; this will work for me, but since we are planning to have Docker Desktop on WSL use seccomp-notify very soon, we are hoping for a fix so that our customers can get to use it without having to worry about what version of WSL they have.

Thanks again for the help so far!

[kelsey-steele]

@ctalledo appreciate the extra information! Sounds like the code that introduced this issue has been isolated and there should be a fix out soon. This is being handled on the WSL side (I'm kernel side), though I'll keep an eye on this and post back with updates.

[ctalledo]

Perfect, thanks @kelsey-steele, much appreciated!

[kelsey-steele]

Hi @ctalledo! A 1.1.3 WSL Pre-Release was put out yesterday. Can you please verify your issue is resolved with this version? You should be able to switch to it with an update if you're still on the windows pre-release channel or manually install the release with the same instructions a few comments back for rolling the WSL version back.

[ctalledo]

Thanks @kelsey-steele! Let me try that tomorrow and get back to you.

[kelsey-steele]

Hi @ctalledo! Following up to see if you were able to test the WSL 1.1.3 pre-release?

[ctalledo]

Hi @kelsey-steele, was just about to report; yes, things work with WSL 1.1.3.

I can see that in a WSL distro, PID 1 is no longer under a seccomp filter:

$ cat /proc/1/status | grep -i seccomp
Seccomp:        0
Seccomp_filters:        0

Moreover, the container runtime we are developing can now insert seccomp-notify filters in the WSL distro as expected?

Thanks so much for the help with this!

Should we mark this issue as resolved, or should we wait for WSL 1.1.3 to be formally released.

BTW, do you know when that would be?

Thanks again!

[kelsey-steele]

Great news! I'll go ahead and close this. The release date depends on how testing is going on the latest pre-release, so I don't have a direct answer for that. Hopefully it's going well and we'll see the next release soon! :)

[ctalledo]

Got it, thanks @kelsey-steele for all the help with this issue, much appreciated!

[MrElectronics]

Occurring again, could not re-open this issue, hence opened a new one: #11434 initially:

WSL version: 2.1.5.0
Kernel version: 5.15.146.1-2
WSLg version: 1.0.60
MSRDC version: 1.2.5105
Direct3D version: 1.611.1-81528511
DXCore version: 10.0.25131.1002-220531-1700.rs-onecore-base2-hyp
Windows version: 10.0.22631.3296
Docker Desktop for Windows 4.27.2 (with Sysbox-runc)

Running any container results into an error message:

docker run -it --rm --runtime sysbox-runc <image-hash> /bin/bash
docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: container_linux.go:427: starting container process caused: init_linux.go:632: loading seccomp notification rules caused: error loading seccomp filter into kernel: error loading seccomp filter: device or resource busy: unknown.

Checking init process properties reveals the expected output:

> cat /proc/1/status | grep -i seccomp
Seccomp:        2
Seccomp_filters:        1

Thanks for looking into this again.

[ctalledo]

Hi @MrElectronics,

I am not able to repro on WSL (same version as yours). However, a user commented in this Sysbox issue that having networkingMode=mirrored in their %USERPROFILE%\.wslconfig was causing the problem for him.

When the user removed networkingMode=mirrored, then Sysbox was able to insert the seccomp-notify filter into the WSL kernel.

[felipecrs]

I'm the user @ctalledo mentioned and I can reliably reproduce the problem, using latest WSL pre-release to date.

I'm happy to provide any information necessary.

[felipecrs]

Issue remains with WSL 2.3.11.

❯ uname -r
6.6.36.3-microsoft-standard-WSL2

❯ docker run --rm -it --runtime=sysbox-runc ghcr.io/felipecrs/devcontainer:2
docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: container_linux.go:439: starting container process caused: init_linux.go:663: loading seccomp notification rules caused: error loading seccomp filter into kernel: error loading seccomp filter: device or resource busy: unknown.

[kelsey-steele]

When investigating this last time, the fix was provided on the WSL side instead of the kernel. Removing my assignment here.

@craigloewen-msft @benhillis pinging for attention and help here please.

[ctalledo]

Hi @craigloewen-msft, @benhillis,

This is Cesar Talledo from Docker. Wondering if there had been any more investigation into this issue, as a Docker Desktop customer that uses WSL with networkingMode=mirrored is bumping into this problem.

$ docker run -it --rm alpine
docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: container_linux.go:439: starting container process caused: init_linux.go:663: loading seccomp notification rules caused: error loading seccomp filter into kernel: error loading seccomp filter: device or resource busy: unknown.

The problem is caused by Docker Desktop trying to insert a seccomp-unotify filter into the Linux kernel in the WSL distro, but this fails because there's already a similar filter unexpectedly installed in the distro.

For example, as soon as I start a new WSL distro (when using networkingMode=mirrored in the .wslconfig file), I see:

cat /proc/1/status | grep -i seccomp
Seccomp:        2
Seccomp_filters:        1               <<< UNEXPECTED

That's unexpected, there should be no seccomp filters installed on a virgin WSL distro.

If I remove the networkingMode=mirrored from the .wslconfig file, then the seccomp filter output is correct:

$ cat /proc/1/status | grep -i sec
Seccomp:        0
Seccomp_filters:        0              <<< GOOD

Would really appreciate any insights into this, thanks!

[ctalledo]

Hi @craigloewen-msft, @benhillis, sorry to bother but any feedback on my prior comment? Thanks!

[felipecrs]

Issue still happens with WSL 2.5.1 (Linux 6.6.75).

[CatalinFetoiu]

hello @ctalledo, sorry for the delay following up
the presence of the seccomp filters is expected when networkingMode=mirrored

can you please share more details about the Docker scenario/s affected by this

thanks

[ctalledo]

Hi @CatalinFetoiu thanks for the response.

can you please share more details about the Docker scenario/s affected by this

Sure; Docker (and possibly other runtimes too) sometimes use a seccomp notify filter to trap system calls inside a container.

The Linux kernel only allows one such filter per process. And filters are inherited by child processes.

Turns out that when networkingMode=mirrored is active, apparently WSL inserts a seccomp notify filter on PID 1 of the WSL distro. That filter is inherited by all processes in the distro (since they are all descendants of PID 1), and therefore it's not possible to insert seccomp notify filters from within the distro (doing son returns EBUSY).

Ideally, we are hoping a WSL distro behaves much like a VM, where there are no seccomp filters on PID 1.

FYI we had this same problem in the past with WSL (original description of this PR), but the WSL developers were able to fix it and remove the seccomp notify filter from PID 1. However it appears to still be occurring when the mirrored networking mode is enabled unfortunately.