Slow COPY with fuse-overlayfs

[sebhoss]

We are trying to switch our kaniko based image builds to buildah. We are using Gitlab CI with their Docker executor running on Ubuntu 20.04 with Docker installed. The CI jobs use quay.io/buildah/stable:latest, they run as privileged, and use the overlay storage driver. Our observation is that kaniko builds that finish in less than 10mins take more than 60mins with buildah (we have a timeout of 60mins so we don't exactly know how long the entire build would take). Most of that time is spend on COPY commands - even when only copying a single very small file.

Since we running buildah inside docker, I'm guessing that this is expected because we cannot run overlay in overlay natively but have to use fuse? Is there something we can do to improve build speeds?

Output of buildah info is:

$ whoami
root
$ buildah info
{
    "host": {
        "CgroupVersion": "v1",
        "Distribution": {
            "distribution": "fedora",
            "version": "41"
        },
        "MemFree": 623775744,
        "MemTotal": 33650003968,
        "OCIRuntime": "crun",
        "SwapFree": 0,
        "SwapTotal": 0,
        "arch": "amd64",
        "cpus": 8,
        "hostname": "runner-wpb2vjgg-project-498-concurrent-1",
        "kernel": "5.14.21-150500.55.83-default",
        "os": "linux",
        "rootless": false,
        "uptime": "3332h 42m 3.76s (Approximately 138.83 days)",
        "variant": ""
    },
    "store": {
        "ContainerStore": {
            "number": 0
        },
        "GraphDriverName": "overlay",
        "GraphOptions": [
            "overlay.imagestore=/var/lib/shared",
            "overlay.imagestore=/usr/lib/containers/storage",
            "overlay.mount_program=/usr/bin/fuse-overlayfs",
            "overlay.mountopt=nodev,fsync=0"
        ],
        "GraphRoot": "/var/lib/containers/storage",
        "GraphStatus": {
            "Backing Filesystem": "xfs",
            "Native Overlay Diff": "false",
            "Supports d_type": "true",
            "Supports shifting": "true",
            "Supports volatile": "true",
            "Using metacopy": "false"
        },
        "ImageStore": {
            "number": 0
        },
        "RunRoot": "/run/containers/storage"
    }
}

[nalind]

I would try removing the overlay.mount_program setting and try removing the fsync=0 from the overlay.mountopt setting here, because given the kernel version I would not expect to need fuse-overlay for mounting overlay filesystems inside of the container. It might even be time to do that by default in https://github.com/containers/image_build

[rhatdan]

You will also need to do a reset of storage, because containers/storage will remember if you ever use fuse-overlay.fs.

[sebhoss]

Thanks! This did improve our build times greatly - we can now finish our build with buildah within 15mins. For reference, buildah info shows:

{
    "host": {
        "CgroupVersion": "v1",
        "Distribution": {
            "distribution": "fedora",
            "version": "41"
        },
        "MemFree": 8249450496,
        "MemTotal": 33650003968,
        "OCIRuntime": "crun",
        "SwapFree": 0,
        "SwapTotal": 0,
        "arch": "amd64",
        "cpus": 8,
        "hostname": "runner-wpb2vjgg-project-43-concurrent-1",
        "kernel": "5.14.21-150500.55.83-default",
        "os": "linux",
        "rootless": false,
        "uptime": "3343h 36m 31.35s (Approximately 139.29 days)",
        "variant": ""
    },
    "store": {
        "ContainerStore": {
            "number": 0
        },
        "GraphDriverName": "overlay",
        "GraphOptions": [
            "overlay.imagestore=/usr/lib/containers/storage",
            "overlay.mountopt=nodev"
        ],
        "GraphRoot": "/var/lib/containers/storage",
        "GraphStatus": {
            "Backing Filesystem": "xfs",
            "Native Overlay Diff": "true",
            "Supports d_type": "true",
            "Supports shifting": "false",
            "Supports volatile": "true",
            "Using metacopy": "false"
        },
        "ImageStore": {
            "number": 0
        },
        "RunRoot": "/run/containers/storage"
    }
}