Running anaconda from within livemedia-creator gets stuck in the ostree deploy command

[flaper87]

Greetings,

I'm trying to create a live iso for a rhel distribution using a rhel9 VM. When running the livemedia-creator command, I see anaconda properly coming up but it gets stuck in the ostree deploy command. I managed to get the logs from ostree as well.

From the ostree logs we can see that ostree is failing to complete. The interesting thing is that things work fine when I run the ostree command manually, from the shell.

Wondering if there's anything I'm missing in terms of permissions, settings, etc.

This is the lmc command I'm running:

$ livemedia-creator --nomacboot --no-virt --iso-only --make-iso --ks kickstart.ks --anaconda-arg="--debug" --anaconda-arg="--product Red Hat Enterprise Linux" --dracut-arg=--debug --iso 72cd3681-bd00-412a-8419-45d2149726bd-installer.iso --iso-name r4e-live-x86_64.iso --resultdir /root/livemedia-creator --logfile=/root/livemedia.log --image-size-align 10000

livemedia.log

2022-11-07 03:22:48,383 DEBUG livemedia-creator: Namespace(make_iso=True, make_disk=False, make_fsimage=False, make_appliance=False, make_ami=False, make_tar=False, make_tar_disk=False, make_pxe_live=False, make_ostree_live=False, make_oci=False, make_vagrant=False, iso='/home/cloud-user/72cd3681-bd00-412a-8419-45d2149726bd-installer.iso', iso_only=True, iso_name='r4e-live-x86_64.iso', ks=['/home/cloud-user/kickstart.ks'], image_only=False, no_virt=True, proxy=None, anaconda_args=['--debug', '--product Red Hat Enterprise Linux'], armplatform=None, location=None, logfile='/root/livemedia.log', lorax_templates=None, tmp='/var/tmp', result_dir='/root/livemedia-creator', domacboot=False, extra_boot_args='', disk_image=None, keep_image=False, fs_image=None, image_name=None, tar_disk_name=None, fs_label='Anaconda', image_size_align=10000, image_type=None, qemu_args=[], qcow2=False, compression='xz', compress_args=[], app_name=None, app_template=None, app_file='appliance.xml', ram=2048, vcpus=None, vnc=None, arch=None, kernel_args=None, ovmf_path='/usr/share/edk2/ovmf/', virt_uefi=False, no_kvm=False, with_rng='/dev/random', dracut_conf=None, dracut_args=['--debug'], live_rootfs_size=0, live_rootfs_keep_size=False, oci_config=None, oci_runtime=None, vagrant_metadata=None, vagrantfile=None, project='Red Hat Enterprise Linux', releasever='9', volid=None, squashfs_only=False, timeout=None)
2022-11-07 03:22:48,383 INFO livemedia-creator: livemedia-creator v34.9.14-1
2022-11-07 03:22:48,383 INFO pylorax: selinux is enabled and in Permissive mode
2022-11-07 03:22:48,435 INFO pylorax: disk_img = /root/livemedia-creator/lmc-disk-2rb3thef.img
2022-11-07 03:22:48,435 INFO pylorax: Using disk size of 10000MiB
2022-11-07 03:22:48,585 DEBUG pylorax.imgutils: make tmp mountdir /var/tmp/lorax.imgutils.b2wrf4mf
2022-11-07 03:22:48,609 DEBUG pylorax.imgutils: remove tmp mountdir /var/tmp/lorax.imgutils.b2wrf4mf
2022-11-07 03:22:48,636 INFO pylorax: Running anaconda.
2022-11-07 03:22:51,077 INFO livemedia-creator: Processing logs from ('127.0.0.1', 44118)
2022-11-07 03:22:51,142 INFO pylorax: Starting installer, one moment...
2022-11-07 03:22:51,142 INFO pylorax: terminal size detection failed, using default width
2022-11-07 03:22:51,142 INFO pylorax: anaconda 34.25.0.29-1.el9_0 for Red Hat Enterprise Linux 9 (pre-release) started.
2022-11-07 03:22:51,142 INFO pylorax: 03:22:50 Not asking for VNC because of an automated install
2022-11-07 03:22:51,642 INFO pylorax: Starting automated install
2022-11-07 03:22:51,643 INFO pylorax: ================================================================================
2022-11-07 03:22:51,643 INFO pylorax: ================================================================================
2022-11-07 03:22:51,643 INFO pylorax: Installation
2022-11-07 03:22:51,643 INFO pylorax: 
2022-11-07 03:22:51,643 INFO pylorax: 1) [x] Language settings                 2) [x] Time settings
2022-11-07 03:22:51,643 INFO pylorax:        (English (United States))                (UTC timezone)
2022-11-07 03:22:51,643 INFO pylorax: 3) [x] Shell                             4) [ ] User creation
2022-11-07 03:22:51,643 INFO pylorax:        (Start shell)                            (No user will be created)
2022-11-07 03:22:51,643 INFO pylorax: 
2022-11-07 03:22:51,643 INFO pylorax: ================================================================================
2022-11-07 03:22:51,643 INFO pylorax: ================================================================================
2022-11-07 03:22:51,643 INFO pylorax: Progress
2022-11-07 03:22:51,643 INFO pylorax: 
2022-11-07 03:22:51,643 INFO pylorax: Setting up the installation environment
2022-11-07 03:22:51,643 INFO pylorax: Configuring storage
2022-11-07 03:22:52,644 INFO pylorax: ..
2022-11-07 03:22:52,645 INFO pylorax: Running pre-installation scripts
2022-11-07 03:22:52,645 INFO pylorax: .
2022-11-07 03:22:52,645 INFO pylorax: Running pre-installation tasks
2022-11-07 03:22:52,645 INFO pylorax: ...
2022-11-07 03:22:52,645 INFO pylorax: Installing.
2022-11-07 03:22:53,646 INFO pylorax: Starting pull of rhel/8/x86_64/edge from edge
2022-11-07 03:22:54,647 INFO pylorax: Writing objects
2022-11-07 03:22:55,649 INFO pylorax: Writing objects
2022-11-07 03:22:55,649 INFO pylorax: Preparing deployment of rhel/8/x86_64/edge
2022-11-07 03:23:05,663 INFO pylorax: Deployment starting: rhel/8/x86_64/edge

ostree logs

OT: Didn't find $sysroot/boot/loader symlink; assuming bootversion 0
OT: Didn't find $sysroot/ostree/boot.0 symlink; assuming subbootversion 0
OT: Preparing transaction in repository 0x55925a240140
OT: Pushing lock non-blocking with timeout 30
OT: Opening repo lock file
OT: Push lock: state=unlocked, depth=0
OT: Locking repo shared
OT: Using new tmpdir staging-3470f58f-7cec-4c35-b82a-a92470d1b924-gSlmOP
OT: Committing transaction in repository 0x55925a240140
OT: txn commit staging-3470f58f-7cec-4c35-b82a-a92470d1b924-gSlmOP
OT: Popping lock non-blocking with timeout 30
OT: Pop lock: state=shared, depth=1
OT: Unlocking repo
Regex version mismatch, expected: 10.37 2021-05-26 actual: 10.32 2018-09-10
Regex version mismatch, expected: 10.37 2021-05-26 actual: 10.32 2018-09-10
note: Deploying commit 031cdee1ba34753c862d3da109a6b6532540235977036865f80298dbcd0af905 which contains content in /var/lib that will be ignored.
bwrap: open /proc/192/ns/ns failed: No such file or directory

I've also filed an issue in the ostree repo as I don't think the command should get stuck but rather exit when hitting failures like this.

@bcl has been helping me so far but I think we ran into a dead end at this point and I'm afraid there's something else going on now.

[flaper87]

looks like this was an issue in ostree and it was fixed there: ostreedev/ostree#2758