gok vm --arch arm64: sometimes throws Synchronous Exception

[damdo]

OS

darwin / macOS Sonoma 14.7.2

Arch:

Apple M3 Pro (arm64)

gok

$ gok version
https://github.com/gokrazy/tools/commit/70c48e594cf1

qemu

# installed via nix
$ qemu-system-aarch64 -version
QEMU emulator version 9.0.2
Copyright (c) 2003-2024 Fabrice Bellard and the QEMU Project developers

Sometimes, like ~50% of the times the following command

GOPROXY=direct GOOS=linux GOARCH=arm64 ./gok -i gokrazy-dummy-arm64 --parent_dir=./.. vm run --arch arm64 --graphic=false

yields this exception:

GOPROXY=direct GOOS=linux GOARCH=arm64 ./gok -i gokrazy-dummy-arm64 --parent_dir=./.. vm run --arch arm64 --graphic=false
2025/01/26 16:54:16 building disk image
gokrazy gok g70c48e on GOARCH=arm64 GOOS=darwin

Build target: CGO_ENABLED=0 GOARCH=arm64 GOARCH=arm64 GOOS=linux GOOS=linux
Build timestamp: 2025-01-26T16:54:16+01:00
Loading system CA certificates from bundled Mozilla CA list
Building 4 Go packages:

[...]
[done] in 6.22s

Including extra files for Go packages:

  github.com/gokrazy/serial-busybox
    will include extra files in the root file system
      from /Users/ddonati/go/pkg/mod/github.com/gokrazy/[email protected]/_gokrazy/extrafiles_arm64.tar
      last modified: 2025-01-19T15:59:16+01:00 (168h55m9s ago)

  github.com/gokrazy/breakglass
    will include extra files in the root file system
      from /Users/ddonati/src/private/infra/metal/gokrazy/gokrazy-dummy-arm64/breakglass.authorized_keys
      last modified: 2025-01-26T16:31:34+01:00 (22m50s ago)

Including loadable kernel modules from:
/Users/ddonati/go/pkg/mod/github.com/gokrazy/[email protected]/lib/modules

Feature summary:
  use GPT: true
  use PARTUUID: true
  use GPT PARTUUID: true

Creating boot file system
[creating boot file system]
Kernel directory: /Users/ddonati/go/pkg/mod/github.com/gokrazy/[email protected]
EEPROM update summary:
  pieeprom.upd (sig 23d672cce2)
  vl805.bin (sig 548581c70a)
  recovery.bin
[done] in 0.70s
MBR summary:
  LBAs: vmlinuz=8300 cmdline.txt=112200
  PARTUUID: 9205e2b8

Creating root file system
[done] in 0.37s
If your applications need to store persistent data, create a file system using e.g.:
        /sbin/mkfs.ext4 -F -E offset=1157627904 /var/folders/23/sd8wmjm93wl5895s_zk2kvfw0000gn/T/gokrazy-vm854963094 98295

To boot gokrazy, copy /var/folders/23/sd8wmjm93wl5895s_zk2kvfw0000gn/T/gokrazy-vm854963094 to an SD card and plug it into a supported device (see https://gokrazy.org/platforms/)


Build complete!

[...]
2025/01/26 16:54:27 running QEMU
[qemu-system-aarch64 -name gokrazy-dummy-arm64 -boot order=d -drive file=/var/folders/23/sd8wmjm93wl5895s_zk2kvfw0000gn/T/gokrazy-vm854963094,format=raw -device i6300esb,id=watchdog0 -watchdog-action reset -smp 12 -device e1000,netdev=net0 -netdev user,id=net0,hostfwd=tcp::8080-:80,hostfwd=tcp::8022-:22 -m 1024 -machine virt,highmem=off -cpu cortex-a72 -bios /var/folders/23/sd8wmjm93wl5895s_zk2kvfw0000gn/T/gokrazy-vm918204012/arm64-QEMU_EFI.fd -accel hvf -nographic]
UEFI firmware (version  built at 20:43:50 on Feb 12 2024)


Synchronous Exception at 0x000000007FAF1A14

This is my current config.json:

{
    "Hostname": "gokrazy-dummy-arm64",
    "Update": {
        "HTTPPassword": "xxxxxxxxx"
    },
    "Packages": [
        "github.com/gokrazy/fbstatus",
        "github.com/gokrazy/hello",
        "github.com/gokrazy/serial-busybox",
        "github.com/gokrazy/breakglass"
    ],
    "KernelPackage": "github.com/gokrazy/kernel.arm64",
    "FirmwarePackage": "github.com/gokrazy/kernel.arm64",
    "PackageConfig": {
        "github.com/gokrazy/breakglass": {
            "CommandLineFlags": [
                "-authorized_keys=/etc/breakglass.authorized_keys"
            ],
            "ExtraFilePaths": {
                "/etc/breakglass.authorized_keys": "breakglass.authorized_keys"
            }
        },
        "github.com/gokrazy/gokrazy/cmd/randomd": {
            "ExtraFileContents": {
                "/etc/machine-id": "ced622c745ca42aa929b49c8e7588b59\n"
            }
        }
    },
    "SerialConsole": "disabled",
    "InternalCompatibilityFlags": {}
}

---

Edit: 1

This issue only happens when running arm64 vms, so with gok vm --arch arm64 and it doesn't with gok vm --arch amd64 (with of course a changed config.json and the correct kernel and env var flags).

[damdo]

Could it be due to a version discrepancy between the qemu version I'm running qemu 9.0.2, and the gok's embedded QEMU_EFI.fd (2022.11-6, which I think comes on debian with qemu 7.2.15)?

Is this related https://gitlab.archlinux.org/archlinux/packaging/packages/edk2/-/issues/4 ?

[damdo]

Found this in the QEMU 9.0 release notes (source):

Note that when the virt board uses the non-secure EL2 virtual timer IRQ, this will trip a bug in older versions of the EDK2 guest firmware, which causes EDK2 to assert on bootup with "ASSERT [ArmTimerDxe] /home/kraxel/projects/qemu/roms/edk2/ArmVirtPkg/Library/ArmVirtTimerFdtClientLib/ArmVirtTimerFdtClientLib.c(72): PropSize == 36 || PropSize == 48". If you see that assertion you should do one of:

update your EDK2 binaries to edk2-stable202311 or newer
use the 'virt-8.2' versioned machine type
not use 'virtualization=on'

And I have a hunch this might be what's causing issue, considering it is a pulsing one, the theory that it is triggered by a bug migth explain this.

Should we update the QEMU_EFI to be after 202311 ? Will this force a minumum the version of QEMU?
WDYT @stapelberg ?

[stapelberg]

Generally, UEFI should be compatible in both directions, but apparently not always :D

Yeah, upgrading to a (probably very recent) edk2 seems fine to me. Do you want to give it a try and see if it solves the problem for you?

[damdo]

Yeah, that would make sense, it sounds like they didn't meant to break that.

Anyway here is the PR to bump things: #81

Please let me know if it works for older versions of qemu if you are running one (e.g. 7.y or 8.y) Thanks!