Starting instances in a domain other than root fails,

[alsko-icom]

• In Cloudstack v4.20 I created a new domain and a subdomain under it.
• Then I created a domain admin user for the domain and again a domain admin for the subdomain. Same username and password.
• I logged in to the subdomain to create an instance there.
• The instance during creation does not have the option to configure which cluster/pod/host it will use, cool so far (I would dedicate something while in ROOT if I wanted).
• After creation, I cannot start it with error "no suitable host found for VM". So I log off and log on to root.
• When I try to start the instance while in root domain, I am asked to which pod/cluster/host to start it on and whether to remember last host.
• After selecting whatever options, the VM starts normally.
• Why? I am obviously forgetting something but could not read enough through it. Some configuration I am missing? In ROOT (or subdomain) can I somehow choose pod/cluster/host once so i don't get a selection prompt every time i want to start it?
• I have checked that zone, pod, cluster and hosts are not dedicated somewhere.
• I also changed Global Settings Vm allocation algorithm (vm.allocation.algorithm) from random to FirstFit.
• Accounts are not nearing quota.

Thanks for any help!

[alsko-icom]

When I search in the /var/log/cloudstack/management/management-server.log i see entries of the VM with id "xxxxxxxx..." and the error "No suitable host found for VM" what words should I search for in the log that will determine the reason?

[winterhazel]

Before No suitable host found for VM (on the same line), there will be a part containing logid:<something>. Could you copy this part, execute a grep 'logid:<something>' /var/log/cloudstack/management/management-server.log and share the result? It will probably include the reason why the deployment failed.

[alsko-icom]

Thanks for ponting it out, I hope you can make something of the output. The log lines containing the logid of the failure are the below.

[root@cloudstack1 ~]# cat /var/log/cloudstack/management/management-server.log |grep 'logid:738bcad5'

2025-03-06 09:17:08,584 DEBUG [c.c.a.ApiServlet] (qtp1390913202-168842:[ctx-8e9f0fbc]) (logid:738bcad5) Two factor authentication is already verified for the user 12, so skipping
2025-03-06 09:17:08,586 DEBUG [c.c.a.ApiServer] (qtp1390913202-168842:[ctx-8e9f0fbc, ctx-2e9d9008]) (logid:738bcad5) CIDRs from which account 'Account [{"accountName":"alsko-VF9","id":15,"uuid":"b0f41028-9b78-4d71-bac2-ef0bcedb8d26"}]' is allowed to perform API calls: 0.0.0.0/0,::/0

[DaanHoogland]

This fragment does not contain the error @alsko-icom , are you sure you used the right logid?

[winterhazel]

These logs are unrelated to the error. As seen in the first line, they are related to a call to to the listVirtualMachines API (command=listVirtualMachines).

You need to grep the logid from the same line that contains the No suitable host found for VM you found.

[alsko-icom]

Apologies I may have correlated the VM uuid instead of the "No suitable host"! After capturing the correct lines this time it appears as the VM has a UEFI tag which the hosts do not. If I get this right, how do I overcome? Its a simple selection of a UEFI VM, not something exotic with manually creating tags for it. Pasting the logs below, 50~ish lines. I am pasting both the tail of the error and the cat of the logid.

[root@cloudstack1 ~]# tail -f /var/log/cloudstack/management/management-server.log |grep 'No suitable host found'

2025-03-06 15:17:06,708 ERROR [c.c.a.m.a.i.FirstFitRoutingAllocator] (API-Job-Executor-44:[ctx-7c27e5ec, job-825, ctx-99e4d63b, FirstFitRoutingAllocator]) (logid:c51b4fab) No suitable host found for vm [VM instance {"id":39,"instanceName":"i-15-39-VM","type":"User","uuid":"68cbdc53-28f2-4a9a-8e97-54796be4ed19"}] with tags [null].

[root@cloudstack1 ~]# cat /var/log/cloudstack/management/management-server.log |grep 'logid:c51b4fab'

2025-03-06 15:17:06,706 INFO [c.c.a.m.a.i.FirstFitRoutingAllocator] (API-Job-Executor-44:[ctx-7c27e5ec, job-825, ctx-99e4d63b, FirstFitRoutingAllocator]) (logid:c51b4fab) Guest VM is requested with Custom[UEFI] Boot Type true

2025-03-06 15:17:06,706 DEBUG [c.c.a.m.a.i.FirstFitRoutingAllocator] (API-Job-Executor-44:[ctx-7c27e5ec, job-825, ctx-99e4d63b, FirstFitRoutingAllocator]) (logid:c51b4fab) Looking for hosts in zone [1], pod [1], cluster [1]

2025-03-06 15:17:06,707 DEBUG [c.c.a.m.a.i.FirstFitRoutingAllocator] (API-Job-Executor-44:[ctx-7c27e5ec, job-825, ctx-99e4d63b, FirstFitRoutingAllocator]) (logid:c51b4fab) Hosts with tag 'UEFI' are:[]

2025-03-06 15:17:06,708 ERROR [c.c.a.m.a.i.FirstFitRoutingAllocator] (API-Job-Executor-44:[ctx-7c27e5ec, job-825, ctx-99e4d63b, FirstFitRoutingAllocator]) (logid:c51b4fab) No suitable host found for vm [VM instance {"id":39,"instanceName":"i-15-39-VM","type":"User","uuid":"68cbdc53-28f2-4a9a-8e97-54796be4ed19"}] with tags [null].

[winterhazel]

After capturing the correct lines this time it appears as the VM has a UEFI tag which the hosts do not. If I get this right, how do I overcome?

Yup, that's an issue with UEFI, not with tags. You need to setup UEFI support manually in your hosts:

1. Install ovmf
2. Create a file in /etc/cloudstack/agent/uefi.properties with the following content (this is for Ubuntu hosts. You may need to change the values depending on your OS):

guest.nvram.template.secure=/usr/share/OVMF/OVMF_VARS_4M.ms.fd
guest.nvram.template.legacy=/usr/share/OVMF/OVMF_VARS_4M.fd
guest.loader.secure=/usr/share/OVMF/OVMF_CODE_4M.ms.fd
guest.loader.legacy=/usr/share/OVMF/OVMF_CODE_4M.fd
guest.nvram.path=/var/lib/libvirt/qemu/nvram/

3. Restart cloudstack-agent

With this, the deployment of VMs using UEFI should work.

[alsko-icom]

Thank you very much for the for the suggestion, it solved the UEFI issue! Now the instance goes to starting for some moments. After that it fails with another message: Unable to start a VM due to [Unable to create a deployment for VM instance {"id":39,"instanceName":"i-15-39-VM","type":"User","uuid":"68cbdc53-28f2-4a9a-8e97-54796be4ed19"}].

The logs with the relevant logid are now 590 lines, maybe I should disable debugging? Can you suggest a filter to locate the problem now?
logid25079cf7.txt

[winterhazel]

CloudStack is unable to find /usr/share/OVMF/OVMF_VARS_4M.ms.fd on ucs-kvm2.straylight.gr.

2025-03-07 11:33:07,846 INFO  [c.c.v.ClusteredVirtualMachineManagerImpl] (Work-Job-Executor-47:[ctx-9f1d333a, job-831/job-832, ctx-d65f9ceb]) (logid:25079cf7) Unable to start VM on Host {"id":6,"name":"ucs-kvm2.straylight.gr","type":"Routing","uuid":"05d1cd35-6004-47d8-a2dc-4131e0c1b822"} due to Failed to open file '/usr/share/OVMF/OVMF_VARS_4M.ms.fd': No such file or directory

Could you list the files at /usr/share/OVMF/? You may need to change the values from uefi.properties depending on the existing files.

The logs with the relevant logid are now 590 lines, maybe I should disable debugging?

I usually leave debug enabled. Sometimes info unfortunately does not contain enough information to identify the cause of an issue.

[alsko-icom]

I checked the dir contents and they are a bit different in my 4 rhel8 kvm servers.

lrwxrwxrwx. 1 root root 33 Oct 29 17:21 OVMF_CODE.secboot.fd -> ../edk2/ovmf/OVMF_CODE.secboot.fd
lrwxrwxrwx. 1 root root 33 Oct 29 17:21 OVMF_VARS.secboot.fd -> ../edk2/ovmf/OVMF_VARS.secboot.fd
lrwxrwxrwx. 1 root root 26 Oct 29 17:21 UefiShell.iso -> ../edk2/ovmf/UefiShell.iso
lrwxrwxrwx. 1 root root 25 Oct 29 17:21 OVMF_VARS.fd -> ../edk2/ovmf/OVMF_VARS.fd

So as per the above fles i changed the uefi.properties contents to look like: (I am not sure, just trying out)

guest.nvram.template.secure=/usr/share/OVMF/OVMF_VARS.secboot.fd
guest.nvram.template.legacy=/usr/share/OVMF/OVMF_VARS.fd
guest.loader.secure=/usr/share/OVMF/OVMF_CODE.secboot.fd
guest.loader.legacy=/usr/share/OVMF/OVMF_CODE.fd
guest.nvram.path=/var/lib/libvirt/qemu/nvram/

Now, VMs start but cannot boot, even VMs that were previously booting. In boot manager there is no disk. If I rename uefi.properties and restart agent, I am back to VMs that boot normaly when i start them from root domain and will not start from the subdomain. Maybe the uefi.properties needs more entries to support the disk?

[winterhazel]

Was the template that you're using created using BIOS? If yes, that's probably why no devices are found when booting using UEFI 😅

Could you try using BIOS instead? Remove the UEFI setting from the template and from existing VMs that use it, and ensure that BIOS is selected when deploying a new VM.

If you wish to use UEFI, your hosts are configured almost properly (would be good to add a symlink to /usr/share/edk2/ovmf/OVMF_CODE.fd in /usr/share/OVMF/OVMF_CODE.fd so that deployment using UEFI LEGACY does not fail), but you will need to create new templates that support UEFI (either by reinstalling the OS or converting the existing ones to use UEFI).

[alsko-icom]

-My testing environment is new but my server hardware is old (2012 and 2014). All my VMs and templates are based on ubuntu20-22, debian12, windows 2022, all using uefi secure. Nowhere bios except for the system VM template (centos5.5).

-Cannot find how to change to bios. Only options available are name, display name, os type, dynamically scalable, ha enabled, group, delete protection. Plus a few under instance settings. (like UEFI secure or legacy). I think since they were created using uefi and secure boot, they cannot switch to bios now based on the boot partition within the VM, no?

-I created a symlink for OVMF_CODE.cc.fd and added it to the uefi.properties so all entries have a working value! (My file ends with cc.fd, line 4)

guest.nvram.template.secure=/usr/share/OVMF/OVMF_VARS.secboot.fd
guest.nvram.template.legacy=/usr/share/OVMF/OVMF_VARS.fd
guest.loader.secure=/usr/share/OVMF/OVMF_CODE.secboot.fd
guest.loader.legacy=/usr/share/OVMF/OVMF_CODE.cc.fd
guest.nvram.path=/var/lib/libvirt/qemu/nvram/

I will re-install VMs with uefi (i am thinking not secure boot this time) and copy them to templates. (help! how can i delete old templates? no delete button...)

After enabling the uefi-properties file and restarting the agent, some old centos system template based VM starts fine. All the VMs I created based on my iso and templates either cannot start or start but cannot boot. They are all based on uefi firmware and secure boot. Enabling the uefi.properties what does it really do? Do i need it? It is the only way for a tenant (subdomain) admin or user to start uefi secure or insecure VMs? I am asking because all my uefi instances worked fine prior to enabling it. Also do you think I need a uefi storage pool as defined for Alma while i am running rhel8? (https://www.siberoloji.com/set-uefi-boot-kvm-virtual-machines-almalinux/)

Thanks again for all your time and clarifications!

[winterhazel]

Enabling the uefi.properties what does it really do? Do i need it? It is the only way for a tenant (subdomain) admin or user to start uefi secure or insecure VMs?

Yes, this is needed. This file tells CloudStack where the UEFI firmware is located at, so that it can create VMs using that firmware.

-Cannot find how to change to bios. Only options available are name, display name, os type, dynamically scalable, ha enabled, group, delete protection. Plus a few under instance settings. (like UEFI secure or legacy).

Under instance settings, delete the UEFI setting. The VM will boot using BIOS when this setting is not defined.

All my VMs and templates are based on ubuntu20-22, debian12, windows 2022, all using uefi secure. Nowhere bios except for the system VM template (centos5.5).

After enabling the uefi-properties file and restarting the agent, some old centos system template based VM starts fine. All the VMs I created based on my iso and templates either cannot start or start but cannot boot.

I am asking because all my uefi instances worked fine prior to enabling it.

Your VMs were booting using BIOS (not UEFI) before creating the uefi.properties. Even if there is the UEFI setting under their instance settings, CloudStack configures existing VMs to boot using BIOS instead of UEFI if no uefi.properties is found in a host.

I assume that your templates were created from MBR disks with BIOS-compatible bootloaders because you were using BIOS when installing the OS. Therefore, now that the VMs are actually using UEFI, they cannot boot because there is no EFI partition with a bootloader.

Could you configure the VMs to boot using BIOS, as described above, and check whether they start successfully?

(help! how can i delete old templates? no delete button...)

You can delete a template by accessing its details page and going to the zones tab. There will be a button to delete the template from each zone that it is installed in.

Also do you think I need a uefi storage pool as defined for Alma while i am running rhel8? (https://www.siberoloji.com/set-uefi-boot-kvm-virtual-machines-almalinux/)

You don't need to create this pool.

[alsko-icom]

your clarifications made me feel smarter today, thanks!