VM Disk performance issue

[akrasnov-drv]

Hi,
We see degraded performance in our VMs in CloudStack (4.20.0.0) during compilation and other heavy I/O operations.
I set our compute offerings to use local disk, nevertheless when there are 20-40 VMs per host, it does not work well.
As I understand it happens because of shared base disk architecture used by CloudStack with QCOW images.

According to information I found for QCOW:
Case-Based Limits:
• Light workloads: 10-20 images sharing one base can work fine.
• Heavy workloads (e.g., database, high IOPS): Keep it below 5 per base.
• Cloud setups (OpenStack, CloudStack): Optimized storage backends (Ceph, LVM) are preferable.

Please advise if

• there is a way to limit number of images per shared base (or completely change this to independent disks)
• there is a way to define compute offering initially with 2 disks - 1st root with shared base from template, 2nd independent and empty.

Thanks,
Alex.

[NuxRo]

Hello,

I have never noticed issues using linked-clones and I suspect the culprit may be something else, perhaps environmental, otherwise we would have had people complain. Linked-clones has always been the default.

To answer your question, you cannot limit the number of images per shared image (ie how many VMs you can spawn from a given template).
Also, you cannot define a compute offering which contains 2 disks, you'd have to create the 2nd disk as part of the creation procedure.

It'd be worth doing some in-depth monitoring of the hypervisor, especially I/O metrics and see what is going on when you feel like it's slow etc.

[akrasnov-drv]

Hello @NuxRo,
Thank you for your reply.
I hoped to get some other answer, e.g. that I missed some way to do the desired changes.

Till now we used own VMs with dedicated disks and pinned cores (core pass-through with no reuse, grouped per numa). These VMs accomplish our tasks significantly faster than CloudStack VMs with the same resources on the same servers. For now I don't think the difference could rise just because of CPU cores (I also configured cpu pass-through, though without numa optimization), so only the disks are left.
I'll try to experiment with this more. For now I believe the problem is in layered disks.

If anybody gets ideas on optimizing disks in CloudStack, please share.
Thanks,
Alex.

[rajujith]

@akrasnov-drv Could you create a new compute offering with the storage provisioning type 'Fat provisioning'? this should create dedicated disks for your instances created by CloudStack. Will that make the instance configuration close to the external VMs so that you can compare the performance?

[akrasnov-drv]

Oh great! Thanks @rajujith
I'll try and test.
I've got hope :)

[akrasnov-drv]

Thank you.
Fat disk really does what I wanted and solves the issue (don't know how I overlooked it)