General way of exposing values from k8s objects into metrics

[Omrigan]

Problem description / Motivation

We have lots of useful info available on VM/VMM/other objects. But once the object is deleted from k8s, we have no way of looking at what happened to an object.

One specific example I had in mind is exposing migration info, so that we can calculate stats:

autoscaling/pkg/neonvm/controllers/vmmigration_controller.go

Lines 548 to 556 in e3b2b76

	migration.Status.Info.Status = migrationInfo.Status
	migration.Status.Info.TotalTimeMs = migrationInfo.TotalTimeMs
	migration.Status.Info.SetupTimeMs = migrationInfo.SetupTimeMs
	migration.Status.Info.DowntimeMs = migrationInfo.DowntimeMs
	migration.Status.Info.Ram.Transferred = migrationInfo.Ram.Transferred
	migration.Status.Info.Ram.Remaining = migrationInfo.Ram.Remaining
	migration.Status.Info.Ram.Total = migrationInfo.Ram.Total
	migration.Status.Info.Compression.CompressedSize = migrationInfo.Compression.CompressedSize
	migration.Status.Info.Compression.CompressionRate = int64(math.Round(migrationInfo.Compression.CompressionRate))

Feature idea(s) / DoD

We have a clear way of adding new values to be exposed, and we understand what guarantees we have around it.

Implementation ideas

We already have per-VM metrics in agent, do we want to keep using it, and improve?

Maybe there is a standard way to do it in k8s already?

[Omrigan]

@sharnoff suggests we can look into https://github.com/kubernetes/kube-state-metrics/blob/main/docs/metrics/extend/customresourcestate-metrics.md