RFC: Trustee support High Availability

[huoqifeng]

There are two kinds of resources are saved in trustee today:

• secrets/resources used by CoCo Pod
• policies used by Trustee/KBS itself

I think KBS supported HA already if the secrets/resources are saved in KMS and if trustee get the resource from KMS plugin. But if the resources are saved in KBS directly, it's not HA yet. We'd better have an approach to enable HA if user select to save resources in KBS directly.

For policy files, it's not HA yet because a damaged/recreated trustee Pod will cause all the policies files missing.

Maybe, there are two options here:

1. Use persistent storage to save all the resources and policies file. The storage should. be encrypted
2. Use ConfigMap in k8s to save all the resources and policies file.

I created this issue to track and discussion the approach.
@fitzthum @Xynnn007 @mkulke

[fitzthum]

I am in favor of this. @mkulke has been a big proponent.

One thing to keep in mind is that we also have this issue confidential-containers/trustee-operator#70 where we are trying to figure out how to deploy Trustee in CoCo.

This conflicts slightly with some of the k8s-based approaches to handling resource, policies, etc. since we don't want to Trust the control plane.

We might just need to support two different paradigms.

[Xynnn007]

Let's try to decouple this problem. On the one hand, it is the high availability of trustee storage secrets and policies, and on the other hand, it is the data security of CoCo deployment.

It should be emphasized that my discussion does not assume that Trustee is deployed through CoCo, ordinary K8S is also fine, as long as it is in the user's trusted domain. Therefore, we do not have a strong need to encrypt a Trustee resource that has been deployed in the user's trusted domain.

First, let’s talk about the high availability of trustee. Store the secret in the KMS plug-in, or put it in a Volume that is highly available, and then bind it to the KBS container to achieve high availability. As for the policies, the same strategy can be used, but Trustee semantics supports writing, so we have to additionally deal with the issue of write consistency. I believe this problem has been solved in the traditional k8s ecosystem (for example, using file systems such as Ceph as storage plug-ins) and not new for CoCo.

Secondly, about data confidentiality of CoCo’s deployment. The principle of persistent encrypted data security should be similar to kata-containers/kata-containers#9999, which further supports transparent encryption of persistent Volumes.

[huoqifeng]

Thanks @fitzthum and @Xynnn007 for the inputs. @mkulke I think you have ideas or made some progress already as what @fitzthum mentioned?

[mkulke]

I agree mostly with the approach that @Xynnn007 suggest, let's decouple of HA/state from the TEE requirements.

however, I would be careful to assume writeable storage. While it's possible to use NFS or CEPH as k8s storage backend so the api server can write to the fs. I think that's a rather uncommon way to architect a greenfield cloud-native applications.

In the short-term I would attempt to make KBS/AS run stateless first

• I think we have in-memory state in the RCAR protocol in the form of session cookies.
• We should have a toggle to turn off operations that cause side-effects (like allowing only the GET method or disabling any admin endpoint)
• policies and resources can be backed (and managed) by configmaps/secret or CSI integrations

I think that would be easy to test if we turn the replication of a KBS deployment to 2+ in the respective test suite.

In the longer/mid term, I would expect a service like KBS/AS to store its data in a database (can be sqlite for dev deployments), not on the fs. A bit later on the maturity curve there should also be boilerplate like a restful CRUD API to manage resources with RBAC and IAM integration, telemetry and all that.

KBS/AS being written in Rust, there are not always off-the-shelf solutions for features like this, so this might require some effort. The charm of making KBS/AS stateless is that we can defer topics like state and user access to the lower layers of K8S or managed cloud offerings until we implement those things.