GCP TDX attests failed in RHEL VM

[yuxisun1217]

Describe the bug

Attest GCP TDX VM failed in RHEL-9.6:

# kbs-client --url http://trusteeserver:8080 attest --tee-key-file test/tee_key.pem  > test/attestation_token
[2025-01-21T08:29:18Z WARN  attester::tdx] Read AA Eventlog failed: Os { code: 2, kind: NotFound, message: "No such file or directory" }
[2025-01-21T08:29:18Z WARN  kbs_protocol::client::rcar_client] RCAR handshake failed: RcarHandshake("KBS Server Internal Failed, Response: \"payload reached size limit\""), retry 1...
[2025-01-21T08:30:19Z WARN  kbs_protocol::client::rcar_client] RCAR handshake failed: RcarHandshake("error sending request for url (http://trusteeserver:8080/kbs/v0/auth)"), retry 2...
[2025-01-21T08:30:20Z WARN  attester::tdx] Read AA Eventlog failed: Os { code: 2, kind: NotFound, message: "No such file or directory" }
[2025-01-21T08:30:20Z WARN  kbs_protocol::client::rcar_client] RCAR handshake failed: RcarHandshake("KBS Server Internal Failed, Response: \"payload reached size limit\""), retry 3...
[2025-01-21T08:31:21Z WARN  kbs_protocol::client::rcar_client] RCAR handshake failed: RcarHandshake("error sending request for url (http://trusteeserver:8080/kbs/v0/auth)"), retry 4...
[2025-01-21T08:31:22Z WARN  attester::tdx] Read AA Eventlog failed: Os { code: 2, kind: NotFound, message: "No such file or directory" }
Error: RCAR handshake failed: Unable to get token. RCAR handshake retried 5 times. Final attempt failed with: RcarHandshake("KBS Server Internal Failed, Response: \"payload reached size limit\"")

Not see such issue in GCP SNP VM.

How to reproduce

Create a RHEL-9.6 TDX VM in GCP:

gcloud compute instances create rhel-96-tdx-client --confidential-compute-type=TDX --machine-type=c3-standard-4 --maintenance-policy=TERMINATE --zone=us-central1-c --image=rhel-guest-image-9-6-20250116-5-x86-64 --image-project=virt-qe

And compile install all-attesters from source code. Then try to attest it:

kbs-client --url http://trusteeserver:8080 attest --tee-key-file test/tee_key.pem  > test/attestation_token

CoCo version information

trustee main branch
RHEL-9.6

What TEE are you seeing the problem on

Tdx

Failing command and relevant log output

[mythi]

[2025-01-21T08:29:18Z WARN kbs_protocol::client::rcar_client] RCAR handshake failed: RcarHandshake("KBS Server Internal Failed, Response: \"payload reached size limit\""), retry 1...

Haven't seen this error before. Would you be able to share the Trustee server logs too?

[yuxisun1217]

Hi @mythi ,
Here are the server logs:

[2025-01-27T03:34:48Z INFO  actix_server::server] starting service: "actix-web-service-127.0.0.1:8080", workers: 4, listening on: 127.0.0.1:8080
[2025-01-27T03:34:48Z INFO  actix_server::server] starting service: "actix-web-service-10.128.0.32:8080", workers: 4, listening on: 10.128.0.32:8080
[2025-01-27T03:34:59Z INFO  actix_web::middleware::logger] 10.128.0.3 "POST /kbs/v0/auth HTTP/1.1" 200 74 "-" "attestation-agent-kbs-client/0.1.0" 0.000167
[2025-01-27T03:34:59Z INFO  actix_web::middleware::logger] 10.128.0.3 "POST /kbs/v0/attest HTTP/1.1" 413 26 "-" "attestation-agent-kbs-client/0.1.0" 0.000028
[2025-01-27T03:36:01Z INFO  actix_web::middleware::logger] 10.128.0.3 "POST /kbs/v0/auth HTTP/1.1" 200 74 "-" "attestation-agent-kbs-client/0.1.0" 0.000117
[2025-01-27T03:36:01Z INFO  actix_web::middleware::logger] 10.128.0.3 "POST /kbs/v0/attest HTTP/1.1" 413 26 "-" "attestation-agent-kbs-client/0.1.0" 0.000024

[mythi]

@yuxisun1217 thanks. Are you using the docker-compose setup or the kubernetes deployment for Trustee? I'd be helpful if you could run KBS (and AS if they are separate) using RUST_LOG=debug to get more verbose logs.

[yuxisun1217]

Hi @mythi ,
Sorry for late. I run trustee in RHEL VM. Here are the debug logs:

kbs-server.log
kbs-client.log

[mythi]

@yuxisun1217 thanks!

The reason seems obvious: TDX Evidence is bigger than what the KBS server default max payload size is (according to Actix web docs: 256KiB)

we might need either confidential-containers/guest-components#575 (or similar) or some Trustee adjustments on the payload size.

[fitzthum]

hmm yeah we probably want to adjust that from the Trustee side, especially given that we will soon be passing the init-data claims to Trustee as well.