Really confused about lack of layer cache reuse

[moltar]

I have been using Docker for ten years, and I have never experienced something unexplainable and hard to figure out.

I'm running a build inside AWS CodeBuild CI. Using S3 caching as well, as well as ECR for publishing final images.

The build happens in two stages, inside two different CodeBuild runners.

1. base stage
2. app stage

When the app builds, it reuses all layers correctly from the base layer (pulls from S3/ECR), but the last layer always gets rebuilt.

I've ran two builds, via the pipeline, by triggering the exact same commit hash on both builds, so it should, in theory always result in the exact same output.

When I pulled these images from ECR to my local, and examined them with dive tool, I see that every layer up to the last one, is always reused, and the digest remains the same. But then, the last layer is always rebuilt and results in a different digest.

How's that even possible?

Under which conditions would the last layer invalidate, given that the layer just before it is identical?

The build command is:

docker buildx build --target="workspace" \
  --build-arg="FOO_ENV=local" --build-arg="FOO_WORKSPACE=bar" \
  --tag="$FOO_DOCKER_IMAGE_TAG" \
  --cache-to="type=s3,region=eu-west-1,bucket=my-bucket,name=foo,upload_parallelism=16,mode=max" \
  --cache-from="type=s3,region=eu-west-1,bucket=my-bucket,name=foo" \
  --push --progress=plain .