Artifacts cannot be stored in object-lock enabled S3 buckets #11182

akloss-cibo · 2023-06-05T20:07:17Z

Pre-requisites

I have double-checked my configuration
I can confirm the issues exists when I tested with :latest
I'd like to contribute the fix myself (see contributing guide)

What happened/what you expected to happen?

Artifacts cannot be stored into AWS S3 buckets with Object Lock enabled.

~~I believe this may be fixed by minio/minio#15433~~

Version

3.4.7 and latest (and probably others)

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  annotations:
    iam.amazonaws.com/role: AWSIAMROLENAME
    workflows.argoproj.io/pod-name-format: v1
  name: wf-artifact-test
spec:
  entrypoint: wf-artifact-test
  templates:
  - name: wf-artifact-test
    metadata:
      annotations:
        iam.amazonaws.com/role: '{{workflow.annotations.iam.amazonaws.com/role}}'
    podSpecPatch: |
      containers:
      - name: wait
        image: argoproj/argoexec:latest
    container:
      name: wf-artifact-test
      image: busybox
      command:
      - /bin/sh
      - -c
      - |
        echo '{}' > /out/test.json
        sleep 1
        cat /out/test.json
      resources:
        requests:
          cpu: 100m
          memory: &memory 300Mi
        limits:
          memory: *memory
      volumeMounts:
      - name: out
        mountPath: /out
    volumes:
    - name: out
      emptyDir: {}
    outputs:
      artifacts:
      - archive:
          none: {}
        name: test-artifact
        path: /out/test.json
        s3:
          bucket: object-lock-enabled-bucket
          endpoint: s3.amazonaws.com
          key: test/test-artifact.json

Logs from the workflow controller

(none)

Logs from in your workflow's wait container

time="2023-06-05T20:02:44.014Z" level=warning msg="Non-transient error: Content-MD5 OR x-amz-checksum- HTTP header is required for Put Object requests with Object Lock parameters"
time="2023-06-05T20:02:44.014Z" level=error msg="executor error: failed to put file: Content-MD5 OR x-amz-checksum- HTTP header is required for Put Object requests with Object Lock parameters"
time="2023-06-05T20:02:44.114Z" level=fatal msg="failed to put file: Content-MD5 OR x-amz-checksum- HTTP header is required for Put Object requests with Object Lock parameters"

The text was updated successfully, but these errors were encountered:

juliev0 · 2023-06-08T17:30:11Z

@akloss-cibo do you want to try updating to that version of minio in that case and contribute the fix if it's working?

akloss-cibo · 2023-06-08T20:21:58Z

It looks like dependabot keeps minio up to date (see commit 760299f). I tried using argoproj/argoexec@sha256:e237c40aadcfd5d3d6b8325d4eba79e53fd7085cccd8dc6eefd48049966d873c and it still fails. Perhaps Argo Workflows is using one of the cases that minio doesn't handle correctly.

minio/minio#16480 seems related as well.

juliev0 · 2023-06-08T21:23:31Z

thanks for checking

harshavardhana · 2023-08-21T08:20:38Z

Client must set Content-MD5 in this case and the problem is solved for all S3 providers. Is there a specific reason argo-cd is trying to look for the exact error returned in this case?

harshavardhana · 2023-08-21T08:30:20Z

This is what mc does if set the content-md5 if the bucket has ObjectLockEnabled.

tooptoop4 · 2024-11-18T20:39:17Z

would u like to raise PR @harshavardhana @akloss-cibo ?

akloss-cibo · 2025-01-16T18:52:22Z

I have hacked together a solution that does seem to work for us. The only required change is

diff --git a/s3/s3.go b/s3/s3.go
index e7fda4b..7ebd146 100644
--- a/s3/s3.go
+++ b/s3/s3.go
@@ -219,7 +219,7 @@ func (s *s3client) PutFile(bucket, key, path string) error {
                return err
        }

-       _, err = s.minioClient.FPutObject(s.ctx, bucket, key, path, minio.PutObjectOptions{ServerSideEncryption: encOpts})
+       _, err = s.minioClient.FPutObject(s.ctx, bucket, key, path, minio.PutObjectOptions{SendContentMd5: true, ServerSideEncryption: encOpts})
        if err != nil {
                return err
        }
%

in argoprog/pkg. This is not a very elegant solution.

To my eye, augmenting S3ClientOpts to include an option for this seems reasonable. I'm not a skilled golang engineer so if anyone wants to send me down a different path first, please speak up.

This checksum is required if a bucket has Object Lock enabled by default. addresses argoproj/argo-workflows#11182

This checksum is required if a bucket has Object Lock enabled by default. addresses argoproj/argo-workflows#11182 Signed-off-by: Alec Kloss <[email protected]>

akloss-cibo · 2025-01-22T16:46:30Z

I'm not quite sure how to get someone's attention to look at argoproj/pkg#790... help?

This checksum is required if a bucket has Object Lock enabled by default. addresses argoproj/argo-workflows#11182 Signed-off-by: Alec Kloss <[email protected]>

akloss-cibo · 2025-01-24T05:03:40Z

I have this working on-site with some images I built but I'm stuck in the make pre-commit -B; this bit of the docker part exit code 1s with no other messages:

% docker run --rm --user 501:20 -v /Users/alec/go/src/github.com/argoproj/argo-workflows/sdks/java/client:/wd --workdir /wd openapitools/openapi-generator-cli:v5.2.1 \
                generate -v \
                -i /wd/swagger.json \
                -g java \
                -o /wd \
                -p hideGenerationTimestamp=true \
                -p serializationLibrary=jsonb \
                -p dateLibrary=java8 \
                --api-package io.argoproj.workflow.apis \
                --invoker-package io.argoproj.workflow \
                --model-package io.argoproj.workflow.models \
                --skip-validate-spec \
                --group-id io.argoproj.workflow \
                --artifact-id argo-client-java \
                --artifact-version 0.0.0-SNAPSHOT \
                --import-mappings Time=java.time.Instant \
                --import-mappings Affinity=io.kubernetes.client.openapi.models.V1Affinity \
                --import-mappings ConfigMapKeySelector=io.kubernetes.client.openapi.models.V1ConfigMapKeySelector \
                --import-mappings Container=io.kubernetes.client.openapi.models.V1Container \
                --import-mappings ContainerPort=io.kubernetes.client.openapi.models.V1ContainerPort \
                --import-mappings EnvFromSource=io.kubernetes.client.openapi.models.V1EnvFromSource \
                --import-mappings EnvVar=io.kubernetes.client.openapi.models.V1EnvVar \
                --import-mappings HostAlias=io.kubernetes.client.openapi.models.V1HostAlias \
                --import-mappings Lifecycle=io.kubernetes.client.openapi.models.V1Lifecycle \
                --import-mappings ListMeta=io.kubernetes.client.openapi.models.V1ListMeta \
                --import-mappings LocalObjectReference=io.kubernetes.client.openapi.models.V1LocalObjectReference \
                --import-mappings ObjectMeta=io.kubernetes.client.openapi.models.V1ObjectMeta \
                --import-mappings ObjectReference=io.kubernetes.client.openapi.models.V1ObjectReference \
                --import-mappings PersistentVolumeClaim=io.kubernetes.client.openapi.models.V1PersistentVolumeClaim \
                --import-mappings PodDisruptionBudgetSpec=io.kubernetes.client.openapi.models.V1beta1PodDisruptionBudgetSpec \
                --import-mappings PodDNSConfig=io.kubernetes.client.openapi.models.V1PodDNSConfig \
                --import-mappings PodSecurityContext=io.kubernetes.client.openapi.models.V1PodSecurityContext \
                --import-mappings Probe=io.kubernetes.client.openapi.models.V1Probe \
                --import-mappings ResourceRequirements=io.kubernetes.client.openapi.models.V1ResourceRequirements \
                --import-mappings SecretKeySelector=io.kubernetes.client.openapi.models.V1SecretKeySelector \
                --import-mappings SecurityContext=io.kubernetes.client.openapi.models.V1SecurityContext \
                --import-mappings Toleration=io.kubernetes.client.openapi.models.V1Toleration \
                --import-mappings Volume=io.kubernetes.client.openapi.models.V1Volume \
                --import-mappings VolumeDevice=io.kubernetes.client.openapi.models.V1VolumeDevice \
                --import-mappings VolumeMount=io.kubernetes.client.openapi.models.V1VolumeMount \
                --generate-alias-as-model
% echo $?
1
%

Help?

This change will allow objects to be written to buckets with S3 Object Lock enabled. fixes argoproj#11182 Signed-off-by: Alec Kloss <[email protected]>

akloss-cibo added the type/bug label Jun 5, 2023

juliev0 added the P3 Low priority label Jun 8, 2023

This comment was marked as resolved.

Sign in to view

stale bot added the problem/stale This has not had a response in some time label Sep 17, 2023

terrytangyuan removed the problem/stale This has not had a response in some time label Sep 20, 2023

agilgur5 added the area/artifacts S3/GCP/OSS/Git/HDFS etc label Oct 5, 2023

akloss-cibo added a commit to cibotech/argoproj-pkg that referenced this issue Jan 21, 2025

add flag to enable MD5 checksum for S3 upload

d002fe1

This checksum is required if a bucket has Object Lock enabled by default. addresses argoproj/argo-workflows#11182

akloss-cibo mentioned this issue Jan 21, 2025

add flag to enable MD5 checksum for S3 upload argoproj/pkg#790

Merged

akloss-cibo linked a pull request Jan 31, 2025 that will close this issue

fix: add content-md5 header for artifacts #14140

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Artifacts cannot be stored in object-lock enabled S3 buckets #11182

Artifacts cannot be stored in object-lock enabled S3 buckets #11182

akloss-cibo commented Jun 5, 2023 •

edited

Loading

juliev0 commented Jun 8, 2023

akloss-cibo commented Jun 8, 2023

juliev0 commented Jun 8, 2023 •

edited by agilgur5

Loading

harshavardhana commented Aug 21, 2023

harshavardhana commented Aug 21, 2023 •

edited

Loading

This comment was marked as resolved.

tooptoop4 commented Nov 18, 2024

akloss-cibo commented Jan 16, 2025

akloss-cibo commented Jan 22, 2025

akloss-cibo commented Jan 24, 2025 •

edited

Loading

Artifacts cannot be stored in object-lock enabled S3 buckets #11182

Artifacts cannot be stored in object-lock enabled S3 buckets #11182

Comments

akloss-cibo commented Jun 5, 2023 • edited Loading

Pre-requisites

What happened/what you expected to happen?

Version

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

Logs from the workflow controller

Logs from in your workflow's wait container

juliev0 commented Jun 8, 2023

akloss-cibo commented Jun 8, 2023

juliev0 commented Jun 8, 2023 • edited by agilgur5 Loading

harshavardhana commented Aug 21, 2023

harshavardhana commented Aug 21, 2023 • edited Loading

This comment was marked as resolved.

tooptoop4 commented Nov 18, 2024

akloss-cibo commented Jan 16, 2025

akloss-cibo commented Jan 22, 2025

akloss-cibo commented Jan 24, 2025 • edited Loading

akloss-cibo commented Jun 5, 2023 •

edited

Loading

juliev0 commented Jun 8, 2023 •

edited by agilgur5

Loading

harshavardhana commented Aug 21, 2023 •

edited

Loading

akloss-cibo commented Jan 24, 2025 •

edited

Loading