Add download step from csds

[billfreeman44]

No description provided.

[razor-x]

@billfreeman44 There are not tested, but should be all we need to setup a basic bucket config.

days_to_keep_data_on_s3 = 30
account_id = boto3.client("sts").get_caller_identity().get("Account")
bucket_name = "-".join([account_id, "pureskillgg", "dataset"])
bucket_region = boto3.session.Session().region_name

client = boto3.client("s3")

client.create_bucket(
    Bucket=bucket_name,
    ACL="private",
    CreateBucketConfiguration={"LocationConstraint": bucket_region},
)

client.put_bucket_policy(
    Bucket=bucket_name,
    Policy=json.dumps(
        {
            "Version": "2012-10-17",
            "Statement": [
                {
                    "Effect": "Allow",
                    "Principal": {"Service": "dataexchange.amazonaws.com"},
                    "Action": ["s3:PutObject", "s3:PutObjectAcl"],
                    "Resource": f"arn:aws:s3:::{bucket_name}/*",
                    "Condition": {"StringEquals": {"aws:SourceAccount": account_id}},
                }
            ],
        }
    ),
)

client.put_bucket_lifecycle_configuration(
    Bucket=bucket_name,
    LifecycleConfiguration={
        "Rules": [
            {
                "Expiration": {
                    "Days": days_to_keep_data_on_s3,
                },
                "Status": "Enabled",
            },
        ],
    },
)

[razor-x]

We should add a check at the top of the cell incase the bucket exists. There are three cases for this call

client.head_bucket(
  Bucket='string',
  ExpectedBucketOwner=account_id
)

1. Bucket exists and is owned by account_id so no error and we can continue without creating the bucket / acl / lifecycle (assume it's good to go).
2. Bucket exists and is NOT owned by account_id, so they need a different bucket name or are on the wrong account.
3. Bucket does not exist so we can go ahead and create it.

[billfreeman44]

Should be good to go @razor-x

[razor-x]

Ok I'll check it one more time after these updates, but I have to head off for now. I recommend batching the suggestions instead of single commits!