Multipart upload for large files without using credentials

[rgc183]

I am trying to achieve multipart upload. I have chunks coming in and I want to upload each chunk separately for a large file. I do not have credentials, so I will be using presigned URL for this, the only input to a program is bucket name and object name. I was looking into multi-part upload approach. Here is what I have:

#include <aws/core/Aws.h>
#include <aws/s3/S3Client.h>
#include <aws/s3/model/CompleteMultipartUploadRequest.h>
#include <aws/s3/model/CreateMultipartUploadRequest.h>
#include <aws/s3/model/UploadPartRequest.h>
#include <curl/curl.h>
#include <gflags/gflags.h>
#include <fstream>
#include <iostream>
#include <string>
#include <vector>

DEFINE_string(input_file, "", "Input file path to be written in cloud storage");
const size_t CHUNK_SIZE = 5 * 1024 * 1024;  // 5 MB chunks

size_t ReadCallback(char *buffer, size_t size, size_t nitems,
                    std::ifstream *readFile) {
  if (!readFile->eof()) {
    readFile->read(buffer, size * nitems);
    return readFile->gcount();
  }
  return 0;
}

// Function to generate a presigned URL for uploading a part
std::string generatePresignedUrl(
    Aws::S3::S3Client &s3Client, const std::string &bucketName,
    const std::string &objectKey, const std::string &uploadId, int partNumber,
    int expirationSeconds = 3600) {  // Default expiration: 1 hour
  Aws::Http::URI uri;
  auto expiration = Aws::Utils::DateTime::Now() + std::chrono::hours(1);
  Aws::String presignedUrl = s3Client.GeneratePresignedUrl(
      bucketName, objectKey, Aws::Http::HttpMethod::HTTP_PUT,
      expirationSeconds);

  // Append query parameters for multipart upload
  uri.SetQueryString("?partNumber=" + std::to_string(partNumber) +
                     "&uploadId=" + uploadId);
  return uri.GetURIString();
}

bool UploadPart(const std::string &presignedUrl, std::ifstream &file,
                size_t partSize) {
  CURL *curl = curl_easy_init();
  if (!curl) {
    std::cerr << "Failed to initialize CURL" << std::endl;
    return false;
  }

  curl_easy_setopt(curl, CURLOPT_URL, presignedUrl.c_str());
  curl_easy_setopt(curl, CURLOPT_UPLOAD, 1L);
  curl_easy_setopt(curl, CURLOPT_READFUNCTION, ReadCallback);
  curl_easy_setopt(curl, CURLOPT_READDATA, &file);
  curl_easy_setopt(curl, CURLOPT_INFILESIZE_LARGE,
                   static_cast<curl_off_t>(partSize));

  CURLcode res = curl_easy_perform(curl);
  curl_easy_cleanup(curl);

  return (res == CURLE_OK);
}

bool MultipartUploadS3Object(Aws::S3::S3Client &s3_client,
                             const std::string &bucketName,
                             const std::string &objectKey,
                             const std::string &filePath) {
  std::ifstream file(filePath, std::ios::binary | std::ios::ate);
  if (!file.is_open()) {
    std::cerr << "Failed to open file: " << filePath << std::endl;
    return false;
  }

  std::streamsize fileSize = file.tellg();
  file.seekg(0, std::ios::beg);

  // Initiate multipart upload
  Aws::S3::Model::CreateMultipartUploadRequest createMultipartUploadRequest;
  createMultipartUploadRequest.SetBucket(bucketName);
  createMultipartUploadRequest.SetKey(objectKey);

  auto createOutcome =
      s3_client.CreateMultipartUpload(createMultipartUploadRequest);
  if (!createOutcome.IsSuccess()) {
    std::cerr << "Failed to initiate multipart upload: "
              << createOutcome.GetError().GetMessage() << std::endl;
    return false;
  }

  std::string uploadId = createOutcome.GetResult().GetUploadId();

  // Upload parts
  std::vector<Aws::S3::Model::CompletedPart> completedParts;
  int partNumber = 1;
  for (std::streamsize offset = 0; offset < fileSize;
       offset += CHUNK_SIZE, ++partNumber) {
    size_t partSize =
        std::min(static_cast<std::streamsize>(CHUNK_SIZE), fileSize - offset);

    Aws::S3::Model::UploadPartRequest uploadPartRequest;
    uploadPartRequest.SetBucket(bucketName);
    uploadPartRequest.SetKey(objectKey);
    uploadPartRequest.SetUploadId(uploadId);
    uploadPartRequest.SetPartNumber(partNumber);
    uploadPartRequest.SetContentLength(partSize);
    auto expiration = Aws::Utils::DateTime::Now() + std::chrono::hours(1);
    auto presignedUrl = generatePresignedUrl(s3_client, bucketName, objectKey,
                                             uploadId, partNumber, 3600);
    // auto presignedUrl = s3_client.GeneratePresignedUrl(
    //    uploadPartRequest, Aws::Http::HttpMethod::HTTP_PUT, expiration);

    if (UploadPart(presignedUrl, file, partSize)) {
      Aws::S3::Model::CompletedPart completedPart;
      completedPart.SetPartNumber(partNumber);
      completedPart.SetETag(
          "");  // ETag is not available with presigned URL, leave it empty
      completedParts.push_back(completedPart);
    } else {
      std::cerr << "Failed to upload part " << partNumber << std::endl;
      return false;
    }
  }

  // Complete multipart upload
  Aws::S3::Model::CompleteMultipartUploadRequest completeMultipartUploadRequest;
  completeMultipartUploadRequest.SetBucket(bucketName);
  completeMultipartUploadRequest.SetKey(objectKey);
  completeMultipartUploadRequest.SetUploadId(uploadId);

  Aws::S3::Model::CompletedMultipartUpload completedMultipartUpload;
  completedMultipartUpload.SetParts(completedParts);
  completeMultipartUploadRequest.SetMultipartUpload(completedMultipartUpload);

  auto completeOutcome =
      s3_client.CompleteMultipartUpload(completeMultipartUploadRequest);
  if (!completeOutcome.IsSuccess()) {
    std::cerr << "Failed to complete multipart upload: "
              << completeOutcome.GetError().GetMessage() << std::endl;
    return false;
  }

  return true;
}

int main(int argc, char *argv[]) {
  Aws::SDKOptions options;
  Aws::InitAPI(options);
  gflags::SetUsageMessage("Video Effects Service usage");
  gflags::ParseCommandLineFlags(&argc, &argv, true);

  {
    Aws::Client::ClientConfiguration config;
    config.region = "ap-south-1";  // Replace with your region

    Aws::S3::S3Client s3_client(config);

    std::string bucketName = "streaming-nim-files";
    std::string objectKey = "your-object-key";
    // std::string filePath = "/path/to/your/file.txt";

    if (MultipartUploadS3Object(s3_client, bucketName, objectKey,
                                FLAGS_input_file)) {
      std::cout << "File uploaded successfully." << std::endl;
    } else {
      std::cerr << "Failed to upload file." << std::endl;
    }
  }

  Aws::ShutdownAPI(options);
  return 0;
}

Above code fails at line:

  auto createOutcome =
      s3_client.CreateMultipartUpload(createMultipartUploadRequest);
  if (!createOutcome.IsSuccess()) {
    std::cerr << "Failed to initiate multipart upload: "
              << createOutcome.GetError().GetMessage() << std::endl;
    return false;
  }

with error message: "Failed to initiate multipart upload: Anonymous users cannot initiate multipart uploads. Please authenticate"

It wont fail if I create client with authentication params. So seems like the approach I am trying with multipart upload will not work. How do I solve this problem?

[DmitriyMusatkin]

CreateMPU requires auth with aws and there is no workaround to call it if you dont have auth.
Note: using presigned url is not a magical way to avoid not using credentials. presinged urls are still signed with the credentials provided to the sdk. s3 does have a few operations that can be called without creds (if the bucket is public for example), but you can do so without using presigned urls

[rgc183]

thanks @DmitriyMusatkin for taking time to answer this. Regarding credentials, presigned url does not contain aws_secret_access_key, however when I generate AWS CLI token externally with account login, it has 3 components: aws_access_key_id, aws_secret_access_key and aws_session_token. When client is default constructed, it reads 3 components from ENV config file. Can you specify how do we initialize S3 client with fields extracted from pre-signed url, what all components it need to be extracted?

My use-case is I have client application has input: presigned URL of the output file. Now data is coming in the stream, so as soon as chunk is received instead of accumulating it in RAM/disk, I want to send it to S3.
Is this achievable with Aws::S3::S3Client or Aws::S3::HttpClient ? I was looking into curl APIs, however it was giving an error if use content-range in CURLOPT_HTTPHEADER so couldn't get it working with curl either.

[DmitriyMusatkin]

you cannot extract creds back from signed url. singing is a one way operation.

cpp sdk does not support using presigned urls to execute requests, so you will have use curl directly. Note that presigned url already has all the headers presigned and moved to query params, so adding more headers will lead to signature mismatches.

[rgc183]

@DmitriyMusatkin Is there any sample code/example which demonstrates how to use curl with above use case? If there is not any existing sample, can you share steps how to implement it?