AWS v2.30 SDK InputStream behavior changes

[rphilippen]

Describe the bug

Starting from AWS SDK version 2.30.0, the multipart upload calls ContentStreamProvider::newStream() twice on the provider passed to RequestBody::fromContentProvider(ContentStreamProvider provider, long contentLength, String mimeType). This leads to two InputStream instances being created, but only the second one gets closed. If code using the SDK creates an InputSteam that is tied to an OS handle (like reading from file), the OS handle leaks.

For quite a few streams in Java, it doesn't really matter if you close them or not. Typically this goes for "in-memory" streams, such as ByteArrayInputStream; after all, there is no underlying handle to release. However, if you provide an InputStream that has has a native OS handle attached to it (such as a file handle), the container may run out of file handles and crash.

For our use case, this issue is blocking us from upgrading to version 2.30, simply because it breaks our exports feature. We unload data (chunks) onto S3. We then load these chunks (one-by-one, in order) from our service to run them through a zip stream, and upload the result as parts. This process has been running fine for years with the AWS sdk up to version 2.29.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

The AWS SDK should close any InputStream's it asks, preferably only once.
Preferably, the SDK should also only ask for the InputStream once, as producing the stream may be an expensive operation.

Current Behavior

The SDK invokes newStream() on the provided ContentStreamProvider twice, but only closes the second stream. This has been observed by adding logging in our application, which produced this output (heavily simplified):

Requested newStream() -> InputStream@49022cf9 (active streams: 1)
Requested newStream() -> InputStream@79fe40f (active streams: 2)
Closed InputStream@79fe40f (active streams: 1)

(No more logs related to this)

I'm pretty confident this issue was first introduced in #4492. Specifically, the AwsChunkedV4PayloadSigner::beforeSigning method calls newStream(), but does not close it.

    public void beforeSigning(SdkHttpRequest.Builder request, ContentStreamProvider payload) {
        long encodedContentLength = 0;
        long contentLength = moveContentLength(request, payload != null ? payload.newStream() : new StringInputStream(""));

It doesn't close it, because the SignerUtils::moveContentLength method takes an InputStream, but never closes it.

Reproduction Steps

Writing out a realistic example for a multipart upload is quite complicated, especially with the requirement of any non-last parts to have a size > 5mb. In short, to trace the issue, start a multipart upload (we use the synchronous S3Client), and upload a part using a ContentStreamProvider that has been patched to track "open" (ContentStreamProvider::newStream()) and close (InputStream::close()) actions. We're providing the content-length (as we know it), as well as a mime-type.

Expand for an extremely simplified example of the issue.

package example;

import java.io.*;
import java.util.*;
import java.util.concurrent.atomic.AtomicInteger;

import software.amazon.awssdk.core.sync.RequestBody;
import software.amazon.awssdk.http.ContentStreamProvider;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.model.*;

public class MultiPartUploadIssueExample {
  /**
   * Tiny wrapper around {@link ByteArrayInputStream}, adding logic on {@link #close()}: calls the
   * provided callback and resets the reference to the shared buffer.
   */
  private static class ByteArrayInputStreamWithCallback extends ByteArrayInputStream {
    private final Runnable closeCallback;

    public ByteArrayInputStreamWithCallback(byte[] buf, Runnable closeCallback) {
      super(buf);
      this.closeCallback = Objects.requireNonNull(closeCallback);
    }

    @Override
    public synchronized void close() throws IOException {
      super.close();
      closeCallback.run();
    }
  }

  private static class CountingContentProvider implements ContentStreamProvider {
    private final byte[] data;
    private final AtomicInteger count;

    public CountingContentProvider(byte[] data) {
      this.data = Objects.requireNonNull(data);
      this.count = new AtomicInteger();
    }

    @Override
    public InputStream newStream() {
      count.incrementAndGet();
      return new ByteArrayInputStreamWithCallback(data, count::decrementAndGet);
    }

    public long contentLength() {
      return data.length;
    }

    public int getOpenCount() {
      return count.get();
    }
  }

  private final S3Client s3Client;

  public MultiPartUploadIssueExample(S3Client s3Client) {
    this.s3Client = s3Client;
  }

  public CompleteMultipartUploadResponse executeExample(String bucketName, String targetFile, String singlePart) {
    CreateMultipartUploadRequest createMultipartUploadRequest =
        CreateMultipartUploadRequest.builder()
            .bucket(bucketName)
            .key(targetFile)
            .build();

    final CreateMultipartUploadResponse pendingMultipartUpload =
        s3Client.createMultipartUpload(createMultipartUploadRequest);

    try {
      List<CompletedPart> uploadedParts = new ArrayList<>(1);

      // This would normally be in a loop
      int partNumber = 1;
      CountingContentProvider provider = new CountingContentProvider(singlePart.getBytes());

      RequestBody requestBody = RequestBody.fromContentProvider(provider, provider.contentLength(), "application/octet-stream");

      UploadPartRequest.Builder uploadPartRequestBuilder =
          UploadPartRequest.builder()
              .bucket(bucketName)
              .key(targetFile)
              .uploadId(pendingMultipartUpload.uploadId())
              .partNumber(partNumber);
      requestBody.optionalContentLength().ifPresent(uploadPartRequestBuilder::contentLength);

      UploadPartResponse uploadPartResponse =
          s3Client.uploadPart(uploadPartRequestBuilder.build(), requestBody);

      // Theoretically, this should report 0; but in practice it reports 1 open stream.
      System.out.println("Open stream handles after upload: " + provider.getOpenCount());

      uploadedParts.add(CompletedPart.builder().partNumber(partNumber).eTag(uploadPartResponse.eTag()).build());
      // End of hypothetical loop

      CompletedMultipartUpload completedMultipartUpload =
          CompletedMultipartUpload.builder().parts(uploadedParts).build();
      CompleteMultipartUploadRequest completeMultipartUploadRequest =
          CompleteMultipartUploadRequest.builder()
              .bucket(bucketName)
              .key(targetFile)
              .uploadId(pendingMultipartUpload.uploadId())
              .multipartUpload(completedMultipartUpload)
              .build();

      return s3Client.completeMultipartUpload(completeMultipartUploadRequest);
    } catch (Exception e) {
      s3Client.abortMultipartUpload(
          AbortMultipartUploadRequest.builder()
              .bucket(bucketName)
              .key(targetFile)
              .uploadId(pendingMultipartUpload.uploadId())
              .build());
      throw e;
    }
  }
}

Possible Solution

Obviously: close all user-provided streams the SDK asks for. There is some nuance to this though: the SDK should also be mindful of memory usage when working with data to upload.

The use case we are using the multipart upload with is basically:

• an external data base unloads data into S3 (often spread over multiple files).
• we iterate the chunks sequentially, reading them from S3 (one-by-one), streaming the data into a zip stream.
• The output of the zip stream is uploaded in chunks, so we are more resilient against network issues (only have to re-do a part, not the whole large file, if a part fails).

We used to have a naïve process that would keep all data in memory. It is probably no surprise that our service tended to Out-of-Memory on larger exports (think 1gb range). About two years ago, we re-wrote the implementation to be fully streaming. The data is streamed directly from S3 (the getObject response) into the zip output stream. The output stream is a custom byte buffer that follows an "exclusive write, multiple read" pattern; and it enforces this. To write to the stream, no outstanding readers may be active. It is allowed to read multiple times from the same buffer though. With this set-up, an export takes about 20Mb of ram to process; regardless of how big the actual export is. There are some read buffers, and we have a 16Mb chunk buffer that is written to, and read from, repeatedly.

It was the sanity check of this shared buffer class that alerted us to the issue in sdk version 2.30.0 and upwards. Once the first part was done uploading to S3, the streaming zip operation requested write access to the buffer, but that flagged (via an exception) that there was still a reader open.

I want to pinpoint a specific unfortunate abstraction. Look at StreamManagingStage::ClosingStreamProvider and ask yourself: what happens if newStream() is called multiple times, before closeCurrentStream() is invoked? The answer is of course: any non-latest currentStream value is overwritten, and those streams are leaked.

Additional Information/Context

To add some unsolicited advice:
I think it's best to take a good look at the SDK's usage of streams, and revisit how input data is handled. This summary may be too simple (I may be missing something), but to my knowledge, the SDK needs to know:

1. The content-length of the body about to be uploaded.
2. A checksum of the body (requires iterating the content).
3. The body itself, for the http request.

I want to clarify that I grasp the complexity of all the possible use-cases the SDK has to support. If your input is just an InputStream, you'll have to consume the stream at least twice; once to determine the length and checksum, and once more to actually stream the content to the http client (during upload). I think the SDK could do a better job of abstracting the common issues away. But at its core, the problem is also fuzzy ownership (and Java doesn't help there). Once you take ownership of an InputStream, you need to close it. Sharing an InputStream between multiple readers is dangerous, and requires sanity check to ensure the stream is not read concurrently by multiple parties.

What I think the SDK should do, to be both memory efficient and clear in its usage pattern, is either:

1. Only support byte[] inputs (and clarifying ownership of the byte[] is transferred to the SDK, e.g. no more writes should be performed).
2. Be smart about InputStream: if it supports resetting, use resetting when needing to read multiple times. Only copy a stream's contents to an in-memory buffer if it doesn't support resetting.

I notice attempts to avoid shared ownership issues were made in #5837 and #5841, by simply copying the data. Blatantly copying all data is an anti-pattern though. As I described in "Possible solution", we've worked hard to make our export process memory efficient. With the AWS SDK just copying data "to be safe", you've basically taken away any control from end-users to avoid going Out-of-Memory on large uploads. I can't resist pointing out it didn't take long for that to happen, as evidenced by issue #5850.

AWS Java SDK version used

2.30.0

JDK version used

openjdk 21.0.1 2023-10-17 LTS

Operating System and version

Windows 11 24H2, build 26100.2894

[zoewangg]

Taking a look

[zoewangg]

Hi @rphilippen, thank you for your report and feedback!

Let me first address the use of newStream() and how it should be closed. It is by design that newStream() may be invoked multiple times throughout the lifecycle of a request, for instance, first calculating content-length for S3 putObject if it's not already provided and then sending it over the wire, as you mentioned.

Per the contract of ContentStreamProvider, the user is responsible for closing the streams unless it's the last attempt. This is because the user may intend to return the same stream that is resettable instead of a new stream every time newStream() is invoked.

Copy-pasting the Javadoc for ContentStreamProvider here,

Each call to the newStream() method must result in a stream whose position is at the beginning of the content. Implementations may return a new stream or the same stream for each call. If returning a new stream, the implementation must ensure to close() and free any resources acquired by the previous stream. The last stream returned by newStream() will be closed by the SDK.

However, there are places where we can avoid calling newStream(). For example, if content length is known, there is no need to call payload.newStream() in this line moveContentLength(request, payload != null ? payload.newStream() : new StringInputStream(""));. We will do a check in all places where newStream() is invoked and remove unnecessary invocations.

Second, wrt buffering and memory usage, we understand buffering everything in memory is not a good practice, and in hindsight, we should not have supported streaming with unknown content length by buffering in the first place, and we can’t remove the support at this point unfortunately. That is why we recommend that users provide a resettable stream and use RequestBody#fromInputStream for streaming with known content length. For streaming with unknown content length, we recommend following this Dev Guide. We will improve our documentation to make it more discoverable. On a side note, we have a plan to support streaming with unknown content length w/o buffering.

Finally, wrt your suggestions:

Only support byte[] inputs (and clarifying ownership of the byte[] is transferred to the SDK, e.g. no more writes should be performed).

Unfortunately, we can't remove the existing APIs due to backwards compatibility reasons.

Be smart about InputStream: if it supports resetting, use resetting when needing to read multiple times. Only copy a stream's contents to an in-memory buffer if it doesn't support resetting.

This is what we intend to do in the latest code, as you can see here. If an InputStream supports resetting, we don't buffer the content. #5850 is a bug and not a side effect [Edit: not an intended side effect for SDK ContentStreamProvider implementation] of recent change, and a fix has been released in 2.30.13.

Please let us know if you have further questions or suggestions.

[rphilippen]

Only supporting byte[] (or any "equivalent" like SdkBytes) would be a severe step down of functionality. Still, it can be an API choice. I only added it as option to say: if AWS SDK doesn't want to deal with InputStream's need for lifetime management (needing to close()), it's perhaps a better option to just not support it, than dealing with it in a weird way. I personally think the SDK should strive to support InputStream, even if it is complicated.

Each call to the newStream() method must result in a stream whose position is at the beginning of the content. Implementations may return a new stream or the same stream for each call. If returning a new stream, the implementation must ensure to close() and free any resources acquired by the previous stream. The last stream returned by newStream() will be closed by the SDK.

This is an extremely weird ownership set-up. It requires users of the AWS SDK, that provide the streams, to know when they can close the stream. It means all user code handing out streams needs to track them, so they can close them when the AWS SDK asks for the next one. That seems error-prone to me, for something that I personally think the AWS SDK should take responsibility for: closing any streams it asks for.

Also note that if we follow this JavaDoc as to how the AWS SDK should behave regarding streams, there is no need to copy data from streams as was done in #5837 and #5841. Following that documented behavior, the AWS SDK should just call newStream() again, and those PRs should have been rejected.

I already alluded to this in my description, but we switched to a streaming set-up for file uploads, as our service producing the exports ran out of memory for large files. We've coded a byte buffered output stream that shares its byte[] with reader input streams, but also does lifetime checks to guarantee nothing writes to the buffer while somebody still has read access. (For this reason, readers need to close, to signal they're done). For our export job we use a buffer of 16Mb, and that single byte buffer is re-used for every part. With this set-up, our service has been running smoothly for a year of two, using many versions of the AWS SDK, producing large export files while using only about ~20Mb of RAM per export job.

#5850 is a bug and not a side effect of recent change, and a fix has been released in 2.30.13.

I'm sorry, but this statement ("not a side effect of recent change") is just false. The issue report literally starts with "Starting with v2.30.9, …". Besides that, this issue is clearly introduced by #5841, and your fix in #5855 partially "reverts" that change, by avoiding the copy behavior for an internal use-case. What worries me about the original change, combined with the subsequent fix you did, is that it still copies the stream for anything external to the AWS SDK. That means it practically breaks our export job, as we get back the risk of going Out-of-Memory, because the AWS SDK is copying our InputStream.

Currently, BufferingContentStreamProvider (introduced in #5837) is even doing that copying in a memory inefficient way. The class always creates the buffered stream with default initial capacity (8192 bytes, code is here), and proceeds to mark with a limit of Integer.MAX_VALUE. This means the internal buffer will have to grow up to the input length during read, very likely through multiple iterations. That is very wasteful if you already know the length up front (e.g. expectedLength != null).

I know the last point is easily fixed, but then newer versions of the AWS SDK are still copying our 16Mb byte[] backed InputStream. That's completely unnecessary, as the byte[] backed InputStream we provide can be reset to any offset, or even be read multiple times, without any copying of data.

[zoewangg]

I hear your concerns and feedback. Unfortunately, there is not a lot we can do due to backwards compatibility reasons. If users return a new stream every time newStream is invoked, it is recommended to close the stream when newStream is invoked (except the first time before the stream is created).

"not a side effect of recent change"

Sorry, let me clarify, I meant to say it was not an intended side effect for SDK ContentStreamProvider implementations. For your use-case, are you able to use RequestBody#fromInputStream and provide a resettable stream?

We will look into optimizing BufferingContentStreamProvider.

As a side note, #5866 is submitted to avoid newStream() call if content-length is known.

[rphilippen]

Let me start by saying: I realize the title of this issue is was incorrect; the thing I am complaining about is actually documented behavior (on ContentStreamProvider::newStream()). I want to stress that in practice, the AWS SDK behaved as you'd hope for up to version 2.30.0. Also, the recent addition of BufferedContentStreamProvider was a breaking change regarding the behavior of RequestBody.fromContentProvider. I think there's an opportunity to make the SDK more well behaved with regards to InputStream handling.

For your use-case, are you able to use RequestBody#fromInputStream and provide a resettable stream?

Yes, we can, our stream is resettable. The reason I went for fromContentStreamProvider over fromInputStream while working on our export feature, is because fromInputStream and fromContentProvider looked like this in the AWS SDK version I built the feature against at the time, 2.17.251:

    /**
     * Creates a {@link RequestBody} from an input stream. {@value Header#CONTENT_LENGTH} must
     * be provided so that the SDK does not have to make two passes of the data.
     * <p>
     * The stream will not be closed by the SDK. It is up to to caller of this method to close the stream. The stream
     * should not be read outside of the SDK (by another thread) as it will change the state of the {@link InputStream} and
     * could tamper with the sending of the request.
     * <p>
     * To support resetting via {@link ContentStreamProvider}, this uses {@link InputStream#reset()} and uses a read limit of
     * 128 KiB. If you need more control, use {@link #fromContentProvider(ContentStreamProvider, long, String)} or
     * {@link #fromContentProvider(ContentStreamProvider, String)}.
     *
     * @param inputStream   Input stream to send to the service. The stream will not be closed by the SDK.
     * @param contentLength Content length of data in input stream.
     * @return RequestBody instance.
     */
    public static RequestBody fromInputStream(InputStream inputStream, long contentLength) {
        IoUtils.markStreamWithMaxReadLimit(inputStream);
        InputStream nonCloseable = nonCloseableInputStream(inputStream);
        return fromContentProvider(() -> {
            if (nonCloseable.markSupported()) {
                invokeSafely(nonCloseable::reset);
            }
            return nonCloseable;
        }, contentLength, Mimetype.MIMETYPE_OCTET_STREAM);
    }

    /**
     * Creates a {@link RequestBody} from the given {@link ContentStreamProvider}.
     *
     * @param provider The content provider.
     * @param contentLength The content length.
     * @param mimeType The MIME type of the content.
     *
     * @return The created {@code RequestBody}.
     */
    public static RequestBody fromContentProvider(ContentStreamProvider provider, long contentLength, String mimeType) {
        return new RequestBody(provider, contentLength, mimeType);
    }

(For reference, the links to RequestBody and ContentStreamProvider of version 2.17.251)

I overlooked the detail in the JavaDoc of ContentStreamProvider, about it only closing the last stream. However, I want to note that the JavaDoc on fromInputStream strongly suggested that the stream was only consumed twice if no Content-Length is provided; but we do provide that. So the choice was between a method that doesn't close the stream it gets, or a method that in practice does. RequestBody.fromContentProvider always called newStream() only once, and closed the stream it requested (well probably the http client code does that).

So in practice the AWS SDK behaved exactly as desired, by using RequestBody.fromContentProvider with provided content length (at least up to version 2.30.0):

• It only called newStream() once.
• It almost always closed the stream it asked for.
  If it didn't, it was a sign that something went wrong. Simply ignoring that and closing the streams ourselves would lead to the upload failing at the end, complaining that part had 0 bytes. So we check if the stream is closed in our code. A quick search in our production system logs shows this hasn't happened in the past 3 months, so it looks like this issue was fixed over time. 🎉

To give some more background: our export code runs using two threads: a producer and a consumer. The producer thread reads the input streams (from S3), and writes to a zip stream that is wrapping a PipeOutputStream. On the consumer thread, we fill the buffer from the connected PipeInputStream. Once the buffer is full, the consumer thread switches to uploading that part to S3. We use this "exclusive write, shared read" buffer for this. The nice thing about this set-up is:

• The producer thread will automatically halt ("pause") if the buffers are full, because the consumer is behind.
• The consumer thread will automatically halt ("wait") if the buffers are empty, because the producer is behind.

Of course, the whole synchronization depends on knowing when the consumer can release the shared buffer so the producer thread can write to it again. Why am I sharing these details? To establish that closing the InputStream is part of a sane contract to say "I'm done with this resource". Conversely, asking for a stream and then claiming the provider needs to close it for you, is not a sane approach in my opinion. In a single-threaded set-up it might be relatively safe to assume the stream is consumed after the request method returns, but it's still a fragile set-up.

---

Version 2.30.0 was a behavioral breaking change with regards to how streams are dealt with. This is not an "accusation" or in any way meant to sound as a negative, it's just the reality of what happened. I think I understand why: due to the checksum updates. You need to stream the content to determine the checksum, and you need to stream it again to actually send it. Then, a few versions later (2.30.8 and 2.30.9), @dagnir did a behavior breaking changes to add buffering to all InputStreams. Again, not "accusing", simply ascertaining this changed the AWS SDK behavior in a significant way.

The problem the AWS SDK faces, is that it often (but not always) needs to iterate content twice. Optionally to determine the content length and/or a checksum of the data, and obviously it needs to actually stream the content into an http request. For any InputStream, there is no surefire way to tell if you can re-read the full stream. If it's an in-memory buffer, you can, but with a file or network stream, it may not be that simple.

I understand the way the AWS SDK tried to solve it before v2.30, although I also think it did so (and does that) with flawed behavior:

• When users provide an InputStream, it doesn't close it.
  Why? Because internally, it may need to read the InputStream multiple times. That is an SDK internal problem, but the solution chosen puts the burden of releasing the stream on users of the SDK.
• The AWS SDK defines a ContentStreamProvider (basically a Supplier<InputStream>), so it can ask for the content stream multiple times. The idea there is that users can be "smart" about this by returning the same stream. The API contract has hard-coded the assumption that streams are going to be re-used (same instance), by saying: we won't close anything but the last requested stream for you.

So what are typical use-cases for end-users?

• In-memory content (as example, ByteArrayInputStream). Resetting the stream completely to position 0 is arbitrary, as all data is accessible. Closing the stream can release the memory in specialized implementations (like our shared memory buffer), but usually is mostly a no-op.
• External content
  • File (as example, FileInputStream). For large files, re-opening a file stream may be less costly and better than caching the full content in-memory. Closing the stream is a must, not closing it leaks a (file) OS handle.
  • Network (as example, AWS's ResponseInputStream<TResponse>). Here the trade-off between "re-opening" (recreating the connection to the same resource) or copying the contents in-memory is not so simple, it's heavily use-case dependent; e.g. is the connection to a network-local resource, or over the internet. Closing the stream is important, otherwise the connection may remain open (although it'll probably time out at some point).
• Generated content
  An example of generated content is our zip process, which transforms input into something else that is costly to recreate. However, the generated content may also be trivial to regenerate, it depends heavily on the use-case.

Now lets see what options the AWS SDK provides us to deal with these scenarios. In-memory content is easy, the stream is resettable, and creating a new input stream on the same in-memory content is also cheap. For external and generated content, we always have the option to turn it into in-memory content (by putting a BufferedInputStream around it). However, for large content, users may want to simply re-open a file handle for example. Now the SDK behaves poorly for us: we open a file handle on newStream(), but we also have to track it, so we can close it in case the SDK doesn't do that. The point I'm trying to make here: the SDK should always close any InputStream it receives, because it is so much safer and reliable than "optimizing" for the use-case where SDK users want to return the same stream. (Arguably, those use cases should have pointed people to RequestBody.fromInputStream, which didn't close the stream).

If users return a new stream every time newStream is invoked, it is recommended to close the stream when newStream is invoked (except the first time before the stream is created).

With the recent change to always use BufferingContentStreamProvider, the whole idea behind ContentStreamProvider has been nerfed completely. The SDK will now always copy the input to be in-memory, even if it already is. Or, in other words: the SDK will now only call newStream() once, taking away any control over the choice to re-stream data, instead of copying it in-memory.

---

Of course it's easy to point out flaws, but how do you make the AWS SDK "well-behaved" in all use cases, without unnecessary copying of data? Here is my proposal for behavior that I think covers all use cases well, and makes the SDK predictable. Note that the SDK already has all the API definitions in place, it just needs to change its behavior.

• RequestBody.fromInputStream should wrap the input stream in a BufferedInputStream if markSupported == false when it needs to read multiple times, documenting this behavior. For streams that can be reset, it should just use them as-is.
  The SDK should always close the source input stream once it's done reading. Whether to close or reset the stream is an SDK internal problem that should not be "pushed outside" to users of the AWS SDK.
• RequestBody.fromContentProvider should just blatantly call newStream() every time anything needs to iterate the content. (To be clear: without an in-memory buffer between it). Every iteration closes the stream it asked for once it's done. It is the user's responsibility that each returned stream yields exactly the same content, otherwise the checksum and/or length will fail.

This leaves end-users with guaranteed correct behavior, and the option to optimize memory usage in case necessary:

• If users naïvely provide a "OS resource holding" stream like FileInputStream, the SDK optionally buffers it, and will always close it. Regardless if it is provided directly, or created on newStream().
• For in-memory content, the SDK will never copy unnecessarily.
• For advanced use cases, users have the option to buffer in-memory themselves (like we do), or to use ContentStreamProvider and read from source multiple times.

If the AWS SDK wants to be mindful of memory usage, and thus only copy an InputStream if absolutely necessary, you basically need to know if the code requesting a stream (SDK internal) is the last one to do so. I think you know, when giving the stream to the HTTP client to provide the body it's always the last in the chain. Any other step like determining the length and/or a checksum of the content is intermediate. If you are at an intermediate step, and the input stream is non-resettable, only then do you need to copy its contents into an in-memory buffer. If the source stream is resettable, the SDK can provide intermediate parties with a wrapper stream that wraps the source stream, but only resets the source stream on close(), instead of actually closing it.

As a bonus, if the SDK always closes every stream it asks/receives, there are also numerous opportunities in the code base to use Java's try-with-resources feature, making it much easier to guarantee the AWS SDK doesn't leak resources, both on success and exception paths.

And yes, I realize this is a behavior breaking change that may have consequences for code relying on the "the SDK won't close my stream" quirk. I'd like to point out that 2.30.9 was also a behavior breaking change. Instead of potentially closing a stream that may be re-used though, it just removes any hope for streaming content that is too large to fit in memory (as it always copies).

---

Do with this feedback what you will. I've said my piece, I've made my arguments (hopefully clear). We can switch to the RequestBody.fromInputStream variant for now, and close the stream for the AWS SDK, as it documents it won't do that itself.

[steveloughran]

Am I understanding from this bug report that the latest version of the SDK now reads in and buffers all data provided by a content provider just so it can calculate a checksum?

That is going to destroy memory management where we upload data directly from files so we can have multiple threads uploading multiple 64 MB blocks at a time.

If we turn off this new checksum feature of #5801 -will the buffering go away?

[zoewangg]

To provide an update, the latest version 2.30.18 (as of today) does not buffer data if you use a custom content provider AND provide the content length

The SDK only buffers data if you use RequestBody#fromContentProvider w/o providing content length. This behavior is not configurable.

To upload objects with unknown content-length, we recommend using AWS CRT-based S3 client. https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/crt-based-s3-client.html

[steveloughran]

@zoewangg thanks. Reviewed all our uses of fromContentProvider() and length is passed in.