-
|
I feel like I already know the answer to this, and it's not one I want, but is there any such thing as a Stable Version in this project? It just seems like it's a constant stream of new tags, every day or so, with no formal or consistent release notes or any real high-level indication of what's changed and whether a particular release is good or bad. Our product has used this SDK for many years, 1.3.10, 1.4.76, 1.7.83, 1.7.123, 1.7.301, 1.9.335, and back to 1.7.301 for unclear reasons (developer since departed). I inherited the long-overdue dependencies update task, and about six months ago I tried some newer versions, most recently 1.11.376, but it seems that there was an S3 access performance cliff somewhere before that point, so I literally did a brute-force bisection and found a subjective ~25% performance drop between 1.11.211 and 1.11.212, although looking at the (substantial) diff between those two versions gave no real insight as to what might have caused it. Even 1.11.211 was noticeably slower than 1.7.301 in high-level profiling. Obviously I'm aware that any profiling of S3 access is subject to the whims of the Internet, but I tried multiple runs at different times of day and the mean difference was still clear. There have been nearly 150 new versions since that point, now that I'm looking at it again. I'm about to build with the latest 1.11.517 and compare the performance again with my most recent fave, 1.11.211, and with our current 1.7.301. It's also not clear what version of Curl we should be using underneath it, plus we have Apache Arrow in the mix too, wanting to update that from 9 to at least 16. Our unit tests that go through Arrow didn't seem as affected by the later versions, so I tried stealing the Endpoint Provider Cache implementation from Arrow for our own non-Arrow S3 access code paths, which caches Any and all suggestions warmly welcomed. |
Beta Was this translation helpful? Give feedback.
Replies: 4 comments 29 replies
-
|
hey @simoneves thanks for reaching out and sorry about any frustration you may have, lets see if we cant work through it.
the way versioning works on this project is each version number is 1.11.x -> receives daily updates so the latest version of 1.11.x will receive model updates, and bug fixes, the older versions will not. So if you want the freshest version of the SDK 1.11.x is the latest version and will have all AWS models and bug fixes. So to answer your question, there is no "stable version" there is more so a "currently worked on version".
Each day we release new model for aws services. these models add new functionality for services. For instance last fridays added several features for different AWS services. for instance one of them is
which you can see reflected in that diff. This diff is not very helpful though and we plan to move to something like how go-v2 does it, but I cannot give a time frame on when we will get to that. all releases should be "good" and go through a daily CI.
The version that you are speaking on specifically introduced support for s3 express which we had to do some work for to support. But we've also pushed bug fixes for it and pushed several s3 features for it since, so really the best baseline is against the current HEAD of the project. Also you don't mention what the performance drop is in? what are you seeing a performance drop with? specifically with a reproducible example what are you doing where you see a performance drop
We dont have have a recommended version, its more or less bring what version you want to or that your system supports. I would defer to the curl project to determine the best version. That said there are some known buggy versions of curl on mac specifically that we have called out in the past.
Are you creating and destroying lots of S3Clients? |
Beta Was this translation helpful? Give feedback.
-
Hi Sam. Thank you for getting back to me. I apologize for my original message being somewhat combative.
OK, understood. I figured that's what the middle number meant, at least.
Thanks.
To clarify, I first tried the latest-at-the-time 1.11.376, and found this to be much slower than our previous 1.7.301. I bisected back, and found a small intermediate performance drop-off introduced at 1.11.212, but 1.11.211 is still a lot slower than 1.7.301, as (IIRC) are several other earlier 1.11.x and 1.10.x versions. I did not previously manage to bisect to that point, but I might try again, perhaps starting at the other end. I have not yet been able to pinpoint the actual SDK usage that is "slower", which I accept is something I need to concentrate on. I am simply referring to the overall runtime of some of our GoogleTest-based unit test executables for various types of database import/export operations which take up to 5x longer than they used to (e.g. 32 minutes vs 6 minutes). Obviously these include MANY tests of various types, although I am filtering to just those tests which use S3 in this context. Obviously there have also been code changes on our side, some by the previous (now departed) developer, some by be, to allow us to build against your newer versions at all. It's entirely possible that these changes introduced a bug or inefficiency, and the slow-down is not on your side at all.
Understood. This is all on Linux, Ubuntu 22 and RHEL 9, compiling with GCC 11 or 13, on x86 and ARM, although new S3 testing has only been on x86 so far.
My initial testing and profiling six months ago seemed to indicate that S3Client creation was one area where the actual SDK usage appeared to be called out as slower in the profiler. I have no frame of reference as to whether we are creating "too many", as I was not involved in the original code design and have no previous experience with the SDK. I will go away and do more experiments and analysis, and report back when I have something more constructive to say. Thank you again for your assistance! :) Simon |
Beta Was this translation helpful? Give feedback.
-
No you're fine! we love finding areas where we can help people, and if we're doing something bad we like hearing about it before we do more bad!
So this is something that is known and is actually something that pyarrow has actually called out #2880 to us before. tl;dr the logic of how endpoints are created each time is kind of expensive, so avoid creating lots of clients. If they share a endpoint provider like how pyarrow does it, this hit can be minimized to a single time. One general piece of advice i can give is that if possible, avoid calling client creation too many times. The client is thread safe and one should prefer to call the async methods on the client, instead of trying to manage many clients.
Yeah this might be tricky to find. If you can narrow down the usage that is slower we would love to take a look at it and hopefully fix it. where we spend a lot of time monitoring performance is the hot paths like GetObject or PutObject, which we should have caught. but if we didnt, more than happy to sort it out. I know this complicates a simple update task but if you want to try to use the s3-crt client instead of the normal "S3Client" you will almost certainly see a performance increase. It is essentially the same s3 client except GetObject, PutObject, and CopyObject are super charged through the CRT, which customers have usually found to be better. FWIW the big change between 1.7/1.8 and 1.9 was the introduction of the CRT libraries into the SDK which is why the old engineer may have downgraded in the past, as they can be tricky to work with, but should be a drop in replacement.. |
Beta Was this translation helpful? Give feedback.
-
So it seems we do create quite a few S3Clients, but the vast majority of the creations are well sub-millisecond. There are some which seem to take about 4 seconds, with both the old (1.7.301) and newer (1.11.211) SDKs. Not sure why that would be, but it's consistent. After belatedly activating some debug timer functionality, I have found something very fundamentally curious, though. At least in the context of the one specific test which for some reason causes the ~4 second create-client operations, it seems that all the major S3 operations (create client, list files, list objects, get object) ALL take almost exactly 2 seconds longer with the newer SDK than they do with the older one. Attached are two log snippets which show the same set of operations with the old and new SDK, and you can see that the creates take ~4 seconds with the old, and ~6 seconds with the new, and all the other operations take 100-200ms with the old, but 2100-2200ms with the new. Note that the logs are hierarchical (each group of lines at indent N are totaled by the line above at indent N-1, if that's not obvious). There are no client side code changes other than the minimal required for handling both SDK versions which is mostly to do with how we create S3Client objects, hence my concentration on that before. But the rest of the client-side code (that for all the other operations mentioned above) is identical. I will keep digging and report what I find, but the above is so curious I thought I'd mention it separately. What could possible cause a ~2 second delay on all those operations? |
Beta Was this translation helpful? Give feedback.
-
The 2-seconds thing seems so consistent and so cumulative I am wondering if there is some kind of silent timeout/retry behavior going on, no matter how unintuitive that seems. Google seems to think that the default timeout for at least the Java version of the SDK is 2 seconds, but the C++ docs and headers don't agree. Some seem to be 1000ms, others 3000ms. But I'm sure it's more complex/cleverer than that. Anyway, I tried messing with the three timeout values in the Turns out that the minimum version required by Arrow 16 (our desired version of that) is 1.10.55 so I tried that (no code changes needed), but that's even slower, subjectively the same as 1.11.212 and later! The log for that run shows that the Again, the only code difference on our side (other than some logging changes to do with the field type changes in New: Old: ...where (in my tests) |
Beta Was this translation helpful? Give feedback.
-
|
After belatedly realizing that the old-style Still has the two-second thing. I then tried 1.10.55, which is the version still recommended for Arrow 16.1 which is our chosen new version of that. Also still has the two-second thing. 1.7.301 continues to work perfectly. I guess I'll bisect back through 1.9 and 1.8 to try to find the break. |
Beta Was this translation helpful? Give feedback.
-
|
Note also that S3 access through Arrow (e.g. reading Parquet files) works normally with all AWS SDK versions 1.7 thru 1.11. |
Beta Was this translation helpful? Give feedback.
-
So before going further how are you actually building and distributing the old version, from our end it doesnt compile with warnings enabled on current platforms. So how are you actually building it/installing it/using it? i.e. FROM public.ecr.aws/amazonlinux/amazonlinux:2023
RUN yum groupinstall "Development Tools" -y
RUN yum install -y curl-devel openssl-devel ninja-build cmake3
# build and install SDK
RUN git clone -b 1.7.301 --recurse-submodules https://github.com/aws/aws-sdk-cpp && \
cd aws-sdk-cpp && \
mkdir build && \
cd build && \
cmake -DBUILD_ONLY="s3" -DCMAKE_INSTALL_PREFIX=/sdk-install -DAUTORUN_UNIT_TESTS=OFF .. && \
cmake --build . && \
cmake --install .So im even having a hard time trying to profile a diff between the two. That aside when you talk about measuring different things, its really hard to know what you are measuring and how, so lets focus on one example and build from there. heres a googlebenchmark test that profiles calling get object 20 times on a 100MB file. What performance difference do you see while using that CMakeLists.txt cmake_minimum_required(VERSION 3.13)
project(sdk_benchmark)
set(CMAKE_CXX_STANDARD 20)
include(FetchContent)
FetchContent_Declare(gbench
GIT_REPOSITORY https://github.com/google/benchmark
GIT_TAG v1.8.3
)
FetchContent_MakeAvailable(gbench)
find_package(AWSSDK REQUIRED COMPONENTS s3)
add_executable(${PROJECT_NAME} "src/main.cpp")
target_link_libraries(${PROJECT_NAME} benchmark::benchmark ${AWSSDK_LINK_LIBRARIES})src/main.cpp #include <aws/core/Aws.h>
#include <aws/s3/S3Client.h>
#include <aws/s3/model/ListBucketsRequest.h>
#include <aws/s3/model/ListObjectsV2Request.h>
#include <aws/s3/model/CreateBucketRequest.h>
#include <aws/s3/model/PutObjectRequest.h>
#include <aws/s3/model/GetObjectRequest.h>
#include <benchmark/benchmark.h>
#include <fstream>
using namespace Aws;
using namespace Aws::S3;
using namespace Aws::S3::Model;
static const char *BUCKET_NAME = "your-bucket-name-unique";
static const char *KEY = "key";
static const char *LOG_TAG = "s3benchmark";
static const int ITERATIONS = 20;
static SDKOptions s_options;
struct TestFilePair {
const char *file_name;
StorageClass storage_class;
};
static void CreateBucketUploadTestObjects() {
S3Client client{};
// Create the bucket if it doesnt exist
auto list_buckets_outcome = client.ListBuckets(ListBucketsRequest());
assert(list_buckets_outcome.IsSuccess());
Vector<Bucket> buckets{};
auto listed_buckets = list_buckets_outcome.GetResult().GetBuckets();
buckets.insert(buckets.end(), listed_buckets.begin(), listed_buckets.end());
while (!list_buckets_outcome.GetResult().GetContinuationToken().empty()) {
list_buckets_outcome = client.ListBuckets(ListBucketsRequest().WithContinuationToken(list_buckets_outcome.GetResult().GetContinuationToken()));
listed_buckets = list_buckets_outcome.GetResult().GetBuckets();
assert(list_buckets_outcome.IsSuccess());
buckets.insert(buckets.end(), listed_buckets.begin(), listed_buckets.end());
}
if (std::ranges::find_if(buckets, [](const Bucket &bucket) { return bucket.GetName() == BUCKET_NAME; }) == buckets.end()) {
auto create_bucket_output = client.CreateBucket(CreateBucketRequest().WithBucket(BUCKET_NAME));
assert(create_bucket_output.IsSuccess());
benchmark::DoNotOptimize(create_bucket_output);
}
// Create Objects if they dont exist
auto list_objects_outcome = client.ListObjectsV2(ListObjectsV2Request().WithBucket(BUCKET_NAME));
assert(list_objects_outcome.IsSuccess());
Vector<Object> objects{};
objects.insert(objects.end(), list_objects_outcome.GetResult().GetContents().begin(), list_objects_outcome.GetResult().GetContents().end());
while (!list_objects_outcome.GetResult().GetContinuationToken().empty()) {
list_objects_outcome = client.ListObjectsV2(ListObjectsV2Request().WithBucket(BUCKET_NAME).WithContinuationToken(list_objects_outcome.GetResult().GetContinuationToken()));
assert(list_objects_outcome.IsSuccess());
objects.insert(objects.end(), list_objects_outcome.GetResult().GetContents().begin(), list_objects_outcome.GetResult().GetContents().end());
}
if (std::ranges::find_if(objects, [](const Object& object) { return object.GetKey() == KEY; }) == objects.end()) {
auto put_object_request = PutObjectRequest().WithBucket(BUCKET_NAME).WithKey(KEY);
std::shared_ptr<IOStream> body = Aws::MakeShared<FStream>(LOG_TAG, "test-files/100mb.txt", std::ios::binary | std::ios::in);
put_object_request.SetBody(body);
auto put_object_response = client.PutObject(put_object_request);
assert(put_object_response.IsSuccess());
benchmark::DoNotOptimize(put_object_response);
}
}
static void DoSetup(const benchmark::State &state) {
InitAPI(s_options);
{
CreateBucketUploadTestObjects();
}
}
static void DoTeardown(const benchmark::State &state) {
ShutdownAPI(s_options);
}
static void BM_S3StandardTierFile(benchmark::State& state) {
S3Client client{};
for (auto _ : state) {
auto get_object_output = client.GetObject(GetObjectRequest().WithBucket(BUCKET_NAME).WithKey(KEY));
assert(get_object_output.IsSuccess());
benchmark::DoNotOptimize(get_object_output);
}
}
BENCHMARK(BM_S3StandardTierFile)
->Setup(DoSetup)
->Teardown(DoTeardown)
->MeasureProcessCPUTime()
->Iterations(ITERATIONS)
->Unit(benchmark::kMillisecond)
->UseRealTime();
BENCHMARK_MAIN();Also goes without saying but appreciate you looking into it indepth |
Beta Was this translation helpful? Give feedback.
-
|
@sbiscigl I'm going to just rewrite this code to instead just read known files from our buckets. Meanwhile, forgive me for asking again, but please, is there really nothing you can think of that might be being accidentally invoked that could cause such an obvious almost-exactly-2-second hitch in these operations? Some silent time-out/retry mechanism? Something with proxy config? For example, here is our low-level function for doing a get-object, which was deliberately packaged like this some time ago in order to make use of our scope-based debug timer mechanism, for our own benchmarking purposes. This is what generates the lines of the form... ...in the logs I posted before. That's the new-SDK log, c/w the equivalent line in the old-SDK log. Barring some fundamental misbehavior in our debug timer mechanism (which, respectfully, I doubt) I don't see how this can indicate anything other than the ...and yet it always succeeds and everything works, tests pass etc. I have played with the timeout values in the |
Beta Was this translation helpful? Give feedback.
-
|
sorry let me get caught up on this briefly |
Beta Was this translation helpful? Give feedback.
-
nah gotta replace with a bucket you own, it will upload and then download from that bucket.
No i would reccomend against using a public bucket, private bucket is fine, just choose your own name and the rest of the benchmark should just run.
no nothing comes to mind, but almost exact timing each time does seem off, and sounds kind of weird
yeah i dont think so either, fwiw the next steps as im seeing it are:
|
Beta Was this translation helpful? Give feedback.
-
|
Agreed. I have located an equivalent slow-down in our older simpler S3 code path, so I'm profiling that now. More when I know it. Thank you again so much! :) |
Beta Was this translation helpful? Give feedback.
-
No Problem! alight after the conversation yesterday i created a short benchmark test that is more so batteries included that should help us get closer to at least "we get a common test that shows a discrepancy that we both can see". sorry about that before i should have given better instructions one what it did and what needed to change. https://github.com/sbiscigl/aws-sdk-cpp-s3-benchmark/tree/main so you should be able to run this with
given curl/zlib/openssl are installed, and your current session has valid aws crednetials. where after running you can see a result like you should hopefully be able to move the version around with having too much trouble. If you could update that profile test to work/fail for a given tag, we can start taking a look from there. I dont think that 1.7.x will work out of the box with it, but let me know if it doesnt, and if we can find a way to update it to make it work. |
Beta Was this translation helpful? Give feedback.
-
|
@sbiscigl thank you for the above, but... I added some more profiling, and it seems that even just constructing a After some wailing and gnashing of teeth with my remote colleagues, one of them then came across this... ...and the related section of this... https://github.com/aws/aws-sdk-cpp/wiki/What%E2%80%99s-New-in-AWS-SDK-for-CPP-Version-1.8 To cut a long story short, if one does indeed set Even the build with 1.7.301 runs WAY faster, presumably due to the expensive client inits I was seeing even with that now also being quick. The particular unit test which used to take 67s, and which increased to 122s with 1.11.211 and 215s with 1.11.212 and later, now runs in just under 6s. This is both excellent AND very frustrating, but oh well. I guess my question now is, what would be the OFFICIAL C++ API way of disabling the automatic EC2 region lookup, rather than having to do a It also begs the question of why But frankly, I don't care, because even that I will report back again when I've done some more testing. |
Beta Was this translation helpful? Give feedback.
-
Yes that is what IMDS is doing, IMDS is fetching the EC2 IAM Role. if you shut off IMDS that will stop working. but it would seem that you werent using it before?
no likely you will have to set the env var, as apache arrow doesnt know where it is running ahead of time so it needs to check all.
So IMDS is the instance metadata service for EC2 instances, it provides various data about a running instance. it can provide region, it can provide credentials. in client configuration it provides region. in the credentials provider chain we also call it looking for credentials. this is based on defaults for all SDKs. so to override the chain, yes you have to override it in the constructor for instance #include <aws/core/Aws.h>
#include <aws/core/auth/AWSCredentialsProvider.h>
#include <aws/s3/S3Client.h>
using namespace Aws;
using namespace Aws::Auth;
using namespace Aws::Client;
using namespace Aws::Utils;
using namespace S3;
namespace {
const char * LOG_TAG = "test-app";
}
auto main() -> int {
SDKOptions options{};
options.loggingOptions.logLevel = Logging::LogLevel::Debug;
InitAPI(options);
{
auto credProvider = Aws::MakeShared<ProfileConfigFileAWSCredentialsProvider>(LOG_TAG);
S3Client client{std::move(credProvider)};
}
ShutdownAPI(options);
return 0;
}which would only use the profile credentials provider. i think though that in the end this can really be summarized as "i dont want to use IMDS crednetials unless im on a EC2 instance". which isnt something that we can do automatically. that is something that will have to be configured when you run your applicaiton |
Beta Was this translation helpful? Give feedback.
-
I assume that only works within an EC2 setup. Obviously it's not a valid global public IP, and doing that Digesting the rest. Thank you. |
Beta Was this translation helpful? Give feedback.
-
|
To clarify, we don't actually know that it's slow within an EC2 setup. It may well be fine. It's just that all our dev and CI and testing is obviously NOT happening in an EC2 setup! :) |
Beta Was this translation helpful? Give feedback.
-
|
169.254.169.254 is an ip that very often used for communication with metadata services on local network and its not a valid public ip, so makes sense that its hanging. @sbiscigl imds connect timeout should be 1s, so 2s seems a bit strange. Are we doing a couple rounds of timeouts? In general you want to specify as much as possible to avoid having client to look it up from environment. in this case if you want to have no creds (i.e. anonymous calls to s3) its probably better to specify that upfront instead of going through cred chain |
Beta Was this translation helpful? Give feedback.
-
|
We have decided to stick with the env var technique, thus also solving the issue for the Arrow S3 access without us having to patch the Arrow code. We already have config options for controlling authentication modes, so although we default to disabling IMDS, you can turn it back on again if you know you're running on an EC2 instance where the lookup should be quick. I believe this solves all our issues with using the latest version. Thank you both very much for all your help. Much appreciated. |
Beta Was this translation helpful? Give feedback.
@sbiscigl thank you for the above, but...
I added some more profiling, and it seems that even just constructing a
ClientConfiguration(or these days, anS3ClientConfiguration) was what was taking the two seconds. Literally a low-level timer around the constructor line showed that.After some wailing and gnashing of teeth with my remote colleagues, one of them then came across this...
#1410
...and the related section of this...
https://github.com/aws/aws-sdk-cpp/wiki/What%E2%80%99s-New-in-AWS-SDK-for-CPP-Version-1.8
To cut a long story short, if one does indeed set
AWS_EC2_METADATA_DISABLED=trueeverything then runs blazing fast.Even the build with 1.7.301 runs WAY faster, presumably due …