GH-43057: [C++] Thread-safe AesEncryptor / AesDecryptor

[EnricoMi]

Rationale for this change

OpenSSL encryption / decryption is wrapped by AesEncryptor / AesDencryptor, which is used by multiple threads of a single scanner or by multiple concurrent scanners when scanning a dataset. Some thread may call WipeOut while other threads still use the instance.

What changes are included in this PR?

• Remove the WipeOut methods and related datastructures entirely.
• Each call into CtrEncrypt / CtrDecrypt and GcmEncrypt / GcmDecrypt uses its own EVP_CIPHER_CTX instance, making this thread-safe.

After fixing this "AesDecryptor was wiped out" issue, two other segmentation faults surfaced: GH-44988. This has also been addressed here as it can only be exposed after fixing the wipe-out issue.

Fixes GH-43057.
Fixes GH-44852.
Fixes GH-44988.

Are these changes tested?

A unit test that scans a dataset concurrently reproduced the initial issue in 30% of the test runs.

Are there any user-facing changes?

No.

• GitHub Issue: [CI][C++] test-ubuntu-20.04-cpp fails sometimes with a segmentation fault on arrow-dataset-file-parquet-encryption-test #43057

[adamreeve]

The test failures look like they might be caused by openssl/openssl#21955

The failing ASAN test run is using OpenSSL 3.02, but builds with other versions are passing. Eg. the Ubuntu 20.04 build uses OpenSSL 1.1.1f and the Conda build has OpenSSL 3.3.2.

The fix for this was backported to the 3.0 branch and released in 3.0.13 (openssl/openssl#23102), but Ubuntu 22.04 still uses 3.0.2. If there's not an easy way around this, maybe we need a slower code path for 3.0.2 that creates a new context each time?

[pitrou]

If there's not an easy way around this, maybe we need a slower code path for 3.0.2 that creates a new context each time?

Let's use the "slower" code path everywhere? I'm skeptical it will be that slow...

[pitrou]

Thanks a lot for stepping in and submitting this @EnricoMi . You'll find a number of comments below.

[EnricoMi]

I have addressed all comments except for the test related ones. Will go over them tomorrow.

Thanks for your input, that is highly appreciated!

[adamreeve]

I tested concurrent scans of a larger dataset with uniform encryption, and this change doesn't completely fix that scenario:

--- a/cpp/src/arrow/dataset/file_parquet_encryption_test.cc
+++ b/cpp/src/arrow/dataset/file_parquet_encryption_test.cc
@@ -289,6 +289,8 @@ TEST_F(DatasetEncryptionTest, ReadSingleFile) {
 // processing encrypted datasets over 2^15 rows in multi-threaded mode.
 class LargeRowEncryptionTest : public DatasetEncryptionTestBase {
  public:
+  LargeRowEncryptionTest() : DatasetEncryptionTestBase(true) {}
+
   // The dataset is partitioned using a Hive partitioning scheme.
   void PrepareTableAndPartitioning() override {
     // Specifically chosen to be greater than batch size for triggering prefetch.
@@ -307,7 +309,7 @@ class LargeRowEncryptionTest : public DatasetEncryptionTestBase {
 
 // Test for writing and reading encrypted dataset with large row count.
 TEST_F(LargeRowEncryptionTest, ReadEncryptLargeRows) {
-  ASSERT_NO_FATAL_FAILURE(TestScanDataset());
+  ASSERT_NO_FATAL_FAILURE(TestScanDataset(true));
 }
 
 }  // namespace dataset

This gives errors like:

failed with IOError: Failed decryption finalization

I believe this is due to the decryptor AAD being mutable. It's updated for each data page read, so concurrent use will result in incorrect AADs being used in some threads.

Rather than mutating the decryptor state, it might be possible to refactor this so that the AAD values are passed in to the decrypt method as a parameter.

I think this should probably be addressed as a separate PR though as the changes should be orthogonal.

[EnricoMi]

failed with IOError: Failed decryption finalization

Confirmed in ec9b054. Multi-threaded uniform encryption read fails.

[EnricoMi]

failed with IOError: Failed decryption finalization

Confirmed in ec9b054. Multi-threaded uniform encryption read fails.

Attempt to fix multi-threaded AAD update: d77a081

The PageReader uses its own copy of the data_decryptor / data_decryptor. Happy to move into separate PR.

[mapleFU]

Any updates?

[pitrou]

@EnricoMi Sorry for the delay and thanks a lot for persevering on this. I think the fix is still not 100% correct, see comments below. But we're nearing a solution.

Also, I can help on this PR if you want.

[pitrou]

I pushed some more cleanups, in particular I switched to std::unique_ptr for non-shareable objects Decryptor and AesDecryptor.

[wgtmac]

+1

[pitrou]

I think at some point a large cleanup and reorganization phase would be deserved on the Parquet encryption code:

• header and source file naming
• object and method naming
• reorganizing responsabilities across objects
• wacky implementation details (e.g. why exactly do we need the mutable CryptoContext::start_decrypt_with_dictionary_page?)

[pitrou]

Hmm, the Lint CI step is failing for unrelated reasons (see #45521).

[pitrou]

@github-actions crossbow submit -g cpp

[github-actions]

Revision: ed76e98

Submitted crossbow builds: ursacomputing/crossbow @ actions-5fc2004b28

Task	Status
example-cpp-minimal-build-static
example-cpp-minimal-build-static-system-dependency
example-cpp-tutorial
test-alpine-linux-cpp
test-build-cpp-fuzz
test-conda-cpp
test-conda-cpp-valgrind
test-cuda-cpp-ubuntu-20.04-cuda-11.2.2
test-cuda-cpp-ubuntu-22.04-cuda-11.7.1
test-debian-12-cpp-amd64
test-debian-12-cpp-i386
test-fedora-39-cpp
test-ubuntu-20.04-cpp
test-ubuntu-20.04-cpp-bundled
test-ubuntu-22.04-cpp
test-ubuntu-22.04-cpp-20
test-ubuntu-22.04-cpp-emscripten
test-ubuntu-22.04-cpp-no-threading
test-ubuntu-24.04-cpp
test-ubuntu-24.04-cpp-bundled-offline
test-ubuntu-24.04-cpp-gcc-13-bundled
test-ubuntu-24.04-cpp-gcc-14
test-ubuntu-24.04-cpp-minimal-with-formats
test-ubuntu-24.04-cpp-thread-sanitizer

[raulcd]

@pitrou I took a moment to put your comment above into an enhancement issue here:

• [Parquet][C++] Cleanup and reorganization for Parquet encryption code #45525

[pitrou]

Oh, thank you @raulcd !

[pitrou]

Well, there's an interesting segfault on the Windows C++ build...

Edit: it looks like a non-deterministic bug, but it might be due to the WipeoutDecryptionKeys call in the InternalFileDecryptor destructor. Will try to investigate more. Actually, it also happened without the WipeoutDecryptionKeys call when retrying CI jobs on another commit. It might not be related to this PR at all, as it is happening in the single-threaded Parquet encryption tests, not in the multi-threaded Dataset sets.

[EnricoMi]

@pitrou quite some substantial refactoring, LGTM! Thanks for stepping in!

[pitrou]

After several retries, it seems the Windows segfaults can happen randomly in various places, even when writing an encrypted file. Example logs (two segfaults, including one in TestEncryptionKeyManagement.WrapLocally):
https://github.com/pitrou/arrow/actions/runs/13314446458/job/37184921711#step:10:266

[pitrou]

Given the symptoms, I'm not sure the Windows segfaults are directly caused by this PR's changes. I might try to reproduce on a Windows VM, though (unless someone beats me to it?).