HADOOP-19471. ABFS: Support Fixed SAS Token at Container Level #7461
+176
−20
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
manika137
changed the title
|
💔 -1 overall
This message was automatically generated. |
manika137
force-pushed
the
HADOOP-19471_SASTokenSupport
branch
from
c5b14bb to
4a182ac
Compare
|
🎊 +1 overall
This message was automatically generated. |
| passchars = rawConfig.getPassword(accountConf(key)); | ||
| if(passchars == null){ | ||
| passchars = rawConfig.getPassword(key); | ||
| } |
There was a problem hiding this comment.
can be simplified as :- char[] passchars = rawConfig.getPassword(containerConf(key)) != null ?
rawConfig.getPassword(containerConf(key)) :
rawConfig.getPassword(accountConf(key)) != null ?
rawConfig.getPassword(accountConf(key)) :
rawConfig.getPassword(key);
| @@ -493,7 +497,7 @@ public AbfsConfiguration(final Configuration rawConfig, | |||
| */ | |||
| public AbfsConfiguration(final Configuration rawConfig, String accountName) | |||
There was a problem hiding this comment.
Ideally we should add a separate constructor with this new param
| * Helper method to get the Fixed SAS token value | ||
| */ | ||
| private String getFixedSASToken(AbfsConfiguration config) throws Exception { | ||
| return config.getSASTokenProvider().getSASToken(this.getAccountName(), this.getFileSystemName(), getMethodName(), "read"); |
There was a problem hiding this comment.
nit: use constants for string
|
🎊 +1 overall
This message was automatically generated. |
Test Results============================================================
|
manika137
force-pushed
the
HADOOP-19471_SASTokenSupport
branch
from
b2cb26d to
6915d0e
Compare
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
| @@ -150,7 +150,9 @@ public void testBothProviderFixedTokenConfigured() throws Exception { | |||
| * Helper method to get the Fixed SAS token value | |||
| */ | |||
| private String getFixedSASToken(AbfsConfiguration config) throws Exception { | |||
| return config.getSASTokenProvider().getSASToken(this.getAccountName(), this.getFileSystemName(), getMethodName(), "read"); | |||
| String readPermission = "read"; | |||
There was a problem hiding this comment.
this should be a static string used as a constant.
anmolanmol1234
approved these changes
Mar 11, 2025
| testAbfsConfig.set(FS_AZURE_SAS_FIXED_TOKEN, accountSAS); | ||
|
|
||
| // Assert that Container Specific Fixed SAS is used | ||
| Assertions.assertThat(getFixedSASToken(testAbfsConfig)).contains("sr=c"); |
There was a problem hiding this comment.
Please add description for this.
|
|
||
| // Assert that Account Specific Fixed SAS is used if container SAS isn't set | ||
| testAbfsConfig.unset(containerProperty(FS_AZURE_SAS_FIXED_TOKEN, this.getFileSystemName(), this.getAccountName())); | ||
| Assertions.assertThat(getFixedSASToken(testAbfsConfig)).contains("ss=bf"); |
There was a problem hiding this comment.
Same as above, please add description in all the assert.
| //Assert that Account-Agnostic fixed SAS is used if no other fixed SAS configs are set. | ||
| // The token is the same as the Account Specific Fixed SAS. | ||
| testAbfsConfig.unset(accountProperty(FS_AZURE_SAS_FIXED_TOKEN, this.getAccountName())); | ||
| Assertions.assertThat(getFixedSASToken(testAbfsConfig)).contains("ss=bf"); |
| * Helper method to get the Fixed SAS token value | ||
| */ | ||
| private String getFixedSASToken(AbfsConfiguration config) throws Exception { | ||
| return config.getSASTokenProvider().getSASToken(this.getAccountName(), this.getFileSystemName(), getMethodName(), |
There was a problem hiding this comment.
Please review the format. This line has too many characters.
| @@ -322,6 +322,10 @@ public static String accountProperty(String property, String account) { | |||
| return property + "." + account; | |||
| } | |||
|
|
|||
| public static String containerProperty(String property, String fsName, String account) { | |||
| return property + "." + fsName + "." + account; | |||
There was a problem hiding this comment.
We can use DOT constant here (AbfsHttpConstants)
| * @return Container-specific configuration key | ||
| */ | ||
| public String containerConf(String key) { | ||
| return key + "." + fsName + "." + accountName; |
There was a problem hiding this comment.
We can use DOT constant here (AbfsHttpConstants)
manika137
force-pushed
the
HADOOP-19471_SASTokenSupport
branch
from
0969033 to
f4c6458
Compare
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
anujmodi2021
requested changes
Mar 12, 2025
| * @throws IOException | ||
| */ | ||
| @Test | ||
| public void testFixedTokenPreference() throws Exception { |
There was a problem hiding this comment.
Nit: `testFixedSASTokenProviderPreference'
| testAbfsConfig.set(FS_AZURE_SAS_FIXED_TOKEN, accountSAS); | ||
|
|
||
| // Assert that Container Specific Fixed SAS is used | ||
| Assertions.assertThat(getFixedSASToken(testAbfsConfig)) |
There was a problem hiding this comment.
This seems good but you can also assert on the whole SAS Token string.
| testAbfsConfig.set( | ||
| containerProperty(FS_AZURE_SAS_FIXED_TOKEN, this.getFileSystemName(), | ||
| this.getAccountName()), containerSAS); | ||
| testAbfsConfig.set( |
There was a problem hiding this comment.
We are setting same SAS Token as both account specific and account agnostic. We should separate them out.
There was a problem hiding this comment.
Makes sense, taken.
| accountProperty(FS_AZURE_SAS_FIXED_TOKEN, this.getAccountName())); | ||
| Assertions.assertThat(getFixedSASToken(testAbfsConfig)) | ||
| .describedAs("Account-agnostic fixed SAS should've been used.") | ||
| .contains("ss=bf"); |
There was a problem hiding this comment.
This assert should be different from other else it won't be able to catch regressions
manika137
force-pushed
the
HADOOP-19471_SASTokenSupport
branch
from
22851b7 to
a800cda
Compare
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of PR
The ABFS driver currently lacks support for multiple SAS tokens for the same storage account across different containers.
Introducing this support through this PR.
To use fixed SAS token at container level the configuration to be used is:
fs.azure.sas.fixed.token.<container-name>.<storage-account-name>How was this patch tested?
Adding the test results in the comments below.