HADOOP-19389: Optimize shell -text command I/O with multi-byte read. #7291
+561
−167
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
🎊 +1 overall
This message was automatically generated. |
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Display.java
Show resolved
Hide resolved
...p-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/shell/TestTextCommand.java
Show resolved
Hide resolved
|
Hey @cnauroth! |
steveloughran
requested changes
Jan 16, 2025
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Display.java
Outdated
Show resolved
Hide resolved
| @@ -67,39 +75,194 @@ public void testDisplayForAvroFiles() throws Exception { | |||
| assertEquals(expectedOutput, output); | |||
| } | |||
|
|
|||
| @Test | |||
| public void testEmptyAvroFile() throws Exception { | |||
There was a problem hiding this comment.
I'm going to suggest something different: add an abstract contract test for this command and implement it in file:, hdfs, s3a and azure -then in gcs once that is in.
That way we can see how well/badly those stores cope with this
There was a problem hiding this comment.
I'm unclear about bringing this into a contract test. These tests are covering semantics very specific to hadoop fs -text. The expectation is that the command can decode Avro and sequence files and print them in JSON/text representation. At the FileSystem layer, it would just be returning the raw bytes. Existing contract tests cover "input bytes == output bytes". If hdfs/s3a/azure/gcs get that part right, then the hadoop fs -text behavior falls into place regardless of the FS.
However, maybe something we can do is expand contract coverage of reads in a more generic way. For example, we could cover things like the bad argument checks and "same data returned from read() vs. read(byte[]...)duringAbstractContractOpenTest. Maybe some updates to the FSDataInputStream` specification docs too.
I'd recommend driving all this in a separate patch though.
...p-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/shell/TestTextCommand.java
Outdated
Show resolved
Hide resolved
| @Test | ||
| public void testAvroFileInputStreamZeroLengthRead() throws Exception { | ||
| createFile(AVRO_FILENAME, generateWeatherAvroBinaryData()); | ||
| URI uri = new URI(AVRO_FILENAME); |
There was a problem hiding this comment.
create a field for this as it is used everywhere...fix in setup along with avro filename, which will now have to be on the target fs of the contract test
There was a problem hiding this comment.
It sounds like you would like to see something here lifted to a @Before. The uri gets initialized to different values though in different tests, so I'm not sure I can make that comm in a @Before.
...p-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/shell/TestTextCommand.java
Outdated
Show resolved
Hide resolved
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Display.java
Outdated
Show resolved
Hide resolved
|
|
||
| private static final String SEPARATOR = System.getProperty("line.separator"); | ||
|
|
||
| @Rule |
There was a problem hiding this comment.
if you make this a contract test, there's a rule in the superclass along with some thread naming
cnauroth
force-pushed
the
HADOOP-19389-trunk
branch
2 times, most recently
from
30b8c89 to
124509c
Compare
ashutoshcipher
approved these changes
Jan 17, 2025
Thanks, @steveloughran ! I entered a question about the contract test feedback. For all other feedback, I've updated the patch. |
|
🎊 +1 overall
This message was automatically generated. |
cnauroth
force-pushed
the
HADOOP-19389-trunk
branch
from
124509c to
46b95de
Compare
|
I pushed one more update for checkstyle warnings. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
The remaining checkstyle issues getting flagged are "variable must be private and have accessor methods." These are on inner classes, and I think only getting flagged due to me doing some refactoring improvements, so I'm not planning on addressing these. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of PR
hadoop fs -textreads Avro files and sequence files by internally wrapping the stream inAvroFileInputStreamorTextRecordInputStream. These classes implement the required single-byteread(), but not the optional multi-byte bufferedread(byte[], int, int). The default implementation in the JDK is a loop over single-byte read, which causes sub-optimal I/O and method call overhead. We can optimize this by overriding the multi-byte read method.How was this patch tested?
Multiple new unit tests cover expectations for both single-byte and multi-byte read. I ran these tests both before and after the code change to make sure there is no unexpected behavioral change.
Here is some benchmarking I ran to test the effects of this patch. This shows a ~1.7x throughput improvement for sequence files.
The improvement is more modest for Avro, where there are unrelated overheads from deserializing and reserializing to JSON:
For code changes:
LICENSE,LICENSE-binary,NOTICE-binaryfiles?