Skip to content

v2.0.0
Compare
Choose a tag to compare
@RamazanKara RamazanKara released this 06 Jan 22:39
· 13 commits to main since this release
v2.0.0
f05b114

v2.0.0

Version 2.0 of the One Billion Row Challenge Processor introduces significant optimizations, leading to a substantial reduction in processing time. This release focuses on enhancing concurrency handling and reducing contention, along with other performance improvements.

Performance Enhancements

  • Concurrent Map Implementation: Introduced a sharded concurrent map to reduce lock contention. This allows for more efficient updates to the data structure in a multi-threaded environment.
  • Hash-Based Sharding: Implemented hash-based sharding for distributing data across multiple shards, further reducing the chance of lock conflicts.
  • Optimized String Processing: Refined the string handling logic to minimize overhead during file parsing.
  • Buffer Size Adjustments: Tuned the buffer sizes for channels to balance throughput and memory usage.
  • Efficient Data Aggregation: Streamlined the data aggregation process for improved efficiency.

Processing Time 5m19s. Tested with a Ryzen 5800x3d

Assets 2
Loading