-
-
Notifications
You must be signed in to change notification settings - Fork 282
2025
💻 Zoom link: https://us06web.zoom.us/j/89601195963
📆 Meeting calendar invite.
Note
Please provide time estimates for each agenda item.
• —– ٠ ✤ ٠ —–· · • —– ٠ ✤ ٠ —– • ·· • —– ٠ ✤ ٠ —–· · • —– ٠ ✤ ٠ —– • ·· • —– ٠ ✤ ٠ —–· · • —– ٠ ✤ ٠ —– •
- Facilitator/time-keeper: Allen Byrne
- Note-taker/recorder: AI
- [] Scot will check whether collaborators can create branches within the HDF Group organization on GitHub.
- [] The HDF Group will present its vision for community collaboration at the next meeting.
- [] The HDF Group will provide an update on the HEP (HDF5 Enhancement Proposal) process at the next meeting.
- [] Quincey will initiate discussions about the accelerator native storage and sharded storage proposals in the forum within the next month, in light of a more formal HEP framework.
- []
• —– ٠ ✤ ٠ —–· · • —– ٠ ✤ ٠ —– • ·· • —– ٠ ✤ ٠ —–· · • —– ٠ ✤ ٠ —– • ·· • —– ٠ ✤ ٠ —–· · • —– ٠ ✤ ٠ —– •
• —– ٠ ✤ ٠ —–· · • —– ٠ ✤ ٠ —– • ·· • —– ٠ ✤ ٠ —–· · • —– ٠ ✤ ٠ —– • ·· • —– ٠ ✤ ٠ —–· · • —– ٠ ✤ ٠ —– •
- Facilitator/time-keeper: Neil Fortner
- Note-taker/recorder: Scot Breitenfeld/AI
(Presumably, none.)
- Changes to the agenda?
- Ideas for how to enable community collaboration (Quincey,20 min)
- Branch management, technical discussions, etc.
- Possibly create Github org, with HEPs and other collaborations
- NVIDIA roadmap collaboration opportunities (Quincey, 40 min)
- Accelerator-enabled I/O operations
- Sharded storage
Quick Recap
The HDF5 Working Group meeting focused on the potential for establishing a separate organization for community discussions and collaborations and the need for increased community involvement in two upcoming NVIDIA projects. The meeting also explored the current CPU-GPU node architecture, emphasizing the importance of concentrating on the accelerator components and the potential of implementing an actual metadata database for quicker operations. Additionally, the group discussed the design of the HDF5 architecture, the advantages of the Zarr format over HDF5, and two technical proposals for GPU accelerators: storage and sharded storage.
Summary
HDF5 Working Group Meeting Agenda
Neil, the facilitator, shared the agenda and invited any additions. Scot informed the group about two methods for receiving meeting cancellation notices: via the mailing list and calendar invites. He recommended using these automated methods instead of relying on forum posts for updates.
Neutral Platform for Community Discussions
Quincey proposed creating a separate, broader, and independent organization for community discussions and collaborations, suggesting it could serve as a neutral space not explicitly organized by the HDF Group. He highlighted that this could benefit Nvidia and AMD collaborations, managing branches, and reviewing documents. Steve questioned the necessity of this organization apart from the HDF Group, while Gerd raised concerns about its scope and neutrality. Quincey emphasized the need for a discussion platform that could operate independently from HDF Group meetings. Gerd suggested that the HDF Group could be a neutral platform for such discussions.
Revenue Generation for HDF Group
Scot and Steve discussed the importance of generating revenue for the HDF Group, with Steve expressing concern about the time and resources spent on initiatives that do not produce revenue. Scot emphasized the need for a stable outlook on HDF5 and their services and the potential for increased revenue through outreach. Quincey suggested that the HDF Group allow collaborators to create branches, which Scot agreed to investigate. The conversation concluded with Quincey seeking clarity on Steve's comments regarding the HDF Group's responsibilities.
Increased Community Involvement in Projects
Quincey discussed the need for greater community involvement in two projects: one focusing on accelerator native storage and the other on sharded storage. He desired increased participation from management and other community members in these projects. Quincey mentioned that he would begin inviting people to participate if necessary, and he planned to wrap up multi-threading discussions in the next couple of weeks.
New GPU Architecture for Data Transfer
Quincey spoke about the current CPU-GPU node architecture, where data is cached in CPU memory before being transferred to the GPU. He proposed a new architecture where the GPU would handle most operations, with data transferred in and out more efficiently. Quincey emphasized the importance of establishing a vendor-neutral mechanism for GPU-related tasks and encouraged participation from others. He also suggested integrating this architecture with MPI for collective I/O operations, proposing that type conversion could be incorporated into the new I/O pipeline.
Improving HDF5 Compatibility and Design
Quincey stressed the need to focus on the accelerator components and enhance the techniques introduced by Zarr to ensure HDF5 compatibility with POSIX and object stores. He proposed a design that includes a directory resembling a container, sharding out the dataset storage and utilizing databases for metadata management. He encouraged feedback from a variety of stakeholders to enhance the design. Aleksandar questioned how this approach differs from Zarr and the existing HDF5 schema. Quincey clarified that there are indeed distinctions.
Metadata Database for Faster Operations
Quincey discussed using an actual metadata database for faster operations, which could provide an advantage over Zarr. Aleksandar agreed with Quincey’s points but emphasized the importance of understanding and accessibility for scientists. He expressed concern about HDF5's complexity and the lack of alternatives, stating that these factors should be considered in their decision-making processes.
GPU Memory Storage and Compatibility During the meeting, Joe Lee and Quincey discussed the potential for storing key-value pairs in GPU memory. Quincey confirmed this is feasible but stressed the importance of an abstract and pluggable interface. They also discussed using Nvidia's compression algorithm, nvCOMP. Joe Lee asked about the openness of the instruction set for the H200 chip, to which Quincey admitted he did not know the answer. Furthermore, they discussed the need for a vendor-neutral interface between HDF5 and Nvidia GPUs, as well as the use of NVIDIA GPUDirect® Storage (GDS) APIs to communicate with them.
HDF5 Architecture and Acceleration Discussion Quincey presented the design of the HDF5 architecture, emphasizing the significance of source and destination data buffers on accelerators. He proposed a vendor-neutral approach to enhance performance and suggested collaborating with the HDF5 GPU VFD. Joe Lee inquired about benchmarking HDF5 GPU VFD against other I/O libraries using an AI application that Nvidia can showcase at GTC 2026; Quincey responded that the key metric is the acceleration of I/O for HDF5-based applications. He noted that his components demonstrate improvements, although the final product is not yet built. Gerd sought clarification regarding Alexander's remark about scientists understanding storage concepts, and Alexander explained that these individuals are early adopters of storage software.
Zarr’s Advantages Over HDF5 Aleksandar outlined the advantages of the Zarr format compared to HDF5, highlighting its simplicity and ease of implementation. He pointed out that scientists have adopted Zarr's direct implementation in various programming languages. Aleksandar also mentioned that Zarr is now adding features, such as chunks in a file, which were previously lacking. Quincey proposed that the interface for interacting with the metadata database should enable interaction with a JSON plain text metadata file, which could serve as another plugin for the metadata.
On-Node Storage and Sharded Proposals Quincey introduced two technical proposals regarding GPU storage and sharded storage, seeking interest from participating organizations. Aleksandar expressed interest in the sharded proposal, while Neil indicated an interest in both proposals but highlighted funding constraints. Steve from Lifeboat raised concerns about aligning community and commercial interests. Quincey plans to begin sketching designs for the proposals but mentioned that thread safety work is currently consuming his time. The group agreed to reconvene in two weeks to continue the discussion.
- [] Scot will check whether collaborators can create branches within the HDF Group organization on GitHub.
- [] The HDF Group will present its vision for community collaboration at the next meeting.
- [] The HDF Group will provide an update on the HEP (HDF5 Enhancement Proposal) process at the next meeting.
- [] Quincey will initiate discussions about the accelerator native storage and sharded storage proposals in the forum within the next month, in light of a more formal HEP framework.
• —– ٠ ✤ ٠ —–· · • —– ٠ ✤ ٠ —– • ·· • —– ٠ ✤ ٠ —–· · • —– ٠ ✤ ٠ —– • ·· • —– ٠ ✤ ٠ —–· · • —– ٠ ✤ ٠ —– •