Release v0.1.2

Release Notes

Features

• Optimized Data Transfer: Change broadcast-based data transfer into gather-scatter for better performance.
• Refactored Master Worker: Provide better code readability and support asyncio package with uvloop.
• Support Tensorboard Logging: Support CLI options to enable Tensorboard logging on the master worker.

Documentation

• Fix Estimated Training Time: Fixed the estimated training time of 7B experiments in README.

Release v0.1.1

Feature

• User-friendly Launch Tutorials: Updated tutorials and scripts to enable one-click startup of training workflows for faster setup and experimentation.
• Loss Scale Normalization: Normalized loss scaling by token count across micro-batches to stabilize training.
• Configurable Loss Scaling: Added CLI options to customize loss scale window size and initial scaling values.
• Micro-Batch Splitting Optimization: Improved micro-batch splitting logic to ensure balanced workload distribution and enhance training efficiency.

Bug Fixes

• Dataloader Seed Reproducibility: Fixed an issue where dataloaders reused identical random seeds across epochs, ensuring proper shuffling and reproducibility.
• Math Verification Stability: Resolved timeout errors in mathematical verification steps during training.

Documentation

• Update README: Updated 7B-zero model performance figures.