Commits

Breadcrumbs

History for

onmain

User selector

All users

Datepicker

All time

Commit History

Commits on Jul 19, 2024

• acquire device when required (#420)

  Show description for 3af2e25
  polisettyvarma

  authoredJul 19, 2024
  3af2e25

  Copy full SHA for 3af2e25

Commits on Jul 18, 2024

• add support to run custom Hf tokenizer for training and dataset pre-processing (#421)

  Show description for 7d23e33
  polisettyvarma

  authoredJul 18, 2024
  7d23e33

  Copy full SHA for 7d23e33

Commits on Jun 28, 2023

• Make --workers required in preprocess_data.py

  

  Jon Barker

  committedJun 28, 2023
  23329e2

  Copy full SHA for 23329e2

Commits on Jun 26, 2023

• Replace redundant preprocess_data.py with preprocess_data_partitions.py

  jon-barker

  committedJun 26, 2023
  d1a36e5

  Copy full SHA for d1a36e5

Commits on Apr 20, 2023

• Adding NullTokenzier that takes in pure integers on preprocessing, training and inference so user can do there own tokenization/detokenization out of the loop

  

  rprenger

  committedApr 20, 2023
  2e9a3fb

  Copy full SHA for 2e9a3fb

Commits on Apr 6, 2023

• Missed some changes from next-best-lm branch.

  jaredcasper

  committedApr 6, 2023
  3207c19

  Copy full SHA for 3207c19

• Add support for GPTSentencePieceTokenizer and related fixes.

  jaredcasper

  committedApr 6, 2023
  46ffb75

  Copy full SHA for 46ffb75

Commits on Nov 17, 2022

• Data Preprocessing Optimizations

  

  kvareddy

  authored and

  Mike Chrzanowski

  committedNov 17, 2022
  7fc9611

  Copy full SHA for 7fc9611

Commits on Sep 23, 2022

• Clean up licensing.

  Show description for fabad46
  jaredcasper

  committedSep 23, 2022
  fabad46

  Copy full SHA for fabad46

Commits on May 12, 2022

• Data preprocessing testing changes + fixes

  Show description for a2c5e6c
  

  John Kamalu

  committedMay 12, 2022
  a2c5e6c

  Copy full SHA for a2c5e6c

Commits on May 14, 2021

• Add arg needed for tokenizer.

  jaredcasper

  committedMay 14, 2021
  2c247af

  Copy full SHA for 2c247af

Commits on Feb 1, 2021

• Handle empty documents in preprocess_data.

  jaredcasper

  committedFeb 1, 2021
  09d220c

  Copy full SHA for 09d220c

Commits on Nov 12, 2020

• Intra-layer MP -> Tensor MP, Inter-layer MP -> Pipeline MP

  deepakn94

  committedNov 12, 2020
  52a5f2f

  Copy full SHA for 52a5f2f

• Pipeline parallelism implementation with periodic full-pipeline syncs

  Show description for 7abd3e9
  deepakn94

  committedNov 12, 2020
  7abd3e9

  Copy full SHA for 7abd3e9

Commits on Apr 24, 2020

• added case wordpiece arguments

  

  Raul Puri

  committedApr 24, 2020
  ef5b2f0

  Copy full SHA for ef5b2f0

Commits on Apr 16, 2020

• changed licence 2019 to 2020

  shoeybi

  committedApr 16, 2020
  463d125

  Copy full SHA for 463d125

Commits on Apr 13, 2020

• refactored merge_mp_partitions.py

  shoeybi

  committedApr 13, 2020
  03d0d93

  Copy full SHA for 03d0d93

Commits on Apr 8, 2020

• Add sentence or document suffix to output filename of preprocess_data.py

  jaredcasper

  committedApr 8, 2020
  1e28344

  Copy full SHA for 1e28344

• New tools directory.

  jaredcasper

  committedApr 8, 2020
  3500d03

  Copy full SHA for 3500d03

Loading