ValueError: offset must be non-negative and no greater than buffer length

[LiYixuan727]

Hi,
I'm training the fairseq with the following script and get the error ValueError: offset must be non-negative and no greater than buffer length.

fairseq-train data-bin --arch transformer
--max-epoch 10
--max-tokens 2048
--num-workers 20
--max-sentences 5000
--fp16
--optimizer adam --lr-scheduler inverse_sqrt --lr 0.0007
--criterion label_smoothed_cross_entropy

[LiYixuan727]

And here is the whole traceback:

2024-09-23 14:53:13 | INFO | fairseq_cli.train | task: TranslationTask
2024-09-23 14:53:13 | INFO | fairseq_cli.train | model: TransformerModel
2024-09-23 14:53:13 | INFO | fairseq_cli.train | criterion: LabelSmoothedCrossEntropyCriterion
2024-09-23 14:53:13 | INFO | fairseq_cli.train | num. shared model params: 22,480,862,208 (num. trained: 22,480,862,208)
2024-09-23 14:53:13 | INFO | fairseq_cli.train | num. expert model params: 0 (num. trained: 0)
2024-09-23 14:53:13 | INFO | fairseq.data.data_utils | loaded 51,352 examples from: data-bin/valid.en-es.en
2024-09-23 14:53:13 | INFO | fairseq.data.data_utils | loaded 51,352 examples from: data-bin/valid.en-es.es
2024-09-23 14:53:13 | INFO | fairseq.tasks.translation | data-bin valid en-es 51352 examples
2024-09-23 14:53:45 | INFO | fairseq.utils | CUDA enviroments for all 1 workers
2024-09-23 14:53:45 | INFO | fairseq.utils | rank 0: capabilities = 8.6 ; total memory = 47.431 GB ; name = NVIDIA RTX A6000
2024-09-23 14:53:45 | INFO | fairseq.utils | CUDA enviroments for all 1 workers
2024-09-23 14:53:45 | INFO | fairseq_cli.train | training on 1 devices (GPUs/TPUs)
2024-09-23 14:53:45 | INFO | fairseq_cli.train | max tokens per device = 4096 and max sentences per device = 5000
2024-09-23 14:53:45 | INFO | fairseq.trainer | Preparing to load checkpoint checkpoints/checkpoint_last.pt
2024-09-23 14:53:45 | INFO | fairseq.trainer | No existing checkpoint found checkpoints/checkpoint_last.pt
2024-09-23 14:53:45 | INFO | fairseq.trainer | loading train data for epoch 1
2024-09-23 14:53:49 | INFO | fairseq.data.data_utils | loaded 51,249,574 examples from: data-bin/train.en-es.en
2024-09-23 14:53:53 | INFO | fairseq.data.data_utils | loaded 51,249,574 examples from: data-bin/train.en-es.es
2024-09-23 14:53:53 | INFO | fairseq.tasks.translation | data-bin train en-es 51249574 examples
Traceback (most recent call last):
File "/home/ag/.local/bin/fairseq-train", line 8, in
sys.exit(cli_main())
File "/home/ag/.local/lib/python3.10/site-packages/fairseq_cli/train.py", line 557, in cli_main
distributed_utils.call_main(cfg, main)
File "/home/ag/.local/lib/python3.10/site-packages/fairseq/distributed/utils.py", line 369, in call_main
main(cfg, **kwargs)
File "/home/ag/.local/lib/python3.10/site-packages/fairseq_cli/train.py", line 164, in main
extra_state, epoch_itr = checkpoint_utils.load_checkpoint(
File "/home/ag/.local/lib/python3.10/site-packages/fairseq/checkpoint_utils.py", line 272, in load_checkpoint
epoch_itr = trainer.get_train_iterator(
File "/home/ag/.local/lib/python3.10/site-packages/fairseq/trainer.py", line 719, in get_train_iterator
self.reset_dummy_batch(batch_iterator.first_batch)
File "/home/ag/.local/lib/python3.10/site-packages/fairseq/data/iterators.py", line 368, in first_batch
return self.collate_fn([self.dataset[i] for i in self.frozen_batches[0]])
File "/home/ag/.local/lib/python3.10/site-packages/fairseq/data/iterators.py", line 368, in
return self.collate_fn([self.dataset[i] for i in self.frozen_batches[0]])
File "/home/ag/.local/lib/python3.10/site-packages/fairseq/data/language_pair_dataset.py", line 305, in getitem
tgt_item = self.tgt[index] if self.tgt is not None else None
File "/home/ag/.local/lib/python3.10/site-packages/fairseq/data/indexed_dataset.py", line 523, in getitem
np_array = np.frombuffer(
ValueError: offset must be non-negative and no greater than buffer length (6711936916)

[Herostomo]

I wanted to offer my assistance regarding the ValueError: offset must be non-negative and no greater than buffer length error you encountered while training with Fairseq.

Summary of the Issue:
The error occurs during the training process, specifically when the code attempts to access an index in the dataset that is out of range. This typically indicates a potential issue with the dataset formatting or indexing.

Approach :
Verify Dataset Integrity
Check Data Loading and Indexing
Consistency Between Datasets
Adjust Worker Count
Check Configuration Parameters
Inspect Data Paths

[dtamayo-nlp]

Hi!

In my case this problem appeared because of a problem with integer precision when processing long files in the binarization of the corpus. It can be solved by adding here the following line:

sizes = [np.int64(el) for el in sizes]
address = np.int64(0)

And processing again the corpus with fairseq-preprocess.

You could also avoid this problem by splitting your big files in smaller ones.