About multi-GPU training error

[chinagalaxy2002]

01/11 13:39:38 - mmengine - INFO - Epoch(train) [1][ 50/601] lr: 1.0000e-04 eta: 1 day, 20:57:31 time: 2.1053 data_time: 0.1393 memory: 5030 grad_norm: 245.5285 loss: nan inst_loss: nan seg_loss: 3.2195 merge_loss_tmp: 2.9625
Traceback (most recent call last):
File "tools/train.py", line 135, in
main()
File "tools/train.py", line 131, in main
runner.train()
File "/home/zh/miniconda3/envs/ESAM/lib/python3.8/site-packages/mmengine/runner/runner.py", line 1777, in train
model = self.train_loop.run() # type: ignore
File "/home/zh/miniconda3/envs/ESAM/lib/python3.8/site-packages/mmengine/runner/loops.py", line 98, in run
self.run_epoch()
File "/home/zh/miniconda3/envs/ESAM/lib/python3.8/site-packages/mmengine/runner/loops.py", line 115, in run_epoch
self.run_iter(idx, data_batch)
File "/home/zh/miniconda3/envs/ESAM/lib/python3.8/site-packages/mmengine/runner/loops.py", line 131, in run_iter
outputs = self.runner.model.train_step(
File "/home/zh/miniconda3/envs/ESAM/lib/python3.8/site-packages/mmengine/model/wrappers/distributed.py", line 121, in train_step
losses = self._run_forward(data, mode='loss')
File "/home/zh/miniconda3/envs/ESAM/lib/python3.8/site-packages/mmengine/model/wrappers/distributed.py", line 161, in _run_forward
results = self(**data, mode=mode)
File "/home/zh/miniconda3/envs/ESAM/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/home/zh/miniconda3/envs/ESAM/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 994, in forward
if torch.is_grad_enabled() and self.reducer._rebuild_buckets():
RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one. This error indicates that your module has parameters that were not used in producing loss. You can enable unused parameter detection by passing the keyword argument find_unused_parameters=True to torch.nn.parallel.DistributedDataParallel, and by
making sure all forward function outputs participate in calculating loss.
If you already have done the above, then the distributed data parallel module wasn't able to locate the output tensors in the return value of your module's forward function. Please include the loss function and the structure of the return value of forward of your module when reporting this issue (e.g. list, dict, iterable).
Parameter indices which did not receive grad for rank 0: 317 318 319 320 321 322 323 324
In addition, you can set the environment variable TORCH_DISTRIBUTED_DEBUG to either INFO or DETAIL to print out information about which particular parameters did not receive gradient on this rank as part of this error
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 150039 closing signal SIGTERM
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 150036) of binary: /home/zh/miniconda3/envs/ESAM/bin/python
Traceback (most recent call last):
File "/home/zh/miniconda3/envs/ESAM/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/home/zh/miniconda3/envs/ESAM/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/zh/miniconda3/envs/ESAM/lib/python3.8/site-packages/torch/distributed/launch.py", line 193, in
main()
File "/home/zh/miniconda3/envs/ESAM/lib/python3.8/site-packages/torch/distributed/launch.py", line 189, in main
launch(args)
File "/home/zh/miniconda3/envs/ESAM/lib/python3.8/site-packages/torch/distributed/launch.py", line 174, in launch
run(args)
File "/home/zh/miniconda3/envs/ESAM/lib/python3.8/site-packages/torch/distributed/run.py", line 752, in run
elastic_launch(
File "/home/zh/miniconda3/envs/ESAM/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 131, in call
return launch_agent(self._config, self._entrypoint, list(args))
File "/home/zh/miniconda3/envs/ESAM/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 245, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:

tools/train.py FAILED

Failures:
<NO_OTHER_FAILURES>

Root Cause (first observed failure):
[0]:
time : 2025-01-11_13:39:53
host : dell-PowerEdge-T640
rank : 0 (local_rank: 0)
exitcode : 1 (pid: 150036)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html