B1223 Nemo训练时报错

我们在使用自行录音的语音数据集从头进行训练时遇到了一下报错信息:
Traceback (most recent call last):
File “nemo_hackathon_final_3.28.py”, line 87, in
trainer.fit(first_asr_model)
File “/opt/conda/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py”, line 445, in fit
results = self.accelerator_backend.train()
File “/opt/conda/lib/python3.6/site-packages/pytorch_lightning/accelerators/ddp_spawn_accelerator.py”, line 74, in train
mp.spawn(self.ddp_train, nprocs=self.nprocs, args=(self.mp_queue, model,))
File “/opt/conda/lib/python3.6/site-packages/torch/multiprocessing/spawn.py”, line 200, in spawn
return start_processes(fn, args, nprocs, join, daemon, start_method=‘spawn’)
File “/opt/conda/lib/python3.6/site-packages/torch/multiprocessing/spawn.py”, line 149, in start_processes
process.start()
File “/opt/conda/lib/python3.6/multiprocessing/process.py”, line 105, in start
self._popen = self._Popen(self)
File “/opt/conda/lib/python3.6/multiprocessing/context.py”, line 284, in _Popen
return Popen(process_obj)
File “/opt/conda/lib/python3.6/multiprocessing/popen_spawn_posix.py”, line 32, in init
super().init(process_obj)
File “/opt/conda/lib/python3.6/multiprocessing/popen_fork.py”, line 19, in init
self._launch(process_obj)
File “/opt/conda/lib/python3.6/multiprocessing/popen_spawn_posix.py”, line 47, in _launch
reduction.dump(process_obj, fp)
File “/opt/conda/lib/python3.6/multiprocessing/reduction.py”, line 60, in dump
ForkingPickler(file, protocol).dump(obj)
AttributeError: Can’t pickle local object ‘FilterbankFeatures.init..’

查询Google发现github上有人提出过issue,根据提示更改为单GPU运行但依然报错:AttributeError: Can't pickle local object 'FilterbankFeatures.__init__.<locals>.<lambda>' · Issue #1194 · NVIDIA/NeMo · GitHub
但之后尝试直接使用迁移学习模型训练,则没有遇到报错。请问该如何解决?(使用的运行文件在附件中)
PS: 使用的设备和环境是DGX Station, Ubuntu 18.04, Nemo Docker

请先检查你的数据集语音格式是否为单声道 44100Hz .wav格式?

处理过的了,已再次确认,且直接使用迁移学习模型是可以正确输出模型的

你的意思是你的数据集没问题 ,迁移学习模型跑通了 ,只是从头训练部分报错吗?

是的,使用的代码就是训练营提供的ipynb转的py,已经上传附件了

你有几个GPU啊? gpus=[2,3]这个位置设置是不是可以调整调整参数试试? 我用你的代码调整成我的GPU数量1即"gpus=1" 是可以正常运行的

DGX有4个,前两个在跑别的模型,所以我指定了剩下两个[2, 3]

老师你好,我已经自己解决了,需要在trainer里多加一个参数 accelerator=‘ddp’ 就可以了,参考Pytorch Lightning文档:Trainer — PyTorch Lightning 2.2.0.rc0 documentation

真棒!

训练营提供的ipynb转的py,和视频讲解的不太一致吧?源文件从哪里获取的?谢谢!