Model training error

[Yeek99]

File "E:\Retrieval-based-Voice-Conversion-WebUI\infer\modules\train\train.py", line 256, in run net_g.module.load_state_dict( File "E:\Retrieval-based-Voice-Conversion-WebUI\venv\Lib\site-packages\torch\nn\modules\module.py", line 2593, in load_state_dict raise RuntimeError( RuntimeError: Error(s) in loading state_dict for SynthesizerTrnMs768NSFsid: Missing key(s) in state_dict: "enc_p.emb_pitch.weight", "dec.m_source.l_linear.weight", "dec.m_source.l_linear.bias", "dec.noise_convs.0.weight", "dec.noise_convs.0.bias", "dec.noise_convs.1.weight", "dec.noise_convs.1.bias", "dec.noise_convs.2.weight", "dec.noise_convs.2.bias", "dec.noise_convs.3.weight", "dec.noise_convs.3.bias".

Unable to train due to this error during One-click training.

OS: Win11
Python env: 3.11.9
torch: 2.0.1+cu117

[acely]

Same errors for me:

DEBUG:__main__:Write filelist done
INFO:__main__:Use gpus:
INFO:__main__:Execute: "python" infer/modules/train/train.py -e "arcpr" -sr 40k -f0 1 -bs 1 -te 20 -se 5 -pg "assets/pretrained_v2/G40k.pth" -pd "assets/pretrained_v2/D40k.pth" -l 0 -c 0 -sw 0 -v v2 -a ""
DEBUG:faiss.loader:Environment variable FAISS_OPT_LEVEL is not set, so let's pick the instruction set according to the current CPU
INFO:faiss.loader:Loading faiss.
INFO:faiss.loader:Successfully loaded faiss.
DEBUG:faiss.loader:Environment variable FAISS_OPT_LEVEL is not set, so let's pick the instruction set according to the current CPU
INFO:faiss.loader:Loading faiss.
INFO:faiss.loader:Successfully loaded faiss.
INFO:arcpr:{'data': {'filter_length': 2048, 'hop_length': 400, 'max_wav_value': 32768.0, 'mel_fmax': None, 'mel_fmin': 0.0, 'n_mel_channels': 125, 'sampling_rate': 40000, 'win_length': 2048, 'training_files': './logs/arcpr/filelist.txt'}, 'model': {'filter_channels': 768, 'gin_channels': 256, 'hidden_channels': 192, 'inter_channels': 192, 'kernel_size': 3, 'n_heads': 2, 'n_layers': 6, 'p_dropout': 0, 'resblock': '1', 'resblock_dilation_sizes': [[1, 3, 5], [1, 3, 5], [1, 3, 5]], 'resblock_kernel_sizes': [3, 7, 11], 'spk_embed_dim': 109, 'upsample_initial_channel': 512, 'upsample_kernel_sizes': [16, 16, 4, 4], 'upsample_rates': [10, 10, 2, 2], 'use_spectral_norm': False}, 'train': {'batch_size': 1, 'betas': [0.8, 0.99], 'c_kl': 1.0, 'c_mel': 45, 'epochs': 20000, 'eps': 1e-09, 'fp16_run': False, 'init_lr_ratio': 1, 'learning_rate': 0.0001, 'log_interval': 200, 'lr_decay': 0.999875, 'seed': 1234, 'segment_size': 12800, 'warmup_epochs': 0}, 'model_dir': './logs/arcpr', 'experiment_dir': './logs/arcpr', 'save_every_epoch': 5, 'name': 'arcpr', 'total_epoch': 20, 'pretrainG': 'assets/pretrained_v2/G40k.pth', 'pretrainD': 'assets/pretrained_v2/D40k.pth', 'version': 'v2', 'gpus': '0', 'sample_rate': '40k', 'if_f0': 1, 'if_latest': 0, 'save_every_weights': '0', 'if_cache_data_in_gpu': 0, 'author': ''}
/Users/ec2-user/Documents/torch311/lib/python3.11/site-packages/torch/nn/utils/weight_norm.py:144: FutureWarning: `torch.nn.utils.weight_norm` is deprecated in favor of `torch.nn.utils.parametrizations.weight_norm`.
  WeightNorm.apply(module, name, dim)
INFO:arcpr:loaded pretrained assets/pretrained_v2/G40k.pth
Process Process-1:
Traceback (most recent call last):
  File "/Users/ec2-user/Documents/Retrieval-based-Voice-Conversion-WebUI/infer/modules/train/train.py", line 235, in run
    utils.latest_checkpoint_path(hps.model_dir, "D_*.pth"), net_d, optim_d
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ec2-user/Documents/Retrieval-based-Voice-Conversion-WebUI/infer/lib/train/utils.py", line 120, in latest_checkpoint_path
    x = f_list[-1]
        ~~~~~~^^^^
IndexError: list index out of range

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/ec2-user/Documents/torch311/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/Users/ec2-user/Documents/torch311/lib/python3.11/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/ec2-user/Documents/Retrieval-based-Voice-Conversion-WebUI/infer/modules/train/train.py", line 255, in run
    net_g.module.load_state_dict(
  File "/Users/ec2-user/Documents/torch311/lib/python3.11/site-packages/torch/nn/modules/module.py", line 2624, in load_state_dict
    raise RuntimeError(
RuntimeError: Error(s) in loading state_dict for SynthesizerTrnMs768NSFsid:
	Missing key(s) in state_dict: "enc_p.emb_pitch.weight", "dec.m_source.l_linear.weight", "dec.m_source.l_linear.bias", "dec.noise_convs.0.weight", "dec.noise_convs.0.bias", "dec.noise_convs.1.weight", "dec.noise_convs.1.bias", "dec.noise_convs.2.weight", "dec.noise_convs.2.bias", "dec.noise_convs.3.weight", "dec.noise_convs.3.bias".
DEBUG:matplotlib.pyplot:Loaded backend macosx version unknown.

OS: macos 13 arm
python: 3.11

pretrained model downloaded from https://github.com/fumiama/RVC-Models-Downloader

Both 40k and 48k have this error.
Has the model structure changed?

[acely]

找到解决办法了，给infer/modules/train/train.py这个文件修改，load_state_dict函数加一个strict=False参数就行，加4个：

except:  # 如果首次不能加载，加载pretrain
      # traceback.print_exc()
      epoch_str = 1
      global_step = 0
      if hps.pretrainG != "":
          if rank == 0:
              logger.info("loaded pretrained %s" % (hps.pretrainG))
          if hasattr(net_g, "module"):
              logger.info(
                  net_g.module.load_state_dict(
                      torch.load(
                          hps.pretrainG, map_location="cpu", weights_only=True
                      )["model"], strict=False
                  )
              )  ##测试不加载优化器
          else:
              logger.info(
                  net_g.load_state_dict(
                      torch.load(
                          hps.pretrainG, map_location="cpu", weights_only=True
                      )["model"], strict=False
                  )
              )  ##测试不加载优化器
      if hps.pretrainD != "":
          if rank == 0:
              logger.info("loaded pretrained %s" % (hps.pretrainD))
          if hasattr(net_d, "module"):
              logger.info(
                  net_d.module.load_state_dict(
                      torch.load(
                          hps.pretrainD, map_location="cpu", weights_only=True
                      )["model"], strict=False
                  )
              )
          else:
              logger.info(
                  net_d.load_state_dict(
                      torch.load(
                          hps.pretrainD, map_location="cpu", weights_only=True
                      )["model"], strict=False
                  )
              )

[larpedd]

INFO:uiiai:Training is done. The program is closed.
/usr/local/lib/python3.12/dist-packages/torch/nn/utils/weight_norm.py:144: FutureWarning: `torch.nn.utils.weight_norm` is deprecated in favor of `torch.nn.utils.parametrizations.weight_norm`.
  WeightNorm.apply(module, name, dim)
INFO:uiiai:saving final ckpt:Traceback (most recent call last):
  File "/content/RV2/infer/lib/train/process_ckpt.py", line 51, in save_small_model
    h = model_hash_ckpt(opt)
        ^^^^^^^^^^^^^^^^^^^^
  File "/content/RV2/infer/modules/vc/hash.py", line 147, in model_hash_ckpt
    h = model_hash(config, tgt_sr, net_g, if_f0, version)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/content/RV2/infer/modules/vc/hash.py", line 100, in model_hash
    hbt = load_hubert(config.device, config.is_half)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/content/RV2/infer/modules/vc/utils.py", line 24, in load_hubert
    models, _, _ = checkpoint_utils.load_model_ensemble_and_task(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/fairseq/checkpoint_utils.py", line 432, in load_model_ensemble_and_task
    state = load_checkpoint_to_cpu(filename, arg_overrides)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/fairseq/checkpoint_utils.py", line 322, in load_checkpoint_to_cpu
    state = torch.load(f, map_location=torch.device("cpu"))
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/serialization.py", line 1529, in load
    raise pickle.UnpicklingError(_get_wo_message(str(e))) from None
_pickle.UnpicklingError: Weights only load failed. This file can still be loaded, to do so you have two options, do those steps only if you trust the source of the checkpoint. 
	(1) In PyTorch 2.6, we changed the default value of the `weights_only` argument in `torch.load` from `False` to `True`. Re-running `torch.load` with `weights_only` set to `False` will likely succeed, but it can result in arbitrary code execution. Do it only if you got the file from a trusted source.
	(2) Alternatively, to load with `weights_only=True` please check the recommended steps in the following error message.
	WeightsUnpickler error: Unsupported global: GLOBAL fairseq.data.dictionary.Dictionary was not an allowed global by default. Please use `torch.serialization.add_safe_globals([fairseq.data.dictionary.Dictionary])` or the `torch.serialization.safe_globals([fairseq.data.dictionary.Dictionary])` context manager to allowlist this global if you trust this class/function.

Check the documentation of torch.load to learn more about types accepted by default with weights_only https://pytorch.org/docs/stable/generated/torch.load.html.

/usr/lib/python3.12/multiprocessing/resource_tracker.py:279: UserWarning: resource_tracker: There appear to be 21 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '

i have a different error this time, related to training