RuntimeError: Distributed package doesn't have NCCL built in

[runshengdu]

any way to fix this issue?

(dlt) PS C:\Users\skyne\omages> python -m src.trainer --opts src/models/omages64_DiT/cfgs/pipeline_N2G2M.yaml --gpus 0 --mode 'test'
**** INFO **** Choosing GPUS: [0]
**** INFO **** Running single opt file
{'full_opt': {'project_name': 'omg', 'expr_name': 'omages64/pipeline_N2G2M', 'pltrainer_opt': {'fast_dev_run': False, 'gpus': [0], 'max_epochs': 100, 'min_epochs': 1, 'max_time': None, 'copy_ckpt_from': '', 'resume_from': 'resume', 'check_val_every_n_epoch': 3, 'checkpoint_monitor': 'val/loss', 'early_stop_patience': 5, 'disable_auto_lr_scale': False, 'logger': 'wandb', 'logger_kwargs': {'mode': 'disabled'}, 'wandb_silent': False, 'auto_lr_find': False, 'gradient_clip_val': None, 'accelerator': 'auto', 'strategy': 'ddp_find_unused_parameters_true', 'precision': '32-true', 'use_tensorcores': True, 'seed': 314, 'strict_loading': True, 'extra_kwargs': {'num_sanity_val_steps': 0, 'enable_model_summary': True, 'precision': '32-true'}}, 'pl_model_opt': {'target': 'src.models.omages64_DiT.pl_N2G2M_pipeline.Model'}, 'datamodule_opt': {'target': 'src.datamodule.DataModule', 'num_workers': 4, 'batch_size': 4, 'test_batch_size': 4, 'trainset_opt': {'target': 'src.data.abo_datasets.omg_dataset.N2G_Label_OmgABO', 'mode': 'fullsplit', 'cates': ['chair'], 'duplicate': 1, 'dset_df': 'datasets/ABO/omages/df_p64_m02_res64'}, 'testset_opt': {'target': 'src.data.abo_datasets.omg_dataset.N2G_Label_OmgABO', 'mode': 'fullsplit', 'cates': ['chair'], 'duplicate': 1, 'dset_df': 'datasets/ABO/omages/df_p64_m02_res64'}}, 'callbacks': {'vis': {'target': 'src.models.omages64_DiT.pl_N2G2M_pipeline.N2G2MCallback', 'no_sanity_check': False, 'every_n_epoch': 1, 'if_load_saved_compute': [True, True, False], 'visual_indices': [0, 1, 2, 3, 4, 5, 6, 7], 'N2G_cfg': 'src/models/omages64_DiT/cfgs/null2geo.yaml', 'N2G_ckpt': 'experiments/omages64/null2geo/checkpoints/N2G_dit_allcate.ckpt', 'G2M_cfg': 'src/models/omages64_DiT/cfgs/geo2mat_imagen.yaml', 'G2M_ckpt': 'experiments/omages64/geo2mat_imagen/checkpoints/G2M_imagen_allcate.ckpt'}}, 'meta_info': {'src_dir': 'C:\Users\skyne\omages\src', 'datasets_dir': 'C:\Users\skyne\omages\datasets/', 'experiments_dir': 'C:\Users\skyne\omages\experiments/', 'expr_dir': 'C:\Users\skyne\omages\experiments/omages64/pipeline_N2G2M', 'logs_dir': 'C:\Users\skyne\omages\experiments/omages64/pipeline_N2G2M\logs', 'checkpoints_dir': 'C:\Users\skyne\omages\experiments/omages64/pipeline_N2G2M\checkpoints', 'results_dir': 'C:\Users\skyne\omages\experiments/omages64/pipeline_N2G2M\results', 'session_name': 'omages64/pipeline_N2G2M_240825_2157'}}, 'full_cfg': {'project_name': 'omg', 'expr_name': 'omages64/pipeline_N2G2M', 'pltrainer_opt': {'fast_dev_run': False, 'gpus': [0], 'max_epochs': 100, 'min_epochs': 1, 'max_time': None, 'copy_ckpt_from': '', 'resume_from': 'resume', 'check_val_every_n_epoch': 3, 'checkpoint_monitor': 'val/loss', 'early_stop_patience': 5, 'disable_auto_lr_scale': False, 'logger': 'wandb', 'logger_kwargs': {'mode': 'disabled'}, 'wandb_silent': False, 'auto_lr_find': False, 'gradient_clip_val': None, 'accelerator': 'auto', 'strategy': 'ddp_find_unused_parameters_true', 'precision': '32-true', 'use_tensorcores': True, 'seed': 314, 'strict_loading': True, 'extra_kwargs': {'num_sanity_val_steps': 0, 'enable_model_summary': True, 'precision': '32-true'}}, 'pl_model_opt': {'target': 'src.models.omages64_DiT.pl_N2G2M_pipeline.Model', 'full_opt': {'project_name': 'omg', 'expr_name': 'omages64/pipeline_N2G2M', 'pltrainer_opt': {'fast_dev_run': False, 'gpus': [0], 'max_epochs': 100, 'min_epochs': 1, 'max_time': None, 'copy_ckpt_from': '', 'resume_from': 'resume', 'check_val_every_n_epoch': 3, 'checkpoint_monitor': 'val/loss', 'early_stop_patience': 5, 'disable_auto_lr_scale': False, 'logger': 'wandb', 'logger_kwargs': {'mode': 'disabled'}, 'wandb_silent': False, 'auto_lr_find': False, 'gradient_clip_val': None, 'accelerator': 'auto', 'strategy': 'ddp_find_unused_parameters_true', 'precision': '32-true', 'use_tensorcores': True, 'seed': 314, 'strict_loading': True, 'extra_kwargs': {'num_sanity_val_steps': 0, 'enable_model_summary': True, 'precision': '32-true'}}, 'pl_model_opt': {'target': 'src.models.omages64_DiT.pl_N2G2M_pipeline.Model'}, 'datamodule_opt': {'target': 'src.datamodule.DataModule', 'num_workers': 4, 'batch_size': 4, 'test_batch_size': 4, 'trainset_opt': {'target': 'src.data.abo_datasets.omg_dataset.N2G_Label_OmgABO', 'mode': 'fullsplit', 'cates': ['chair'], 'duplicate': 1, 'dset_df': 'datasets/ABO/omages/df_p64_m02_res64'}, 'testset_opt': {'target': 'src.data.abo_datasets.omg_dataset.N2G_Label_OmgABO', 'mode': 'fullsplit', 'cates': ['chair'], 'duplicate': 1, 'dset_df': 'datasets/ABO/omages/df_p64_m02_res64'}}, 'callbacks': {'vis': {'target': 'src.models.omages64_DiT.pl_N2G2M_pipeline.N2G2MCallback', 'no_sanity_check': False, 'every_n_epoch': 1, 'if_load_saved_compute': [True, True, False], 'visual_indices': [0, 1, 2, 3, 4, 5, 6, 7], 'N2G_cfg': 'src/models/omages64_DiT/cfgs/null2geo.yaml', 'N2G_ckpt': 'experiments/omages64/null2geo/checkpoints/N2G_dit_allcate.ckpt', 'G2M_cfg': 'src/models/omages64_DiT/cfgs/geo2mat_imagen.yaml', 'G2M_ckpt': 'experiments/omages64/geo2mat_imagen/checkpoints/G2M_imagen_allcate.ckpt'}}, 'meta_info': {'src_dir': 'C:\Users\skyne\omages\src', 'datasets_dir': 'C:\Users\skyne\omages\datasets/', 'experiments_dir': 'C:\Users\skyne\omages\experiments/', 'expr_dir': 'C:\Users\skyne\omages\experiments/omages64/pipeline_N2G2M', 'logs_dir': 'C:\Users\skyne\omages\experiments/omages64/pipeline_N2G2M\logs', 'checkpoints_dir': 'C:\Users\skyne\omages\experiments/omages64/pipeline_N2G2M\checkpoints', 'results_dir': 'C:\Users\skyne\omages\experiments/omages64/pipeline_N2G2M\results', 'session_name': 'omages64/pipeline_N2G2M_240825_2157'}}}, 'datamodule_opt': {'target': 'src.datamodule.DataModule', 'num_workers': 4, 'batch_size': 4, 'test_batch_size': 4, 'trainset_opt': {'target': 'src.data.abo_datasets.omg_dataset.N2G_Label_OmgABO', 'mode': 'fullsplit', 'cates': ['chair'], 'duplicate': 1, 'dset_df': 'datasets/ABO/omages/df_p64_m02_res64'}, 'testset_opt': {'target': 'src.data.abo_datasets.omg_dataset.N2G_Label_OmgABO', 'mode': 'fullsplit', 'cates': ['chair'], 'duplicate': 1, 'dset_df': 'datasets/ABO/omages/df_p64_m02_res64'}}, 'callbacks': {'vis': {'target': 'src.models.omages64_DiT.pl_N2G2M_pipeline.N2G2MCallback', 'no_sanity_check': False, 'every_n_epoch': 1, 'if_load_saved_compute': [True, True, False], 'visual_indices': [0, 1, 2, 3, 4, 5, 6, 7], 'N2G_cfg': 'src/models/omages64_DiT/cfgs/null2geo.yaml', 'N2G_ckpt': 'experiments/omages64/null2geo/checkpoints/N2G_dit_allcate.ckpt', 'G2M_cfg': 'src/models/omages64_DiT/cfgs/geo2mat_imagen.yaml', 'G2M_ckpt': 'experiments/omages64/geo2mat_imagen/checkpoints/G2M_imagen_allcate.ckpt'}}, 'meta_info': {'src_dir': 'C:\Users\skyne\omages\src', 'datasets_dir': 'C:\Users\skyne\omages\datasets/', 'experiments_dir': 'C:\Users\skyne\omages\experiments/', 'expr_dir': 'C:\Users\skyne\omages\experiments/omages64/pipeline_N2G2M', 'logs_dir': 'C:\Users\skyne\omages\experiments/omages64/pipeline_N2G2M\logs', 'checkpoints_dir': 'C:\Users\skyne\omages\experiments/omages64/pipeline_N2G2M\checkpoints', 'results_dir': 'C:\Users\skyne\omages\experiments/omages64/pipeline_N2G2M\results', 'session_name': 'omages64/pipeline_N2G2M_240825_2157'}}}
{'num_workers': 4, 'batch_size': 4, 'test_batch_size': 4, 'trainset_opt': {'target': 'src.data.abo_datasets.omg_dataset.N2G_Label_OmgABO', 'mode': 'fullsplit', 'cates': ['chair'], 'duplicate': 1, 'dset_df': 'datasets/ABO/omages/df_p64_m02_res64'}, 'testset_opt': {'target': 'src.data.abo_datasets.omg_dataset.N2G_Label_OmgABO', 'mode': 'fullsplit', 'cates': ['chair'], 'duplicate': 1, 'dset_df': 'datasets/ABO/omages/df_p64_m02_res64'}}
{'data_dir': 'C:\Users\skyne\omages\experiments/omages64/pipeline_N2G2M\results\N2G2MCallback', 'target': 'src.models.omages64_DiT.pl_N2G2M_pipeline.N2G2MCallback', 'no_sanity_check': False, 'every_n_epoch': 1, 'if_load_saved_compute': [True, True, False], 'visual_indices': [0, 1, 2, 3, 4, 5, 6, 7], 'N2G_cfg': 'src/models/omages64_DiT/cfgs/null2geo.yaml', 'N2G_ckpt': 'experiments/omages64/null2geo/checkpoints/N2G_dit_allcate.ckpt', 'G2M_cfg': 'src/models/omages64_DiT/cfgs/geo2mat_imagen.yaml', 'G2M_ckpt': 'experiments/omages64/geo2mat_imagen/checkpoints/G2M_imagen_allcate.ckpt', 'num_gpus': 1}
VI [0, 1, 2, 3, 4, 5, 6, 7]
{'baseDiT_cfg': {'target': 'src.models.omages64_DiT.dit_models.OmgDiT', 'input_size': 64, 'in_channels': 4, 'condimg_channels': None, 'num_classes': 63, 'patch_size': 1, 'depth': 12, 'hidden_size': 768, 'num_heads': 12, 'learn_sigma': True}, 'infer_steps': 250}
{'input_size': 64, 'in_channels': 4, 'condimg_channels': None, 'num_classes': 63, 'patch_size': 1, 'depth': 12, 'hidden_size': 768, 'num_heads': 12, 'learn_sigma': True}
!!!!!!!!!! 0.999 4.128422482196914e-05
!!!!!!!!!! 0.999 4.128422482196914e-05
{'init_from_ckpt': None, 'omage_channels': 5, 'cond_omage_channels': 4, 'dim': 256}
Callbacks [<src.models.omages64_DiT.pl_N2G2M_pipeline.N2G2MCallback object at 0x0000024EEFC18430>]
??? resume test
{'src_dir': 'C:\Users\skyne\omages\src', 'datasets_dir': 'C:\Users\skyne\omages\datasets/', 'experiments_dir': 'C:\Users\skyne\omages\experiments/', 'expr_dir': 'C:\Users\skyne\omages\experiments/omages64/pipeline_N2G2M', 'logs_dir': 'C:\Users\skyne\omages\experiments/omages64/pipeline_N2G2M\logs', 'checkpoints_dir': 'C:\Users\skyne\omages\experiments/omages64/pipeline_N2G2M\checkpoints', 'results_dir': 'C:\Users\skyne\omages\experiments/omages64/pipeline_N2G2M\results', 'session_name': 'omages64/pipeline_N2G2M_240825_2157'} ckpts
Directory C:\Users\skyne\omages\experiments/omages64/pipeline_N2G2M\checkpoints has no checkpoints
====== Max epochs ===== 100
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
{'mode': 'fullsplit', 'cates': ['chair'], 'duplicate': 1, 'dset_df': 'datasets/ABO/omages/df_p64_m02_res64', 'split': 'train'}
{'mode': 'fullsplit', 'cates': ['chair'], 'duplicate': 1, 'dset_df': 'datasets/ABO/omages/df_p64_m02_res64', 'split': 'test'}
{'src_dir': 'C:\Users\skyne\omages\src', 'datasets_dir': 'C:\Users\skyne\omages\datasets/', 'experiments_dir': 'C:\Users\skyne\omages\experiments/', 'expr_dir': 'C:\Users\skyne\omages\experiments/omages64/pipeline_N2G2M', 'logs_dir': 'C:\Users\skyne\omages\experiments/omages64/pipeline_N2G2M\logs', 'checkpoints_dir': 'C:\Users\skyne\omages\experiments/omages64/pipeline_N2G2M\checkpoints', 'results_dir': 'C:\Users\skyne\omages\experiments/omages64/pipeline_N2G2M\results', 'session_name': 'omages64/pipeline_N2G2M_240825_2157'} ckpts
Directory C:\Users\skyne\omages\experiments/omages64/pipeline_N2G2M\checkpoints has no checkpoints
Test resume from None
Initializing distributed: GLOBAL_RANK: 0, MEMBER: 1/1
[W C:\cb\pytorch_1000000000000\work\torch\csrc\distributed\c10d\socket.cpp:601] [c10d] The client socket has failed to connect to [kubernetes.docker.internal]:63997 (system error: 10049 - 在其上下文中，该请求的地址无效。).
[W C:\cb\pytorch_1000000000000\work\torch\csrc\distributed\c10d\socket.cpp:601] [c10d] The client socket has failed to connect to [kubernetes.docker.internal]:63997 (system error: 10049 - 在其上下文中，该请求的地址无效。).
Traceback (most recent call last):
File "C:\Users\skyne\anaconda3\envs\dlt\lib\runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Users\skyne\anaconda3\envs\dlt\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
File "C:\Users\skyne\omages\src\trainer.py", line 387, in
func()
File "C:\Users\skyne\omages\src\trainer.py", line 280, in test
self.trainer.test(
File "C:\Users\skyne\anaconda3\envs\dlt\lib\site-packages\lightning\pytorch\trainer\trainer.py", line 754, in test
return call._call_and_handle_interrupt(
File "C:\Users\skyne\anaconda3\envs\dlt\lib\site-packages\lightning\pytorch\trainer\call.py", line 43, in _call_and_handle_interrupt
return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
File "C:\Users\skyne\anaconda3\envs\dlt\lib\site-packages\lightning\pytorch\strategies\launchers\subprocess_script.py", line 105, in launch
return function(*args, **kwargs)
File "C:\Users\skyne\anaconda3\envs\dlt\lib\site-packages\lightning\pytorch\trainer\trainer.py", line 794, in _test_impl
results = self._run(model, ckpt_path=ckpt_path)
File "C:\Users\skyne\anaconda3\envs\dlt\lib\site-packages\lightning\pytorch\trainer\trainer.py", line 943, in _run
self.strategy.setup_environment()
File "C:\Users\skyne\anaconda3\envs\dlt\lib\site-packages\lightning\pytorch\strategies\ddp.py", line 154, in setup_environment
self.setup_distributed()
File "C:\Users\skyne\anaconda3\envs\dlt\lib\site-packages\lightning\pytorch\strategies\ddp.py", line 203, in setup_distributed
_init_dist_connection(self.cluster_environment, self._process_group_backend, timeout=self._timeout)
File "C:\Users\skyne\anaconda3\envs\dlt\lib\site-packages\lightning\fabric\utilities\distributed.py", line 291, in _init_dist_connection
torch.distributed.init_process_group(torch_distributed_backend, rank=global_rank, world_size=world_size, **kwargs)
File "C:\Users\skyne\anaconda3\envs\dlt\lib\site-packages\torch\distributed\distributed_c10d.py", line 907, in init_process_group
default_pg = _new_process_group_helper(
File "C:\Users\skyne\anaconda3\envs\dlt\lib\site-packages\torch\distributed\distributed_c10d.py", line 1013, in _new_process_group_helper
raise RuntimeError("Distributed package doesn't have NCCL " "built in")
RuntimeError: Distributed package doesn't have NCCL built in

[QhelDIV]

It seems pytorch lightning's distributed training/testing is not supported on Windows.
Maybe you can try to add a line 'strategy: "dp"' to the
'pltrainer_opt' config in the pipeline_N2G2M.yaml file?
You can check the src/trainer.py to see what this does.
Basically, the default strategy is to use pytorch ddp, which is a distributed training strategy.

By changing the 'strategy' to 'dp', no distributed environment is needed.
Or you can try 'auto', or SingleDeviceStrategy

To see the difference of 'ddp' and 'dp', please refer to the pytorch (lightning) documents.