Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bug report related to e4s-cl init #67

Open
thomas-robinson opened this issue May 27, 2022 · 1 comment
Open

Bug report related to e4s-cl init #67

thomas-robinson opened this issue May 27, 2022 · 1 comment
Assignees
@spoutn1k

Comments

@thomas-robinson
Copy link

I am running on a cloud cluster set up on a parallel works platform. I was trying to run e4s-cl init, but I ran into an error, and after looking at documentation, I set up a profile

e4s-cl profile detect -p pw srun --mpi=pmi2 -N 3 -n 9  ./c_hello.x

Then I tried to run e4s-cl with my profile and got an error saying I should report a bug here

$ e4s-cl init --launcher srun --backend singularity --profile pw
icc: warning #10315: specifying -lm before files may supersede the Intel(R) math library and affect performance
Simulating MPI execution using:
Compiler: /apps/oneapi/mpi/2021.5.1/bin/mpicc
Launcher /usr/bin/srun
The target launcher /usr/bin/srun uses a single host by default, which may tamper with the library discovery. Consider running `e4s-cl profile detect` using mpirun specifying multiple hosts.
Failed to list opened files of /tmp/tmpgwhuuqtz: [Errno 2] No such file or directory
srun: error: thomasrobinson-fv3-00009-1-0007: task 0: Exited with exit code 156
Failed to determine necessary libraries: program exited with code 156
MPI execution failed.
An unexpected KeyError exception was raised:

2

Traceback (most recent call last):
  File "/contrib/bin/e4s-cl", line 12, in <module>
    sys.exit(cli_main_cmd.main(sys.argv[1:]))
  File "/contrib/conda/lib/python3.9/site-packages/e4s_cl/cli/commands/__main__.py", line 200, in main
    return cli.execute_command([cmd], cmd_args)
  File "/contrib/conda/lib/python3.9/site-packages/e4s_cl/cli/__init__.py", line 314, in execute_command
    return find_command(cmd).main(cmd_args)
  File "/contrib/conda/lib/python3.9/site-packages/e4s_cl/cli/commands/init.py", line 436, in main
    controller.update({'name': requested_name}, profile.eid)
  File "/contrib/conda/lib/python3.9/site-packages/e4s_cl/model/profile.py", line 102, in update
    super().update(data, keys)
  File "/contrib/conda/lib/python3.9/site-packages/e4s_cl/mvc/controller.py", line 236, in update
    database.update(data, keys, table_name=self.model.name)
  File "/contrib/conda/lib/python3.9/site-packages/e4s_cl/cf/storage/local_file.py", line 459, in update
    table.update(fields, doc_ids=[keys])
  File "/contrib/conda/lib/python3.9/site-packages/tinydb/table.py", line 340, in update
    self._update_table(updater)
  File "/contrib/conda/lib/python3.9/site-packages/tinydb/table.py", line 709, in _update_table
    updater(table)
  File "/contrib/conda/lib/python3.9/site-packages/tinydb/table.py", line 337, in updater
    perform_update(table, doc_id)
  File "/contrib/conda/lib/python3.9/site-packages/tinydb/table.py", line 326, in perform_update
    table[doc_id].update(fields)
KeyError: 2

This is a bug in E4S Container Launcher.
Please raise an issue on Github with the contents of '/home/Thomas.Robinson/.local/e4s_cl/logs/debug_log'.

Here's the contents of the file

$ cat /home/Thomas.Robinson/.local/e4s_cl/logs/debug_log
[Debug root:491] Unlink /home/Thomas.Robinson/.local/e4s_cl/logs/latest failed: [Errno 2] No such file or directory: '/home/Thomas.Robinson/.local/e4s_cl/logs/latest'
[Debug root:503] 
################################################################################################################
E4S CONTAINER LAUNCHER LOGGING INITIALIZED

Timestamp         : 2022-05-27 12:21:03.378903
Hostname          : mgmt-thomasrobinson-fv3-00009
Platform          : Linux-3.10.0-1160.59.1.el7.x86_64-x86_64-with-glibc2.17
Version           : 1.0rc6
Python Version    : 3.9.12
Working Directory : /contrib
Terminal Size     : 112x26
Frozen            : False
Log ID            : fd02aa3ff3e94da26c65add3747862b90677a8053e9498c33c7214f7cf2e804b
################################################################################################################

[Debug e4s_cl.cf.wi4mpi:26] Getting WI4MPI root failed
[Debug e4s_cl.cf.storage.local_file:146] Initialized system filesystem prefix '/contrib/system'
[Debug e4s_cl.cli.commands.__main__:77] e4s-cl args: Namespace(command='init', options=['--launcher', 'srun', '--backend', 'singularity'], dry_run=None)
[Debug e4s_cl.cli.commands.__main__:195] Arguments: Namespace(command='init', options=['--launcher', 'srun', '--backend', 'singularity'], dry_run=None)
[Debug e4s_cl.cli.commands.__main__:196] Verbosity level: INFO
[Debug e4s_cl.cf.storage.local_file:50] '/contrib/system/system.json' opened read-write
[Debug e4s_cl.cf.storage.local_file:170] Initialized system database '/contrib/system/system.json'
[Debug e4s_cl.cf.storage.local_file:50] '/home/Thomas.Robinson/.local/e4s_cl/user.json' opened read-write
[Debug e4s_cl.cf.storage.local_file:170] Initialized user database '/home/Thomas.Robinson/.local/e4s_cl/user.json'
[Debug e4s_cl.cli.commands.init:77] e4s-cl init args: Namespace(launcher='/usr/bin/srun', backend='singularity')
[Debug e4s_cl.cli.commands.init:185] MPI vendor not supported by precompiled binary initialisation
Proceeding with legacy initialisation
[Debug e4s_cl.cli.commands.init:121] Compiling with: '/apps/oneapi/mpi/2021.5.1/bin/mpicc -o /tmp/tmpkv6f6q4s -lm /tmp/tmpwaazorcm.c'
Simulating MPI execution using:
Compiler: /apps/oneapi/mpi/2021.5.1/bin/mpicc
Launcher /usr/bin/srun
The target launcher /usr/bin/srun uses a single host by default, which may tamper with the library discovery. Consider
running `e4s-cl profile detect` using mpirun specifying multiple hosts.
[Debug e4s_cl.cli.commands.profile.detect:77] e4s-cl profile detect args: Namespace(profile_name=None, cmd=['/usr/bin/srun', '/tmp/tmpkv6f6q4s'])
Failed to determine necessary libraries: program exited with code 156
MPI execution failed.
[Debug root:503] 
################################################################################################################
E4S CONTAINER LAUNCHER LOGGING INITIALIZED

Timestamp         : 2022-05-27 12:27:00.260572
Hostname          : mgmt-thomasrobinson-fv3-00009
Platform          : Linux-3.10.0-1160.59.1.el7.x86_64-x86_64-with-glibc2.17
Version           : 1.0rc6
Python Version    : 3.9.12
Working Directory : /contrib
Terminal Size     : 112x26
Frozen            : False
Log ID            : 2ed65e431bfbcab3168ae4f4590fc2a6ccc1448853abf8174ff0fced595e54c8
################################################################################################################

[Debug e4s_cl.cf.wi4mpi:26] Getting WI4MPI root failed
[Debug e4s_cl.cli.commands.__main__:77] e4s-cl args: Namespace(command='init', options=['--launcher', 'srun', '--backend', 'singularity'], dry_run=None)
[Debug e4s_cl.cli.commands.__main__:195] Arguments: Namespace(command='init', options=['--launcher', 'srun', '--backend', 'singularity'], dry_run=None)
[Debug e4s_cl.cli.commands.__main__:196] Verbosity level: INFO
[Debug e4s_cl.cf.storage.local_file:50] '/contrib/system/system.json' opened read-write
[Debug e4s_cl.cf.storage.local_file:170] Initialized system database '/contrib/system/system.json'
[Debug e4s_cl.cf.storage.local_file:50] '/home/Thomas.Robinson/.local/e4s_cl/user.json' opened read-write
[Debug e4s_cl.cf.storage.local_file:170] Initialized user database '/home/Thomas.Robinson/.local/e4s_cl/user.json'
[Debug e4s_cl.cli.commands.init:77] e4s-cl init args: Namespace(launcher='/usr/bin/srun', backend='singularity')
[Debug e4s_cl.cli.commands.init:185] MPI vendor not supported by precompiled binary initialisation
Proceeding with legacy initialisation
[Debug e4s_cl.cli.commands.init:121] Compiling with: '/apps/oneapi/mpi/2021.5.1/bin/mpicc -o /tmp/tmp50exhn5p -lm /tmp/tmprztzimz9.c'
Simulating MPI execution using:
Compiler: /apps/oneapi/mpi/2021.5.1/bin/mpicc
Launcher /usr/bin/srun
The target launcher /usr/bin/srun uses a single host by default, which may tamper with the library discovery. Consider
running `e4s-cl profile detect` using mpirun specifying multiple hosts.
[Debug e4s_cl.cli.commands.profile.detect:77] e4s-cl profile detect args: Namespace(profile_name=None, cmd=['/usr/bin/srun', '/tmp/tmp50exhn5p'])
Failed to determine necessary libraries: program exited with code 156
MPI execution failed.
[Debug root:503] 
################################################################################################################
E4S CONTAINER LAUNCHER LOGGING INITIALIZED

Timestamp         : 2022-05-27 12:43:09.562829
Hostname          : mgmt-thomasrobinson-fv3-00009
Platform          : Linux-3.10.0-1160.59.1.el7.x86_64-x86_64-with-glibc2.17
Version           : 1.0rc6
Python Version    : 3.9.12
Working Directory : /contrib
Terminal Size     : 112x26
Frozen            : False
Log ID            : 4000e36d81b7ad1886b0430b9bccdc4f973f7d488b98f11a1528e1d2f8009167
################################################################################################################

[Debug e4s_cl.cf.wi4mpi:26] Getting WI4MPI root failed
[Debug e4s_cl.cli.commands.__main__:77] e4s-cl args: Namespace(command='init', options=['--launcher', 'srun', '--backend', 'singularity'], dry_run=None)
[Debug e4s_cl.cli.commands.__main__:195] Arguments: Namespace(command='init', options=['--launcher', 'srun', '--backend', 'singularity'], dry_run=None)
[Debug e4s_cl.cli.commands.__main__:196] Verbosity level: INFO
[Debug e4s_cl.cf.storage.local_file:50] '/contrib/system/system.json' opened read-write
[Debug e4s_cl.cf.storage.local_file:170] Initialized system database '/contrib/system/system.json'
[Debug e4s_cl.cf.storage.local_file:50] '/home/Thomas.Robinson/.local/e4s_cl/user.json' opened read-write
[Debug e4s_cl.cf.storage.local_file:170] Initialized user database '/home/Thomas.Robinson/.local/e4s_cl/user.json'
[Debug e4s_cl.cli.commands.init:77] e4s-cl init args: Namespace(launcher='/usr/bin/srun', backend='singularity')
[Debug e4s_cl.cli.commands.init:185] MPI vendor not supported by precompiled binary initialisation
Proceeding with legacy initialisation
[Debug e4s_cl.cli.commands.init:121] Compiling with: '/apps/oneapi/mpi/2021.5.1/bin/mpicc -o /tmp/tmp3k608a5f -lm /tmp/tmp7wm2qne0.c'
Simulating MPI execution using:
Compiler: /apps/oneapi/mpi/2021.5.1/bin/mpicc
Launcher /usr/bin/srun
The target launcher /usr/bin/srun uses a single host by default, which may tamper with the library discovery. Consider
running `e4s-cl profile detect` using mpirun specifying multiple hosts.
[Debug e4s_cl.cli.commands.profile.detect:77] e4s-cl profile detect args: Namespace(profile_name=None, cmd=['/usr/bin/srun', '/tmp/tmp3k608a5f'])
Failed to determine necessary libraries: program exited with code 156
MPI execution failed.
[Debug root:503] 
################################################################################################################
E4S CONTAINER LAUNCHER LOGGING INITIALIZED

Timestamp         : 2022-05-27 12:48:33.328930
Hostname          : mgmt-thomasrobinson-fv3-00009
Platform          : Linux-3.10.0-1160.59.1.el7.x86_64-x86_64-with-glibc2.17
Version           : 1.0rc6
Python Version    : 3.9.12
Working Directory : /contrib
Terminal Size     : 112x26
Frozen            : False
Log ID            : 80bee8cfe4305db665a79590bafc927ad89865710fad0e93acbc59a26de0f525
################################################################################################################

[Debug e4s_cl.cf.wi4mpi:26] Getting WI4MPI root failed
[Debug e4s_cl.cli.commands.__main__:77] e4s-cl args: Namespace(command='profile', options=['detect'], dry_run=None)
[Debug e4s_cl.cli.commands.__main__:195] Arguments: Namespace(command='profile', options=['detect'], dry_run=None)
[Debug e4s_cl.cli.commands.__main__:196] Verbosity level: INFO
[Debug e4s_cl.cli.commands.profile:77] e4s-cl profile args: Namespace(subcommand='detect', options=[])
[Debug e4s_cl.cli.commands.profile.detect:77] e4s-cl profile detect args: Namespace(profile_name=None, cmd=[])
[Debug root:503] 
################################################################################################################
E4S CONTAINER LAUNCHER LOGGING INITIALIZED

Timestamp         : 2022-05-27 12:49:27.375315
Hostname          : mgmt-thomasrobinson-fv3-00009
Platform          : Linux-3.10.0-1160.59.1.el7.x86_64-x86_64-with-glibc2.17
Version           : 1.0rc6
Python Version    : 3.9.12
Working Directory : /contrib
Terminal Size     : 112x26
Frozen            : False
Log ID            : 1370b8b30920376a8de2c875365a7c79ec2168c1e968882b4dbe18d8f5fa6064
################################################################################################################

[Debug e4s_cl.cf.wi4mpi:26] Getting WI4MPI root failed
[Debug e4s_cl.cli.commands.__main__:77] e4s-cl args: Namespace(command='init', options=['--launcher', 'srun', '--backend', 'singularity'], dry_run=None)
[Debug e4s_cl.cli.commands.__main__:195] Arguments: Namespace(command='init', options=['--launcher', 'srun', '--backend', 'singularity'], dry_run=None)
[Debug e4s_cl.cli.commands.__main__:196] Verbosity level: INFO
[Debug e4s_cl.cf.storage.local_file:50] '/contrib/system/system.json' opened read-write
[Debug e4s_cl.cf.storage.local_file:170] Initialized system database '/contrib/system/system.json'
[Debug e4s_cl.cf.storage.local_file:50] '/home/Thomas.Robinson/.local/e4s_cl/user.json' opened read-write
[Debug e4s_cl.cf.storage.local_file:170] Initialized user database '/home/Thomas.Robinson/.local/e4s_cl/user.json'
[Debug e4s_cl.cli.commands.init:77] e4s-cl init args: Namespace(launcher='/usr/bin/srun', backend='singularity')
[Debug e4s_cl.cli.commands.init:185] MPI vendor not supported by precompiled binary initialisation
Proceeding with legacy initialisation
[Debug e4s_cl.cli.commands.init:121] Compiling with: '/apps/oneapi/mpi/2021.5.1/bin/mpicc -o /tmp/tmp5tum_jh1 -lm /tmp/tmpxlk4p3nz.c'
Simulating MPI execution using:
Compiler: /apps/oneapi/mpi/2021.5.1/bin/mpicc
Launcher /usr/bin/srun
The target launcher /usr/bin/srun uses a single host by default, which may tamper with the library discovery. Consider
running `e4s-cl profile detect` using mpirun specifying multiple hosts.
[Debug e4s_cl.cli.commands.profile.detect:77] e4s-cl profile detect args: Namespace(profile_name=None, cmd=['/usr/bin/srun', '/tmp/tmp5tum_jh1'])
Failed to determine necessary libraries: program exited with code 156
MPI execution failed.
[Debug root:503] 
################################################################################################################
E4S CONTAINER LAUNCHER LOGGING INITIALIZED

Timestamp         : 2022-05-27 12:56:16.392146
Hostname          : mgmt-thomasrobinson-fv3-00009
Platform          : Linux-3.10.0-1160.59.1.el7.x86_64-x86_64-with-glibc2.17
Version           : 1.0rc6
Python Version    : 3.9.12
Working Directory : /contrib
Terminal Size     : 112x26
Frozen            : False
Log ID            : b68d0d3b050bc18c09ed12970cec9dcf9d9b8c3816b2c77b4a66cce73167bbd7
################################################################################################################

[Debug e4s_cl.cf.wi4mpi:26] Getting WI4MPI root failed
[Debug e4s_cl.cli.commands.__main__:77] e4s-cl args: Namespace(command='init', options=['--launcher', 'srun -N6', '--backend', 'singularity'], dry_run=None)
[Debug e4s_cl.cli.commands.__main__:195] Arguments: Namespace(command='init', options=['--launcher', 'srun -N6', '--backend', 'singularity'], dry_run=None)
[Debug e4s_cl.cli.commands.__main__:196] Verbosity level: INFO
[Debug e4s_cl.cf.storage.local_file:50] '/contrib/system/system.json' opened read-write
[Debug e4s_cl.cf.storage.local_file:170] Initialized system database '/contrib/system/system.json'
[Debug e4s_cl.cf.storage.local_file:50] '/home/Thomas.Robinson/.local/e4s_cl/user.json' opened read-write
[Debug e4s_cl.cf.storage.local_file:170] Initialized user database '/home/Thomas.Robinson/.local/e4s_cl/user.json'
usage: e4s-cl init [-h] [--system machine] [--launcher launcher]
                   [--launcher_args launcher_args] [--mpi /path/to/mpi]
                   [--source script] [--image /path/to/image] [--backend technology]
                   [--profile profile_name] [--wi4mpi path] [--wi4mpi_options opts]
e4s-cl init: error: argument --launcher: Launcher argument 'srun -N6' could not be resolved to a binary
[Debug root:503] 
################################################################################################################
E4S CONTAINER LAUNCHER LOGGING INITIALIZED

Timestamp         : 2022-05-27 12:59:12.937213
Hostname          : mgmt-thomasrobinson-fv3-00009
Platform          : Linux-3.10.0-1160.59.1.el7.x86_64-x86_64-with-glibc2.17
Version           : 1.0rc6
Python Version    : 3.9.12
Working Directory : /contrib/test
Terminal Size     : 112x26
Frozen            : False
Log ID            : f8be31d20e3ad8fe2ea5d3b133dbc6f021d2492d714ae70f3b07940e11a1dfd5
################################################################################################################

[Debug e4s_cl.cf.wi4mpi:26] Getting WI4MPI root failed
[Debug e4s_cl.cli.commands.__main__:77] e4s-cl args: Namespace(command='profile', options=['detect', '-p', 'pw', 'srun', '-N', '3', '-n', '3', './a.out'], dry_run=None)
[Debug e4s_cl.cli.commands.__main__:195] Arguments: Namespace(command='profile', options=['detect', '-p', 'pw', 'srun', '-N', '3', '-n', '3', './a.out'], dry_run=None)
[Debug e4s_cl.cli.commands.__main__:196] Verbosity level: INFO
[Debug e4s_cl.cli.commands.profile:77] e4s-cl profile args: Namespace(subcommand='detect', options=['-p', 'pw', 'srun', '-N', '3', '-n', '3', './a.out'])
[Debug e4s_cl.cli.commands.profile.detect:77] e4s-cl profile detect args: Namespace(profile_name='pw', cmd=['srun', '-N', '3', '-n', '3', './a.out'])
Failed to determine necessary libraries: program exited with code 156
[Debug root:503] 
################################################################################################################
E4S CONTAINER LAUNCHER LOGGING INITIALIZED

Timestamp         : 2022-05-27 13:16:57.733090
Hostname          : mgmt-thomasrobinson-fv3-00009
Platform          : Linux-3.10.0-1160.59.1.el7.x86_64-x86_64-with-glibc2.17
Version           : 1.0rc6
Python Version    : 3.9.12
Working Directory : /contrib/test
Terminal Size     : 112x26
Frozen            : False
Log ID            : 862ab45a8cf14653ac482ca977a491d90bde64fae9b391f1ad1cb12c9908efb4
################################################################################################################

[Debug e4s_cl.cf.wi4mpi:26] Getting WI4MPI root failed
[Debug e4s_cl.cli.commands.__main__:77] e4s-cl args: Namespace(command='profile', options=['detect', '-p', 'pw', 'srun', '--mpi=pmi2', '-N', '3', '-n', '3', './c_hello.x'], dry_run=None)
[Debug e4s_cl.cli.commands.__main__:195] Arguments: Namespace(command='profile', options=['detect', '-p', 'pw', 'srun', '--mpi=pmi2', '-N', '3', '-n', '3', './c_hello.x'], dry_run=None)
[Debug e4s_cl.cli.commands.__main__:196] Verbosity level: INFO
[Debug e4s_cl.cli.commands.profile:77] e4s-cl profile args: Namespace(subcommand='detect', options=['-p', 'pw', 'srun', '--mpi=pmi2', '-N', '3', '-n', '3', './c_hello.x'])
[Debug e4s_cl.cli.commands.profile.detect:77] e4s-cl profile detect args: Namespace(profile_name='pw', cmd=['srun', '--mpi=pmi2', '-N', '3', '-n', '3', './c_hello.x'])
Received keyboard interrupt. Exiting.
[Debug root:503] 
################################################################################################################
E4S CONTAINER LAUNCHER LOGGING INITIALIZED

Timestamp         : 2022-05-27 13:17:22.943267
Hostname          : mgmt-thomasrobinson-fv3-00009
Platform          : Linux-3.10.0-1160.59.1.el7.x86_64-x86_64-with-glibc2.17
Version           : 1.0rc6
Python Version    : 3.9.12
Working Directory : /contrib/test
Terminal Size     : 112x26
Frozen            : False
Log ID            : faa959952f2b5fbb27253e987303baba50cf930b7a424fb5e1bd79974c39def9
################################################################################################################

[Debug e4s_cl.cf.wi4mpi:26] Getting WI4MPI root failed
[Debug e4s_cl.cli.commands.__main__:77] e4s-cl args: Namespace(command='profile', options=['detect', '-p', 'pw', 'srun', '--mpi=pmi2', '-N', '3', '-n', '164', './c_hello.x'], dry_run=None)
[Debug e4s_cl.cli.commands.__main__:195] Arguments: Namespace(command='profile', options=['detect', '-p', 'pw', 'srun', '--mpi=pmi2', '-N', '3', '-n', '164', './c_hello.x'], dry_run=None)
[Debug e4s_cl.cli.commands.__main__:196] Verbosity level: INFO
[Debug e4s_cl.cli.commands.profile:77] e4s-cl profile args: Namespace(subcommand='detect', options=['-p', 'pw', 'srun', '--mpi=pmi2', '-N', '3', '-n', '164', './c_hello.x'])
[Debug e4s_cl.cli.commands.profile.detect:77] e4s-cl profile detect args: Namespace(profile_name='pw', cmd=['srun', '--mpi=pmi2', '-N', '3', '-n', '164', './c_hello.x'])
Failed to determine necessary libraries: program exited with code 1
[Debug root:503] 
################################################################################################################
E4S CONTAINER LAUNCHER LOGGING INITIALIZED

Timestamp         : 2022-05-27 13:17:39.959255
Hostname          : mgmt-thomasrobinson-fv3-00009
Platform          : Linux-3.10.0-1160.59.1.el7.x86_64-x86_64-with-glibc2.17
Version           : 1.0rc6
Python Version    : 3.9.12
Working Directory : /contrib/test
Terminal Size     : 112x26
Frozen            : False
Log ID            : d6737d8a717d7421c5ca98fb8ab422465f16c489e0b2e351b881ee905c8383b2
################################################################################################################

[Debug e4s_cl.cf.wi4mpi:26] Getting WI4MPI root failed
[Debug e4s_cl.cli.commands.__main__:77] e4s-cl args: Namespace(command='profile', options=['detect', '-p', 'pw', 'srun', '--mpi=pmi2', '-n', '164', './c_hello.x'], dry_run=None)
[Debug e4s_cl.cli.commands.__main__:195] Arguments: Namespace(command='profile', options=['detect', '-p', 'pw', 'srun', '--mpi=pmi2', '-n', '164', './c_hello.x'], dry_run=None)
[Debug e4s_cl.cli.commands.__main__:196] Verbosity level: INFO
[Debug e4s_cl.cli.commands.profile:77] e4s-cl profile args: Namespace(subcommand='detect', options=['-p', 'pw', 'srun', '--mpi=pmi2', '-n', '164', './c_hello.x'])
[Debug e4s_cl.cli.commands.profile.detect:77] e4s-cl profile detect args: Namespace(profile_name='pw', cmd=['srun', '--mpi=pmi2', '-n', '164', './c_hello.x'])
Received keyboard interrupt. Exiting.
[Debug root:503] 
################################################################################################################
E4S CONTAINER LAUNCHER LOGGING INITIALIZED

Timestamp         : 2022-05-27 13:18:26.050153
Hostname          : mgmt-thomasrobinson-fv3-00009
Platform          : Linux-3.10.0-1160.59.1.el7.x86_64-x86_64-with-glibc2.17
Version           : 1.0rc6
Python Version    : 3.9.12
Working Directory : /contrib/test
Terminal Size     : 112x26
Frozen            : False
Log ID            : 4e6b9d8703ee647fa200ec47dda7a9dd807aee6690e8e05a5545d8aed37b6413
################################################################################################################

[Debug e4s_cl.cf.wi4mpi:26] Getting WI4MPI root failed
[Debug e4s_cl.cli.commands.__main__:77] e4s-cl args: Namespace(command='profile', options=['detect', '-p', 'pw', 'srun', '--mpi=pmi2', '-N', '3', '-n', '96', './c_hello.x'], dry_run=None)
[Debug e4s_cl.cli.commands.__main__:195] Arguments: Namespace(command='profile', options=['detect', '-p', 'pw', 'srun', '--mpi=pmi2', '-N', '3', '-n', '96', './c_hello.x'], dry_run=None)
[Debug e4s_cl.cli.commands.__main__:196] Verbosity level: INFO
[Debug e4s_cl.cli.commands.profile:77] e4s-cl profile args: Namespace(subcommand='detect', options=['-p', 'pw', 'srun', '--mpi=pmi2', '-N', '3', '-n', '96', './c_hello.x'])
[Debug e4s_cl.cli.commands.profile.detect:77] e4s-cl profile detect args: Namespace(profile_name='pw', cmd=['srun', '--mpi=pmi2', '-N', '3', '-n', '96', './c_hello.x'])
Failed to determine necessary libraries: program exited with code 244
[Debug root:503] 
################################################################################################################
E4S CONTAINER LAUNCHER LOGGING INITIALIZED

Timestamp         : 2022-05-27 13:23:17.684747
Hostname          : mgmt-thomasrobinson-fv3-00009
Platform          : Linux-3.10.0-1160.59.1.el7.x86_64-x86_64-with-glibc2.17
Version           : 1.0rc6
Python Version    : 3.9.12
Working Directory : /contrib/test
Terminal Size     : 112x26
Frozen            : False
Log ID            : 063a179af243a7467b11d2e2a95c37ecdd23a59cbb46420f8fd007f33addc0d9
################################################################################################################

[Debug e4s_cl.cf.wi4mpi:26] Getting WI4MPI root failed
[Debug e4s_cl.cli.commands.__main__:77] e4s-cl args: Namespace(command='profile', options=['detect', '-p', 'pw', 'srun', '--mpi=pmi2', '-N', '3', '-n', '96', './c_hello.x'], dry_run=None)
[Debug e4s_cl.cli.commands.__main__:195] Arguments: Namespace(command='profile', options=['detect', '-p', 'pw', 'srun', '--mpi=pmi2', '-N', '3', '-n', '96', './c_hello.x'], dry_run=None)
[Debug e4s_cl.cli.commands.__main__:196] Verbosity level: INFO
[Debug e4s_cl.cli.commands.profile:77] e4s-cl profile args: Namespace(subcommand='detect', options=['-p', 'pw', 'srun', '--mpi=pmi2', '-N', '3', '-n', '96', './c_hello.x'])
[Debug e4s_cl.cli.commands.profile.detect:77] e4s-cl profile detect args: Namespace(profile_name='pw', cmd=['srun', '--mpi=pmi2', '-N', '3', '-n', '96', './c_hello.x'])
[Debug e4s_cl.cf.storage.local_file:50] '/home/Thomas.Robinson/.local/e4s_cl/user.json' opened read-write
[Debug e4s_cl.cf.storage.local_file:170] Initialized user database '/home/Thomas.Robinson/.local/e4s_cl/user.json'
[Debug root:503] 
################################################################################################################
E4S CONTAINER LAUNCHER LOGGING INITIALIZED

Timestamp         : 2022-05-27 13:28:35.830914
Hostname          : mgmt-thomasrobinson-fv3-00009
Platform          : Linux-3.10.0-1160.59.1.el7.x86_64-x86_64-with-glibc2.17
Version           : 1.0rc6
Python Version    : 3.9.12
Working Directory : /contrib/test
Terminal Size     : 112x26
Frozen            : False
Log ID            : 50f37f205e994b6ca01871c9c7064573903390a37ec908596a7e2c811fffe009
################################################################################################################

[Debug e4s_cl.cf.wi4mpi:26] Getting WI4MPI root failed
[Debug e4s_cl.cli.commands.__main__:77] e4s-cl args: Namespace(command='profile', options=['detect', '-p', 'pw', 'srun', '--mpi=pmi2', '-N', '3', '-n', '9', './c_hello.x'], dry_run=None)
[Debug e4s_cl.cli.commands.__main__:195] Arguments: Namespace(command='profile', options=['detect', '-p', 'pw', 'srun', '--mpi=pmi2', '-N', '3', '-n', '9', './c_hello.x'], dry_run=None)
[Debug e4s_cl.cli.commands.__main__:196] Verbosity level: INFO
[Debug e4s_cl.cli.commands.profile:77] e4s-cl profile args: Namespace(subcommand='detect', options=['-p', 'pw', 'srun', '--mpi=pmi2', '-N', '3', '-n', '9', './c_hello.x'])
[Debug e4s_cl.cli.commands.profile.detect:77] e4s-cl profile detect args: Namespace(profile_name='pw', cmd=['srun', '--mpi=pmi2', '-N', '3', '-n', '9', './c_hello.x'])
[Debug e4s_cl.cf.storage.local_file:50] '/home/Thomas.Robinson/.local/e4s_cl/user.json' opened read-write
[Debug e4s_cl.cf.storage.local_file:170] Initialized user database '/home/Thomas.Robinson/.local/e4s_cl/user.json'
[Debug root:503] 
################################################################################################################
E4S CONTAINER LAUNCHER LOGGING INITIALIZED

Timestamp         : 2022-05-27 13:30:16.899563
Hostname          : mgmt-thomasrobinson-fv3-00009
Platform          : Linux-3.10.0-1160.59.1.el7.x86_64-x86_64-with-glibc2.17
Version           : 1.0rc6
Python Version    : 3.9.12
Working Directory : /contrib/test
Terminal Size     : 112x26
Frozen            : False
Log ID            : d274bdec7f5d2ed4e9c1150e2ddd1f7930fdcf021231c23c2631ce57a563c353
################################################################################################################

[Debug e4s_cl.cf.wi4mpi:26] Getting WI4MPI root failed
[Debug e4s_cl.cli.commands.__main__:77] e4s-cl args: Namespace(command='profile', options=['detect', '-p', 'pw', 'srun', '--mpi=pmi2', '-N', '3', '-n', '9', './c_hello.x'], dry_run=None)
[Debug e4s_cl.cli.commands.__main__:195] Arguments: Namespace(command='profile', options=['detect', '-p', 'pw', 'srun', '--mpi=pmi2', '-N', '3', '-n', '9', './c_hello.x'], dry_run=None)
[Debug e4s_cl.cli.commands.__main__:196] Verbosity level: INFO
[Debug e4s_cl.cli.commands.profile:77] e4s-cl profile args: Namespace(subcommand='detect', options=['-p', 'pw', 'srun', '--mpi=pmi2', '-N', '3', '-n', '9', './c_hello.x'])
[Debug e4s_cl.cli.commands.profile.detect:77] e4s-cl profile detect args: Namespace(profile_name='pw', cmd=['srun', '--mpi=pmi2', '-N', '3', '-n', '9', './c_hello.x'])
[Debug e4s_cl.cf.storage.local_file:50] '/home/Thomas.Robinson/.local/e4s_cl/user.json' opened read-write
[Debug e4s_cl.cf.storage.local_file:170] Initialized user database '/home/Thomas.Robinson/.local/e4s_cl/user.json'
[Debug root:503] 
################################################################################################################
E4S CONTAINER LAUNCHER LOGGING INITIALIZED

Timestamp         : 2022-05-27 13:30:52.826689
Hostname          : mgmt-thomasrobinson-fv3-00009
Platform          : Linux-3.10.0-1160.59.1.el7.x86_64-x86_64-with-glibc2.17
Version           : 1.0rc6
Python Version    : 3.9.12
Working Directory : /contrib/test
Terminal Size     : 112x26
Frozen            : False
Log ID            : dae948381ffebe02644147d6fbd13a77c094c577909aa8e5038c6e917eed7ccc
################################################################################################################

[Debug e4s_cl.cf.wi4mpi:26] Getting WI4MPI root failed
[Debug e4s_cl.cli.commands.__main__:77] e4s-cl args: Namespace(command='init', options=['--launcher', 'srun', '--backend', 'singularity', '--profile', 'pw'], dry_run=None)
[Debug e4s_cl.cli.commands.__main__:195] Arguments: Namespace(command='init', options=['--launcher', 'srun', '--backend', 'singularity', '--profile', 'pw'], dry_run=None)
[Debug e4s_cl.cli.commands.__main__:196] Verbosity level: INFO
[Debug e4s_cl.cf.storage.local_file:50] '/contrib/system/system.json' opened read-write
[Debug e4s_cl.cf.storage.local_file:170] Initialized system database '/contrib/system/system.json'
[Debug e4s_cl.cf.storage.local_file:50] '/home/Thomas.Robinson/.local/e4s_cl/user.json' opened read-write
[Debug e4s_cl.cf.storage.local_file:170] Initialized user database '/home/Thomas.Robinson/.local/e4s_cl/user.json'
[Debug e4s_cl.cli.commands.init:77] e4s-cl init args: Namespace(launcher='/usr/bin/srun', backend='singularity', profile_name='pw')
[Debug e4s_cl.cli.commands.init:185] MPI vendor not supported by precompiled binary initialisation
Proceeding with legacy initialisation
[Debug e4s_cl.cli.commands.init:121] Compiling with: '/apps/oneapi/mpi/2021.5.1/bin/mpicc -o /tmp/tmpgwhuuqtz -lm /tmp/tmphp09k8es.c'
Simulating MPI execution using:
Compiler: /apps/oneapi/mpi/2021.5.1/bin/mpicc
Launcher /usr/bin/srun
The target launcher /usr/bin/srun uses a single host by default, which may tamper with the library discovery. Consider
running `e4s-cl profile detect` using mpirun specifying multiple hosts.
[Debug e4s_cl.cli.commands.profile.detect:77] e4s-cl profile detect args: Namespace(profile_name=None, cmd=['/usr/bin/srun', '/tmp/tmpgwhuuqtz'])
Failed to determine necessary libraries: program exited with code 156
MPI execution failed.
[Debug e4s_cl.error:168] Traceback (most recent call last):
  File "/contrib/bin/e4s-cl", line 12, in <module>
    sys.exit(cli_main_cmd.main(sys.argv[1:]))
  File "/contrib/conda/lib/python3.9/site-packages/e4s_cl/cli/commands/__main__.py", line 200, in main
    return cli.execute_command([cmd], cmd_args)
  File "/contrib/conda/lib/python3.9/site-packages/e4s_cl/cli/__init__.py", line 314, in execute_command
    return find_command(cmd).main(cmd_args)
  File "/contrib/conda/lib/python3.9/site-packages/e4s_cl/cli/commands/init.py", line 436, in main
    controller.update({'name': requested_name}, profile.eid)
  File "/contrib/conda/lib/python3.9/site-packages/e4s_cl/model/profile.py", line 102, in update
    super().update(data, keys)
  File "/contrib/conda/lib/python3.9/site-packages/e4s_cl/mvc/controller.py", line 236, in update
    database.update(data, keys, table_name=self.model.name)
  File "/contrib/conda/lib/python3.9/site-packages/e4s_cl/cf/storage/local_file.py", line 459, in update
    table.update(fields, doc_ids=[keys])
  File "/contrib/conda/lib/python3.9/site-packages/tinydb/table.py", line 340, in update
    self._update_table(updater)
  File "/contrib/conda/lib/python3.9/site-packages/tinydb/table.py", line 709, in _update_table
    updater(table)
  File "/contrib/conda/lib/python3.9/site-packages/tinydb/table.py", line 337, in updater
    perform_update(table, doc_id)
  File "/contrib/conda/lib/python3.9/site-packages/tinydb/table.py", line 326, in perform_update
    table[doc_id].update(fields)
KeyError: 2

An unexpected KeyError exception was raised:

2

Traceback (most recent call last):
  File "/contrib/bin/e4s-cl", line 12, in <module>
    sys.exit(cli_main_cmd.main(sys.argv[1:]))
  File "/contrib/conda/lib/python3.9/site-packages/e4s_cl/cli/commands/__main__.py", line 200, in main
    return cli.execute_command([cmd], cmd_args)
  File "/contrib/conda/lib/python3.9/site-packages/e4s_cl/cli/__init__.py", line 314, in execute_command
    return find_command(cmd).main(cmd_args)
  File "/contrib/conda/lib/python3.9/site-packages/e4s_cl/cli/commands/init.py", line 436, in main
    controller.update({'name': requested_name}, profile.eid)
  File "/contrib/conda/lib/python3.9/site-packages/e4s_cl/model/profile.py", line 102, in update
    super().update(data, keys)
  File "/contrib/conda/lib/python3.9/site-packages/e4s_cl/mvc/controller.py", line 236, in update
    database.update(data, keys, table_name=self.model.name)
  File "/contrib/conda/lib/python3.9/site-packages/e4s_cl/cf/storage/local_file.py", line 459, in update
    table.update(fields, doc_ids=[keys])
  File "/contrib/conda/lib/python3.9/site-packages/tinydb/table.py", line 340, in update
    self._update_table(updater)
  File "/contrib/conda/lib/python3.9/site-packages/tinydb/table.py", line 709, in _update_table
    updater(table)
  File "/contrib/conda/lib/python3.9/site-packages/tinydb/table.py", line 337, in updater
    perform_update(table, doc_id)
  File "/contrib/conda/lib/python3.9/site-packages/tinydb/table.py", line 326, in perform_update
    table[doc_id].update(fields)
KeyError: 2

This is a bug in E4S Container Launcher.
Please raise an issue on Github with the contents of '/home/Thomas.Robinson/.local/e4s_cl/logs/debug_log'.

Let me know if there's any other information you need
@spoutn1k spoutn1k self-assigned this May 28, 2022
@spoutn1k
Copy link
Collaborator

Hi Thomas !

What the init does is wrap profile detect and makes assumptions about the system that are prone to errors. This KeyError exception should however not happen, and we'll take a look at this.

Did the profile detect command succeed ? What error did you encounter while running the profile ? The error you show is from an init command, that should just create a profile.

@spoutn1k spoutn1k mentioned this issue May 31, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@spoutn1k spoutn1k

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@spoutn1k @thomas-robinson