Docker mode fails with Paramiko error when the host ssh the remote machine for the first time after login

[takuma-yoneda]

This doesn't happen if I manually run ssh [email protected] first.

$ lmn run elm -- python -m my_transformer.train_deconly_mul my_transformer/sweep/sweep.jsonl -l 0 --debug 
INFO:lmn: Project name     : my_transformer
INFO:lmn: Project directory: /Users/yoneda/workspace/ripl/my-transformer
[email protected]'s password: 
INFO:lmn: Syncing files (/Users/yoneda/workspace/ripl/my-transformer/ to [email protected]:/scratch/takuma/rmx/takuma/my_transformer/code/)
INFO:lmn: Sync finished!
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.8/bin/lmn", line 8, in <module>
    sys.exit(main())
  File "/Users/yoneda/workspace/takuma-yoneda/lmn/lmn/cli/__init__.py", line 85, in main
    core(args)
  File "/Users/yoneda/workspace/takuma-yoneda/lmn/lmn/cli/__init__.py", line 77, in core
    parsed.handler(project, remote_conf, parsed, preset_conf)
  File "/Users/yoneda/workspace/takuma-yoneda/lmn/lmn/cli/run.py", line 219, in handler
    client = DockerClient(base_url=base_url)  # dockerpty hangs with use_ssh_client=True
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/docker/client.py", line 45, in __init__
    self.api = APIClient(*args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/docker/api/client.py", line 171, in __init__
    self._custom_adapter = SSHHTTPAdapter(
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/docker/transport/sshconn.py", line 176, in __init__
    self._connect()
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/docker/transport/sshconn.py", line 222, in _connect
    self.ssh_client.connect(**self.ssh_params)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/paramiko/client.py", line 450, in connect
    self._auth(
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/paramiko/client.py", line 781, in _auth
    raise saved_exception
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/paramiko/client.py", line 757, in _auth
    self._transport.auth_publickey(username, key)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/paramiko/transport.py", line 1635, in auth_publickey
    return self.auth_handler.wait_for_response(my_event)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/paramiko/auth_handler.py", line 259, in wait_for_response
    raise e
paramiko.ssh_exception.AuthenticationException: Authentication failed.