Cannot create Composer environments on Mac: Error while fetching server API version: Not supported URL scheme http+docker (docker==6.1.3)

[datasci-iopsy]

Hello there,

When attempting to run a local Airflow environment with Composer Local Development CLI tool, I continue to receive the same error when trying to create a local Airflow environment from a Cloud Composer environment.

My development environment is running Python 3.11.9 (a prerequisite) and after installing composer-local-dev via pip, my list can be found in the Appendices. If I run the following Python script using docker==6.1.3:

import docker

try:
    client = docker.from_env()
    print(client.info())
except Exception as e:
    print(f"Error: {e}")

I receive this error:

Error: Error while fetching server API version: Not supported URL scheme http+docker

This is mitigated completely when I use an upgrade version of the docker package within python (e.g., 7.1.0) and the desired output is received; however, the newest version of docker==7.1.0 is incompatible with composer-local-dev as it requires docker=6.*.

I ran the Python script because when creating a Composer instance I encounter the same error. All recommendations to verify that Docker GUI is functioning, sockets, configs, etc. have been checked but to no avail. Here is a breakdown of how I'm creating the Composer instance:

#!/bin/bash

echo -e "Creating a new Composer environment."

export LOCAL_ENV_NAME=local-cmpsr-dev-gren
export SRC_ENV_NAME=cmpsr-dev-green
export SRC_LOCATION=us-east4
export PROJECT_ID=place_holder_project
export WEB_SERVER_PORT=8081
export LOCAL_DAGS_PATH=./dags

composer-dev create ${LOCAL_ENV_NAME} --debug \
    --from-source-environment ${SRC_ENV_NAME} \
    --location ${SRC_LOCATION} \
    --project ${PROJECT_ID} \
    --port ${WEB_SERVER_PORT} \
    --dags-path ${DAG_PATH}}

Any assistance would be greatly appreciated!

Appendices

cachetools==5.5.0
certifi==2024.12.14
charset-normalizer==3.4.1
click==8.1.8
composer-dev @ file:///Users/greend/bitbucket-aeo/people-analytics/local-dags/composer-local-dev
docker==6.1.3
google-api-core==2.24.0
google-auth==2.27.0
google-cloud-artifact-registry==1.14.0
google-cloud-orchestration-airflow==1.16.1
googleapis-common-protos==1.66.0
grpc-google-iam-v1==0.14.0
grpcio==1.69.0
grpcio-status==1.69.0
idna==3.10
markdown-it-py==3.0.0
mdurl==0.1.2
packaging==24.2
proto-plus==1.25.0
protobuf==5.29.3
pyasn1==0.6.1
pyasn1_modules==0.4.1
Pygments==2.19.1
requests==2.32.3
rich==13.9.4
rich-click==1.4
rsa==4.9
urllib3==2.3.0
websocket-client==1.8.0

My machine set up:

Software:

    System Software Overview:

      System Version: macOS 14.7.2 (23H311)
      Kernel Version: Darwin 23.6.0
      Secure Virtual Memory: Enabled
      System Integrity Protection: Enabled
      Time since boot: 2 hours, 49 minutes

Hardware:

    Hardware Overview:

      Model Name: MacBook Pro
      Model Identifier: MacBookPro18,1
      Model Number: MK193LL/A
      Chip: Apple M1 Pro
      Total Number of Cores: 10 (8 performance and 2 efficiency)
      Memory: 16 GB
      System Firmware Version: 11881.61.3
      OS Loader Version: 10151.140.19.700.2

[datasci-iopsy]

Here is the error I receive when running the Composer instance creation script:

DEBUG    docker.utils.config:Trying paths: ['/Users/greend/.docker/config.json', '/Users/greend/.dockercfg']                                                         config.py:21
                    DEBUG    docker.utils.config:Found file at path: /Users/greend/.docker/config.json                                                                                   config.py:25
                    DEBUG    docker.auth:Found 'credsStore' section                                                                                                                       auth.py:187
                    DEBUG    root:Docker not found.                                                                                                                                environment.py:470
                             Traceback (most recent call last):                                                                                                                                      
                               File "/Users/greend/.pyenv/versions/venv-py311-local-dags/lib/python3.11/site-packages/requests/adapters.py", line 633, in send                                       
                                 conn = self.get_connection_with_tls_context(                                                                                                                        
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                        
                               File "/Users/greend/.pyenv/versions/venv-py311-local-dags/lib/python3.11/site-packages/requests/adapters.py", line 489, in                                            
                             get_connection_with_tls_context                                                                                                                                         
                                 conn = self.poolmanager.connection_from_host(                                                                                                                       
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                       
                               File "/Users/greend/.pyenv/versions/venv-py311-local-dags/lib/python3.11/site-packages/urllib3/poolmanager.py", line 303, in connection_from_host                     
                                 return self.connection_from_context(request_context)                                                                                                                
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                
                               File "/Users/greend/.pyenv/versions/venv-py311-local-dags/lib/python3.11/site-packages/urllib3/poolmanager.py", line 325, in                                          
                             connection_from_context                                                                                                                                                 
                                 raise URLSchemeUnknown(scheme)                                                                                                                                      
                             urllib3.exceptions.URLSchemeUnknown: Not supported URL scheme http+docker                                                                                               
                                                                                                                                                                                                     
                             During handling of the above exception, another exception occurred:                                                                                                     
                                                                                                                                                                                                     
                             Traceback (most recent call last):                                                                                                                                      
                               File "/Users/greend/.pyenv/versions/venv-py311-local-dags/lib/python3.11/site-packages/docker/api/client.py", line 214, in _retrieve_server_version                   
                                 return self.version(api_version=False)["ApiVersion"]                                                                                                                
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                              
                               File "/Users/greend/.pyenv/versions/venv-py311-local-dags/lib/python3.11/site-packages/docker/api/daemon.py", line 181, in version                                    
                                 return self._result(self._get(url), json=True)                                                                                                                      
                                                     ^^^^^^^^^^^^^^                                                                                                                                  
                               File "/Users/greend/.pyenv/versions/venv-py311-local-dags/lib/python3.11/site-packages/docker/utils/decorators.py", line 46, in inner                                 
                                 return f(self, *args, **kwargs)                                                                                                                                     
                                        ^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                     
                               File "/Users/greend/.pyenv/versions/venv-py311-local-dags/lib/python3.11/site-packages/docker/api/client.py", line 237, in _get                                       
                                 return self.get(url, **self._set_request_timeout(kwargs))                                                                                                           
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                           
                               File "/Users/greend/.pyenv/versions/venv-py311-local-dags/lib/python3.11/site-packages/requests/sessions.py", line 602, in get                                        
                                 return self.request("GET", url, **kwargs)                                                                                                                           
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                           
                               File "/Users/greend/.pyenv/versions/venv-py311-local-dags/lib/python3.11/site-packages/requests/sessions.py", line 589, in request                                    
                                 resp = self.send(prep, **send_kwargs)                                                                                                                               
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                               
                               File "/Users/greend/.pyenv/versions/venv-py311-local-dags/lib/python3.11/site-packages/requests/sessions.py", line 703, in send                                       
                                 r = adapter.send(request, **kwargs)                                                                                                                                 
                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                 
                               File "/Users/greend/.pyenv/versions/venv-py311-local-dags/lib/python3.11/site-packages/requests/adapters.py", line 637, in send                                       
                                 raise InvalidURL(e, request=request)                                                                                                                                
                             requests.exceptions.InvalidURL: Not supported URL scheme http+docker                                                                                                    
                                                                                                                                                                                                     
                             During handling of the above exception, another exception occurred:                                                                                                     
                                                                                                                                                                                                     
                             Traceback (most recent call last):                                                                                                                                      
                               File "/Users/greend/.pyenv/versions/venv-py311-local-dags/lib/python3.11/site-packages/composer_local_dev/environment.py", line 468, in get_client                    
                                 return docker.from_env()                                                                                                                                            
                                        ^^^^^^^^^^^^^^^^^                                                                                                                                            
                               File "/Users/greend/.pyenv/versions/venv-py311-local-dags/lib/python3.11/site-packages/docker/client.py", line 96, in from_env                                        
                                 return cls(                                                                                                                                                         
                                        ^^^^                                                                                                                                                         
                               File "/Users/greend/.pyenv/versions/venv-py311-local-dags/lib/python3.11/site-packages/docker/client.py", line 45, in __init__                                        
                                 self.api = APIClient(*args, **kwargs)                                                                                                                               
                                            ^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                               
                               File "/Users/greend/.pyenv/versions/venv-py311-local-dags/lib/python3.11/site-packages/docker/api/client.py", line 197, in __init__                                   
                                 self._version = self._retrieve_server_version()                                                                                                                     
                                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                     
                               File "/Users/greend/.pyenv/versions/venv-py311-local-dags/lib/python3.11/site-packages/docker/api/client.py", line 221, in _retrieve_server_version                   
                                 raise DockerException(                                                                                                                                              
                             docker.errors.DockerException: Error while fetching server API version: Not supported URL scheme http+docker                                                            
╭─ Error ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ Docker not available or failed to start. Please ensure docker service is installed and running. Error: Error while fetching    │
│ server API version: Not supported URL scheme http+docker                                                                       │
│                                                                                                                                │
│ To print debug messages please add --debug flag.                                                                               │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯