Bucket Policy upload fails due to GetBucketRequestPayment not being implemented (Ceph)

[sj14]

Summary

When I try to upload a bucket policy to Hetzner Object Storage (Ceph), it fails due to GetBucketRequestPayment not being implemented. The module documentation states it should not fail when requester_pays is set to false, but this also doesn't seem to work for me. Am I missing something?

Issue Type

Bug Report

Component Name

s3_bucket

Ansible Version

$ ansible --version
ansible [core 2.18.1]
  config file = None
  configured module search path = ['/Users/simon/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /opt/homebrew/Cellar/ansible/11.1.0_1/libexec/lib/python3.13/site-packages/ansible
  ansible collection location = /Users/simon/.ansible/collections:/usr/share/ansible/collections
  executable location = /opt/homebrew/bin/ansible
  python version = 3.13.1 (main, Dec  3 2024, 17:59:52) [Clang 16.0.0 (clang-1600.0.26.4)] (/opt/homebrew/Cellar/ansible/11.1.0_1/libexec/bin/python)
  jinja version = 3.1.5
  libyaml = True

Collection Versions

$ ansible-galaxy collection list

# /Users/simon/.ansible/collections/ansible_collections
Collection                               Version
---------------------------------------- -------
amazon.aws                               9.1.0

# /opt/homebrew/Cellar/ansible/11.1.0_1/libexec/lib/python3.13/site-packages/ansible_collections
Collection                               Version
---------------------------------------- -------
amazon.aws                               9.0.0
ansible.netcommon                        7.1.0
ansible.posix                            1.6.2
ansible.utils                            5.1.2
ansible.windows                          2.5.0
arista.eos                               10.0.1
awx.awx                                  24.6.1
azure.azcollection                       3.1.0
check_point.mgmt                         6.2.1
chocolatey.chocolatey                    1.5.3
cisco.aci                                2.10.1
cisco.asa                                6.0.0
cisco.dnac                               6.25.0
cisco.intersight                         2.0.20
cisco.ios                                9.0.3
cisco.iosxr                              10.2.2
cisco.ise                                2.9.6
cisco.meraki                             2.18.3
cisco.mso                                2.9.0
cisco.nxos                               9.2.1
cisco.ucs                                1.14.0
cloud.common                             4.0.0
cloudscale_ch.cloud                      2.4.0
community.aws                            9.0.0
community.ciscosmb                       1.0.9
community.crypto                         2.22.3
community.digitalocean                   1.27.0
community.dns                            3.1.0
community.docker                         4.1.0
community.general                        10.1.0
community.grafana                        2.1.0
community.hashi_vault                    6.2.0
community.hrobot                         2.0.2
community.library_inventory_filtering_v1 1.0.2
community.libvirt                        1.3.0
community.mongodb                        1.7.8
community.mysql                          3.11.0
community.network                        5.1.0
community.okd                            4.0.0
community.postgresql                     3.9.0
community.proxysql                       1.6.0
community.rabbitmq                       1.3.0
community.routeros                       3.1.0
community.sap_libs                       1.4.2
community.sops                           2.0.0
community.vmware                         5.2.0
community.windows                        2.3.0
community.zabbix                         3.2.0
containers.podman                        1.16.2
cyberark.conjur                          1.3.1
cyberark.pas                             1.0.30
dellemc.enterprise_sonic                 2.5.1
dellemc.openmanage                       9.9.0
dellemc.powerflex                        2.5.0
dellemc.unity                            2.0.0
f5networks.f5_modules                    1.32.1
fortinet.fortimanager                    2.8.2
fortinet.fortios                         2.3.8
google.cloud                             1.4.1
grafana.grafana                          5.6.0
hetzner.hcloud                           4.2.2
ibm.qradar                               4.0.0
ibm.spectrum_virtualize                  2.0.0
ibm.storage_virtualize                   2.5.0
ieisystem.inmanage                       3.0.0
infinidat.infinibox                      1.4.5
infoblox.nios_modules                    1.7.1
inspur.ispim                             2.2.3
junipernetworks.junos                    9.1.0
kaytus.ksmanage                          2.0.0
kubernetes.core                          5.0.0
kubevirt.core                            2.1.0
lowlydba.sqlserver                       2.3.4
microsoft.ad                             1.7.1
netapp.cloudmanager                      21.24.0
netapp.ontap                             22.13.0
netapp.storagegrid                       21.13.0
netapp_eseries.santricity                1.4.1
netbox.netbox                            3.20.0
ngine_io.cloudstack                      2.5.0
openstack.cloud                          2.3.0
ovirt.ovirt                              3.2.0
purestorage.flasharray                   1.32.0
purestorage.flashblade                   1.19.1
sensu.sensu_go                           1.14.0
splunk.es                                4.0.0
telekom_mms.icinga_director              2.2.1
theforeman.foreman                       4.2.0
vmware.vmware                            1.7.1
vmware.vmware_rest                       4.3.0
vultr.cloud                              1.13.0
vyos.vyos                                5.0.0
wti.remote                               1.0.10

AWS SDK versions

$ pip3 show boto boto3 botocore
WARNING: Package(s) not found: boto, boto3, botocore

Configuration

$ ansible-config dump --only-changed
CONFIG_FILE() = None
PAGER(env: PAGER) = less

GALAXY_SERVERS:

OS / Environment

MacOS 15.2 (arm64)

Steps to Reproduce

- name: s3
  hosts: localhost
  connection: local
  gather_facts: false
  tasks:
    - amazon.aws.s3_bucket:
        aws_access_key: "<REMOVED>"
        aws_secret_key: "<REMOVED>"
        endpoint_url: https://fsn1.your-objectstorage.com
        region: fsn1
        name: <bucket-name>
        state: present
        policy: "<anything>"
        debug_botocore_endpoint_logs: true
        ceph: true
        requester_pays: false

$ ansible-playbook -vvv s3.yaml

Expected Results

The playbook should be applied successfully

Actual Results

ansible-playbook [core 2.18.1]
  config file = None
  configured module search path = ['/Users/simon/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /opt/homebrew/Cellar/ansible/11.1.0_1/libexec/lib/python3.13/site-packages/ansible
  ansible collection location = /Users/simon/.ansible/collections:/usr/share/ansible/collections
  executable location = /opt/homebrew/bin/ansible-playbook
  python version = 3.13.1 (main, Dec  3 2024, 17:59:52) [Clang 16.0.0 (clang-1600.0.26.4)] (/opt/homebrew/Cellar/ansible/11.1.0_1/libexec/bin/python)
  jinja version = 3.1.5
  libyaml = True
No config file found; using defaults
host_list declined parsing /etc/ansible/hosts as it did not pass its verify_file() method
Skipping due to inventory source not existing or not being readable by the current user
script declined parsing /etc/ansible/hosts as it did not pass its verify_file() method
auto declined parsing /etc/ansible/hosts as it did not pass its verify_file() method
Skipping due to inventory source not existing or not being readable by the current user
yaml declined parsing /etc/ansible/hosts as it did not pass its verify_file() method
Skipping due to inventory source not existing or not being readable by the current user
ini declined parsing /etc/ansible/hosts as it did not pass its verify_file() method
Skipping due to inventory source not existing or not being readable by the current user
toml declined parsing /etc/ansible/hosts as it did not pass its verify_file() method
Skipping callback 'default', as we already have a stdout callback.
Skipping callback 'minimal', as we already have a stdout callback.
Skipping callback 'oneline', as we already have a stdout callback.

PLAYBOOK: s3.yaml **************************************************************
1 plays in s3.yaml

PLAY [s3] **********************************************************************

TASK [amazon.aws.s3_bucket] ****************************************************
task path: /Users/simon/Desktop/ansible/s3.yaml:6
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: simon
<127.0.0.1> EXEC /bin/sh -c 'echo ~simon && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /Users/simon/.ansible/tmp `"&& mkdir "` echo /Users/simon/.ansible/tmp/ansible-tmp-1736430222.468454-36778-102655395157911 `" && echo ansible-tmp-1736430222.468454-36778-102655395157911="` echo /Users/simon/.ansible/tmp/ansible-tmp-1736430222.468454-36778-102655395157911 `" ) && sleep 0'
Using module file /Users/simon/.ansible/collections/ansible_collections/amazon/aws/plugins/modules/s3_bucket.py
<127.0.0.1> PUT /Users/simon/.ansible/tmp/ansible-local-36774dk50z9fy/tmpha6ndb0t TO /Users/simon/.ansible/tmp/ansible-tmp-1736430222.468454-36778-102655395157911/AnsiballZ_s3_bucket.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /Users/simon/.ansible/tmp/ansible-tmp-1736430222.468454-36778-102655395157911/ /Users/simon/.ansible/tmp/ansible-tmp-1736430222.468454-36778-102655395157911/AnsiballZ_s3_bucket.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '/opt/homebrew/Cellar/ansible/11.1.0_1/libexec/bin/python /Users/simon/.ansible/tmp/ansible-tmp-1736430222.468454-36778-102655395157911/AnsiballZ_s3_bucket.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'rm -f -r /Users/simon/.ansible/tmp/ansible-tmp-1736430222.468454-36778-102655395157911/ > /dev/null 2>&1 && sleep 0'
The full traceback is:
Traceback (most recent call last):
  File "/var/folders/68/c0sn4zgx5nqdyy8ll80850cm0000gn/T/ansible_amazon.aws.s3_bucket_payload_3ozacv4t/ansible_amazon.aws.s3_bucket_payload.zip/ansible_collections/amazon/aws/plugins/modules/s3_bucket.py", line 643, in handle_bucket_requester_pays
    requester_pays_status = get_bucket_request_payment(s3_client, name)
  File "/var/folders/68/c0sn4zgx5nqdyy8ll80850cm0000gn/T/ansible_amazon.aws.s3_bucket_payload_3ozacv4t/ansible_amazon.aws.s3_bucket_payload.zip/ansible_collections/amazon/aws/plugins/module_utils/cloud.py", line 119, in _retry_wrapper
    return _retry_func(
        func=partial_func,
    ...<5 lines>...
        base_class=cls.base_class,
    )
  File "/var/folders/68/c0sn4zgx5nqdyy8ll80850cm0000gn/T/ansible_amazon.aws.s3_bucket_payload_3ozacv4t/ansible_amazon.aws.s3_bucket_payload.zip/ansible_collections/amazon/aws/plugins/module_utils/cloud.py", line 68, in _retry_func
    return func()
  File "/var/folders/68/c0sn4zgx5nqdyy8ll80850cm0000gn/T/ansible_amazon.aws.s3_bucket_payload_3ozacv4t/ansible_amazon.aws.s3_bucket_payload.zip/ansible_collections/amazon/aws/plugins/modules/s3_bucket.py", line 1565, in get_bucket_request_payment
    return s3_client.get_bucket_request_payment(Bucket=bucket_name).get("Payer")
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
  File "/var/folders/68/c0sn4zgx5nqdyy8ll80850cm0000gn/T/ansible_amazon.aws.s3_bucket_payload_3ozacv4t/ansible_amazon.aws.s3_bucket_payload.zip/ansible_collections/amazon/aws/plugins/module_utils/retries.py", line 107, in deciding_wrapper
    return unwrapped(*args, **kwargs)
  File "/opt/homebrew/Cellar/ansible/11.1.0_1/libexec/lib/python3.13/site-packages/botocore/client.py", line 569, in _api_call
    return self._make_api_call(operation_name, kwargs)
           ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/ansible/11.1.0_1/libexec/lib/python3.13/site-packages/botocore/client.py", line 1023, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (501) when calling the GetBucketRequestPayment operation: Not Implemented
fatal: [localhost]: FAILED! => {
    "boto3_version": "1.35.87",
    "botocore_version": "1.35.87",
    "changed": false,
    "error": {
        "code": "501",
        "message": "Not Implemented"
    },
    "invocation": {
        "module_args": {
            "accelerate_enabled": null,
            "access_key": "<REMOVED>",
            "acl": null,
            "aws_access_key": "<REMOVED>",
            "aws_ca_bundle": null,
            "aws_config": null,
            "aws_secret_key": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
            "bucket_key_enabled": null,
            "ceph": true,
            "debug_botocore_endpoint_logs": true,
            "delete_object_ownership": false,
            "delete_public_access": false,
            "dualstack": false,
            "encryption": null,
            "encryption_key_id": null,
            "endpoint_url": "https://fsn1.your-objectstorage.com",
            "force": false,
            "inventory": null,
            "name": "<REMOVED>",
            "object_lock_default_retention": null,
            "object_lock_enabled": null,
            "object_ownership": null,
            "policy": "<anything>",
            "profile": null,
            "public_access": null,
            "purge_tags": true,
            "region": "fsn1",
            "requester_pays": false,
            "secret_key": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
            "session_token": null,
            "state": "present",
            "tags": null,
            "validate_bucket_name": true,
            "validate_certs": true,
            "versioning": null
        }
    },
    "msg": "Failed to get bucket request payment: An error occurred (501) when calling the GetBucketRequestPayment operation: Not Implemented",
    "resource_actions": [
        "fsn1:GetBucketRequestPayment",
        "fsn1:HeadBucket",
        "fsn1:GetBucketVersioning"
    ],
    "response_metadata": {
        "http_headers": {
            "cache-control": "no-cache",
            "content-length": "136",
            "content-type": "text/html"
        },
        "http_status_code": 501,
        "retry_attempts": 0
    }
}

PLAY RECAP *********************************************************************
localhost                  : ok=0    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0   

Code of Conduct

• I agree to follow the Ansible Code of Conduct