Unable to import module 'lambda_function': cannot import name 'DEFAULT_CIPHERS' from 'urllib3.util.ssl

[mahela-aws]

Describe the bug
After updating to the latest verision, we are getting the below error
Unable to import module 'lambda_function': cannot import name 'DEFAULT_CIPHERS' from 'urllib3.util.ssl_

Layer Version ARN:
arn:aws:lambda:eu-central-1:770693421928:layer:Klayers-p39-requests:13

Framework:
terraform

[BMorinDrifter]

Ran into this as well because the version we were using was deprecated, which broke our terraform.

On that note, can klayers stop deprecating layers?

[ryandeivert]

We also were hit by this. The root cause appears to be a dependency conflict with the default urllib3 version that is bundled in the lambda runtime. botocore is not update to support urllib3 v2.0.0, while requests was updated for support (as of v2.30.0).

This thread covers the issue fairly well.

Separately: I noticed that the p310 layers ONLY include the latest requests version that suffers from this issue. I'd suggest yanking that version and maybe somehow deploying version <2.30.0. You may also want to yank the latest version for <=p39 as well.

[nateprewitt]

To quickly clarify, Requests supports both urllib3 1.x and 2.x. You can continue upgrading Requests to get new features/security patches, urllib3 just needs to be pinned urllib3<2 for your layer.

[ryandeivert]

@nateprewitt yes sorry - I was mostly referring to all of this within the context of the version Klayers is using for the layer it is creating. I'm not sure if the build pipeline has the ability to enforce constraints for layer dependencies (eg: via a constraints file), but maybe @keithrozario could chime in?

[keithrozario]

Thanks @ryandeivert for the added info.

This is a tough one for me. It seems requests and boto3 have a dependency conflict -- i.e. they both depend on different versions of urrlib3 and hence are not compatible with each other (at least for requests > 2.30.0)

At the moment the pipeline is relatively rigid (it can only build the latest version automatically), I can probably modify some stuff to fix this but I'm reluctant to do this-- the main one being that at some point they're looking to make boto3 compatible with the latest urllib3 which should fix this issue. Plus the only reason this project has lasted as long as it has is that I haven't had to manually modify stuff for specific packages -- that's a slippery slope in terms of maintenance overhead that I'm a hesitant to go down.

Hence I'm leaning towards doing nothing for now -- if you're using python3.9 you can use the non-latest version, and I'll ensure we don't expire it. Expired layers last 1 year now, so there's room for flex, but I'm also open to extending this beyond 1 year -- but there has to be some deprecation :).

I'm also considering manually creating a requests layer for python3.10, so at least they have something to use.

Thoughts @ryandeivert @nateprewitt @mahela-aws ?

[mahela-aws]

quick workaround we did, we pinned the layer as below, hopefully helps some of the others facing the same issue here,

data "klayers_package_all_versions" "requests" {
  name   = "requests"
  region = var.aws_region
}

# within lambda

layers = [
    # this pins the layer to arn:aws:lambda:eu-central-1:770693421928:layer:Klayers-p39-requests:11
    data.klayers_package_all_versions.requests.arns[1] 
  ]

[keithrozario]

Thanks @mahela-aws

I will remove requests from the list of built packages for p39, p310 and p310-arm64. This ensures we don't build any new versions till the issue is resolved.

[keithrozario]

Does anyone know if the issue here still persists?

[jrobison-sb]

@keithrozario I stumbled into this myself today, so yes, it does seem to still persist.

I was able to reproduce the problem using runtime python3.10 and using layer arn:aws:lambda:us-east-1:770693421928:layer:Klayers-p310-requests:3.