Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Python 3.12 (x86 and arm64) #378

Merged
merged 20 commits into from
Jan 3, 2024
Merged
Changes from 7 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 6 additions & 3 deletions .github/workflows/container_builds.yml
Original file line number Diff line number Diff line change
@@ -17,9 +17,6 @@ jobs:
strategy:
matrix:
include:
- python_version: p39
arch: x86
build_platform: linux/amd64
- python_version: p310
arch: x86
build_platform: linux/amd64
@@ -32,6 +29,12 @@ jobs:
- python_version: p311
arch: arm64
build_platform: linux/arm64
- python_version: p312
arch: x86
build_platform: linux/amd64
- python_version: p312
arch: arm64
build_platform: linux/arm64


permissions:
47 changes: 47 additions & 0 deletions pipeline/Serverless/02_pipeline/pipeline_build_containers.yml
Original file line number Diff line number Diff line change
@@ -67,6 +67,29 @@ build311:
- dynamodb:GetItem
Resource: ${self:custom.dbArn}

build312:
description: Builds the package in a container for python3.12 then uploads to S3
image: ${ssm:/${self:service}/${self:provider.stage}/build/p312/x86/repo}@${ssm:/${self:service}/${self:provider.stage}/build/p312/x86/digest}
timeout: 600
memorySize: 1769 # exactly one vcpu
iamRoleStatementsName: ${self:provider.stage}-buildp312
iamRoleStatements:
- Effect: Allow
Action:
- s3:PutObject
- s3:PutObjectAcl
Resource: ${self:custom.s3LayersArn}/*
- Effect: Allow
Action:
- s3:ListBucket
Resource: ${self:custom.s3LayersArn}
- Effect: Allow
Action:
- dynamodb:PutItem
- dynamodb:UpdateItem
- dynamodb:GetItem
Resource: ${self:custom.dbArn}


build310Arm64:
description: Builds the package in a container for python3.10 then uploads to S3
@@ -99,6 +122,30 @@ build311Arm64:
memorySize: 1769 # exactly one vcpu
iamRoleStatementsName: ${self:provider.stage}-buildp311arm64
iamRoleStatements:
- Effect: Allow
Action:
- s3:PutObject
- s3:PutObjectAcl
Resource: ${self:custom.s3LayersArn}/*
- Effect: Allow
Action:
- s3:ListBucket
Resource: ${self:custom.s3LayersArn}
- Effect: Allow
Action:
- dynamodb:PutItem
- dynamodb:UpdateItem
- dynamodb:GetItem
Resource: ${self:custom.dbArn}
architecture: arm64

build312Arm64:
description: Builds the package in a container for python3.12 then uploads to S3
image: ${ssm:/${self:service}/${self:provider.stage}/build/p312/arm64/repo}@${ssm:/${self:service}/${self:provider.stage}/build/p312/arm64/digest}
timeout: 600
memorySize: 1769 # exactly one vcpu
iamRoleStatementsName: ${self:provider.stage}-buildp312arm64
iamRoleStatements:
- Effect: Allow
Action:
- s3:PutObject
38 changes: 19 additions & 19 deletions pipeline/Serverless/state_machines/02_pipeline.yml
Original file line number Diff line number Diff line change
@@ -29,32 +29,33 @@ definition:
ChoicePythonVersion:
Type: Choice
Choices:
- Variable: $.python_version
StringEquals: "p3.8"
Next: Build38
- Variable: $.python_version
StringEquals: "p3.9"
Next: Build39
- Variable: $.python_version
StringEquals: "p3.10"
Next: Build310
- Variable: $.python_version
StringEquals: "p3.11"
Next: Build311
- Variable: $.python_version
StringEquals: "p3.12"
Next: Build312
- Variable: $.python_version
StringEquals: "p3.10-arm64"
Next: Build310arm64
- Variable: $.python_version
StringEquals: "p3.11-arm64"
Next: Build311arm64
- Variable: $.python_version
StringEquals: "p3.12-arm64"
Next: Build312arm64
Default: UnknownPythonVersion
Build38:
Build310:
Type: Task
Resource:
Fn::GetAtt: [BuildLambdaFunction, Arn]
Fn::GetAtt: [Build310LambdaFunction, Arn]
Next: Deploy
Retry:
- ErrorEquals:
- Lambda.AWSLambdaException
- States.Timeout
- Lambda.ClientExecutionTimeoutException
- Lambda.ServiceException
@@ -67,10 +68,10 @@ definition:
- ErrorEquals:
- States.ALL
Next: BuildFail
Build39:
Build311:
Type: Task
Resource:
Fn::GetAtt: [Build39LambdaFunction, Arn]
Fn::GetAtt: [Build311LambdaFunction, Arn]
Next: Deploy
Retry:
- ErrorEquals:
@@ -87,10 +88,10 @@ definition:
- ErrorEquals:
- States.ALL
Next: BuildFail
Build310:
Build312:
Type: Task
Resource:
Fn::GetAtt: [Build310LambdaFunction, Arn]
Fn::GetAtt: [Build312LambdaFunction, Arn]
Next: Deploy
Retry:
- ErrorEquals:
@@ -107,14 +108,13 @@ definition:
- ErrorEquals:
- States.ALL
Next: BuildFail
Build311:
Build310arm64:
Type: Task
Resource:
Fn::GetAtt: [Build311LambdaFunction, Arn]
Fn::GetAtt: [Build310Arm64LambdaFunction, Arn]
Next: Deploy
Retry:
- ErrorEquals:
- Lambda.AWSLambdaException
- States.Timeout
- Lambda.ClientExecutionTimeoutException
- Lambda.ServiceException
@@ -127,10 +127,10 @@ definition:
- ErrorEquals:
- States.ALL
Next: BuildFail
Build310arm64:
Build311arm64:
Type: Task
Resource:
Fn::GetAtt: [Build310Arm64LambdaFunction, Arn]
Fn::GetAtt: [Build311Arm64LambdaFunction, Arn]
Next: Deploy
Retry:
- ErrorEquals:
@@ -146,10 +146,10 @@ definition:
- ErrorEquals:
- States.ALL
Next: BuildFail
Build311arm64:
Build312arm64:
Type: Task
Resource:
Fn::GetAtt: [Build311Arm64LambdaFunction, Arn]
Fn::GetAtt: [Build312Arm64LambdaFunction, Arn]
Next: Deploy
Retry:
- ErrorEquals:
36 changes: 17 additions & 19 deletions pipeline/Terraform/container_repositories.tf
Original file line number Diff line number Diff line change
@@ -1,24 +1,5 @@
# Container Build Images

## Python 3.9 - x86 (only)
resource "aws_ecr_repository" "p39build_x86" {
name = "p39build"
image_tag_mutability = "MUTABLE"
force_delete = true

image_scanning_configuration {
scan_on_push = true
}
}

resource "aws_ssm_parameter" "p39_build_repo" {
type = "String"
description = "URL for p39 x86 repo"
name = "/${var.app_name}/${local.workspace_full_name}/build/p39/x86/repo"
value = aws_ecr_repository.p39build_x86.repository_url
overwrite = true
}

## Python 3.10 - x86
resource "aws_ecr_repository" "p310build_x86" {
name = "p310build"
@@ -95,3 +76,20 @@ resource "aws_ssm_parameter" "p311_arm64_build_repo" {
overwrite = true
}

## Python 3.12 builds

module "python312_x86_build" {
source = "./container_repository"
app_name = var.app_name
workspace_full_name = local.workspace_full_name
python_version = "p312"
architecture = "x86"
}

module "python312_arm64_build" {
source = "./container_repository"
app_name = var.app_name
workspace_full_name = local.workspace_full_name
python_version = "p312"
architecture = "arm64"
}
23 changes: 23 additions & 0 deletions pipeline/Terraform/container_repository/main.tf
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# Container Build Images
variable app_name{}
variable workspace_full_name{}
variable python_version{}
variable architecture{}

resource "aws_ecr_repository" "build" {
name = "${var.python_version}${var.architecture}build"
image_tag_mutability = "MUTABLE"
force_delete = true

image_scanning_configuration {
scan_on_push = true
}
}

resource "aws_ssm_parameter" "build" {
type = "String"
description = "URL for ${var.python_version} ${var.architecture} repo"
name = "/${var.app_name}/${var.workspace_full_name}/build/${var.python_version}/${var.architecture}/repo"
value = aws_ecr_repository.build.repository_url
overwrite = true
}
14 changes: 7 additions & 7 deletions pipeline/config/config.json
Original file line number Diff line number Diff line change
@@ -1,21 +1,21 @@
{
"python_versions": ["p3.8", "p3.9", "p3.10", "p3.10-arm64", "p3.11", "p3.11-arm64"],
"p3.8": {
"packages_file": "packages_p38.csv"
},
"p3.9": {
"packages_file": "packages_p39.csv"
},
"python_versions": ["p3.10", "p3.10-arm64", "p3.11", "p3.11-arm64", "p3.12", "p3.12-arm64"],
"p3.10": {
"packages_file": "packages_p310.csv"
},
"p3.11":{
"packages_file": "packages_p311.csv"
},
"p3.12":{
"packages_file": "packages_p312.csv"
},
"p3.10-arm64": {
"packages_file": "packages_p310-arm64.csv"
},
"p3.11-arm64":{
"packages_file": "packages_p311-arm64.csv"
},
"p3.12-arm64":{
"packages_file": "packages_p312-arm64.csv"
}
}
18 changes: 18 additions & 0 deletions pipeline/config/packages_p312-arm64.csv
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
Package_Name,License,Authors/Maintainers
aws-xray-sdk,Apache-2.0,AWS
aws-requests-auth,BSD,[email protected]
bcrypt,Apache-2.0,[email protected]
beautifulsoup4,MIT,Leonard Richardson <[email protected]>
boto3,Apache-2.0,AWS
cryptography,Apache-2.0,The Python Cryptographic Authority and individual contributors <[email protected]>
dynamodb-encryption-sdk,Apache-2.0,<[email protected]>
idna,https://github.com/kjd/idna/blob/master/LICENSE.rst,Kim Davis [email protected]
jinja2,BSD,Armin Ronache <[email protected]>; Pallets <[email protected]>
numpy,https://www.numpy.org/license.html,[email protected]
pandas,BSD,<[email protected]>
pyqldb,Apache-2.0,AWS
redshift-connector,Apache License Version 2.0,Amazon Web Services <[email protected]>
requests,Apache-2.0,Kenneth Reitz <[email protected]>
mysql-connector-python,GNU GPLv2,Oracle
aliyun-python-sdk-actiontrail,Apache-2.0,Aliyun <[email protected]>
tabulate,MIT,[email protected]
18 changes: 18 additions & 0 deletions pipeline/config/packages_p312.csv
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
Package_Name,License,Authors/Maintainers
aws-xray-sdk,Apache-2.0,AWS
aws-requests-auth,BSD,[email protected]
bcrypt,Apache-2.0,[email protected]
beautifulsoup4,MIT,Leonard Richardson <[email protected]>
boto3,Apache-2.0,AWS
cryptography,Apache-2.0,The Python Cryptographic Authority and individual contributors <[email protected]>
dynamodb-encryption-sdk,Apache-2.0,<[email protected]>
idna,https://github.com/kjd/idna/blob/master/LICENSE.rst,Kim Davis [email protected]
jinja2,BSD,Armin Ronache <[email protected]>; Pallets <[email protected]>
numpy,https://www.numpy.org/license.html,[email protected]
pandas,BSD,<[email protected]>
pyqldb,Apache-2.0,AWS
redshift-connector,Apache License Version 2.0,Amazon Web Services <[email protected]>
requests,Apache-2.0,Kenneth Reitz <[email protected]>
mysql-connector-python,GNU GPLv2,Oracle
aliyun-python-sdk-actiontrail,Apache-2.0,Aliyun <[email protected]>
tabulate,MIT,[email protected]
104 changes: 0 additions & 104 deletions pipeline/config/packages_p38.csv

This file was deleted.

51 changes: 0 additions & 51 deletions pipeline/config/packages_p39.csv

This file was deleted.

14 changes: 7 additions & 7 deletions pipeline/config/test_config/config.json
Original file line number Diff line number Diff line change
@@ -1,21 +1,21 @@
{
"python_versions": ["p3.8","p3.9", "p3.10", "p3.10-arm64", "p3.11", "p3.11-arm64"],
"p3.8": {
"packages_file": "packages_p38.csv"
},
"p3.9": {
"packages_file": "packages_p39.csv"
},
"python_versions": ["p3.10", "p3.10-arm64", "p3.11", "p3.11-arm64", "p3.12", "p3.12-arm64"],
"p3.10": {
"packages_file": "packages_p310.csv"
},
"p3.11":{
"packages_file": "packages_p311.csv"
},
"p3.12":{
"packages_file": "packages_p312.csv"
},
"p3.10-arm64": {
"packages_file": "packages_p310-arm64.csv"
},
"p3.11-arm64":{
"packages_file": "packages_p311-arm64.csv"
},
"p3.12-arm64":{
"packages_file": "packages_p312-arm64.csv"
}
}
3 changes: 3 additions & 0 deletions pipeline/config/test_config/packages_p312-arm64.csv
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
Package_Name,License,Authors/Maintainers
boto3,Apache-2.0,AWS
cryptography,Apache-2.0,The Python Cryptographic Authority and individual contributors <cryptography-dev@python.org>
3 changes: 3 additions & 0 deletions pipeline/config/test_config/packages_p312.csv
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
Package_Name,License,Authors/Maintainers
boto3,Apache-2.0,AWS
cryptography,Apache-2.0,The Python Cryptographic Authority and individual contributors <cryptography-dev@python.org>
3 changes: 0 additions & 3 deletions pipeline/config/test_config/packages_p39.csv

This file was deleted.

13 changes: 13 additions & 0 deletions pipeline/container_images/build_images/p312_arm64/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
FROM public.ecr.aws/lambda/python:3.12-arm64

COPY build.py ./
RUN pip install --upgrade pip
RUN pip install boto3
RUN pip install aws-lambda-powertools

# Use dnf instead of yum
# https://aws.amazon.com/blogs/compute/python-3-12-runtime-now-available-in-aws-lambda/
RUN dnf update
RUN dnf install -y python-devel

CMD ["build.main"]
13 changes: 13 additions & 0 deletions pipeline/container_images/build_images/p312_x86/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
FROM public.ecr.aws/lambda/python:3.12

COPY build.py ./
RUN pip install --upgrade pip
RUN pip install boto3
RUN pip install aws-lambda-powertools

# Use dnf instead of yum
# https://aws.amazon.com/blogs/compute/python-3-12-runtime-now-available-in-aws-lambda/
RUN dnf update
RUN dnf install -y python-devel

CMD ["build.main"]