Pulling host-container from Private ECR Registry

[mantoine96]

Hello,

I am trying to pull and run a custom host container alongside the default host-containers.

My instance has IAM permissions to pull from my private ECR.

My user-data for my Bottlerocket EC2 looks like this:

[settings.kubernetes]
api-server = "https://xxx.yl4.eu-west-1.eks.amazonaws.com"
cluster-certificate = "LxxxxK"
cluster-name = "test-prepared-polliwog"

[settings.host-containers.teleport]
enabled = true
source = "xxxxxx.dkr.ecr.eu-west-1.amazonaws.com/tools/myimage:v1@sha256:xxxxxx"
superpowered = false
user-data = "IyEvYmluL2Jhc2gKCi9iaW4vY29uZmlndXJlX3RlbGVwb3J0LnNoIHRlbGVwb3J0LmV1LWRldi5hd3NtbWNuLnByaXZhdGUgXCd7XCJOYW1lXCI6IFwiVGVzdFwifVwnCg=="

This wasn't working since authentication to the ECR registry failed.

I looked at the documentation and found this block to configure the container-registry:

[[settings.container-registry.credentials]]
registry = "xxxxxx.dkr.ecr.eu-west-1.amazonaws.com"
user = "AWS"
password = "xxxx"

But what bothers me here is I need to set a static password. Therefore I think I'm missing something. Indeed there must be a way to leverage the IAM credentials of the instance to dynamically generate the ECR password?

The only part I could find mentioning ECR was configuration for the Kubernetes ECR plugin, which I don't think is relevant here.

Am I missing something?

Thanks!

[KCSesh]

Hey @mantoine96,

So long as the EC2 instance has an associated IAM role which has the policies to access your ECR image, things should work.

In fact, I just tested this myself using 2 privately hosted containers in the same account as my Bottlerocket instance, making using of eksctl.

This was my yaml file I used to create my cluster and nodes:

---
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: x86-64-aws-k8s-129
  region: us-west-2
  version: '1.29'

nodeGroups:
  - name: ng-x86-64-aws-k8s-129
    instanceType: m5.large
    desiredCapacity: 4
    amiFamily: Bottlerocket
    disableIMDSv1: true
    iam:
       attachPolicyARNs:
          - arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy
          - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
          - arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly
          - arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
    ssh:
        allow: true
        publicKeyName: "<my-own-key>"
    bottlerocket:
      enableAdminContainer: true
      settings:
        host-containers:
            kcsesh-host:
                enabled: true
                source: "<aws_account_id.dkr.ecr.region.amazonaws.com/my-repository:latest>"
                superpowered: true
        bootstrap-containers:
            kcsesh-bootstrap:
                essential: false
                mode: "always"
                source: "<aws_account_id.dkr.ecr.region.amazonaws.com/my-repository:latest>"

The important policy added is AmazonEC2ContainerRegistryReadOnly, which contains:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ecr:GetAuthorizationToken",
                "ecr:BatchCheckLayerAvailability",
                "ecr:GetDownloadUrlForLayer",
                "ecr:GetRepositoryPolicy",
                "ecr:DescribeRepositories",
                "ecr:ListImages",
                "ecr:DescribeImages",
                "ecr:BatchGetImage",
                "ecr:GetLifecyclePolicy",
                "ecr:GetLifecyclePolicyPreview",
                "ecr:ListTagsForResource",
                "ecr:DescribeImageScanFindings"
            ],
            "Resource": "*"
        }
    ]
}

This resulted in the successful execution of both my bootstrap container and host container which are hosted in my own private ECR from the same account as my Bottlerocket instance.

Could you confirm that your EC2 instance IAM role - does in fact - have access to your ECR image?

Are you doing this cross-account, or is everything in the same account?

[mantoine96]

Hello @KCSesh

Thanks for your reply. My node has the correct permissions, and we pull cross account.

After some more tinkering, it does indeed work out of the box. I assumed that the reason my host container wasn't coming up was because it couldn't pull (as I wasn't able to pull it running ctr --address /run/host-containerd/containerd.sock image pull $myimage). Upon further investigation I found some better ways of troubleshooting (using journalctl -fu host-containers@$mycontainer) and found my issues.

Thanks again so much for this!