How do I use EFS with EKS?

[ryehowell]

A common question is how to use EFS on EKS via the Gruntwork Helm k8s-service. Below is a write up of how this can be accomplished. This can also be extended to other kubernetes services running in a cluster and not only services that are deployed via the k8s-service Helm chart.

Summary of Steps

1. Create a new IRSA (IAM Role for Service Account) with a new policy that allows EFS operations.
2. Install the AWS EFS Driver for EKS
3. Create a new EFS Filesystem (or use an existing one)
4. Create a new K8s StorageClass for EFS
5. Configure the application/service to leverage the new EFS StorageClass

Detailed Steps

Most of this information is taken directly from the official docs. Some content has been updated to align with the Gruntwork ecosystem and tailored for the k8s-service Helm chart. For the official reference see the official docs.

1. Create an IAM Policy AmazonEKS_EFS_CSI_Driver_Policy that will be attached to a new Role (created in the next step). The policy should have the following permissions, based on the kubernetes-sigs/aws-efs-csi-driver repo docs:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "elasticfilesystem:DescribeAccessPoints",
        "elasticfilesystem:DescribeFileSystems",
        "elasticfilesystem:DescribeMountTargets",
        "ec2:DescribeAvailabilityZones"
      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "elasticfilesystem:CreateAccessPoint"
      ],
      "Resource": "*",
      "Condition": {
        "StringLike": {
          "aws:RequestTag/efs.csi.aws.com/cluster": "true"
        }
      }
    },
    {
      "Effect": "Allow",
      "Action": [
        "elasticfilesystem:TagResource"
      ],
      "Resource": "*",
      "Condition": {
        "StringLike": {
          "aws:ResourceTag/efs.csi.aws.com/cluster": "true"
        }
      }
    },
    {
      "Effect": "Allow",
      "Action": "elasticfilesystem:DeleteAccessPoint",
      "Resource": "*",
      "Condition": {
        "StringEquals": {
          "aws:ResourceTag/efs.csi.aws.com/cluster": "true"
        }
      }
    }
  ]
}

2. Create a new IAM Role for Service Account (IRSA). This can be implemented multiple ways, but leveraging the Grunwork Helm Service Module will enable you to: create the IRSA and create the EKS EFS CSI Driver deployment with a single module (Steps 1 - 4).
3. Create the Service Account for the EKS EFS CSI Driver and associate the IRSA Role created in Step 2.
4. Install the AWS EKS EFS CSI Driver via the official Helm chart.

### Deploy the EFS CSI Driver and Create/Associate IRSA with the deployment via the helm-service module
module "efs_csi_driver" {
  source = "git::[email protected]:gruntwork-io/terraform-aws-service-catalog.git//modules/services/helm-service?ref=v0.104.1" 
  helm_repository      = "https://kubernetes-sigs.github.io/aws-efs-csi-driver/"
  helm_chart           = "aws-efs-csi-driver"
  helm_chart_version   = "2.4.1"
  application_name     = "aws-efs-csi-driver"
  namespace            = "kube-system"
  service_account_name = "efs-csi-controller-sa"
  iam_role_name = "AmazonEKS_EFS_CSI_DriverRole"
  iam_role_exists = false 

  eks_iam_role_for_service_accounts_config = var.eks_iam_role_for_service_accounts_config

  iam_policy = {
    statement1 = {
      effect    = "Allow"
      resources = ["*"]
      actions = [
        "elasticfilesystem:DescribeAccessPoints",
        "elasticfilesystem:DescribeFileSystems",
        "elasticfilesystem:DescribeMountTargets",
        "ec2:DescribeAvailabilityZones",
      ]
    },
    statement2 = {
      effect    = "Allow"
      resources = ["*"]
      actions   = ["elasticfilesystem:CreateAccessPoint"]
      condition = {
        test     = "StringLike"
        variable = "aws:RequestTag/efs.csi.aws.com/cluster"
        values   = ["true"]
      }
    },
    statement3 = {
      effect    = "Allow"
      resources = ["*"]
      actions   = ["elasticfilesystem:TagResource"]
      condition = {
        test     = "StringLike"
        variable = "aws:ResourceTag/efs.csi.aws.com/cluster"
        values   = ["true"]
      }
    },
    statement4 = {
      effect    = "Allow"
      resources = ["*"]
      actions   = ["elasticfilesystem:DeleteAccessPoint"]
      condition = {
        test     = "StringEquals"
        variable = "aws:ResourceTag/efs.csi.aws.com/cluster"
        values   = ["true"]
      }
    }
  }

  helm_chart_values = {
    image = {
      # Use correct registry for the region deploying to: https://docs.aws.amazon.com/eks/latest/userguide/add-ons-images.html
      repository = "602401143452.dkr.ecr.us-east-2.amazonaws.com/eks/aws-efs-csi-driver"
    }
    controller = {
      serviceAccount = {
        create = true
        name = "efs-csi-controller-sa"
        annotations = {
          "eks.amazonaws.com/role-arn" = format("arn:aws:iam::%s:role/%s", data.aws_caller_identity.current.account_id, "AmazonEKS_EFS_CSI_DriverRole") #local.iam_role_arn
        }
      }
    }
  }

}

5. Create a new EFS File System OR use and existing EFS File System.

### Very simplistic example to create an EFS Filesystem and mount targets. Not recommended for production use.
resource "aws_efs_file_system" "efs_csi" {
  performance_mode = "generalPurpose"
}

resource "aws_efs_mount_target" "efs_csi" {
  for_each = { for k, v in var.efs_subnet_ids : k => v }

  file_system_id  = aws_efs_file_system.efs_csi.id
  subnet_id       = each.value
  security_groups = var.eks_cluster_worker_security_groups
}

6. Create and deploy a new StorageClass for EFS to be used by applications running on EKS.

# An example of how a StorageClass can be created with Terraform
resource "kubernetes_storage_class_v1" "efs" {
  metadata {
    name = "efs-sc"
  }
  storage_provisioner = "efs.csi.aws.com"
  parameters = {
    provisioningMode = "efs-ap" # Dynamic provisioning
    fileSystemId     = aws_efs_file_system.efs_csi.id
    directoryPerms   = "700"
    gidRangeStart = "1000" # optional
    gidRangeEnd = "2000" # optional
    basePath = "/dynamic_provisioning" # optional
  }
}

7. Configure the application to use the new EFS StorageClass. This step can be extended/modified for other cases, but for this case the configuration will be specific to the Gruntwork k8s-service Helm chart:

# A Persistent Volume Claim will also need to be created prior to leveraging it with the k8s-service
resource "kubernetes_persistent_volume_claim" "sample_app_pvc" {
  metadata {
    name = "efs-claim"
  }
  spec {
    access_modes = ["ReadWriteMany"]
    storage_class_name = "efs-sc"
    resources {
      requests = {
        storage = "50Gi"
      }
    }
  }
}

# When using the Gruntwork Service Catalog and leveraging the k8s-service module with Vanilla Terraform, then:
module "application" {
  source = "git::[email protected]:gruntwork-io/terraform-aws-service-catalog.git//modules/services/k8s-service?ref=v0.104.1"

  # only adding relevant var config for the example
  override_chart_inputs = {
    persistentVolumes = {
      efs-storage = {
       mountPath = "/efs-data"
       claimName = "efs-claim"
      }
    }
  }
}

### ie: k8s-sample-app-backend.hcl
# When using the Gruntwork Service Catalog and leveraging the k8s-service module with Terragrunt, then:
inputs = {

# relevant configuration to configure the service to use the new EFS Storage
override_chart_inputs = {
  persistentVolumes = {
    efs-storage = {
      mountPath = "/efs-data"
      claimName = "efs-claim"
    }
  }
}

---

Tracked in ticket #110130