Karpenter – Documentation

Docs: NodeClasses

Mon, 01 Jan 0001 00:00:00 +0000

Node Classes enable configuration of AWS specific settings. Each NodePool must reference an EC2NodeClass using spec.template.spec.nodeClassRef. Multiple NodePools may point to the same EC2NodeClass.

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
 name: default
spec:
 template:
 spec:
 nodeClassRef:
 group: karpenter.k8s.aws
 kind: EC2NodeClass
 name: default
---
apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
 name: default
spec:
 kubelet:
 podsPerCore: 2
 maxPods: 20
 systemReserved:
 cpu: 100m
 memory: 100Mi
 ephemeral-storage: 1Gi
 kubeReserved:
 cpu: 200m
 memory: 100Mi
 ephemeral-storage: 3Gi
 evictionHard:
 memory.available: 5%
 nodefs.available: 10%
 nodefs.inodesFree: 10%
 evictionSoft:
 memory.available: 500Mi
 nodefs.available: 15%
 nodefs.inodesFree: 15%
 evictionSoftGracePeriod:
 memory.available: 1m
 nodefs.available: 1m30s
 nodefs.inodesFree: 2m
 evictionMaxPodGracePeriod: 60
 imageGCHighThresholdPercent: 85
 imageGCLowThresholdPercent: 80
 cpuCFSQuota: true
 clusterDNS: ["10.0.1.100"]
 # Optional, dictates UserData generation and default block device mappings.
 # May be ommited when using an `alias` amiSelectorTerm, otherwise required.
 amiFamily: AL2

 # Required, discovers subnets to attach to instances
 # Each term in the array of subnetSelectorTerms is ORed together
 # Within a single term, all conditions are ANDed
 subnetSelectorTerms:
 # Select on any subnet that has the "karpenter.sh/discovery: ${CLUSTER_NAME}"
 # AND the "environment: test" tag OR any subnet with ID "subnet-09fa4a0a8f233a921"
 - tags:
 karpenter.sh/discovery: "${CLUSTER_NAME}"
 environment: test
 - id: subnet-09fa4a0a8f233a921

 # Required, discovers security groups to attach to instances
 # Each term in the array of securityGroupSelectorTerms is ORed together
 # Within a single term, all conditions are ANDed
 securityGroupSelectorTerms:
 # Select on any security group that has both the "karpenter.sh/discovery: ${CLUSTER_NAME}" tag
 # AND the "environment: test" tag OR any security group with the "my-security-group" name
 # OR any security group with ID "sg-063d7acfb4b06c82c"
 - tags:
 karpenter.sh/discovery: "${CLUSTER_NAME}"
 environment: test
 - name: my-security-group
 - id: sg-063d7acfb4b06c82c

 # Optional, IAM role to use for the node identity.
 # Must specify one of "role" or "instanceProfile" for Karpenter to launch nodes
 role: "KarpenterNodeRole-${CLUSTER_NAME}"

 # Optional, IAM instance profile to use for the node identity.
 # Must specify one of "role" or "instanceProfile" for Karpenter to launch nodes
 instanceProfile: "KarpenterNodeInstanceProfile-${CLUSTER_NAME}"

 # Each term in the array of amiSelectorTerms is ORed together
 # Within a single term, all conditions are ANDed
 amiSelectorTerms:
 # Select on any AMI that has both the `karpenter.sh/discovery: ${CLUSTER_NAME}`
 # AND `environment: test` tags OR any AMI with the name `my-ami` OR an AMI with
 # ID `ami-123` OR an AMI with ID matching the value of my-custom-parameter
 - tags:
 karpenter.sh/discovery: "${CLUSTER_NAME}"
 environment: test
 - name: my-ami
 - id: ami-123
 - ssmParameter: my-custom-parameter # ssm parameter name or ARN
 # Select EKS optimized AL2023 AMIs with version `v20240703`. This term is mutually
 # exclusive and can't be specified with other terms.
 # - alias: al2023@v20240703

 # Optional, each term in the array of capacityReservationSelectorTerms is ORed together.
 capacityReservationSelectorTerms:
 - tags:
 karpenter.sh/discovery: ${CLUSTER_NAME}
 - id: cr-123
 - instanceMatchCriteria: open

 # Optional, propagates tags to underlying EC2 resources
 tags:
 team: team-a
 app: team-a-app

 # Optional, configures IMDS for the instance
 metadataOptions:
 httpEndpoint: enabled
 httpProtocolIPv6: disabled
 httpPutResponseHopLimit: 1 # This is changed to disable IMDS access from containers not on the host network
 httpTokens: required

 # Optional, configures storage devices for the instance
 blockDeviceMappings:
 - deviceName: /dev/xvda
 ebs:
 volumeSize: 100Gi
 volumeType: gp3
 iops: 10000
 encrypted: true
 kmsKeyID: "1234abcd-12ab-34cd-56ef-1234567890ab"
 deleteOnTermination: true
 throughput: 125
 snapshotID: snap-0123456789
 volumeInitializationRate: 100

 # Optional, use instance-store volumes for node ephemeral-storage
 instanceStorePolicy: RAID0

 # Optional, overrides autogenerated userdata with a merge semantic
 userData: |
 echo "Hello world"

 # Optional, configures detailed monitoring for the instance
 detailedMonitoring: true

 # Optional, configures if the instance should be launched with an associated public IP address.
 # If not specified, the default value depends on the subnet's public IP auto-assign setting.
 associatePublicIPAddress: true
status:
 # Resolved subnets
 subnets:
 - id: subnet-0a462d98193ff9fac
 zone: us-east-2b
 - id: subnet-0322dfafd76a609b6
 zone: us-east-2c
 - id: subnet-0727ef01daf4ac9fe
 zone: us-east-2b
 - id: subnet-00c99aeafe2a70304
 zone: us-east-2a
 - id: subnet-023b232fd5eb0028e
 zone: us-east-2c
 - id: subnet-03941e7ad6afeaa72
 zone: us-east-2a

 # Resolved security groups
 securityGroups:
 - id: sg-041513b454818610b
 name: ClusterSharedNodeSecurityGroup
 - id: sg-0286715698b894bca
 name: ControlPlaneSecurityGroup-1AQ073TSAAPW

 # Resolved AMIs
 amis:
 - id: ami-01234567890123456
 name: custom-ami-amd64
 requirements:
 - key: kubernetes.io/arch
 operator: In
 values:
 - amd64
 - id: ami-01234567890123456
 name: custom-ami-arm64
 requirements:
 - key: kubernetes.io/arch
 operator: In
 values:
 - arm64

 # Capacity Reservations
 capacityReservations:
 - availabilityZone: us-west-2a
 id: cr-01234567890123456
 instanceMatchCriteria: targeted
 instanceType: g6.48xlarge
 interruptible: false
 ownerID: "012345678901"
 reservationType: capacity-block
 state: expiring
 - availabilityZone: us-west-2c
 id: cr-12345678901234567
 instanceMatchCriteria: open
 instanceType: g6.48xlarge
 interruptible: true
 ownerID: "98765432109"
 reservationType: default
 state: active

 # Generated instance profile name from "role"
 instanceProfile: "${CLUSTER_NAME}-0123456778901234567789"
 conditions:
 - lastTransitionTime: "2024-02-02T19:54:34Z"
 status: "True"
 type: InstanceProfileReady
 - lastTransitionTime: "2024-02-02T19:54:34Z"
 status: "True"
 type: SubnetsReady
 - lastTransitionTime: "2024-02-02T19:54:34Z"
 status: "True"
 type: SecurityGroupsReady
 - lastTransitionTime: "2024-02-02T19:54:34Z"
 status: "True"
 type: AMIsReady
 - lastTransitionTime: "2024-02-02T19:54:34Z"
 status: "True"
 type: Ready

Refer to the NodePool docs for settings applicable to all providers. To explore various EC2NodeClass configurations, refer to the examples provided in the Karpenter Github repository.

spec.kubelet

Karpenter provides the ability to specify a few additional Kubelet arguments. These are all optional and provide support for additional customization and use cases. Adjust these only if you know you need to do so. For more details on kubelet settings, see the KubeletConfiguration reference. The implemented fields are a subset of the full list of upstream kubelet configuration arguments.

kubelet:
 podsPerCore: 2
 maxPods: 20
 systemReserved:
 cpu: 100m
 memory: 100Mi
 ephemeral-storage: 1Gi
 kubeReserved:
 cpu: 200m
 memory: 100Mi
 ephemeral-storage: 3Gi
 evictionHard:
 memory.available: 5%
 nodefs.available: 10%
 nodefs.inodesFree: 10%
 evictionSoft:
 memory.available: 500Mi
 nodefs.available: 15%
 nodefs.inodesFree: 15%
 evictionSoftGracePeriod:
 memory.available: 1m
 nodefs.available: 1m30s
 nodefs.inodesFree: 2m
 evictionMaxPodGracePeriod: 60
 imageGCHighThresholdPercent: 85
 imageGCLowThresholdPercent: 80
 cpuCFSQuota: true
 clusterDNS: ["10.0.1.100"]

Note

If you need to specify a field that isn’t present in spec.kubelet, you can set it via custom UserData. For example, if you wanted to configure maxPods and registryPullQPS you would set the former through spec.kubelet and the latter through UserData. The following example achieves this with AL2023:

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
spec:
 amiSelectorTerms:
 - alias: al2023@v20240807
 kubelet:
 maxPods: 42
 userData: |
 apiVersion: node.eks.aws/v1alpha1
 kind: NodeConfig
 spec:
 kubelet:
 config:
 # Configured through UserData since unavailable in `spec.kubelet`
 registryPullQPS: 10

Note that when using the Custom AMIFamily you will need to specify fields both in spec.kubelet and spec.userData.

Pods Per Core

An alternative way to dynamically set the maximum density of pods on a node is to use the .spec.kubelet.podsPerCore value. Karpenter will calculate the pod density during scheduling by multiplying this value by the number of logical cores (vCPUs) on an instance type. This value will also be passed through to the --pods-per-core value on kubelet startup to configure the number of allocatable pods the kubelet can assign to the node instance.

The value generated from podsPerCore cannot exceed maxPods, meaning, if both are set, the minimum of the podsPerCore dynamic pod density and the static maxPods value will be used for scheduling.

Note

maxPods may not be set in the kubelet of an EC2NodeClass, but may still be restricted by the ENI_LIMITED_POD_DENSITY value. You may want to ensure that the podsPerCore value that will be used for instance families associated with the EC2NodeClass will not cause unexpected behavior by exceeding the maxPods value.

Max Pods

For small instances that require an increased pod density or large instances that require a reduced pod density, you can override this default value with .spec.kubelet.maxPods. This value will be used during Karpenter pod scheduling and passed through to --max-pods on kubelet startup.

Note

When using small instance types, it may be necessary to enable prefix assignment mode in the AWS VPC CNI plugin to support a higher pod density per node. Prefix assignment mode was introduced in AWS VPC CNI v1.9 and allows ENIs to manage a broader set of IP addresses. Much higher pod densities are supported as a result.

Windows Support Notice

Presently, Windows worker nodes do not support using more than one ENI. As a consequence, the number of IP addresses, and subsequently, the number of pods that a Windows worker node can support is limited by the number of IPv4 addresses available on the primary ENI. Currently, Karpenter will only consider individual secondary IP addresses when calculating the pod density limit.

Reserved Resources

Karpenter will automatically configure the system and kube reserved resource requests on the fly on your behalf. These requests are used to configure your node and to make scheduling decisions for your pods. If you have specific requirements or know that you will have additional capacity requirements, you can optionally override the --system-reserved configuration defaults with the .spec.kubelet.systemReserved values and the --kube-reserved configuration defaults with the .spec.kubelet.kubeReserved values.

Note

Karpenter considers these reserved resources when computing the allocatable ephemeral storage on a given instance type. If kubeReserved is not specified, Karpenter will compute the default reserved CPU and memory resources for the purpose of ephemeral storage computation. These defaults are based on the defaults on Karpenter’s supported AMI families, which are not the same as the kubelet defaults. You should be aware of the CPU and memory default calculation when using Custom AMI Families. If they don’t align, there may be a difference in Karpenter’s computed allocatable ephemeral storage and the actually ephemeral storage available on the node.

Eviction Thresholds

The kubelet supports eviction thresholds by default. When enough memory or file system pressure is exerted on the node, the kubelet will begin to evict pods to ensure that system daemons and other system processes can continue to run in a healthy manner.

Kubelet has the notion of hard evictions and soft evictions. In hard evictions, pods are evicted as soon as a threshold is met, with no grace period to terminate. Soft evictions, on the other hand, provide an opportunity for pods to be terminated gracefully. They do so by sending a termination signal to pods that are planning to be evicted and allowing those pods to terminate up to their grace period.

Karpenter supports hard evictions through the .spec.kubelet.evictionHard field and soft evictions through the .spec.kubelet.evictionSoft field. evictionHard and evictionSoft are configured by listing signal names with either percentage values or resource values.

kubelet:
 evictionHard:
 memory.available: 500Mi
 nodefs.available: 10%
 nodefs.inodesFree: 10%
 imagefs.available: 5%
 imagefs.inodesFree: 5%
 pid.available: 7%
 evictionSoft:
 memory.available: 1Gi
 nodefs.available: 15%
 nodefs.inodesFree: 15%
 imagefs.available: 10%
 imagefs.inodesFree: 10%
 pid.available: 10%

Supported Eviction Signals

Eviction Signal	Description
memory.available	memory.available := node.status.capacity[memory] - node.stats.memory.workingSet
nodefs.available	nodefs.available := node.stats.fs.available
nodefs.inodesFree	nodefs.inodesFree := node.stats.fs.inodesFree
imagefs.available	imagefs.available := node.stats.runtime.imagefs.available
imagefs.inodesFree	imagefs.inodesFree := node.stats.runtime.imagefs.inodesFree
pid.available	pid.available := node.stats.rlimit.maxpid - node.stats.rlimit.curproc

For more information on eviction thresholds, view the Node-pressure Eviction section of the official Kubernetes docs.

Soft Eviction Grace Periods

Soft eviction pairs an eviction threshold with a specified grace period. With soft eviction thresholds, the kubelet will only begin evicting pods when the node exceeds its soft eviction threshold over the entire duration of its grace period. For example, if you specify evictionSoft[memory.available] of 500Mi and a evictionSoftGracePeriod[memory.available] of 1m30, the node must have less than 500Mi of available memory over a minute and a half in order for the kubelet to begin evicting pods.

Optionally, you can specify an evictionMaxPodGracePeriod which defines the administrator-specified maximum pod termination grace period to use during soft eviction. If a namespace-owner had specified a pod terminationGracePeriodInSeconds on pods in their namespace, the minimum of evictionPodGracePeriod and terminationGracePeriodInSeconds would be used.

kubelet:
 evictionSoftGracePeriod:
 memory.available: 1m
 nodefs.available: 1m30s
 nodefs.inodesFree: 2m
 imagefs.available: 1m30s
 imagefs.inodesFree: 2m
 pid.available: 2m
 evictionMaxPodGracePeriod: 60

Pod Density

By default, the number of pods on a node is limited by both the number of networking interfaces (ENIs) that may be attached to an instance type and the number of IP addresses that can be assigned to each ENI. See IP addresses per network interface per instance type for a more detailed information on these instance types’ limits.

Note

By default, the VPC CNI allocates IPs for a node and pods from the same subnet. With VPC CNI Custom Networking, the pods will receive IP addresses from another subnet dedicated to pod IPs. This approach makes it easier to manage IP addresses and allows for separate Network Access Control Lists (NACLs) applied to your pods. VPC CNI Custom Networking reduces the pod density of a node since one of the ENI attachments will be used for the node and cannot share the allocated IPs on the interface to pods. Karpenter supports VPC CNI Custom Networking and similar CNI setups where the primary node interface is separated from the pods interfaces through a global environment variable RESERVED_ENIS, see Settings. In the common case, RESERVED_ENIS should be set to “1” if using Custom Networking.

Windows Support Notice

It’s currently not possible to specify custom networking with Windows nodes.

spec.amiFamily

AMIFamily dictates the default bootstrapping logic for nodes provisioned through this EC2NodeClass. An amiFamily is only required if you don’t specify a spec.amiSelectorTerms.alias object. For example, if you specify alias: al2023@v20240807, the amiFamily is implicitly AL2023.

AMIFamily does not impact which AMI is discovered, only the UserData generation and default BlockDeviceMappings. To automatically discover EKS optimized AMIs, use the new alias field in amiSelectorTerms.

Ubuntu Support Dropped at v1

Support for the Ubuntu AMIFamily has been dropped at Karpenter v1.0.0. This means Karpenter no longer supports automatic AMI discovery and UserData generation for Ubuntu. To continue using Ubuntu AMIs, you will need to select Ubuntu AMIs using amiSelectorTerms.

Additionally, you will need to either maintain UserData yourself using the Custom AMIFamily, or you can use the AL2 AMIFamily and custom blockDeviceMappings (as shown below). The AL2 family has an identical UserData format, but this compatibility isn’t guaranteed long term. Changes to AL2’s or Ubuntu’s UserData format could result in incompatibility, at which point the Custom AMIFamily must be used.

Ubuntu NodeClass Example:

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
spec:
 amiFamily: AL2
 amiSelectorTerms:
 - id: ami-placeholder
 blockDeviceMappings:
 - deviceName: '/dev/sda1'
 rootVolume: true
 ebs:
 encrypted: true
 volumeType: gp3
 volumeSize: 20Gi

AL2

AL2 support dropped at Kubernetes 1.33

Kubernetes version 1.32 is the last version for which Amazon EKS will release Amazon Linux 2 (AL2) AMIs.

Note

Note that Karpenter will automatically generate a call to the /etc/eks/bootstrap.sh script as part of its generated UserData. When using amiFamily: AL2 you should not call this script yourself in .spec.userData. If you need to, use the Custom AMI family instead.

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"

--//
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash -xe
exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1
/etc/eks/bootstrap.sh 'test-cluster' --apiserver-endpoint 'https://test-cluster' --b64-cluster-ca 'ca-bundle' \
--dns-cluster-ip '10.100.0.10' \
--use-max-pods false \
--kubelet-extra-args '--node-labels=karpenter.sh/capacity-type=on-demand,karpenter.sh/nodepool=test --max-pods=110'
--//--

AL2023

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"

--//
Content-Type: application/node.eks.aws

# Karpenter Generated NodeConfig
apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
 cluster:
 name: test-cluster
 apiServerEndpoint: https://example.com
 certificateAuthority: ca-bundle
 cidr: 10.100.0.0/16
 kubelet:
 config:
 maxPods: 110
 flags:
 - --node-labels=karpenter.sh/capacity-type=on-demand,karpenter.sh/nodepool=test

--//--

Bottlerocket

[settings]
[settings.kubernetes]
api-server = 'https://test-cluster'
cluster-certificate = 'ca-bundle'
cluster-name = 'test-cluster'
cluster-dns-ip = '10.100.0.10'
max-pods = 110

[settings.kubernetes.node-labels]
'karpenter.sh/capacity-type' = 'on-demand'
'karpenter.sh/nodepool' = 'test'

Windows2019

<powershell>
[string]$EKSBootstrapScriptFile = "$env:ProgramFiles\Amazon\EKS\Start-EKSBootstrap.ps1"
& $EKSBootstrapScriptFile -EKSClusterName 'test-cluster' -APIServerEndpoint 'https://test-cluster' -Base64ClusterCA 'ca-bundle' -KubeletExtraArgs '--node-labels="karpenter.sh/capacity-type=on-demand,karpenter.sh/nodepool=test" --max-pods=110' -DNSClusterIP '10.100.0.10'
</powershell>

Windows2022

<powershell>
[string]$EKSBootstrapScriptFile = "$env:ProgramFiles\Amazon\EKS\Start-EKSBootstrap.ps1"
& $EKSBootstrapScriptFile -EKSClusterName 'test-cluster' -APIServerEndpoint 'https://test-cluster' -Base64ClusterCA 'ca-bundle' -KubeletExtraArgs '--node-labels="karpenter.sh/capacity-type=on-demand,karpenter.sh/nodepool=test" --max-pods=110' -DNSClusterIP '10.100.0.10'
</powershell>

Windows2025

<powershell>
[string]$EKSBootstrapScriptFile = "$env:ProgramFiles\Amazon\EKS\Start-EKSBootstrap.ps1"
& $EKSBootstrapScriptFile -EKSClusterName 'test-cluster' -APIServerEndpoint 'https://test-cluster' -Base64ClusterCA 'ca-bundle' -KubeletExtraArgs '--node-labels="karpenter.sh/capacity-type=on-demand,karpenter.sh/nodepool=test" --max-pods=110' -DNSClusterIP '10.100.0.10'
</powershell>

Custom

The Custom AMIFamily ships without any default userData to allow you to configure custom bootstrapping for control planes or images that don’t support the default methods from the other families. For this AMIFamily, kubelet must add the taint karpenter.sh/unregistered:NoExecute via the --register-with-taints flag (flags) or the KubeletConfiguration spec (options and docs). Karpenter will fail to register nodes that do not have this taint.

spec.subnetSelectorTerms

Subnet Selector Terms allow you to specify selection logic for a set of subnet options that Karpenter can choose from when launching an instance from the EC2NodeClass. Karpenter discovers subnets through the EC2NodeClass using ids or tags. When launching nodes, a subnet is automatically chosen that matches the desired zone. If multiple subnets exist for a zone, the one with the most available IP addresses will be used.

This selection logic is modeled as terms, where each term contains multiple conditions that must all be satisfied for the selector to match. Effectively, all requirements within a single term are ANDed together. It’s possible that you may want to select on two different subnets that have unrelated requirements. In this case, you can specify multiple terms which will be ORed together to form your selection logic. The example below shows how this selection logic is fulfilled.

subnetSelectorTerms:
 # Select on any subnet that has the "karpenter.sh/discovery: ${CLUSTER_NAME}"
 # AND the "environment: test" tag OR any subnet with ID "subnet-09fa4a0a8f233a921"
 - tags:
 karpenter.sh/discovery: "${CLUSTER_NAME}"
 environment: test
 - id: subnet-09fa4a0a8f233a921

Tip

Subnets may be specified by any tag, including Name. Selecting tag values using wildcards (*) is supported.

Examples

Select all with a specified tag key:

spec:
 subnetSelectorTerms:
 - tags:
 karpenter.sh/discovery/MyClusterName: '*'

Select by name and tag (all criteria must match):

spec:
 subnetSelectorTerms:
 - tags:
 Name: my-subnet
 MyTag: '' # matches all resources with the tag

Select using multiple tag terms:

spec:
 subnetSelectorTerms:
 - tags:
 Name: "my-subnet-1"
 - tags:
 Name: "my-subnet-2"

Select using wildcards:

spec:
 subnetSelectorTerms:
 - tags:
 Name: "*Public*"

Select using ids:

spec:
 subnetSelectorTerms:
 - id: "subnet-09fa4a0a8f233a921"
 - id: "subnet-0471ca205b8a129ae"

spec.securityGroupSelectorTerms

Security Group Selector Terms allow you to specify selection logic for all security groups that will be attached to an instance launched from the EC2NodeClass. The security group of an instance is comparable to a set of firewall rules. EKS creates at least two security groups by default.

This selection logic is modeled as terms, where each term contains multiple conditions that must all be satisfied for the selector to match. Effectively, all requirements within a single term are ANDed together. It’s possible that you may want to select on two different security groups that have unrelated requirements. In this case, you can specify multiple terms which will be ORed together to form your selection logic. The example below shows how this selection logic is fulfilled.

securityGroupSelectorTerms:
 # Select on any security group that has both the "karpenter.sh/discovery: ${CLUSTER_NAME}" tag
 # AND the "environment: test" tag OR any security group with the "my-security-group" name
 # OR any security group with ID "sg-063d7acfb4b06c82c"
 - tags:
 karpenter.sh/discovery: "${CLUSTER_NAME}"
 environment: test
 - name: my-security-group
 - id: sg-063d7acfb4b06c82c

Tip

Security groups may be specified by any tag, including “Name”. Selecting tags using wildcards (*) is supported.

Note

When launching nodes, Karpenter uses all the security groups that match the selector. If you choose to use the kubernetes.io/cluster/$CLUSTER_NAME tag for discovery, note that this may result in failures using the AWS Load Balancer controller. The Load Balancer controller only supports a single security group having that tag key. See this issue for more details.

To verify if this restriction affects you, run the following commands.

CLUSTER_VPC_ID="$(aws eks describe-cluster --name $CLUSTER_NAME --query cluster.resourcesVpcConfig.vpcId --output text)"

aws ec2 describe-security-groups --filters Name=vpc-id,Values=$CLUSTER_VPC_ID Name=tag-key,Values=kubernetes.io/cluster/$CLUSTER_NAME --query 'SecurityGroups[].[GroupName]' --output text

If multiple securityGroups are printed, you will need more specific securityGroupSelectorTerms. We generally recommend that you use the karpenter.sh/discovery: $CLUSTER_NAME tag selector instead.

Examples

Select all assigned to a cluster:

spec:
 securityGroupSelectorTerms:
 - tags:
 kubernetes.io/cluster/$CLUSTER_NAME: "owned"

Select all with a specified tag key:

spec:
 securityGroupSelectorTerms:
 - tags:
 MyTag: '*'

Select by name and tag (all criteria must match):

spec:
 securityGroupSelectorTerms:
 - name: my-security-group
 tags:
 MyTag: '*' # matches all resources with the tag

Select using multiple tag terms:

spec:
 securityGroupSelectorTerms:
 - tags:
 Name: "my-security-group-1"
 - tags:
 Name: "my-security-group-2"

Select by name using a wildcard:

spec:
 securityGroupSelectorTerms:
 - name: "*Public*"

Select using ids:

spec:
 securityGroupSelectorTerms:
 - id: "sg-063d7acfb4b06c82c"
 - id: "sg-06e0cf9c198874591"

spec.role

Role is an optional field and tells Karpenter which IAM identity nodes should assume. You must specify one of role or instanceProfile when creating a Karpenter EC2NodeClass. If using the Karpenter Getting Started Guide to deploy Karpenter, you can use the KarpenterNodeRole-$CLUSTER_NAME role provisioned by that process.

spec:
 role: "KarpenterNodeRole-$CLUSTER_NAME"

spec.instanceProfile

InstanceProfile is an optional field and tells Karpenter which IAM identity nodes should assume. You must specify one of role or instanceProfile when creating a Karpenter EC2NodeClass. If you use the instanceProfile field instead of role, Karpenter will not manage the InstanceProfile on your behalf; instead, it expects that you have pre-provisioned an IAM instance profile and assigned it a role.

You can provision and assign a role to an IAM instance profile using CloudFormation or by using the aws iam create-instance-profile and aws iam add-role-to-instance-profile commands in the CLI.

Note

For private clusters that do not have access to the public internet, using spec.instanceProfile is required. spec.role cannot be used since Karpenter needs to access IAM endpoints to manage a generated instance profile. IAM doesn’t support private endpoints to enable accessing the service without going to the public internet.

Warning

When using spec.instanceProfile, ensure you are using pre-provisioned instance profiles that you manage yourself.

spec.amiSelectorTerms

AMI Selector Terms are required and are used to configure AMIs for Karpenter to use. AMIs are discovered through alias, id, owner, name, and tags.

This selection logic is modeled as terms, where each term contains multiple conditions that must all be satisfied for the selector to match. Effectively, all requirements within a single term are ANDed together. It’s possible that you may want to select on two different AMIs that have unrelated requirements. In this case, you can specify multiple terms which will be ORed together to form your selection logic. The example below shows how this selection logic is fulfilled.

amiSelectorTerms:
 # Select on any AMI that has both the `karpenter.sh/discovery: ${CLUSTER_NAME}`
 # AND `environment: test` tags OR any AMI with the name `my-ami` OR an AMI with
 # ID `ami-123` OR an AMI with ID matching the value of my-custom-parameter
 - tags:
 karpenter.sh/discovery: "${CLUSTER_NAME}"
 environment: test
 - name: my-ami
 - id: ami-123
 - ssmParameter: my-custom-parameter # ssm parameter name or ARN
 # Select EKS optimized AL2023 AMIs with version `v20240807`. This term is mutually
 # exclusive and can't be specified with other terms.
 # - alias: al2023@v20240807

An alias term can be used to select EKS-optimized AMIs. An alias is formatted as family@version. Family can be one of the following values:

al2
al2023
bottlerocket
windows2019
windows2022
windows2025

The version string can be set to latest, or pinned to a specific AMI using the format of that AMI’s GitHub release tags. For example, AL2 and AL2023 use dates for their release, so they can be pinned as follows:

alias: al2023@v20240703

Bottlerocket uses a semantic version for their releases. You can pin bottlerocket as follows:

alias: bottlerocket@v1.20.4

The Windows family does not support pinning, so only latest is supported.

The following commands can be used to determine the versions availble for an alias in your region:

export K8S_VERSION="1.35"
aws ssm get-parameters-by-path --path "/aws/service/eks/optimized-ami/$K8S_VERSION/amazon-linux-2023/" --recursive | jq -cr '.Parameters[].Name' | grep -v "recommended" | awk -F '/' '{print $10}' | sed -r 's/.*(v[[:digit:]]+)$/\1/' | sort | uniq

export K8S_VERSION="1.35"
aws ssm get-parameters-by-path --path "/aws/service/eks/optimized-ami/$K8S_VERSION/amazon-linux-2/" --recursive | jq -cr '.Parameters[].Name' | grep -v "recommended" | awk -F '/' '{print $8}' | sed -r 's/.*(v[[:digit:]]+)$/\1/' | sort | uniq

export K8S_VERSION="1.35"
aws ssm get-parameters-by-path --path "/aws/service/bottlerocket/aws-k8s-$K8S_VERSION" --recursive | jq -cr '.Parameters[].Name' | grep -v "latest" | awk -F '/' '{print $7}' | sort | uniq

Warning

Karpenter supports automatic AMI selection and upgrades using the latest version pin, but this is not recommended for production environments. When using latest, a new AMI release will cause Karpenter to drift all out-of-date nodes in the cluster, replacing them with nodes running the new AMI. We strongly recommend evaluating new AMIs in a lower environment before rolling them out into a production environment. More details on Karpenter’s recommendations for managing AMIs can be found here.

To select an AMI by name, use the name field in the selector term. To select an AMI by id, use the id field in the selector term. To select AMIs that are not owned by amazon or the account that Karpenter is running in, use the owner field - you can use a combination of account aliases (e.g. self amazon, your-aws-account-name) and account IDs.

If owner is not set for name, it defaults to self,amazon, preventing Karpenter from inadvertently selecting an AMI that is owned by a different account. Tags don’t require an owner as tags can only be discovered by the user who created them.

Tip

AMIs may be specified by any AWS tag, including Name. Selecting by tag or by name using wildcards (*) is supported.

Note

If amiSelectorTerms match more than one AMI, Karpenter will automatically determine which AMI best fits the workloads on the launched worker node under the following constraints:

When launching nodes, Karpenter automatically determines which architecture a custom AMI is compatible with and will use images that match an instanceType’s requirements.
- Unless using an alias, Karpenter cannot detect requirements other than architecture. If you need to specify different AMIs for different kind of nodes (e.g. accelerated GPU AMIs), you should use a separate EC2NodeClass.
If multiple AMIs are found that can be used, Karpenter will choose the latest one.
If no AMIs are found that can be used, then no nodes will be provisioned.

Examples

Select by AMI family and version:

 amiSelectorTerms:
 - alias: al2023@v20240807

Select all with a specified tag:

 amiSelectorTerms:
 - tags:
 karpenter.sh/discovery/MyClusterName: '*'

Select by name:

 amiSelectorTerms:
 - name: my-ami

Select by Name tag:

 amiSelectorTerms:
 - tags:
 Name: my-ami

Select by name and owner:

 amiSelectorTerms:
 - name: my-ami
 owner: self
 - name: my-ami
 owner: "0123456789"

Select by name using a wildcard:

spec:
 amiSelectorTerms:
 - name: "*EKS*"

Select by all under an owner:

spec:
 amiSelectorTerms:
 - name: "*"
 owner: self

Specify using ids:

 amiSelectorTerms:
 - id: "ami-123"
 - id: "ami-456"

Specify using custom ssm parameter name or ARN:

 amiSelectorTerms:
 - ssmParameter: "my-custom-parameter"

Note

When using a custom SSM parameter, you’ll need to expand the ssm:GetParameter permissions on the Karpenter IAM role to include your custom parameter, as the default policy only allows access to the AWS public parameters.

spec.capacityReservationSelectorTerms

Feature State: Beta

Capacity Reservation Selector Terms allow you to select on-demand capacity reservations (ODCRs), which will be made available to NodePools which select the given EC2NodeClass. Karpenter will prioritize utilizing the capacity in these reservations before falling back to on-demand and spot. Capacity reservations can be discovered using ids, tags, or instance match criteria.

This selection logic is modeled as terms. A term can specify an ID, a set of tags, or instance match criteria to select against. When specifying tags, it will select all capacity reservations accessible from the account with matching tags. When specifying instance match criteria, it selects reservations by their matching behavior: open (matches all compatible instances) or targeted (matches only explicitly targeted instances). This can be further restricted by specifying an owner ID.

For more information on utilizing ODCRs with Karpenter, refer to the Utilizing ODCRs Task.

Note

Note that the IAM role Karpenter assumes should have a permissions policy associated with it that grants it permissions to use the ec2:DescribeCapacityReservations action to discover capacity reservations and the ec2:RunInstances action to run instances in those capacity reservations.

Examples

Select the reservations with the given IDs:

spec:
 capacityReservationSelectorTerms:
 - id: cr-123
 - id: cr-456

Select the reservations by tags:

spec:
 capacityReservationSelectorTerms:
 # Select all capacity reservations which have both matching tags
 - tags:
 key1: foo
 key2: bar
 # Additionally, select all capacity reservations with the following matching tag
 - tags:
 key3: foobar

Select by tags and owner ID:

spec:
 # Select all capacity reservations with the matching tags which are also owned by
 # the specified account.
 capacityReservationSelectorTerms:
 - tags:
 key: foo
 ownerID: 012345678901

Select by instance match criteria:

spec:
 capacityReservationSelectorTerms:
 # Select all open capacity reservations
 - instanceMatchCriteria: open

Select by instance match criteria and tags:

spec:
 capacityReservationSelectorTerms:
 # Select targeted capacity reservations with matching tags
 - instanceMatchCriteria: targeted
 tags:
 key: foo

spec.tags

Karpenter adds tags to all resources it creates, including EC2 Instances, EBS volumes, and Launch Templates. The default set of tags are listed below.

Name: <node-name>
karpenter.sh/nodeclaim: <nodeclaim-name>
karpenter.sh/nodepool: <nodepool-name>
karpenter.k8s.aws/ec2nodeclass: <ec2nodeclass-name>
kubernetes.io/cluster/<cluster-name>: owned
eks:eks-cluster-name: <cluster-name>

Additional tags can be added in the tags section, which will be merged with the default tags specified above.

spec:
 tags:
 InternalAccountingTag: 1234
 dev.corp.net/app: Calculator
 dev.corp.net/team: MyTeam

Note

Karpenter allows overrides of the default “Name” tag but does not allow overrides to restricted domains (such as “karpenter.sh”, “karpenter.k8s.aws”, and “kubernetes.io/cluster”). This ensures that Karpenter is able to correctly auto-discover nodes that it owns.

spec.metadataOptions

Control the exposure of Instance Metadata Service on EC2 Instances launched by this EC2NodeClass using a generated launch template.

Refer to recommended, security best practices for limiting exposure of Instance Metadata and User Data to pods.

If metadataOptions are omitted from this EC2NodeClass, the following default settings are applied:

spec:
 metadataOptions:
 httpEndpoint: enabled
 httpProtocolIPv6: disabled
 httpPutResponseHopLimit: 1
 httpTokens: required

spec.blockDeviceMappings

The blockDeviceMappings field in an EC2NodeClass can be used to control the Elastic Block Storage (EBS) volumes that Karpenter attaches to provisioned nodes. Karpenter uses default block device mappings for the AMIFamily specified. For example, the Bottlerocket AMI Family defaults with two block device mappings, one for Bottlerocket’s control volume and the other for container resources such as images and logs.

spec:
 blockDeviceMappings:
 - deviceName: /dev/xvda
 ebs:
 volumeSize: 100Gi
 volumeType: gp3
 iops: 10000
 encrypted: true
 kmsKeyID: "1234abcd-12ab-34cd-56ef-1234567890ab"
 deleteOnTermination: true
 throughput: 125
 snapshotID: snap-0123456789
 volumeInitializationRate: 100

The following blockDeviceMapping defaults are used for each AMIFamily if no blockDeviceMapping overrides are specified in the EC2NodeClass

AL2

spec:
 blockDeviceMappings:
 - deviceName: /dev/xvda
 ebs:
 volumeSize: 20Gi
 volumeType: gp3
 encrypted: true

AL2023

spec:
 blockDeviceMappings:
 - deviceName: /dev/xvda
 ebs:
 volumeSize: 20Gi
 volumeType: gp3
 encrypted: true

Bottlerocket

spec:
 blockDeviceMappings:
 # Root device
 - deviceName: /dev/xvda
 ebs:
 volumeSize: 4Gi
 volumeType: gp3
 encrypted: true
 # Data device: Container resources such as images and logs
 - deviceName: /dev/xvdb
 ebs:
 volumeSize: 20Gi
 volumeType: gp3
 encrypted: true

Windows2019/Windows2022/Windows2025

spec:
 blockDeviceMappings:
 - deviceName: /dev/sda1
 ebs:
 volumeSize: 50Gi
 volumeType: gp3
 encrypted: true

Custom

The Custom AMIFamily ships without any default blockDeviceMappings.

spec.instanceStorePolicy

The instanceStorePolicy field controls how instance-store volumes are handled. By default, Karpenter and Kubernetes will simply ignore them.

RAID0

If you intend to use these volumes for faster node ephemeral-storage, set instanceStorePolicy to RAID0:

spec:
 instanceStorePolicy: RAID0

This will set the allocatable ephemeral-storage of each node to the total size of the instance-store volume(s). This configuration is likely to be useful for workloads that leverage dense storage instance types or require the low latency from instance-stores that are nvme ssd based.

The disks must be formatted & mounted in a RAID0 and be the underlying filesystem for the Kubelet & Containerd. Even if you already configure your volumes with RAID0, Karpenter won’t recognize this by default unless you set the instanceStorePolicy to RAID0. Without this, scheduling workloads that depend on ephemeral-storage from the instance-stores may result in a deadlock due to insufficient storage.

Instructions for each AMI family are listed below:

AL2

On AL2, Karpenter automatically configures the disks through an additional boostrap argument (--local-disks raid0). The device name is /dev/md/0 and its mount point is /mnt/k8s-disks/0. You should ensure any additional disk setup does not interfere with these.

AL2023

On AL2023, Karpenter automatically configures the disks via the generated NodeConfig object. Like AL2, the device name is /dev/md/0 and its mount point is /mnt/k8s-disks/0. You should ensure any additional disk setup does not interfere with these.

Bottlerocket

On Bottlerocket, Karpenter automatically configures the disks by adding the required bootstrap commands to the settings. The ephemeral storage configuration is handled automatically, so you should not manually configure ephemeral storage settings in your userData.

Note

This automatic disk configuration is only available on Bottlerocket v1.22.0 and later.

Others

For all other AMI families, you must configure the disks yourself. Check out the setup-local-disks script in amazon-eks-ami to see how this is done for AL2.

Tip

Since the Kubelet & Containerd will be using the instance-store filesystem, you may consider using a more minimal root volume size.

spec.userData

You can control the UserData that is applied to your worker nodes via this field. This allows you to run custom scripts or pass-through custom configuration to Karpenter instances on start-up.

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
 name: bottlerocket-example
spec:
 ...
 amiFamily: Bottlerocket
 userData: |
 [settings.kubernetes]
 "kube-api-qps" = 30
 "shutdown-grace-period" = "30s"
 "shutdown-grace-period-for-critical-pods" = "30s"
 [settings.kubernetes.eviction-hard]
 "memory.available" = "20%"

This example adds SSH keys to allow remote login to the node (replace my-authorized_keys with your public key file):

Note

Instead of using SSH as set up in this example, you can use Session Manager (SSM) or EC2 Instance Connect to gain shell access to Karpenter nodes. See Node NotReady troubleshooting for an example of starting an SSM session from the command line or EC2 Instance Connect documentation to connect to nodes using SSH.

Also, my-authorized_key key is the public key. See Retrieve the public key material.

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
 name: al2-example
spec:
 ...
 amiFamily: AL2
 userData: |
 #!/bin/bash
 mkdir -p ~ec2-user/.ssh/
 touch ~ec2-user/.ssh/authorized_keys
 cat >> ~ec2-user/.ssh/authorized_keys <<EOF
 {{ insertFile "../my-authorized_keys" | indent 4 }}
 EOF
 chmod -R go-w ~ec2-user/.ssh/authorized_keys
 chown -R ec2-user ~ec2-user/.ssh

Alternatively, you can save the key in your SSM Parameter Store and use the get-parameter command mentioned below to retrieve the key for authorized_keys.

aws ssm get-parameter --name "<parameter-name>" --region <region> --with-decryption --query "Parameter.Value" --output text > /home/ec2-user/.ssh/authorized_keys

For more examples on configuring fields for different AMI families, see the examples here.

Karpenter will merge the userData you specify with the default userData for that AMIFamily. See the AMIFamily section for more details on these defaults. View the sections below to understand the different merge strategies for each AMIFamily.

AL2

Your UserData can be in the MIME multi part archive format.
Karpenter will transform your custom user-data as a MIME part, if necessary, and then merge a final MIME part to the end of your UserData parts which will bootstrap the worker node. Karpenter will have full control over all the parameters being passed to the bootstrap script.
- Karpenter will continue to set MaxPods, ClusterDNS and all other parameters defined in spec.kubeletConfiguration as before.

Consider the following example to understand how your custom UserData will be merged -

Passed-in UserData (bash)

#!/bin/bash
echo "Running custom user data script (bash)"

Merged UserData (bash)

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"

--//
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash
echo "Running custom user data script (bash)"

--//
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash -xe
exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1
/etc/eks/bootstrap.sh 'test-cluster' --apiserver-endpoint 'https://test-cluster' --b64-cluster-ca 'ca-bundle' \
--use-max-pods false \
--kubelet-extra-args '--node-labels=karpenter.sh/capacity-type=on-demand,karpenter.sh/nodepool=test --max-pods=110'
--//--

Passed-in UserData (MIME)

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="BOUNDARY"

--BOUNDARY
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash
echo "Running custom user data script (mime)"

--BOUNDARY--

Merged UserData (MIME)

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"

--//
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash
echo "Running custom user data script (mime)"

--//
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash -xe
exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1
/etc/eks/bootstrap.sh 'test-cluster' --apiserver-endpoint 'https://test-cluster' --b64-cluster-ca 'ca-bundle' \
--use-max-pods false \
--kubelet-extra-args '--node-labels=karpenter.sh/capacity-type=on-demand,karpenter.sh/nodepool=test --max-pods=110'
--//--

Tip

You can set additional kubelet configuration properties, unavailable through spec.kubelet, by updating the kubelet-config.json file:

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
 name: kubelet-config-example
spec:
 amiFamily: AL2
 userData: |
 #!/bin/bash
 echo "$(jq '.kubeAPIQPS=50' /etc/kubernetes/kubelet/kubelet-config.json)" > /etc/kubernetes/kubelet/kubelet-config.json

AL2023

Your UserData may be in one of three formats: a MIME multi part archive, a NodeConfig YAML / JSON string, or a shell script.
Karpenter will transform your custom UserData into a MIME part, if necessary, and then create a MIME multi-part archive. This archive will consist of a generated NodeConfig, containing Karpenter’s default values, followed by the transformed custom UserData. For more information on the NodeConfig spec, refer to the AL2023 EKS Optimized AMI docs.

Warning

Any values configured by the Karpenter generated NodeConfig object will take precedent over values specifed in spec.userData. This includes cluster name, cluster CIDR, cluster endpoint, certificate authority, taints, labels, and any value in spec.kubelet. These fields must be configured natively through Karpenter rather than through UserData.

Passed-in UserData (NodeConfig)

apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
 kubelet:
 config:
 maxPods: 42

Merged UserData (NodeConfig)

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"

--//
Content-Type: application/node.eks.aws

apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
 kubelet:
 config:
 maxPods: 42

--//
Content-Type: application/node.eks.aws

# Karpenter Generated NodeConfig
apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
 cluster:
 apiServerEndpoint: https://test-cluster
 certificateAuthority: cluster-ca
 cidr: 10.100.0.0/16
 name: test-cluster
 kubelet:
 config:
 clusterDNS:
 - 10.100.0.10
 maxPods: 118
 flags:
 - --node-labels="karpenter.sh/capacity-type=on-demand,karpenter.sh/nodepool=default"

--//--

Passed-in UserData (bash)

#!/bin/bash
echo "Hello, AL2023!"

Merged UserData (bash)

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"

--//
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash
echo "Hello, AL2023!"

--//
Content-Type: application/node.eks.aws

# Karpenter Generated NodeConfig
apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
 cluster:
 apiServerEndpoint: https://test-cluster
 certificateAuthority: cluster-ca
 cidr: 10.100.0.0/16
 name: test-cluster
 kubelet:
 config:
 clusterDNS:
 - 10.100.0.10
 maxPods: 118
 flags:
 - --node-labels="karpenter.sh/capacity-type=on-demand,karpenter.sh/nodepool=default"

--//--

Passed-in UserData (MIME)

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"

--//
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash
echo "Hello, AL2023!"

--//
Content-Type: application/node.eks.aws

apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
 kubelet:
 config:
 maxPods: 42
--//

Merged UserData (MIME)

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"

--//
Content-Type: application/node.eks.aws

apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
 kubelet:
 config:
 maxPods: 42
--//
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash
echo "Hello, AL2023!"

--//
Content-Type: application/node.eks.aws

# Karpenter Generated NodeConfig
apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
 cluster:
 apiServerEndpoint: https://test-cluster
 certificateAuthority: cluster-ca
 cidr: 10.100.0.0/16
 name: test-cluster
 kubelet:
 config:
 clusterDNS:
 - 10.100.0.10
 maxPods: 118
 flags:
 - --node-labels="karpenter.sh/capacity-type=on-demand,karpenter.sh/nodepool=default"

--//--

Bottlerocket

Your UserData must be valid TOML.
Unknown TOML fields will be ignored when the final merged UserData is generated by Karpenter.

Warning

Any values configured by Karpenter will take precedent over values specifed in spec.userData. This includes cluster name, cluster endpoint, cluster certificate, taints, labels, and any value in spec.kubelet. These fields must be configured natively through Karpenter rather than through UserData.

Consider the following example to understand how your custom UserData settings will be merged in.

Passed-in UserData

[settings.kubernetes.eviction-hard]
"memory.available" = "12%"
[settings.kubernetes]
"unknown-setting" = "unknown"
[settings.kubernetes.node-labels]
'field.controlled.by/karpenter' = 'will-be-overridden'

Merged UserData

[settings]
[settings.kubernetes]
api-server = 'https://cluster'
cluster-certificate = 'ca-bundle'
cluster-name = 'cluster'

[settings.kubernetes.node-labels]
'karpenter.sh/capacity-type' = 'on-demand'
'karpenter.sh/nodepool' = 'default'

[settings.kubernetes.node-taints]

[settings.kubernetes.eviction-hard]
'memory.available' = '12%%'

Device ownership in Bottlerocket

Bottlerocket v1.30.0+ supports device ownership using the security context provided in the Kubernetes specfile. To enable this, you will need the following user-data configurations:

[settings]
[settings.kubernetes]
device-ownership-from-security-context = true

This allows the container to take ownership of devices allocated to the pod via device-plugins based on the runAsUser and runAsGroup values provided in the spec. For more details on this, see the Kubernetes documentation

This setting helps you enable Neuron workloads on Bottlerocket instances. See Accelerators/GPU Resources for more details.

Windows2019/Windows2022/Windows2025

Your UserData must be specified as PowerShell commands.
The UserData specified will be prepended to a Karpenter managed section that will bootstrap the kubelet.
Karpenter will continue to set ClusterDNS and all other parameters defined in spec.kubeletConfiguration as before.

Consider the following example to understand how your custom UserData settings will be merged in.

Passed-in UserData

Write-Host "Running custom user data script"

Merged UserData

<powershell>
Write-Host "Running custom user data script"
[string]$EKSBootstrapScriptFile = "$env:ProgramFiles\Amazon\EKS\Start-EKSBootstrap.ps1"
& $EKSBootstrapScriptFile -EKSClusterName 'test-cluster' -APIServerEndpoint 'https://test-cluster' -Base64ClusterCA 'ca-bundle' -KubeletExtraArgs '--node-labels="karpenter.sh/capacity-type=spot,karpenter.sh/nodepool=windows2022" --max-pods=110' -DNSClusterIP '10.0.100.10'
</powershell>

Windows Support Notice

Currently, Karpenter does not specify -ServiceCIDR to EKS Windows AMI Bootstrap script. Windows worker nodes will use 172.20.0.0/16 or 10.100.0.0/16 for Kubernetes service IP address ranges based on the IP address of the primary interface. The effective ServiceCIDR can be verified at $env:ProgramData\Amazon\EKS\cni\config\vpc-bridge.conf on the worker node.

Support for the Windows ServiceCIDR argument can be tracked in a Karpenter Github Issue. Currently, if the effective ServiceCIDR is incorrect for your windows worker nodes, you can add the following userData as a workaround.

spec:
 userData: |
 $global:EKSCluster = Get-EKSCluster -Name my-cluster

Custom

No merging is performed, your UserData must perform all setup required of the node to allow it to join the cluster.
Custom UserData must meet the following requirements to work correctly with Karpenter:
- It must ensure the node is registered with the karpenter.sh/unregistered:NoExecute taint (via kubelet configuration field registerWithTaints)
- It must set kubelet config options to match those configured in spec.kubelet

spec.detailedMonitoring

Enabling detailed monitoring controls the EC2 detailed monitoring feature. If you enable this option, the Amazon EC2 console displays monitoring graphs with a 1-minute period for the instances that Karpenter launches.

spec:
 detailedMonitoring: true

spec.associatePublicIPAddress

You can explicitly set AssociatePublicIPAddress: false when you are only launching into private subnets. Previously, Karpenter auto-set associatePublicIPAddress on the primary ENI to false if a user’s subnet options were all private subnets. This value is a boolean field that controls whether instances created by Karpenter for this EC2NodeClass will have an associated public IP address. This overrides the MapPublicIpOnLaunch setting applied to the subnet the node is launched in. If this field is not set, the MapPublicIpOnLaunch field will be respected.

Note

If a NodeClaim requests vpc.amazonaws.com/efa resources, spec.associatePublicIPAddress is respected. However, if this NodeClaim requests multiple EFA resources and the value for spec.associatePublicIPAddress is true, the instance will fail to launch. This is due to an EC2 restriction which requires that the field is only set to true when configuring an instance with a single ENI at launch. When using this field, it is advised that users segregate their EFA workload to use a separate NodePool / EC2NodeClass pair.

spec.ipPrefixCount

This value is a integer field that controls how many ip prefixes will be assigned to NodeClaim. See the EC2 Launch Template Network Interface Spec for more information. Sets ipv4PrefixCount if you are using an IPv4 Cluster, or ipv6PrefixCount if you are using IPv6.

status.subnets

status.subnets contains the resolved id and zone of the subnets that were selected by the spec.subnetSelectorTerms for the node class. The subnets will be sorted by the available IP address count in decreasing order.

Examples

spec:
 subnetSelectorTerms:
 - tags:
 karpenter.sh/discovery: "${CLUSTER_NAME}"
status:
 subnets:
 - id: subnet-0a462d98193ff9fac
 zone: us-east-2b
 - id: subnet-0322dfafd76a609b6
 zone: us-east-2c
 - id: subnet-0727ef01daf4ac9fe
 zone: us-east-2b
 - id: subnet-00c99aeafe2a70304
 zone: us-east-2a
 - id: subnet-023b232fd5eb0028e
 zone: us-east-2c
 - id: subnet-03941e7ad6afeaa72
 zone: us-east-2a

status.securityGroups

status.securityGroups contains the resolved id and name of the security groups that were selected by the spec.securityGroupSelectorTerms for the node class. The subnets will be sorted by the available IP address count in decreasing order.

Examples

spec:
 securityGroupSelectorTerms:
 - tags:
 karpenter.sh/discovery: "${CLUSTER_NAME}"
status:
 securityGroups:
 - id: sg-041513b454818610b
 name: ClusterSharedNodeSecurityGroup
 - id: sg-0286715698b894bca
 name: ControlPlaneSecurityGroup-1AQ073TSAAPW

status.amis

status.amis contains the resolved id, name, requirements, and the deprecated status of either the default AMIs for the spec.amiFamily or the AMIs selected by the spec.amiSelectorTerms if this field is specified. The deprecated status will be shown for resolved AMIs that are deprecated.

Examples

AMIs resolved with an AL2 alias:

spec:
 amiSelectorTerms:
 - alias: al2@v20240807
status:
 amis:
 - id: ami-03c3a3dcda64f5b75
 name: amazon-linux-2-gpu
 requirements:
 - key: kubernetes.io/arch
 operator: In
 values:
 - amd64
 - key: karpenter.k8s.aws/instance-gpu-count
 operator: Exists
 - id: ami-03c3a3dcda64f5b75
 name: amazon-linux-2-gpu
 requirements:
 - key: kubernetes.io/arch
 operator: In
 values:
 - amd64
 - key: karpenter.k8s.aws/instance-accelerator-count
 operator: Exists
 - id: ami-06afb2d101cc4b8bd
 name: amazon-linux-2-arm64
 requirements:
 - key: kubernetes.io/arch
 operator: In
 values:
 - arm64
 - key: karpenter.k8s.aws/instance-gpu-count
 operator: DoesNotExist
 - key: karpenter.k8s.aws/instance-accelerator-count
 operator: DoesNotExist
 - id: ami-0e28b76d768af234e
 name: amazon-linux-2
 requirements:
 - key: kubernetes.io/arch
 operator: In
 values:
 - amd64
 - key: karpenter.k8s.aws/instance-gpu-count
 operator: DoesNotExist
 - key: karpenter.k8s.aws/instance-accelerator-count
 operator: DoesNotExist

AMIs resolved from tags:

spec:
 amiSelectorTerms:
 - tags:
 karpenter.sh/discovery: "${CLUSTER_NAME}"
status:
 amis:
 - id: ami-01234567890123456
 name: custom-ami-amd64
 deprecated: true
 requirements:
 - key: kubernetes.io/arch
 operator: In
 values:
 - amd64
 - id: ami-01234567890123456
 name: custom-ami-arm64
 requirements:
 - key: kubernetes.io/arch
 operator: In
 values:
 - arm64

status.capacityReservations

status.capacityReservations contains the following information for each resolved capacity reservation:

Field	Example	Description
`availabilityZone`	`us-east-1a`	The availability zone the capacity reservation is available in
`id`	`cr-56fac701cc1951b03`	The ID of the capacity reservation
`instanceMatchCriteria`	`open`	The instanceMatchCriteria for the capacity reservation. Can be `open` or `targeted`.
`instanceType`	`m5.large`	The EC2 instance type of the capacity reservation
`ownerID`	`459763720645`	The account ID that owns the capacity reservation
`reservationType`	`default`	The type of the capacity reservation. Can be `default` or `capacity-block`.
`state`	`active`	The state of the capacity reservation. Can be `active` or `expiring`.
`interruptible`	`true` or `false`	Whether the capacity reservation is interruptible.

Examples

status:
 capacityReservations:
 - availabilityZone: us-west-2a
 id: cr-01234567890123456
 instanceMatchCriteria: targeted
 instanceType: g6.48xlarge
 interruptible: false
 ownerID: "012345678901"
 reservationType: capacity-block
 state: expiring
 - availabilityZone: us-west-2c
 id: cr-12345678901234567
 instanceMatchCriteria: open
 instanceType: g6.48xlarge
 interruptible: true
 ownerID: "98765432109"
 reservationType: default
 state: active

status.instanceProfile

status.instanceProfile contains the resolved instance profile generated by Karpenter from the spec.role

spec:
 role: "KarpenterNodeRole-${CLUSTER_NAME}"
status:
 instanceProfile: "${CLUSTER_NAME}-0123456778901234567789"

status.conditions

status.conditions indicates EC2NodeClass readiness. This will be Ready when Karpenter successfully discovers AMIs, Instance Profile, Subnets, Cluster CIDR (AL2023 only) and SecurityGroups for the EC2NodeClass.

NodeClasses have the following status conditions:

Condition Type	Description
SubnetsReady	Subnets are discovered.
SecurityGroupsReady	Security Groups are discovered.
InstanceProfileReady	Instance Profile is discovered.
AMIsReady	AMIs are discovered.
Ready	Top level condition that indicates if the nodeClass is ready. If any of the underlying conditions is `False` then this condition is set to `False` and `Message` on the condition indicates the dependency that was not resolved.

If a NodeClass is not ready, NodePools that reference it through their nodeClassRef will not be considered for scheduling.

Preview: NodeClasses

Mon, 01 Jan 0001 00:00:00 +0000

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
 name: default
spec:
 template:
 spec:
 nodeClassRef:
 group: karpenter.k8s.aws
 kind: EC2NodeClass
 name: default
---
apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
 name: default
spec:
 kubelet:
 podsPerCore: 2
 maxPods: 20
 systemReserved:
 cpu: 100m
 memory: 100Mi
 ephemeral-storage: 1Gi
 kubeReserved:
 cpu: 200m
 memory: 100Mi
 ephemeral-storage: 3Gi
 evictionHard:
 memory.available: 5%
 nodefs.available: 10%
 nodefs.inodesFree: 10%
 evictionSoft:
 memory.available: 500Mi
 nodefs.available: 15%
 nodefs.inodesFree: 15%
 evictionSoftGracePeriod:
 memory.available: 1m
 nodefs.available: 1m30s
 nodefs.inodesFree: 2m
 evictionMaxPodGracePeriod: 60
 imageGCHighThresholdPercent: 85
 imageGCLowThresholdPercent: 80
 cpuCFSQuota: true
 clusterDNS: ["10.0.1.100"]
 # Optional, dictates UserData generation and default block device mappings.
 # May be ommited when using an `alias` amiSelectorTerm, otherwise required.
 amiFamily: AL2

 # Required, discovers subnets to attach to instances
 # Each term in the array of subnetSelectorTerms is ORed together
 # Within a single term, all conditions are ANDed
 subnetSelectorTerms:
 # Select on any subnet that has the "karpenter.sh/discovery: ${CLUSTER_NAME}"
 # AND the "environment: test" tag OR any subnet with ID "subnet-09fa4a0a8f233a921"
 - tags:
 karpenter.sh/discovery: "${CLUSTER_NAME}"
 environment: test
 - id: subnet-09fa4a0a8f233a921

 # Required, discovers security groups to attach to instances
 # Each term in the array of securityGroupSelectorTerms is ORed together
 # Within a single term, all conditions are ANDed
 securityGroupSelectorTerms:
 # Select on any security group that has both the "karpenter.sh/discovery: ${CLUSTER_NAME}" tag
 # AND the "environment: test" tag OR any security group with the "my-security-group" name
 # OR any security group with ID "sg-063d7acfb4b06c82c"
 - tags:
 karpenter.sh/discovery: "${CLUSTER_NAME}"
 environment: test
 - name: my-security-group
 - id: sg-063d7acfb4b06c82c

 # Optional, IAM role to use for the node identity.
 # Must specify one of "role" or "instanceProfile" for Karpenter to launch nodes
 role: "KarpenterNodeRole-${CLUSTER_NAME}"

 # Optional, IAM instance profile to use for the node identity.
 # Must specify one of "role" or "instanceProfile" for Karpenter to launch nodes
 instanceProfile: "KarpenterNodeInstanceProfile-${CLUSTER_NAME}"

 # Each term in the array of amiSelectorTerms is ORed together
 # Within a single term, all conditions are ANDed
 amiSelectorTerms:
 # Select on any AMI that has both the `karpenter.sh/discovery: ${CLUSTER_NAME}`
 # AND `environment: test` tags OR any AMI with the name `my-ami` OR an AMI with
 # ID `ami-123` OR an AMI with ID matching the value of my-custom-parameter
 - tags:
 karpenter.sh/discovery: "${CLUSTER_NAME}"
 environment: test
 - name: my-ami
 - id: ami-123
 - ssmParameter: my-custom-parameter # ssm parameter name or ARN
 # Select EKS optimized AL2023 AMIs with version `v20240703`. This term is mutually
 # exclusive and can't be specified with other terms.
 # - alias: al2023@v20240703

 # Optional, each term in the array of capacityReservationSelectorTerms is ORed together.
 capacityReservationSelectorTerms:
 - tags:
 karpenter.sh/discovery: ${CLUSTER_NAME}
 - id: cr-123
 - instanceMatchCriteria: open

 # Optional, propagates tags to underlying EC2 resources
 tags:
 team: team-a
 app: team-a-app

 # Optional, configures IMDS for the instance
 metadataOptions:
 httpEndpoint: enabled
 httpProtocolIPv6: disabled
 httpPutResponseHopLimit: 1 # This is changed to disable IMDS access from containers not on the host network
 httpTokens: required

 # Optional, configures storage devices for the instance
 blockDeviceMappings:
 - deviceName: /dev/xvda
 ebs:
 volumeSize: 100Gi
 volumeType: gp3
 iops: 10000
 encrypted: true
 kmsKeyID: "1234abcd-12ab-34cd-56ef-1234567890ab"
 deleteOnTermination: true
 throughput: 125
 snapshotID: snap-0123456789
 volumeInitializationRate: 100

 # Optional, configures the network interfaces for the instance
 networkInterfaces:
 - networkCardIndex: 0
 deviceIndex: 0
 interfaceType: "interface"
 - networkCardIndex: 0
 deviceIndex: 1
 interfaceType: "interface"

 # Optional, use instance-store volumes for node ephemeral-storage
 instanceStorePolicy: RAID0

 # Optional, overrides autogenerated userdata with a merge semantic
 userData: |
 echo "Hello world"

 # Optional, configures detailed monitoring for the instance
 detailedMonitoring: true

 # Optional, configures if the instance should be launched with an associated public IP address.
 # If not specified, the default value depends on the subnet's public IP auto-assign setting.
 associatePublicIPAddress: true
status:
 # Resolved subnets
 subnets:
 - id: subnet-0a462d98193ff9fac
 zone: us-east-2b
 - id: subnet-0322dfafd76a609b6
 zone: us-east-2c
 - id: subnet-0727ef01daf4ac9fe
 zone: us-east-2b
 - id: subnet-00c99aeafe2a70304
 zone: us-east-2a
 - id: subnet-023b232fd5eb0028e
 zone: us-east-2c
 - id: subnet-03941e7ad6afeaa72
 zone: us-east-2a

 # Resolved security groups
 securityGroups:
 - id: sg-041513b454818610b
 name: ClusterSharedNodeSecurityGroup
 - id: sg-0286715698b894bca
 name: ControlPlaneSecurityGroup-1AQ073TSAAPW

 # Resolved AMIs
 amis:
 - id: ami-01234567890123456
 name: custom-ami-amd64
 requirements:
 - key: kubernetes.io/arch
 operator: In
 values:
 - amd64
 - id: ami-01234567890123456
 name: custom-ami-arm64
 requirements:
 - key: kubernetes.io/arch
 operator: In
 values:
 - arm64

 # Capacity Reservations
 capacityReservations:
 - availabilityZone: us-west-2a
 id: cr-01234567890123456
 instanceMatchCriteria: targeted
 instanceType: g6.48xlarge
 interruptible: false
 ownerID: "012345678901"
 reservationType: capacity-block
 state: expiring
 - availabilityZone: us-west-2c
 id: cr-12345678901234567
 instanceMatchCriteria: open
 instanceType: g6.48xlarge
 interruptible: true
 ownerID: "98765432109"
 reservationType: default
 state: active

 # Generated instance profile name from "role"
 instanceProfile: "${CLUSTER_NAME}-0123456778901234567789"
 conditions:
 - lastTransitionTime: "2024-02-02T19:54:34Z"
 status: "True"
 type: InstanceProfileReady
 - lastTransitionTime: "2024-02-02T19:54:34Z"
 status: "True"
 type: SubnetsReady
 - lastTransitionTime: "2024-02-02T19:54:34Z"
 status: "True"
 type: SecurityGroupsReady
 - lastTransitionTime: "2024-02-02T19:54:34Z"
 status: "True"
 type: AMIsReady
 - lastTransitionTime: "2024-02-02T19:54:34Z"
 status: "True"
 type: Ready

Refer to the NodePool docs for settings applicable to all providers. To explore various EC2NodeClass configurations, refer to the examples provided in the Karpenter Github repository.

spec.kubelet

kubelet:
 podsPerCore: 2
 maxPods: 20
 systemReserved:
 cpu: 100m
 memory: 100Mi
 ephemeral-storage: 1Gi
 kubeReserved:
 cpu: 200m
 memory: 100Mi
 ephemeral-storage: 3Gi
 evictionHard:
 memory.available: 5%
 nodefs.available: 10%
 nodefs.inodesFree: 10%
 evictionSoft:
 memory.available: 500Mi
 nodefs.available: 15%
 nodefs.inodesFree: 15%
 evictionSoftGracePeriod:
 memory.available: 1m
 nodefs.available: 1m30s
 nodefs.inodesFree: 2m
 evictionMaxPodGracePeriod: 60
 imageGCHighThresholdPercent: 85
 imageGCLowThresholdPercent: 80
 cpuCFSQuota: true
 clusterDNS: ["10.0.1.100"]

Note

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
spec:
 amiSelectorTerms:
 - alias: al2023@v20240807
 kubelet:
 maxPods: 42
 userData: |
 apiVersion: node.eks.aws/v1alpha1
 kind: NodeConfig
 spec:
 kubelet:
 config:
 # Configured through UserData since unavailable in `spec.kubelet`
 registryPullQPS: 10

Note that when using the Custom AMIFamily you will need to specify fields both in spec.kubelet and spec.userData.

Pods Per Core

Note

Max Pods

Note

Windows Support Notice

Reserved Resources

Note

Eviction Thresholds

kubelet:
 evictionHard:
 memory.available: 500Mi
 nodefs.available: 10%
 nodefs.inodesFree: 10%
 imagefs.available: 5%
 imagefs.inodesFree: 5%
 pid.available: 7%
 evictionSoft:
 memory.available: 1Gi
 nodefs.available: 15%
 nodefs.inodesFree: 15%
 imagefs.available: 10%
 imagefs.inodesFree: 10%
 pid.available: 10%

Supported Eviction Signals

Eviction Signal	Description
memory.available	memory.available := node.status.capacity[memory] - node.stats.memory.workingSet
nodefs.available	nodefs.available := node.stats.fs.available
nodefs.inodesFree	nodefs.inodesFree := node.stats.fs.inodesFree
imagefs.available	imagefs.available := node.stats.runtime.imagefs.available
imagefs.inodesFree	imagefs.inodesFree := node.stats.runtime.imagefs.inodesFree
pid.available	pid.available := node.stats.rlimit.maxpid - node.stats.rlimit.curproc

For more information on eviction thresholds, view the Node-pressure Eviction section of the official Kubernetes docs.

Soft Eviction Grace Periods

kubelet:
 evictionSoftGracePeriod:
 memory.available: 1m
 nodefs.available: 1m30s
 nodefs.inodesFree: 2m
 imagefs.available: 1m30s
 imagefs.inodesFree: 2m
 pid.available: 2m
 evictionMaxPodGracePeriod: 60

Pod Density

Note

Windows Support Notice

It’s currently not possible to specify custom networking with Windows nodes.

spec.amiFamily

Ubuntu Support Dropped at v1

Ubuntu NodeClass Example:

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
spec:
 amiFamily: AL2
 amiSelectorTerms:
 - id: ami-placeholder
 blockDeviceMappings:
 - deviceName: '/dev/sda1'
 rootVolume: true
 ebs:
 encrypted: true
 volumeType: gp3
 volumeSize: 20Gi

AL2

AL2 support dropped at Kubernetes 1.33

Kubernetes version 1.32 is the last version for which Amazon EKS will release Amazon Linux 2 (AL2) AMIs.

Note

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"

--//
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash -xe
exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1
/etc/eks/bootstrap.sh 'test-cluster' --apiserver-endpoint 'https://test-cluster' --b64-cluster-ca 'ca-bundle' \
--dns-cluster-ip '10.100.0.10' \
--use-max-pods false \
--kubelet-extra-args '--node-labels=karpenter.sh/capacity-type=on-demand,karpenter.sh/nodepool=test --max-pods=110'
--//--

AL2023

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"

--//
Content-Type: application/node.eks.aws

# Karpenter Generated NodeConfig
apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
 cluster:
 name: test-cluster
 apiServerEndpoint: https://example.com
 certificateAuthority: ca-bundle
 cidr: 10.100.0.0/16
 kubelet:
 config:
 maxPods: 110
 flags:
 - --node-labels=karpenter.sh/capacity-type=on-demand,karpenter.sh/nodepool=test

--//--

Bottlerocket

[settings]
[settings.kubernetes]
api-server = 'https://test-cluster'
cluster-certificate = 'ca-bundle'
cluster-name = 'test-cluster'
cluster-dns-ip = '10.100.0.10'
max-pods = 110

[settings.kubernetes.node-labels]
'karpenter.sh/capacity-type' = 'on-demand'
'karpenter.sh/nodepool' = 'test'

Windows2019

<powershell>
[string]$EKSBootstrapScriptFile = "$env:ProgramFiles\Amazon\EKS\Start-EKSBootstrap.ps1"
& $EKSBootstrapScriptFile -EKSClusterName 'test-cluster' -APIServerEndpoint 'https://test-cluster' -Base64ClusterCA 'ca-bundle' -KubeletExtraArgs '--node-labels="karpenter.sh/capacity-type=on-demand,karpenter.sh/nodepool=test" --max-pods=110' -DNSClusterIP '10.100.0.10'
</powershell>

Windows2022

<powershell>
[string]$EKSBootstrapScriptFile = "$env:ProgramFiles\Amazon\EKS\Start-EKSBootstrap.ps1"
& $EKSBootstrapScriptFile -EKSClusterName 'test-cluster' -APIServerEndpoint 'https://test-cluster' -Base64ClusterCA 'ca-bundle' -KubeletExtraArgs '--node-labels="karpenter.sh/capacity-type=on-demand,karpenter.sh/nodepool=test" --max-pods=110' -DNSClusterIP '10.100.0.10'
</powershell>

Windows2025

<powershell>
[string]$EKSBootstrapScriptFile = "$env:ProgramFiles\Amazon\EKS\Start-EKSBootstrap.ps1"
& $EKSBootstrapScriptFile -EKSClusterName 'test-cluster' -APIServerEndpoint 'https://test-cluster' -Base64ClusterCA 'ca-bundle' -KubeletExtraArgs '--node-labels="karpenter.sh/capacity-type=on-demand,karpenter.sh/nodepool=test" --max-pods=110' -DNSClusterIP '10.100.0.10'
</powershell>

Custom

spec.subnetSelectorTerms

subnetSelectorTerms:
 # Select on any subnet that has the "karpenter.sh/discovery: ${CLUSTER_NAME}"
 # AND the "environment: test" tag OR any subnet with ID "subnet-09fa4a0a8f233a921"
 - tags:
 karpenter.sh/discovery: "${CLUSTER_NAME}"
 environment: test
 - id: subnet-09fa4a0a8f233a921

Tip

Subnets may be specified by any tag, including Name. Selecting tag values using wildcards (*) is supported.

Examples

Select all with a specified tag key:

spec:
 subnetSelectorTerms:
 - tags:
 karpenter.sh/discovery/MyClusterName: '*'

Select by name and tag (all criteria must match):

spec:
 subnetSelectorTerms:
 - tags:
 Name: my-subnet
 MyTag: '' # matches all resources with the tag

Select using multiple tag terms:

spec:
 subnetSelectorTerms:
 - tags:
 Name: "my-subnet-1"
 - tags:
 Name: "my-subnet-2"

Select using wildcards:

spec:
 subnetSelectorTerms:
 - tags:
 Name: "*Public*"

Select using ids:

spec:
 subnetSelectorTerms:
 - id: "subnet-09fa4a0a8f233a921"
 - id: "subnet-0471ca205b8a129ae"

spec.securityGroupSelectorTerms

This selection logic is modeled as terms, where each term contains multiple conditions that must all be satisfied for the selector to match. Effectively, all requirements within a single term are ANDed together. It’s possible that you may want to select on two different security groups that have unrelated requirements. In this case, you can specify multiple terms which will be ORed together to form your selection logic. The example below shows how this selection logic is fulfilled.

securityGroupSelectorTerms:
 # Select on any security group that has both the "karpenter.sh/discovery: ${CLUSTER_NAME}" tag
 # AND the "environment: test" tag OR any security group with the "my-security-group" name
 # OR any security group with ID "sg-063d7acfb4b06c82c"
 - tags:
 karpenter.sh/discovery: "${CLUSTER_NAME}"
 environment: test
 - name: my-security-group
 - id: sg-063d7acfb4b06c82c

Tip

Security groups may be specified by any tag, including “Name”. Selecting tags using wildcards (*) is supported.

Note

To verify if this restriction affects you, run the following commands.

CLUSTER_VPC_ID="$(aws eks describe-cluster --name $CLUSTER_NAME --query cluster.resourcesVpcConfig.vpcId --output text)"

aws ec2 describe-security-groups --filters Name=vpc-id,Values=$CLUSTER_VPC_ID Name=tag-key,Values=kubernetes.io/cluster/$CLUSTER_NAME --query 'SecurityGroups[].[GroupName]' --output text

If multiple securityGroups are printed, you will need more specific securityGroupSelectorTerms. We generally recommend that you use the karpenter.sh/discovery: $CLUSTER_NAME tag selector instead.

Examples

Select all assigned to a cluster:

spec:
 securityGroupSelectorTerms:
 - tags:
 kubernetes.io/cluster/$CLUSTER_NAME: "owned"

Select all with a specified tag key:

spec:
 securityGroupSelectorTerms:
 - tags:
 MyTag: '*'

Select by name and tag (all criteria must match):

spec:
 securityGroupSelectorTerms:
 - name: my-security-group
 tags:
 MyTag: '*' # matches all resources with the tag

Select using multiple tag terms:

spec:
 securityGroupSelectorTerms:
 - tags:
 Name: "my-security-group-1"
 - tags:
 Name: "my-security-group-2"

Select by name using a wildcard:

spec:
 securityGroupSelectorTerms:
 - name: "*Public*"

Select using ids:

spec:
 securityGroupSelectorTerms:
 - id: "sg-063d7acfb4b06c82c"
 - id: "sg-06e0cf9c198874591"

spec.role

spec:
 role: "KarpenterNodeRole-$CLUSTER_NAME"

spec.instanceProfile

You can provision and assign a role to an IAM instance profile using CloudFormation or by using the aws iam create-instance-profile and aws iam add-role-to-instance-profile commands in the CLI.

Note

For private clusters without access to their AWS region’s IAM API endpoint, using spec.instanceProfile is required. spec.role cannot be used since Karpenter needs to access IAM endpoints to manage a generated instance profile. IAM doesn’t support private endpoints to enable accessing the service without going to the public internet.

Warning

When using spec.instanceProfile, ensure you are using pre-provisioned instance profiles that you manage yourself.

spec.amiSelectorTerms

AMI Selector Terms are required and are used to configure AMIs for Karpenter to use. AMIs are discovered through alias, id, owner, name, and tags.

This selection logic is modeled as terms, where each term contains multiple conditions that must all be satisfied for the selector to match. Effectively, all requirements within a single term are ANDed together. It’s possible that you may want to select on two different AMIs that have unrelated requirements. In this case, you can specify multiple terms which will be ORed together to form your selection logic. The example below shows how this selection logic is fulfilled.

amiSelectorTerms:
 # Select on any AMI that has both the `karpenter.sh/discovery: ${CLUSTER_NAME}`
 # AND `environment: test` tags OR any AMI with the name `my-ami` OR an AMI with
 # ID `ami-123` OR an AMI with ID matching the value of my-custom-parameter
 - tags:
 karpenter.sh/discovery: "${CLUSTER_NAME}"
 environment: test
 - name: my-ami
 - id: ami-123
 - ssmParameter: my-custom-parameter # ssm parameter name or ARN
 # Select EKS optimized AL2023 AMIs with version `v20240807`. This term is mutually
 # exclusive and can't be specified with other terms.
 # - alias: al2023@v20240807

An alias term can be used to select EKS-optimized AMIs. An alias is formatted as family@version. Family can be one of the following values:

al2
al2023
bottlerocket
windows2019
windows2022
windows2025

alias: al2023@v20240703

Bottlerocket uses a semantic version for their releases. You can pin bottlerocket as follows:

alias: bottlerocket@v1.20.4

The Windows family does not support pinning, so only latest is supported.

The following commands can be used to determine the versions availble for an alias in your region:

export K8S_VERSION="1.35"
aws ssm get-parameters-by-path --path "/aws/service/eks/optimized-ami/$K8S_VERSION/amazon-linux-2023/" --recursive | jq -cr '.Parameters[].Name' | grep -v "recommended" | awk -F '/' '{print $10}' | sed -r 's/.*(v[[:digit:]]+)$/\1/' | sort | uniq

export K8S_VERSION="1.35"
aws ssm get-parameters-by-path --path "/aws/service/eks/optimized-ami/$K8S_VERSION/amazon-linux-2/" --recursive | jq -cr '.Parameters[].Name' | grep -v "recommended" | awk -F '/' '{print $8}' | sed -r 's/.*(v[[:digit:]]+)$/\1/' | sort | uniq

export K8S_VERSION="1.35"
aws ssm get-parameters-by-path --path "/aws/service/bottlerocket/aws-k8s-$K8S_VERSION" --recursive | jq -cr '.Parameters[].Name' | grep -v "latest" | awk -F '/' '{print $7}' | sort | uniq

Warning

Tip

AMIs may be specified by any AWS tag, including Name. Selecting by tag or by name using wildcards (*) is supported.

Note

If amiSelectorTerms match more than one AMI, Karpenter will automatically determine which AMI best fits the workloads on the launched worker node under the following constraints:

When launching nodes, Karpenter automatically determines which architecture a custom AMI is compatible with and will use images that match an instanceType’s requirements.
- Unless using an alias, Karpenter cannot detect requirements other than architecture. If you need to specify different AMIs for different kind of nodes (e.g. accelerated GPU AMIs), you should use a separate EC2NodeClass.
If multiple AMIs are found that can be used, Karpenter will choose the latest one.
If no AMIs are found that can be used, then no nodes will be provisioned.

Examples

Select by AMI family and version:

 amiSelectorTerms:
 - alias: al2023@v20240807

Select all with a specified tag:

 amiSelectorTerms:
 - tags:
 karpenter.sh/discovery/MyClusterName: '*'

Select by name:

 amiSelectorTerms:
 - name: my-ami

Select by Name tag:

 amiSelectorTerms:
 - tags:
 Name: my-ami

Select by name and owner:

 amiSelectorTerms:
 - name: my-ami
 owner: self
 - name: my-ami
 owner: "0123456789"

Select by name using a wildcard:

spec:
 amiSelectorTerms:
 - name: "*EKS*"

Select by all under an owner:

spec:
 amiSelectorTerms:
 - name: "*"
 owner: self

Specify using ids:

 amiSelectorTerms:
 - id: "ami-123"
 - id: "ami-456"

Specify using custom ssm parameter name or ARN:

 amiSelectorTerms:
 - ssmParameter: "my-custom-parameter"

Note

spec.capacityReservationSelectorTerms

Feature State: Beta

For more information on utilizing ODCRs with Karpenter, refer to the Utilizing ODCRs Task.

Note

Examples

Select the reservations with the given IDs:

spec:
 capacityReservationSelectorTerms:
 - id: cr-123
 - id: cr-456

Select the reservations by tags:

spec:
 capacityReservationSelectorTerms:
 # Select all capacity reservations which have both matching tags
 - tags:
 key1: foo
 key2: bar
 # Additionally, select all capacity reservations with the following matching tag
 - tags:
 key3: foobar

Select by tags and owner ID:

spec:
 # Select all capacity reservations with the matching tags which are also owned by
 # the specified account.
 capacityReservationSelectorTerms:
 - tags:
 key: foo
 ownerID: 012345678901

Select by instance match criteria:

spec:
 capacityReservationSelectorTerms:
 # Select all open capacity reservations
 - instanceMatchCriteria: open

Select by instance match criteria and tags:

spec:
 capacityReservationSelectorTerms:
 # Select targeted capacity reservations with matching tags
 - instanceMatchCriteria: targeted
 tags:
 key: foo

spec.tags

Karpenter adds tags to all resources it creates, including EC2 Instances, EBS volumes, and Launch Templates. The default set of tags are listed below.

Name: <node-name>
karpenter.sh/nodeclaim: <nodeclaim-name>
karpenter.sh/nodepool: <nodepool-name>
karpenter.k8s.aws/ec2nodeclass: <ec2nodeclass-name>
kubernetes.io/cluster/<cluster-name>: owned
eks:eks-cluster-name: <cluster-name>

Additional tags can be added in the tags section, which will be merged with the default tags specified above.

spec:
 tags:
 InternalAccountingTag: 1234
 dev.corp.net/app: Calculator
 dev.corp.net/team: MyTeam

Note

spec.metadataOptions

Control the exposure of Instance Metadata Service on EC2 Instances launched by this EC2NodeClass using a generated launch template.

Refer to recommended, security best practices for limiting exposure of Instance Metadata and User Data to pods.

If metadataOptions are omitted from this EC2NodeClass, the following default settings are applied:

spec:
 metadataOptions:
 httpEndpoint: enabled
 httpProtocolIPv6: disabled
 httpPutResponseHopLimit: 1
 httpTokens: required

spec.blockDeviceMappings

spec:
 blockDeviceMappings:
 - deviceName: /dev/xvda
 ebs:
 volumeSize: 100Gi
 volumeType: gp3
 iops: 10000
 encrypted: true
 kmsKeyID: "1234abcd-12ab-34cd-56ef-1234567890ab"
 deleteOnTermination: true
 throughput: 125
 snapshotID: snap-0123456789
 volumeInitializationRate: 100

The following blockDeviceMapping defaults are used for each AMIFamily if no blockDeviceMapping overrides are specified in the EC2NodeClass

AL2

spec:
 blockDeviceMappings:
 - deviceName: /dev/xvda
 ebs:
 volumeSize: 20Gi
 volumeType: gp3
 encrypted: true

AL2023

spec:
 blockDeviceMappings:
 - deviceName: /dev/xvda
 ebs:
 volumeSize: 20Gi
 volumeType: gp3
 encrypted: true

Bottlerocket

spec:
 blockDeviceMappings:
 # Root device
 - deviceName: /dev/xvda
 ebs:
 volumeSize: 4Gi
 volumeType: gp3
 encrypted: true
 # Data device: Container resources such as images and logs
 - deviceName: /dev/xvdb
 ebs:
 volumeSize: 20Gi
 volumeType: gp3
 encrypted: true

Windows2019/Windows2022/Windows2025

spec:
 blockDeviceMappings:
 - deviceName: /dev/sda1
 ebs:
 volumeSize: 50Gi
 volumeType: gp3
 encrypted: true

Custom

The Custom AMIFamily ships without any default blockDeviceMappings.

spec.networkInterfaces

The networkInterfaces field allows you to configure network interface attachments for instances, including support for EFA (Elastic Fabric Adapter) devices for high-performance computing and machine learning workloads. For more information see the AWS EFA docs.

Configure network interfaces by specifying the network card index, device index, and interface type:

spec:
 networkInterfaces:
 - networkCardIndex: 0
 deviceIndex: 0
 interfaceType: "interface"
 - networkCardIndex: 0
 deviceIndex: 1
 interfaceType: "efa-only"

Interface Types

interface: Standard ENA (Elastic Network Adapter) interface providing IP connectivity
efa-only: EFA interface that provides only the EFA device for RDMA communication without consuming an IP address

spec.instanceStorePolicy

The instanceStorePolicy field controls how instance-store volumes are handled. By default, Karpenter and Kubernetes will simply ignore them.

RAID0

If you intend to use these volumes for faster node ephemeral-storage, set instanceStorePolicy to RAID0:

spec:
 instanceStorePolicy: RAID0

Instructions for each AMI family are listed below:

AL2

AL2023

Bottlerocket

Note

This automatic disk configuration is only available on Bottlerocket v1.22.0 and later.

Others

For all other AMI families, you must configure the disks yourself. Check out the setup-local-disks script in amazon-eks-ami to see how this is done for AL2.

Tip

Since the Kubelet & Containerd will be using the instance-store filesystem, you may consider using a more minimal root volume size.

spec.userData

You can control the UserData that is applied to your worker nodes via this field. This allows you to run custom scripts or pass-through custom configuration to Karpenter instances on start-up.

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
 name: bottlerocket-example
spec:
 ...
 amiFamily: Bottlerocket
 userData: |
 [settings.kubernetes]
 "kube-api-qps" = 30
 "shutdown-grace-period" = "30s"
 "shutdown-grace-period-for-critical-pods" = "30s"
 [settings.kubernetes.eviction-hard]
 "memory.available" = "20%"

This example adds SSH keys to allow remote login to the node (replace my-authorized_keys with your public key file):

Note

Also, my-authorized_key key is the public key. See Retrieve the public key material.

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
 name: al2-example
spec:
 ...
 amiFamily: AL2
 userData: |
 #!/bin/bash
 mkdir -p ~ec2-user/.ssh/
 touch ~ec2-user/.ssh/authorized_keys
 cat >> ~ec2-user/.ssh/authorized_keys <<EOF
 {{ insertFile "../my-authorized_keys" | indent 4 }}
 EOF
 chmod -R go-w ~ec2-user/.ssh/authorized_keys
 chown -R ec2-user ~ec2-user/.ssh

Alternatively, you can save the key in your SSM Parameter Store and use the get-parameter command mentioned below to retrieve the key for authorized_keys.

aws ssm get-parameter --name "<parameter-name>" --region <region> --with-decryption --query "Parameter.Value" --output text > /home/ec2-user/.ssh/authorized_keys

For more examples on configuring fields for different AMI families, see the examples here.

AL2

Your UserData can be in the MIME multi part archive format.
Karpenter will transform your custom user-data as a MIME part, if necessary, and then merge a final MIME part to the end of your UserData parts which will bootstrap the worker node. Karpenter will have full control over all the parameters being passed to the bootstrap script.
- Karpenter will continue to set MaxPods, ClusterDNS and all other parameters defined in spec.kubeletConfiguration as before.

Consider the following example to understand how your custom UserData will be merged -

Passed-in UserData (bash)

#!/bin/bash
echo "Running custom user data script (bash)"

Merged UserData (bash)

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"

--//
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash
echo "Running custom user data script (bash)"

--//
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash -xe
exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1
/etc/eks/bootstrap.sh 'test-cluster' --apiserver-endpoint 'https://test-cluster' --b64-cluster-ca 'ca-bundle' \
--use-max-pods false \
--kubelet-extra-args '--node-labels=karpenter.sh/capacity-type=on-demand,karpenter.sh/nodepool=test --max-pods=110'
--//--

Passed-in UserData (MIME)

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="BOUNDARY"

--BOUNDARY
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash
echo "Running custom user data script (mime)"

--BOUNDARY--

Merged UserData (MIME)

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"

--//
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash
echo "Running custom user data script (mime)"

--//
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash -xe
exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1
/etc/eks/bootstrap.sh 'test-cluster' --apiserver-endpoint 'https://test-cluster' --b64-cluster-ca 'ca-bundle' \
--use-max-pods false \
--kubelet-extra-args '--node-labels=karpenter.sh/capacity-type=on-demand,karpenter.sh/nodepool=test --max-pods=110'
--//--

Tip

You can set additional kubelet configuration properties, unavailable through spec.kubelet, by updating the kubelet-config.json file:

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
 name: kubelet-config-example
spec:
 amiFamily: AL2
 userData: |
 #!/bin/bash
 echo "$(jq '.kubeAPIQPS=50' /etc/kubernetes/kubelet/kubelet-config.json)" > /etc/kubernetes/kubelet/kubelet-config.json

AL2023

Your UserData may be in one of three formats: a MIME multi part archive, a NodeConfig YAML / JSON string, or a shell script.
Karpenter will transform your custom UserData into a MIME part, if necessary, and then create a MIME multi-part archive. This archive will consist of a generated NodeConfig, containing Karpenter’s default values, followed by the transformed custom UserData. For more information on the NodeConfig spec, refer to the AL2023 EKS Optimized AMI docs.

Warning

Passed-in UserData (NodeConfig)

apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
 kubelet:
 config:
 maxPods: 42

Merged UserData (NodeConfig)

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"

--//
Content-Type: application/node.eks.aws

apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
 kubelet:
 config:
 maxPods: 42

--//
Content-Type: application/node.eks.aws

# Karpenter Generated NodeConfig
apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
 cluster:
 apiServerEndpoint: https://test-cluster
 certificateAuthority: cluster-ca
 cidr: 10.100.0.0/16
 name: test-cluster
 kubelet:
 config:
 clusterDNS:
 - 10.100.0.10
 maxPods: 118
 flags:
 - --node-labels="karpenter.sh/capacity-type=on-demand,karpenter.sh/nodepool=default"

--//--

Passed-in UserData (bash)

#!/bin/bash
echo "Hello, AL2023!"

Merged UserData (bash)

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"

--//
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash
echo "Hello, AL2023!"

--//
Content-Type: application/node.eks.aws

# Karpenter Generated NodeConfig
apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
 cluster:
 apiServerEndpoint: https://test-cluster
 certificateAuthority: cluster-ca
 cidr: 10.100.0.0/16
 name: test-cluster
 kubelet:
 config:
 clusterDNS:
 - 10.100.0.10
 maxPods: 118
 flags:
 - --node-labels="karpenter.sh/capacity-type=on-demand,karpenter.sh/nodepool=default"

--//--

Passed-in UserData (MIME)

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"

--//
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash
echo "Hello, AL2023!"

--//
Content-Type: application/node.eks.aws

apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
 kubelet:
 config:
 maxPods: 42
--//

Merged UserData (MIME)

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"

--//
Content-Type: application/node.eks.aws

apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
 kubelet:
 config:
 maxPods: 42
--//
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash
echo "Hello, AL2023!"

--//
Content-Type: application/node.eks.aws

# Karpenter Generated NodeConfig
apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
 cluster:
 apiServerEndpoint: https://test-cluster
 certificateAuthority: cluster-ca
 cidr: 10.100.0.0/16
 name: test-cluster
 kubelet:
 config:
 clusterDNS:
 - 10.100.0.10
 maxPods: 118
 flags:
 - --node-labels="karpenter.sh/capacity-type=on-demand,karpenter.sh/nodepool=default"

--//--

Bottlerocket

Your UserData must be valid TOML.
Unknown TOML fields will be ignored when the final merged UserData is generated by Karpenter.

Warning

Consider the following example to understand how your custom UserData settings will be merged in.

Passed-in UserData

[settings.kubernetes.eviction-hard]
"memory.available" = "12%"
[settings.kubernetes]
"unknown-setting" = "unknown"
[settings.kubernetes.node-labels]
'field.controlled.by/karpenter' = 'will-be-overridden'

Merged UserData

[settings]
[settings.kubernetes]
api-server = 'https://cluster'
cluster-certificate = 'ca-bundle'
cluster-name = 'cluster'

[settings.kubernetes.node-labels]
'karpenter.sh/capacity-type' = 'on-demand'
'karpenter.sh/nodepool' = 'default'

[settings.kubernetes.node-taints]

[settings.kubernetes.eviction-hard]
'memory.available' = '12%%'

Device ownership in Bottlerocket

Bottlerocket v1.30.0+ supports device ownership using the security context provided in the Kubernetes specfile. To enable this, you will need the following user-data configurations:

[settings]
[settings.kubernetes]
device-ownership-from-security-context = true

This setting helps you enable Neuron workloads on Bottlerocket instances. See Accelerators/GPU Resources for more details.

Windows2019/Windows2022/Windows2025

Your UserData must be specified as PowerShell commands.
The UserData specified will be prepended to a Karpenter managed section that will bootstrap the kubelet.
Karpenter will continue to set ClusterDNS and all other parameters defined in spec.kubeletConfiguration as before.

Consider the following example to understand how your custom UserData settings will be merged in.

Passed-in UserData

Write-Host "Running custom user data script"

Merged UserData

<powershell>
Write-Host "Running custom user data script"
[string]$EKSBootstrapScriptFile = "$env:ProgramFiles\Amazon\EKS\Start-EKSBootstrap.ps1"
& $EKSBootstrapScriptFile -EKSClusterName 'test-cluster' -APIServerEndpoint 'https://test-cluster' -Base64ClusterCA 'ca-bundle' -KubeletExtraArgs '--node-labels="karpenter.sh/capacity-type=spot,karpenter.sh/nodepool=windows2022" --max-pods=110' -DNSClusterIP '10.0.100.10'
</powershell>

Windows Support Notice

spec:
 userData: |
 $global:EKSCluster = Get-EKSCluster -Name my-cluster

Custom

No merging is performed, your UserData must perform all setup required of the node to allow it to join the cluster.
Custom UserData must meet the following requirements to work correctly with Karpenter:
- It must ensure the node is registered with the karpenter.sh/unregistered:NoExecute taint (via kubelet configuration field registerWithTaints)
- It must set kubelet config options to match those configured in spec.kubelet

spec.detailedMonitoring

spec:
 detailedMonitoring: true

spec.associatePublicIPAddress

Note

spec.ipPrefixCount

status.subnets

Examples

spec:
 subnetSelectorTerms:
 - tags:
 karpenter.sh/discovery: "${CLUSTER_NAME}"
status:
 subnets:
 - id: subnet-0a462d98193ff9fac
 zone: us-east-2b
 - id: subnet-0322dfafd76a609b6
 zone: us-east-2c
 - id: subnet-0727ef01daf4ac9fe
 zone: us-east-2b
 - id: subnet-00c99aeafe2a70304
 zone: us-east-2a
 - id: subnet-023b232fd5eb0028e
 zone: us-east-2c
 - id: subnet-03941e7ad6afeaa72
 zone: us-east-2a

status.securityGroups

Examples

spec:
 securityGroupSelectorTerms:
 - tags:
 karpenter.sh/discovery: "${CLUSTER_NAME}"
status:
 securityGroups:
 - id: sg-041513b454818610b
 name: ClusterSharedNodeSecurityGroup
 - id: sg-0286715698b894bca
 name: ControlPlaneSecurityGroup-1AQ073TSAAPW

status.amis

Examples

AMIs resolved with an AL2 alias:

spec:
 amiSelectorTerms:
 - alias: al2@v20240807
status:
 amis:
 - id: ami-03c3a3dcda64f5b75
 name: amazon-linux-2-gpu
 requirements:
 - key: kubernetes.io/arch
 operator: In
 values:
 - amd64
 - key: karpenter.k8s.aws/instance-gpu-count
 operator: Exists
 - id: ami-03c3a3dcda64f5b75
 name: amazon-linux-2-gpu
 requirements:
 - key: kubernetes.io/arch
 operator: In
 values:
 - amd64
 - key: karpenter.k8s.aws/instance-accelerator-count
 operator: Exists
 - id: ami-06afb2d101cc4b8bd
 name: amazon-linux-2-arm64
 requirements:
 - key: kubernetes.io/arch
 operator: In
 values:
 - arm64
 - key: karpenter.k8s.aws/instance-gpu-count
 operator: DoesNotExist
 - key: karpenter.k8s.aws/instance-accelerator-count
 operator: DoesNotExist
 - id: ami-0e28b76d768af234e
 name: amazon-linux-2
 requirements:
 - key: kubernetes.io/arch
 operator: In
 values:
 - amd64
 - key: karpenter.k8s.aws/instance-gpu-count
 operator: DoesNotExist
 - key: karpenter.k8s.aws/instance-accelerator-count
 operator: DoesNotExist

AMIs resolved from tags:

spec:
 amiSelectorTerms:
 - tags:
 karpenter.sh/discovery: "${CLUSTER_NAME}"
status:
 amis:
 - id: ami-01234567890123456
 name: custom-ami-amd64
 deprecated: true
 requirements:
 - key: kubernetes.io/arch
 operator: In
 values:
 - amd64
 - id: ami-01234567890123456
 name: custom-ami-arm64
 requirements:
 - key: kubernetes.io/arch
 operator: In
 values:
 - arm64

status.capacityReservations

status.capacityReservations contains the following information for each resolved capacity reservation:

Field	Example	Description
`availabilityZone`	`us-east-1a`	The availability zone the capacity reservation is available in
`id`	`cr-56fac701cc1951b03`	The ID of the capacity reservation
`instanceMatchCriteria`	`open`	The instanceMatchCriteria for the capacity reservation. Can be `open` or `targeted`.
`instanceType`	`m5.large`	The EC2 instance type of the capacity reservation
`ownerID`	`459763720645`	The account ID that owns the capacity reservation
`reservationType`	`default`	The type of the capacity reservation. Can be `default` or `capacity-block`.
`state`	`active`	The state of the capacity reservation. Can be `active` or `expiring`.
`interruptible`	`true` or `false`	Whether the capacity reservation is interruptible.

Examples

status:
 capacityReservations:
 - availabilityZone: us-west-2a
 id: cr-01234567890123456
 instanceMatchCriteria: targeted
 instanceType: g6.48xlarge
 interruptible: false
 ownerID: "012345678901"
 reservationType: capacity-block
 state: expiring
 - availabilityZone: us-west-2c
 id: cr-12345678901234567
 instanceMatchCriteria: open
 instanceType: g6.48xlarge
 interruptible: true
 ownerID: "98765432109"
 reservationType: default
 state: active

status.instanceProfile

status.instanceProfile contains the resolved instance profile generated by Karpenter from the spec.role

spec:
 role: "KarpenterNodeRole-${CLUSTER_NAME}"
status:
 instanceProfile: "${CLUSTER_NAME}-0123456778901234567789"

status.conditions

NodeClasses have the following status conditions:

Condition Type	Description
SubnetsReady	Subnets are discovered.
SecurityGroupsReady	Security Groups are discovered.
InstanceProfileReady	Instance Profile is discovered.
AMIsReady	AMIs are discovered.
Ready	Top level condition that indicates if the nodeClass is ready. If any of the underlying conditions is `False` then this condition is set to `False` and `Message` on the condition indicates the dependency that was not resolved.

If a NodeClass is not ready, NodePools that reference it through their nodeClassRef will not be considered for scheduling.

V1.0: NodeClasses

Mon, 01 Jan 0001 00:00:00 +0000

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
 name: default
spec:
 template:
 spec:
 nodeClassRef:
 group: karpenter.k8s.aws
 kind: EC2NodeClass
 name: default
---
apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
 name: default
spec:
 kubelet:
 podsPerCore: 2
 maxPods: 20
 systemReserved:
 cpu: 100m
 memory: 100Mi
 ephemeral-storage: 1Gi
 kubeReserved:
 cpu: 200m
 memory: 100Mi
 ephemeral-storage: 3Gi
 evictionHard:
 memory.available: 5%
 nodefs.available: 10%
 nodefs.inodesFree: 10%
 evictionSoft:
 memory.available: 500Mi
 nodefs.available: 15%
 nodefs.inodesFree: 15%
 evictionSoftGracePeriod:
 memory.available: 1m
 nodefs.available: 1m30s
 nodefs.inodesFree: 2m
 evictionMaxPodGracePeriod: 60
 imageGCHighThresholdPercent: 85
 imageGCLowThresholdPercent: 80
 cpuCFSQuota: true
 clusterDNS: ["10.0.1.100"]
 # Optional, dictates UserData generation and default block device mappings.
 # May be ommited when using an `alias` amiSelectorTerm, otherwise required.
 amiFamily: AL2

 # Required, discovers subnets to attach to instances
 # Each term in the array of subnetSelectorTerms is ORed together
 # Within a single term, all conditions are ANDed
 subnetSelectorTerms:
 # Select on any subnet that has the "karpenter.sh/discovery: ${CLUSTER_NAME}"
 # AND the "environment: test" tag OR any subnet with ID "subnet-09fa4a0a8f233a921"
 - tags:
 karpenter.sh/discovery: "${CLUSTER_NAME}"
 environment: test
 - id: subnet-09fa4a0a8f233a921

 # Required, discovers security groups to attach to instances
 # Each term in the array of securityGroupSelectorTerms is ORed together
 # Within a single term, all conditions are ANDed
 securityGroupSelectorTerms:
 # Select on any security group that has both the "karpenter.sh/discovery: ${CLUSTER_NAME}" tag
 # AND the "environment: test" tag OR any security group with the "my-security-group" name
 # OR any security group with ID "sg-063d7acfb4b06c82c"
 - tags:
 karpenter.sh/discovery: "${CLUSTER_NAME}"
 environment: test
 - name: my-security-group
 - id: sg-063d7acfb4b06c82c

 # Optional, IAM role to use for the node identity.
 # The "role" field is immutable after EC2NodeClass creation. This may change in the
 # future, but this restriction is currently in place today to ensure that Karpenter
 # avoids leaking managed instance profiles in your account.
 # Must specify one of "role" or "instanceProfile" for Karpenter to launch nodes
 role: "KarpenterNodeRole-${CLUSTER_NAME}"

 # Optional, IAM instance profile to use for the node identity.
 # Must specify one of "role" or "instanceProfile" for Karpenter to launch nodes
 instanceProfile: "KarpenterNodeInstanceProfile-${CLUSTER_NAME}"

 # Each term in the array of amiSelectorTerms is ORed together
 # Within a single term, all conditions are ANDed
 amiSelectorTerms:
 # Select on any AMI that has both the `karpenter.sh/discovery: ${CLUSTER_NAME}`
 # AND `environment: test` tags OR any AMI with the name `my-ami` OR an AMI with
 # ID `ami-123`
 - tags:
 karpenter.sh/discovery: "${CLUSTER_NAME}"
 environment: test
 - name: my-ami
 - id: ami-123
 # Select EKS optimized AL2023 AMIs with version `v20240703`. This term is mutually
 # exclusive and can't be specified with other terms.
 # - alias: al2023@v20240703

 # Optional, propagates tags to underlying EC2 resources
 tags:
 team: team-a
 app: team-a-app

 # Optional, configures IMDS for the instance
 metadataOptions:
 httpEndpoint: enabled
 httpProtocolIPv6: disabled
 httpPutResponseHopLimit: 1 # This is changed to disable IMDS access from containers not on the host network
 httpTokens: required

 # Optional, configures storage devices for the instance
 blockDeviceMappings:
 - deviceName: /dev/xvda
 ebs:
 volumeSize: 100Gi
 volumeType: gp3
 iops: 10000
 encrypted: true
 kmsKeyID: "1234abcd-12ab-34cd-56ef-1234567890ab"
 deleteOnTermination: true
 throughput: 125
 snapshotID: snap-0123456789

 # Optional, use instance-store volumes for node ephemeral-storage
 instanceStorePolicy: RAID0

 # Optional, overrides autogenerated userdata with a merge semantic
 userData: |
 echo "Hello world"

 # Optional, configures detailed monitoring for the instance
 detailedMonitoring: true

 # Optional, configures if the instance should be launched with an associated public IP address.
 # If not specified, the default value depends on the subnet's public IP auto-assign setting.
 associatePublicIPAddress: true
status:
 # Resolved subnets
 subnets:
 - id: subnet-0a462d98193ff9fac
 zone: us-east-2b
 - id: subnet-0322dfafd76a609b6
 zone: us-east-2c
 - id: subnet-0727ef01daf4ac9fe
 zone: us-east-2b
 - id: subnet-00c99aeafe2a70304
 zone: us-east-2a
 - id: subnet-023b232fd5eb0028e
 zone: us-east-2c
 - id: subnet-03941e7ad6afeaa72
 zone: us-east-2a

 # Resolved security groups
 securityGroups:
 - id: sg-041513b454818610b
 name: ClusterSharedNodeSecurityGroup
 - id: sg-0286715698b894bca
 name: ControlPlaneSecurityGroup-1AQ073TSAAPW

 # Resolved AMIs
 amis:
 - id: ami-01234567890123456
 name: custom-ami-amd64
 requirements:
 - key: kubernetes.io/arch
 operator: In
 values:
 - amd64
 - id: ami-01234567890123456
 name: custom-ami-arm64
 requirements:
 - key: kubernetes.io/arch
 operator: In
 values:
 - arm64

 # Generated instance profile name from "role"
 instanceProfile: "${CLUSTER_NAME}-0123456778901234567789"
 conditions:
 - lastTransitionTime: "2024-02-02T19:54:34Z"
 status: "True"
 type: InstanceProfileReady
 - lastTransitionTime: "2024-02-02T19:54:34Z"
 status: "True"
 type: SubnetsReady
 - lastTransitionTime: "2024-02-02T19:54:34Z"
 status: "True"
 type: SecurityGroupsReady
 - lastTransitionTime: "2024-02-02T19:54:34Z"
 status: "True"
 type: AMIsReady
 - lastTransitionTime: "2024-02-02T19:54:34Z"
 status: "True"
 type: Ready

Refer to the NodePool docs for settings applicable to all providers. To explore various EC2NodeClass configurations, refer to the examples provided in the Karpenter Github repository.

spec.kubelet

kubelet:
 podsPerCore: 2
 maxPods: 20
 systemReserved:
 cpu: 100m
 memory: 100Mi
 ephemeral-storage: 1Gi
 kubeReserved:
 cpu: 200m
 memory: 100Mi
 ephemeral-storage: 3Gi
 evictionHard:
 memory.available: 5%
 nodefs.available: 10%
 nodefs.inodesFree: 10%
 evictionSoft:
 memory.available: 500Mi
 nodefs.available: 15%
 nodefs.inodesFree: 15%
 evictionSoftGracePeriod:
 memory.available: 1m
 nodefs.available: 1m30s
 nodefs.inodesFree: 2m
 evictionMaxPodGracePeriod: 60
 imageGCHighThresholdPercent: 85
 imageGCLowThresholdPercent: 80
 cpuCFSQuota: true
 clusterDNS: ["10.0.1.100"]

Note

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
spec:
 amiSelectorTerms:
 - alias: al2023@v20240807
 kubelet:
 maxPods: 42
 userData: |
 apiVersion: node.eks.aws/v1alpha1
 kind: NodeConfig
 spec:
 kubelet:
 config:
 # Configured through UserData since unavailable in `spec.kubelet`
 registryPullQPS: 10

Note that when using the Custom AMIFamily you will need to specify fields both in spec.kublet and spec.userData.

Warning

The Bottlerocket AMIFamily does not support the following fields:

evictionSoft
evictionSoftGracePeriod
evictionMaxPodGracePeriod

If any of these fields are specified on a Bottlerocket EC2NodeClass, they will be ommited from generated UserData and ignored for scheduling purposes. Support for these fields can be tracked via GitHub issue #3722.

Pods Per Core

Note

Max Pods

Note

Windows Support Notice

Reserved Resources

Note

Eviction Thresholds

kubelet:
 evictionHard:
 memory.available: 500Mi
 nodefs.available: 10%
 nodefs.inodesFree: 10%
 imagefs.available: 5%
 imagefs.inodesFree: 5%
 pid.available: 7%
 evictionSoft:
 memory.available: 1Gi
 nodefs.available: 15%
 nodefs.inodesFree: 15%
 imagefs.available: 10%
 imagefs.inodesFree: 10%
 pid.available: 10%

Supported Eviction Signals

Eviction Signal	Description
memory.available	memory.available := node.status.capacity[memory] - node.stats.memory.workingSet
nodefs.available	nodefs.available := node.stats.fs.available
nodefs.inodesFree	nodefs.inodesFree := node.stats.fs.inodesFree
imagefs.available	imagefs.available := node.stats.runtime.imagefs.available
imagefs.inodesFree	imagefs.inodesFree := node.stats.runtime.imagefs.inodesFree
pid.available	pid.available := node.stats.rlimit.maxpid - node.stats.rlimit.curproc

For more information on eviction thresholds, view the Node-pressure Eviction section of the official Kubernetes docs.

Soft Eviction Grace Periods

kubelet:
 evictionSoftGracePeriod:
 memory.available: 1m
 nodefs.available: 1m30s
 nodefs.inodesFree: 2m
 imagefs.available: 1m30s
 imagefs.inodesFree: 2m
 pid.available: 2m
 evictionMaxPodGracePeriod: 60

Pod Density

Note

By default, the VPC CNI allocates IPs for a node and pods from the same subnet. With VPC CNI Custom Networking, the pods will receive IP addresses from another subnet dedicated to pod IPs. This approach makes it easier to manage IP addresses and allows for separate Network Access Control Lists (NACLs) applied to your pods. VPC CNI Custom Networking reduces the pod density of a node since one of the ENI attachments will be used for the node and cannot share the allocated IPs on the interface to pods. Karpenter supports VPC CNI Custom Networking and similar CNI setups where the primary node interface is separated from the pods interfaces through a global environment variable RESERVED_ENIS, see Settings. In the common case, RESERVED_ENIS should be set to "1" if using Custom Networking.

Windows Support Notice

It’s currently not possible to specify custom networking with Windows nodes.

spec.amiFamily

Ubuntu Support Dropped at v1

Ubuntu NodeClass Example:

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
spec:
 amiFamily: AL2
 amiSelectorTerms:
 - id: ami-placeholder
 blockDeviceMappings:
 - deviceName: '/dev/sda1'
 rootVolume: true
 ebs:
 encrypted: true
 volumeType: gp3
 volumeSize: 20Gi

AL2

Note

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"

--//
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash -xe
exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1
/etc/eks/bootstrap.sh 'test-cluster' --apiserver-endpoint 'https://test-cluster' --b64-cluster-ca 'ca-bundle' \
--dns-cluster-ip '10.100.0.10' \
--use-max-pods false \
--kubelet-extra-args '--node-labels=karpenter.sh/capacity-type=on-demand,karpenter.sh/nodepool=test --max-pods=110'
--//--

AL2023

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"

--//
Content-Type: application/node.eks.aws

# Karpenter Generated NodeConfig
apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
 cluster:
 name: test-cluster
 apiServerEndpoint: https://example.com
 certificateAuthority: ca-bundle
 cidr: 10.100.0.0/16
 kubelet:
 config:
 maxPods: 110
 flags:
 - --node-labels=karpenter.sh/capacity-type=on-demand,karpenter.sh/nodepool=test

--//--

Bottlerocket

[settings]
[settings.kubernetes]
api-server = 'https://test-cluster'
cluster-certificate = 'ca-bundle'
cluster-name = 'test-cluster'
cluster-dns-ip = '10.100.0.10'
max-pods = 110

[settings.kubernetes.node-labels]
'karpenter.sh/capacity-type' = 'on-demand'
'karpenter.sh/nodepool' = 'test'

Windows2019

<powershell>
[string]$EKSBootstrapScriptFile = "$env:ProgramFiles\Amazon\EKS\Start-EKSBootstrap.ps1"
& $EKSBootstrapScriptFile -EKSClusterName 'test-cluster' -APIServerEndpoint 'https://test-cluster' -Base64ClusterCA 'ca-bundle' -KubeletExtraArgs '--node-labels="karpenter.sh/capacity-type=on-demand,karpenter.sh/nodepool=test" --max-pods=110' -DNSClusterIP '10.100.0.10'
</powershell>

Windows2022

<powershell>
[string]$EKSBootstrapScriptFile = "$env:ProgramFiles\Amazon\EKS\Start-EKSBootstrap.ps1"
& $EKSBootstrapScriptFile -EKSClusterName 'test-cluster' -APIServerEndpoint 'https://test-cluster' -Base64ClusterCA 'ca-bundle' -KubeletExtraArgs '--node-labels="karpenter.sh/capacity-type=on-demand,karpenter.sh/nodepool=test" --max-pods=110' -DNSClusterIP '10.100.0.10'
</powershell>

Custom

spec.subnetSelectorTerms

subnetSelectorTerms:
 # Select on any subnet that has the "karpenter.sh/discovery: ${CLUSTER_NAME}"
 # AND the "environment: test" tag OR any subnet with ID "subnet-09fa4a0a8f233a921"
 - tags:
 karpenter.sh/discovery: "${CLUSTER_NAME}"
 environment: test
 - id: subnet-09fa4a0a8f233a921

Tip

Subnets may be specified by any tag, including Name. Selecting tag values using wildcards (*) is supported.

Examples

Select all with a specified tag key:

spec:
 subnetSelectorTerms:
 - tags:
 karpenter.sh/discovery/MyClusterName: '*'

Select by name and tag (all criteria must match):

spec:
 subnetSelectorTerms:
 - tags:
 Name: my-subnet
 MyTag: '' # matches all resources with the tag

Select using multiple tag terms:

spec:
 subnetSelectorTerms:
 - tags:
 Name: "my-subnet-1"
 - tags:
 Name: "my-subnet-2"

Select using wildcards:

spec:
 subnetSelectorTerms:
 - tags:
 Name: "*Public*"

Select using ids:

spec:
 subnetSelectorTerms:
 - id: "subnet-09fa4a0a8f233a921"
 - id: "subnet-0471ca205b8a129ae"

spec.securityGroupSelectorTerms

This selection logic is modeled as terms, where each term contains multiple conditions that must all be satisfied for the selector to match. Effectively, all requirements within a single term are ANDed together. It’s possible that you may want to select on two different security groups that have unrelated requirements. In this case, you can specify multiple terms which will be ORed together to form your selection logic. The example below shows how this selection logic is fulfilled.

securityGroupSelectorTerms:
 # Select on any security group that has both the "karpenter.sh/discovery: ${CLUSTER_NAME}" tag
 # AND the "environment: test" tag OR any security group with the "my-security-group" name
 # OR any security group with ID "sg-063d7acfb4b06c82c"
 - tags:
 karpenter.sh/discovery: "${CLUSTER_NAME}"
 environment: test
 - name: my-security-group
 - id: sg-063d7acfb4b06c82c

Tip

Security groups may be specified by any tag, including “Name”. Selecting tags using wildcards (*) is supported.

Note

To verify if this restriction affects you, run the following commands.

CLUSTER_VPC_ID="$(aws eks describe-cluster --name $CLUSTER_NAME --query cluster.resourcesVpcConfig.vpcId --output text)"

aws ec2 describe-security-groups --filters Name=vpc-id,Values=$CLUSTER_VPC_ID Name=tag-key,Values=kubernetes.io/cluster/$CLUSTER_NAME --query 'SecurityGroups[].[GroupName]' --output text

If multiple securityGroups are printed, you will need more specific securityGroupSelectorTerms. We generally recommend that you use the karpenter.sh/discovery: $CLUSTER_NAME tag selector instead.

Examples

Select all assigned to a cluster:

spec:
 securityGroupSelectorTerms:
 - tags:
 kubernetes.io/cluster/$CLUSTER_NAME: "owned"

Select all with a specified tag key:

spec:
 securityGroupSelectorTerms:
 - tags:
 MyTag: '*'

Select by name and tag (all criteria must match):

spec:
 securityGroupSelectorTerms:
 - name: my-security-group
 tags:
 MyTag: '*' # matches all resources with the tag

Select using multiple tag terms:

spec:
 securityGroupSelectorTerms:
 - tags:
 Name: "my-security-group-1"
 - tags:
 Name: "my-security-group-2"

Select by name using a wildcard:

spec:
 securityGroupSelectorTerms:
 - name: "*Public*"

Select using ids:

spec:
 securityGroupSelectorTerms:
 - id: "sg-063d7acfb4b06c82c"
 - id: "sg-06e0cf9c198874591"

spec.role

spec:
 role: "KarpenterNodeRole-$CLUSTER_NAME"

spec.instanceProfile

You can provision and assign a role to an IAM instance profile using CloudFormation or by using the aws iam create-instance-profile and aws iam add-role-to-instance-profile commands in the CLI.

Note

spec.amiSelectorTerms

AMI Selector Terms are required and are used to configure AMIs for Karpenter to use. AMIs are discovered through alias, id, owner, name, and tags.

This selection logic is modeled as terms, where each term contains multiple conditions that must all be satisfied for the selector to match. Effectively, all requirements within a single term are ANDed together. It’s possible that you may want to select on two different AMIs that have unrelated requirements. In this case, you can specify multiple terms which will be ORed together to form your selection logic. The example below shows how this selection logic is fulfilled.

amiSelectorTerms:
 # Select on any AMI that has both the `karpenter.sh/discovery: ${CLUSTER_NAME}`
 # AND `environment: test` tags OR any AMI with the name `my-ami` OR an AMI with
 # ID `ami-123`
 - tags:
 karpenter.sh/discovery: "${CLUSTER_NAME}"
 environment: test
 - name: my-ami
 - id: ami-123
 # Select EKS optimized AL2023 AMIs with version `v20240807`. This term is mutually
 # exclusive and can't be specified with other terms.
 # - alias: al2023@v20240807

An alias term can be used to select EKS-optimized AMIs. An alias is formatted as family@version. Family can be one of the following values:

al2
al2023
bottlerocket
windows2019
windows2022

alias: al2023@v20240703

Bottlerocket uses a semantic version for their releases. You can pin bottlerocket as follows:

alias: bottlerocket@v1.20.4

The Windows family does not support pinning, so only latest is supported.

The following commands can be used to determine the versions availble for an alias in your region:

export K8S_VERSION="1.35"
aws ssm get-parameters-by-path --path "/aws/service/eks/optimized-ami/$K8S_VERSION/amazon-linux-2023/" --recursive | jq -cr '.Parameters[].Name' | grep -v "recommended" | awk -F '/' '{print $10}' | sed -r 's/.*(v[[:digit:]]+)$/\1/' | sort | uniq

export K8S_VERSION="1.35"
aws ssm get-parameters-by-path --path "/aws/service/eks/optimized-ami/$K8S_VERSION/amazon-linux-2/" --recursive | jq -cr '.Parameters[].Name' | grep -v "recommended" | awk -F '/' '{print $8}' | sed -r 's/.*(v[[:digit:]]+)$/\1/' | sort | uniq

export K8S_VERSION="1.35"
aws ssm get-parameters-by-path --path "/aws/service/bottlerocket/aws-k8s-$K8S_VERSION" --recursive | jq -cr '.Parameters[].Name' | grep -v "latest" | awk -F '/' '{print $7}' | sort | uniq

Warning

Tip

AMIs may be specified by any AWS tag, including Name. Selecting by tag or by name using wildcards (*) is supported.

Note

If amiSelectorTerms match more than one AMI, Karpenter will automatically determine which AMI best fits the workloads on the launched worker node under the following constraints:

When launching nodes, Karpenter automatically determines which architecture a custom AMI is compatible with and will use images that match an instanceType’s requirements.
- Unless using an alias, Karpenter cannot detect requirements other than architecture. If you need to specify different AMIs for different kind of nodes (e.g. accelerated GPU AMIs), you should use a separate EC2NodeClass.
If multiple AMIs are found that can be used, Karpenter will choose the latest one.
If no AMIs are found that can be used, then no nodes will be provisioned.

Examples

Select by AMI family and version:

 amiSelectorTerms:
 - alias: al2023@v20240807

Select all with a specified tag:

 amiSelectorTerms:
 - tags:
 karpenter.sh/discovery/MyClusterName: '*'

Select by name:

 amiSelectorTerms:
 - name: my-ami

Select by Name tag:

 amiSelectorTerms:
 - tags:
 Name: my-ami

Select by name and owner:

 amiSelectorTerms:
 - name: my-ami
 owner: self
 - name: my-ami
 owner: "0123456789"

Select by name using a wildcard:

spec:
 amiSelectorTerms:
 - name: "*EKS*"

Select by all under an owner:

spec:
 amiSelectorTerms:
 - name: "*"
 owner: self

Specify using ids:

 amiSelectorTerms:
 - id: "ami-123"
 - id: "ami-456"

spec.tags

Karpenter adds tags to all resources it creates, including EC2 Instances, EBS volumes, and Launch Templates. The default set of tags are listed below.

Name: <node-name>
karpenter.sh/nodeclaim: <nodeclaim-name>
karpenter.sh/nodepool: <nodepool-name>
karpenter.k8s.aws/ec2nodeclass: <ec2nodeclass-name>
kubernetes.io/cluster/<cluster-name>: owned
eks:eks-cluster-name: <cluster-name>

Additional tags can be added in the tags section, which will be merged with the default tags specified above.

spec:
 tags:
 InternalAccountingTag: 1234
 dev.corp.net/app: Calculator
 dev.corp.net/team: MyTeam

Note

spec.metadataOptions

Control the exposure of Instance Metadata Service on EC2 Instances launched by this EC2NodeClass using a generated launch template.

Refer to recommended, security best practices for limiting exposure of Instance Metadata and User Data to pods.

If metadataOptions are omitted from this EC2NodeClass, the following default settings are applied:

spec:
 metadataOptions:
 httpEndpoint: enabled
 httpProtocolIPv6: disabled
 httpPutResponseHopLimit: 1
 httpTokens: required

spec.blockDeviceMappings

spec:
 blockDeviceMappings:
 - deviceName: /dev/xvda
 ebs:
 volumeSize: 100Gi
 volumeType: gp3
 iops: 10000
 encrypted: true
 kmsKeyID: "1234abcd-12ab-34cd-56ef-1234567890ab"
 deleteOnTermination: true
 throughput: 125
 snapshotID: snap-0123456789

The following blockDeviceMapping defaults are used for each AMIFamily if no blockDeviceMapping overrides are specified in the EC2NodeClass

AL2

spec:
 blockDeviceMappings:
 - deviceName: /dev/xvda
 ebs:
 volumeSize: 20Gi
 volumeType: gp3
 encrypted: true

AL2023

spec:
 blockDeviceMappings:
 - deviceName: /dev/xvda
 ebs:
 volumeSize: 20Gi
 volumeType: gp3
 encrypted: true

Bottlerocket

spec:
 blockDeviceMappings:
 # Root device
 - deviceName: /dev/xvda
 ebs:
 volumeSize: 4Gi
 volumeType: gp3
 encrypted: true
 # Data device: Container resources such as images and logs
 - deviceName: /dev/xvdb
 ebs:
 volumeSize: 20Gi
 volumeType: gp3
 encrypted: true

Windows2019/Windows2022

spec:
 blockDeviceMappings:
 - deviceName: /dev/sda1
 ebs:
 volumeSize: 50Gi
 volumeType: gp3
 encrypted: true

Custom

The Custom AMIFamily ships without any default blockDeviceMappings.

spec.instanceStorePolicy

The instanceStorePolicy field controls how instance-store volumes are handled. By default, Karpenter and Kubernetes will simply ignore them.

RAID0

If you intend to use these volumes for faster node ephemeral-storage, set instanceStorePolicy to RAID0:

spec:
 instanceStorePolicy: RAID0

Instructions for each AMI family are listed below:

AL2

AL2023

Others

For all other AMI families, you must configure the disks yourself. Check out the setup-local-disks script in amazon-eks-ami to see how this is done for AL2.

Tip

Since the Kubelet & Containerd will be using the instance-store filesystem, you may consider using a more minimal root volume size.

spec.userData

You can control the UserData that is applied to your worker nodes via this field. This allows you to run custom scripts or pass-through custom configuration to Karpenter instances on start-up.

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
 name: bottlerocket-example
spec:
 ...
 amiFamily: Bottlerocket
 userData: |
 [settings.kubernetes]
 "kube-api-qps" = 30
 "shutdown-grace-period" = "30s"
 "shutdown-grace-period-for-critical-pods" = "30s"
 [settings.kubernetes.eviction-hard]
 "memory.available" = "20%"

This example adds SSH keys to allow remote login to the node (replace my-authorized_keys with your key file):

Note

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
 name: al2-example
spec:
 ...
 amiFamily: AL2
 userData: |
 #!/bin/bash
 mkdir -p ~ec2-user/.ssh/
 touch ~ec2-user/.ssh/authorized_keys
 cat >> ~ec2-user/.ssh/authorized_keys <<EOF
 {{ insertFile "../my-authorized_keys" | indent 4 }}
 EOF
 chmod -R go-w ~ec2-user/.ssh/authorized_keys
 chown -R ec2-user ~ec2-user/.ssh

For more examples on configuring fields for different AMI families, see the examples here.

AL2

Your UserData can be in the MIME multi part archive format.
Karpenter will transform your custom user-data as a MIME part, if necessary, and then merge a final MIME part to the end of your UserData parts which will bootstrap the worker node. Karpenter will have full control over all the parameters being passed to the bootstrap script.
- Karpenter will continue to set MaxPods, ClusterDNS and all other parameters defined in spec.kubeletConfiguration as before.

Consider the following example to understand how your custom UserData will be merged -

Passed-in UserData (bash)

#!/bin/bash
echo "Running custom user data script (bash)"

Merged UserData (bash)

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"

--//
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash
echo "Running custom user data script (bash)"

--//
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash -xe
exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1
/etc/eks/bootstrap.sh 'test-cluster' --apiserver-endpoint 'https://test-cluster' --b64-cluster-ca 'ca-bundle' \
--use-max-pods false \
--kubelet-extra-args '--node-labels=karpenter.sh/capacity-type=on-demand,karpenter.sh/nodepool=test --max-pods=110'
--//--

Passed-in UserData (MIME)

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="BOUNDARY"

--BOUNDARY
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash
echo "Running custom user data script (mime)"

--BOUNDARY--

Merged UserData (MIME)

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"

--//
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash
echo "Running custom user data script (mime)"

--//
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash -xe
exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1
/etc/eks/bootstrap.sh 'test-cluster' --apiserver-endpoint 'https://test-cluster' --b64-cluster-ca 'ca-bundle' \
--use-max-pods false \
--kubelet-extra-args '--node-labels=karpenter.sh/capacity-type=on-demand,karpenter.sh/nodepool=test --max-pods=110'
--//--

Tip

You can set additional kubelet configuration properties, unavailable through spec.kubelet, by updating the kubelet-config.json file:

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
 name: kubelet-config-example
spec:
 amiFamily: AL2
 userData: |
 #!/bin/bash
 echo "$(jq '.kubeAPIQPS=50' /etc/kubernetes/kubelet/kubelet-config.json)" > /etc/kubernetes/kubelet/kubelet-config.json

AL2023

Your UserData may be in one of three formats: a MIME multi part archive, a NodeConfig YAML / JSON string, or a shell script.
Karpenter will transform your custom UserData into a MIME part, if necessary, and then create a MIME multi-part archive. This archive will consist of a generated NodeConfig, containing Karpenter’s default values, followed by the transformed custom UserData. For more information on the NodeConfig spec, refer to the AL2023 EKS Optimized AMI docs.

Warning

Passed-in UserData (NodeConfig)

apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
 kubelet:
 config:
 maxPods: 42

Merged UserData (NodeConfig)

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"

--//
Content-Type: application/node.eks.aws

apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
 kubelet:
 config:
 maxPods: 42

--//
Content-Type: application/node.eks.aws

# Karpenter Generated NodeConfig
apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
 cluster:
 apiServerEndpoint: https://test-cluster
 certificateAuthority: cluster-ca
 cidr: 10.100.0.0/16
 name: test-cluster
 kubelet:
 config:
 clusterDNS:
 - 10.100.0.10
 maxPods: 118
 flags:
 - --node-labels="karpenter.sh/capacity-type=on-demand,karpenter.sh/nodepool=default"

--//--

Passed-in UserData (bash)

#!/bin/bash
echo "Hello, AL2023!"

Merged UserData (bash)

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"

--//
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash
echo "Hello, AL2023!"

--//
Content-Type: application/node.eks.aws

# Karpenter Generated NodeConfig
apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
 cluster:
 apiServerEndpoint: https://test-cluster
 certificateAuthority: cluster-ca
 cidr: 10.100.0.0/16
 name: test-cluster
 kubelet:
 config:
 clusterDNS:
 - 10.100.0.10
 maxPods: 118
 flags:
 - --node-labels="karpenter.sh/capacity-type=on-demand,karpenter.sh/nodepool=default"

--//--

Passed-in UserData (MIME)

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"

--//
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash
echo "Hello, AL2023!"

--//
Content-Type: application/node.eks.aws

apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
 kubelet:
 config:
 maxPods: 42
--//

Merged UserData (MIME)

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"

--//
Content-Type: application/node.eks.aws

apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
 kubelet:
 config:
 maxPods: 42
--//
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash
echo "Hello, AL2023!"

--//
Content-Type: application/node.eks.aws

# Karpenter Generated NodeConfig
apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
 cluster:
 apiServerEndpoint: https://test-cluster
 certificateAuthority: cluster-ca
 cidr: 10.100.0.0/16
 name: test-cluster
 kubelet:
 config:
 clusterDNS:
 - 10.100.0.10
 maxPods: 118
 flags:
 - --node-labels="karpenter.sh/capacity-type=on-demand,karpenter.sh/nodepool=default"

--//--

Bottlerocket

Your UserData must be valid TOML.
Unknown TOML fields will be ignored when the final merged UserData is generated by Karpenter.

Warning

Consider the following example to understand how your custom UserData settings will be merged in.

Passed-in UserData

[settings.kubernetes.eviction-hard]
"memory.available" = "12%"
[settings.kubernetes]
"unknown-setting" = "unknown"
[settings.kubernetes.node-labels]
'field.controlled.by/karpenter' = 'will-be-overridden'

Merged UserData

[settings]
[settings.kubernetes]
api-server = 'https://cluster'
cluster-certificate = 'ca-bundle'
cluster-name = 'cluster'

[settings.kubernetes.node-labels]
'karpenter.sh/capacity-type' = 'on-demand'
'karpenter.sh/nodepool' = 'default'

[settings.kubernetes.node-taints]

[settings.kubernetes.eviction-hard]
'memory.available' = '12%%'

Windows2019/Windows2022

Your UserData must be specified as PowerShell commands.
The UserData specified will be prepended to a Karpenter managed section that will bootstrap the kubelet.
Karpenter will continue to set ClusterDNS and all other parameters defined in spec.kubeletConfiguration as before.

Consider the following example to understand how your custom UserData settings will be merged in.

Passed-in UserData

Write-Host "Running custom user data script"

Merged UserData

<powershell>
Write-Host "Running custom user data script"
[string]$EKSBootstrapScriptFile = "$env:ProgramFiles\Amazon\EKS\Start-EKSBootstrap.ps1"
& $EKSBootstrapScriptFile -EKSClusterName 'test-cluster' -APIServerEndpoint 'https://test-cluster' -Base64ClusterCA 'ca-bundle' -KubeletExtraArgs '--node-labels="karpenter.sh/capacity-type=spot,karpenter.sh/nodepool=windows2022" --max-pods=110' -DNSClusterIP '10.0.100.10'
</powershell>

Windows Support Notice

spec:
 userData: |
 $global:EKSCluster = Get-EKSCluster -Name my-cluster

Custom

No merging is performed, your UserData must perform all setup required of the node to allow it to join the cluster.
Custom UserData must meet the following requirements to work correctly with Karpenter:
- It must ensure the node is registered with the karpenter.sh/unregistered:NoExecute taint (via kubelet configuration field registerWithTaints)
- It must set kubelet config options to match those configured in spec.kubelet

spec.detailedMonitoring

spec:
 detailedMonitoring: true

spec.associatePublicIPAddress

Note

status.subnets

Examples

spec:
 subnetSelectorTerms:
 - tags:
 karpenter.sh/discovery: "${CLUSTER_NAME}"
status:
 subnets:
 - id: subnet-0a462d98193ff9fac
 zone: us-east-2b
 - id: subnet-0322dfafd76a609b6
 zone: us-east-2c
 - id: subnet-0727ef01daf4ac9fe
 zone: us-east-2b
 - id: subnet-00c99aeafe2a70304
 zone: us-east-2a
 - id: subnet-023b232fd5eb0028e
 zone: us-east-2c
 - id: subnet-03941e7ad6afeaa72
 zone: us-east-2a

status.securityGroups

Examples

spec:
 securityGroupSelectorTerms:
 - tags:
 karpenter.sh/discovery: "${CLUSTER_NAME}"
status:
 securityGroups:
 - id: sg-041513b454818610b
 name: ClusterSharedNodeSecurityGroup
 - id: sg-0286715698b894bca
 name: ControlPlaneSecurityGroup-1AQ073TSAAPW

status.amis

status.amis contains the resolved id, name, and requirements of either the default AMIs for the spec.amiFamily or the AMIs selected by the spec.amiSelectorTerms if this field is specified.

Examples

AMIs resolved with an AL2 alias:

spec:
 amiSelectorTerms:
 - alias: al2@v20240807
status:
 amis:
 - id: ami-03c3a3dcda64f5b75
 name: amazon-linux-2-gpu
 requirements:
 - key: kubernetes.io/arch
 operator: In
 values:
 - amd64
 - key: karpenter.k8s.aws/instance-gpu-count
 operator: Exists
 - id: ami-03c3a3dcda64f5b75
 name: amazon-linux-2-gpu
 requirements:
 - key: kubernetes.io/arch
 operator: In
 values:
 - amd64
 - key: karpenter.k8s.aws/instance-accelerator-count
 operator: Exists
 - id: ami-06afb2d101cc4b8bd
 name: amazon-linux-2-arm64
 requirements:
 - key: kubernetes.io/arch
 operator: In
 values:
 - arm64
 - key: karpenter.k8s.aws/instance-gpu-count
 operator: DoesNotExist
 - key: karpenter.k8s.aws/instance-accelerator-count
 operator: DoesNotExist
 - id: ami-0e28b76d768af234e
 name: amazon-linux-2
 requirements:
 - key: kubernetes.io/arch
 operator: In
 values:
 - amd64
 - key: karpenter.k8s.aws/instance-gpu-count
 operator: DoesNotExist
 - key: karpenter.k8s.aws/instance-accelerator-count
 operator: DoesNotExist

AMIs resolved from tags:

spec:
 amiSelectorTerms:
 - tags:
 karpenter.sh/discovery: "${CLUSTER_NAME}"
status:
 amis:
 - id: ami-01234567890123456
 name: custom-ami-amd64
 requirements:
 - key: kubernetes.io/arch
 operator: In
 values:
 - amd64
 - id: ami-01234567890123456
 name: custom-ami-arm64
 requirements:
 - key: kubernetes.io/arch
 operator: In
 values:
 - arm64

status.instanceProfile

status.instanceProfile contains the resolved instance profile generated by Karpenter from the spec.role

spec:
 role: "KarpenterNodeRole-${CLUSTER_NAME}"
status:
 instanceProfile: "${CLUSTER_NAME}-0123456778901234567789"

status.conditions

NodeClasses have the following status conditions:

Condition Type	Description
SubnetsReady	Subnets are discovered.
SecurityGroupsReady	Security Groups are discovered.
InstanceProfileReady	Instance Profile is discovered.
AMIsReady	AMIs are discovered
Ready	Top level condition that indicates if the nodeClass is ready. If any of the underlying conditions is `False` then this condition is set to `False` and `Message` on the condition indicates the dependency that was not resolved.

If a NodeClass is not ready, NodePools that reference it through their nodeClassRef will not be considered for scheduling.

V1.10: NodeClasses

Mon, 01 Jan 0001 00:00:00 +0000

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
 name: default
spec:
 template:
 spec:
 nodeClassRef:
 group: karpenter.k8s.aws
 kind: EC2NodeClass
 name: default
---
apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
 name: default
spec:
 kubelet:
 podsPerCore: 2
 maxPods: 20
 systemReserved:
 cpu: 100m
 memory: 100Mi
 ephemeral-storage: 1Gi
 kubeReserved:
 cpu: 200m
 memory: 100Mi
 ephemeral-storage: 3Gi
 evictionHard:
 memory.available: 5%
 nodefs.available: 10%
 nodefs.inodesFree: 10%
 evictionSoft:
 memory.available: 500Mi
 nodefs.available: 15%
 nodefs.inodesFree: 15%
 evictionSoftGracePeriod:
 memory.available: 1m
 nodefs.available: 1m30s
 nodefs.inodesFree: 2m
 evictionMaxPodGracePeriod: 60
 imageGCHighThresholdPercent: 85
 imageGCLowThresholdPercent: 80
 cpuCFSQuota: true
 clusterDNS: ["10.0.1.100"]
 # Optional, dictates UserData generation and default block device mappings.
 # May be ommited when using an `alias` amiSelectorTerm, otherwise required.
 amiFamily: AL2

 # Required, discovers subnets to attach to instances
 # Each term in the array of subnetSelectorTerms is ORed together
 # Within a single term, all conditions are ANDed
 subnetSelectorTerms:
 # Select on any subnet that has the "karpenter.sh/discovery: ${CLUSTER_NAME}"
 # AND the "environment: test" tag OR any subnet with ID "subnet-09fa4a0a8f233a921"
 - tags:
 karpenter.sh/discovery: "${CLUSTER_NAME}"
 environment: test
 - id: subnet-09fa4a0a8f233a921

 # Required, discovers security groups to attach to instances
 # Each term in the array of securityGroupSelectorTerms is ORed together
 # Within a single term, all conditions are ANDed
 securityGroupSelectorTerms:
 # Select on any security group that has both the "karpenter.sh/discovery: ${CLUSTER_NAME}" tag
 # AND the "environment: test" tag OR any security group with the "my-security-group" name
 # OR any security group with ID "sg-063d7acfb4b06c82c"
 - tags:
 karpenter.sh/discovery: "${CLUSTER_NAME}"
 environment: test
 - name: my-security-group
 - id: sg-063d7acfb4b06c82c

 # Optional, IAM role to use for the node identity.
 # Must specify one of "role" or "instanceProfile" for Karpenter to launch nodes
 role: "KarpenterNodeRole-${CLUSTER_NAME}"

 # Optional, IAM instance profile to use for the node identity.
 # Must specify one of "role" or "instanceProfile" for Karpenter to launch nodes
 instanceProfile: "KarpenterNodeInstanceProfile-${CLUSTER_NAME}"

 # Each term in the array of amiSelectorTerms is ORed together
 # Within a single term, all conditions are ANDed
 amiSelectorTerms:
 # Select on any AMI that has both the `karpenter.sh/discovery: ${CLUSTER_NAME}`
 # AND `environment: test` tags OR any AMI with the name `my-ami` OR an AMI with
 # ID `ami-123` OR an AMI with ID matching the value of my-custom-parameter
 - tags:
 karpenter.sh/discovery: "${CLUSTER_NAME}"
 environment: test
 - name: my-ami
 - id: ami-123
 - ssmParameter: my-custom-parameter # ssm parameter name or ARN
 # Select EKS optimized AL2023 AMIs with version `v20240703`. This term is mutually
 # exclusive and can't be specified with other terms.
 # - alias: al2023@v20240703

 # Optional, each term in the array of capacityReservationSelectorTerms is ORed together.
 capacityReservationSelectorTerms:
 - tags:
 karpenter.sh/discovery: ${CLUSTER_NAME}
 - id: cr-123
 - instanceMatchCriteria: open

 # Optional, propagates tags to underlying EC2 resources
 tags:
 team: team-a
 app: team-a-app

 # Optional, configures IMDS for the instance
 metadataOptions:
 httpEndpoint: enabled
 httpProtocolIPv6: disabled
 httpPutResponseHopLimit: 1 # This is changed to disable IMDS access from containers not on the host network
 httpTokens: required

 # Optional, configures storage devices for the instance
 blockDeviceMappings:
 - deviceName: /dev/xvda
 ebs:
 volumeSize: 100Gi
 volumeType: gp3
 iops: 10000
 encrypted: true
 kmsKeyID: "1234abcd-12ab-34cd-56ef-1234567890ab"
 deleteOnTermination: true
 throughput: 125
 snapshotID: snap-0123456789
 volumeInitializationRate: 100

 # Optional, use instance-store volumes for node ephemeral-storage
 instanceStorePolicy: RAID0

 # Optional, overrides autogenerated userdata with a merge semantic
 userData: |
 echo "Hello world"

 # Optional, configures detailed monitoring for the instance
 detailedMonitoring: true

 # Optional, configures if the instance should be launched with an associated public IP address.
 # If not specified, the default value depends on the subnet's public IP auto-assign setting.
 associatePublicIPAddress: true
status:
 # Resolved subnets
 subnets:
 - id: subnet-0a462d98193ff9fac
 zone: us-east-2b
 - id: subnet-0322dfafd76a609b6
 zone: us-east-2c
 - id: subnet-0727ef01daf4ac9fe
 zone: us-east-2b
 - id: subnet-00c99aeafe2a70304
 zone: us-east-2a
 - id: subnet-023b232fd5eb0028e
 zone: us-east-2c
 - id: subnet-03941e7ad6afeaa72
 zone: us-east-2a

 # Resolved security groups
 securityGroups:
 - id: sg-041513b454818610b
 name: ClusterSharedNodeSecurityGroup
 - id: sg-0286715698b894bca
 name: ControlPlaneSecurityGroup-1AQ073TSAAPW

 # Resolved AMIs
 amis:
 - id: ami-01234567890123456
 name: custom-ami-amd64
 requirements:
 - key: kubernetes.io/arch
 operator: In
 values:
 - amd64
 - id: ami-01234567890123456
 name: custom-ami-arm64
 requirements:
 - key: kubernetes.io/arch
 operator: In
 values:
 - arm64

 # Capacity Reservations
 capacityReservations:
 - availabilityZone: us-west-2a
 id: cr-01234567890123456
 instanceMatchCriteria: targeted
 instanceType: g6.48xlarge
 interruptible: false
 ownerID: "012345678901"
 reservationType: capacity-block
 state: expiring
 - availabilityZone: us-west-2c
 id: cr-12345678901234567
 instanceMatchCriteria: open
 instanceType: g6.48xlarge
 interruptible: true
 ownerID: "98765432109"
 reservationType: default
 state: active

 # Generated instance profile name from "role"
 instanceProfile: "${CLUSTER_NAME}-0123456778901234567789"
 conditions:
 - lastTransitionTime: "2024-02-02T19:54:34Z"
 status: "True"
 type: InstanceProfileReady
 - lastTransitionTime: "2024-02-02T19:54:34Z"
 status: "True"
 type: SubnetsReady
 - lastTransitionTime: "2024-02-02T19:54:34Z"
 status: "True"
 type: SecurityGroupsReady
 - lastTransitionTime: "2024-02-02T19:54:34Z"
 status: "True"
 type: AMIsReady
 - lastTransitionTime: "2024-02-02T19:54:34Z"
 status: "True"
 type: Ready

Refer to the NodePool docs for settings applicable to all providers. To explore various EC2NodeClass configurations, refer to the examples provided in the Karpenter Github repository.

spec.kubelet

kubelet:
 podsPerCore: 2
 maxPods: 20
 systemReserved:
 cpu: 100m
 memory: 100Mi
 ephemeral-storage: 1Gi
 kubeReserved:
 cpu: 200m
 memory: 100Mi
 ephemeral-storage: 3Gi
 evictionHard:
 memory.available: 5%
 nodefs.available: 10%
 nodefs.inodesFree: 10%
 evictionSoft:
 memory.available: 500Mi
 nodefs.available: 15%
 nodefs.inodesFree: 15%
 evictionSoftGracePeriod:
 memory.available: 1m
 nodefs.available: 1m30s
 nodefs.inodesFree: 2m
 evictionMaxPodGracePeriod: 60
 imageGCHighThresholdPercent: 85
 imageGCLowThresholdPercent: 80
 cpuCFSQuota: true
 clusterDNS: ["10.0.1.100"]

Note

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
spec:
 amiSelectorTerms:
 - alias: al2023@v20240807
 kubelet:
 maxPods: 42
 userData: |
 apiVersion: node.eks.aws/v1alpha1
 kind: NodeConfig
 spec:
 kubelet:
 config:
 # Configured through UserData since unavailable in `spec.kubelet`
 registryPullQPS: 10

Note that when using the Custom AMIFamily you will need to specify fields both in spec.kubelet and spec.userData.

Pods Per Core

Note

Max Pods

Note

Windows Support Notice

Reserved Resources

Note

Eviction Thresholds

kubelet:
 evictionHard:
 memory.available: 500Mi
 nodefs.available: 10%
 nodefs.inodesFree: 10%
 imagefs.available: 5%
 imagefs.inodesFree: 5%
 pid.available: 7%
 evictionSoft:
 memory.available: 1Gi
 nodefs.available: 15%
 nodefs.inodesFree: 15%
 imagefs.available: 10%
 imagefs.inodesFree: 10%
 pid.available: 10%

Supported Eviction Signals

Eviction Signal	Description
memory.available	memory.available := node.status.capacity[memory] - node.stats.memory.workingSet
nodefs.available	nodefs.available := node.stats.fs.available
nodefs.inodesFree	nodefs.inodesFree := node.stats.fs.inodesFree
imagefs.available	imagefs.available := node.stats.runtime.imagefs.available
imagefs.inodesFree	imagefs.inodesFree := node.stats.runtime.imagefs.inodesFree
pid.available	pid.available := node.stats.rlimit.maxpid - node.stats.rlimit.curproc

For more information on eviction thresholds, view the Node-pressure Eviction section of the official Kubernetes docs.

Soft Eviction Grace Periods

kubelet:
 evictionSoftGracePeriod:
 memory.available: 1m
 nodefs.available: 1m30s
 nodefs.inodesFree: 2m
 imagefs.available: 1m30s
 imagefs.inodesFree: 2m
 pid.available: 2m
 evictionMaxPodGracePeriod: 60

Pod Density

Note

Windows Support Notice

It’s currently not possible to specify custom networking with Windows nodes.

spec.amiFamily

Ubuntu Support Dropped at v1

Ubuntu NodeClass Example:

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
spec:
 amiFamily: AL2
 amiSelectorTerms:
 - id: ami-placeholder
 blockDeviceMappings:
 - deviceName: '/dev/sda1'
 rootVolume: true
 ebs:
 encrypted: true
 volumeType: gp3
 volumeSize: 20Gi

AL2

AL2 support dropped at Kubernetes 1.33

Kubernetes version 1.32 is the last version for which Amazon EKS will release Amazon Linux 2 (AL2) AMIs.

Note

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"

--//
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash -xe
exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1
/etc/eks/bootstrap.sh 'test-cluster' --apiserver-endpoint 'https://test-cluster' --b64-cluster-ca 'ca-bundle' \
--dns-cluster-ip '10.100.0.10' \
--use-max-pods false \
--kubelet-extra-args '--node-labels=karpenter.sh/capacity-type=on-demand,karpenter.sh/nodepool=test --max-pods=110'
--//--

AL2023

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"

--//
Content-Type: application/node.eks.aws

# Karpenter Generated NodeConfig
apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
 cluster:
 name: test-cluster
 apiServerEndpoint: https://example.com
 certificateAuthority: ca-bundle
 cidr: 10.100.0.0/16
 kubelet:
 config:
 maxPods: 110
 flags:
 - --node-labels=karpenter.sh/capacity-type=on-demand,karpenter.sh/nodepool=test

--//--

Bottlerocket

[settings]
[settings.kubernetes]
api-server = 'https://test-cluster'
cluster-certificate = 'ca-bundle'
cluster-name = 'test-cluster'
cluster-dns-ip = '10.100.0.10'
max-pods = 110

[settings.kubernetes.node-labels]
'karpenter.sh/capacity-type' = 'on-demand'
'karpenter.sh/nodepool' = 'test'

Windows2019

<powershell>
[string]$EKSBootstrapScriptFile = "$env:ProgramFiles\Amazon\EKS\Start-EKSBootstrap.ps1"
& $EKSBootstrapScriptFile -EKSClusterName 'test-cluster' -APIServerEndpoint 'https://test-cluster' -Base64ClusterCA 'ca-bundle' -KubeletExtraArgs '--node-labels="karpenter.sh/capacity-type=on-demand,karpenter.sh/nodepool=test" --max-pods=110' -DNSClusterIP '10.100.0.10'
</powershell>

Windows2022

<powershell>
[string]$EKSBootstrapScriptFile = "$env:ProgramFiles\Amazon\EKS\Start-EKSBootstrap.ps1"
& $EKSBootstrapScriptFile -EKSClusterName 'test-cluster' -APIServerEndpoint 'https://test-cluster' -Base64ClusterCA 'ca-bundle' -KubeletExtraArgs '--node-labels="karpenter.sh/capacity-type=on-demand,karpenter.sh/nodepool=test" --max-pods=110' -DNSClusterIP '10.100.0.10'
</powershell>

Windows2025

<powershell>
[string]$EKSBootstrapScriptFile = "$env:ProgramFiles\Amazon\EKS\Start-EKSBootstrap.ps1"
& $EKSBootstrapScriptFile -EKSClusterName 'test-cluster' -APIServerEndpoint 'https://test-cluster' -Base64ClusterCA 'ca-bundle' -KubeletExtraArgs '--node-labels="karpenter.sh/capacity-type=on-demand,karpenter.sh/nodepool=test" --max-pods=110' -DNSClusterIP '10.100.0.10'
</powershell>

Custom

spec.subnetSelectorTerms

subnetSelectorTerms:
 # Select on any subnet that has the "karpenter.sh/discovery: ${CLUSTER_NAME}"
 # AND the "environment: test" tag OR any subnet with ID "subnet-09fa4a0a8f233a921"
 - tags:
 karpenter.sh/discovery: "${CLUSTER_NAME}"
 environment: test
 - id: subnet-09fa4a0a8f233a921

Tip

Subnets may be specified by any tag, including Name. Selecting tag values using wildcards (*) is supported.

Examples

Select all with a specified tag key:

spec:
 subnetSelectorTerms:
 - tags:
 karpenter.sh/discovery/MyClusterName: '*'

Select by name and tag (all criteria must match):

spec:
 subnetSelectorTerms:
 - tags:
 Name: my-subnet
 MyTag: '' # matches all resources with the tag

Select using multiple tag terms:

spec:
 subnetSelectorTerms:
 - tags:
 Name: "my-subnet-1"
 - tags:
 Name: "my-subnet-2"

Select using wildcards:

spec:
 subnetSelectorTerms:
 - tags:
 Name: "*Public*"

Select using ids:

spec:
 subnetSelectorTerms:
 - id: "subnet-09fa4a0a8f233a921"
 - id: "subnet-0471ca205b8a129ae"

spec.securityGroupSelectorTerms

This selection logic is modeled as terms, where each term contains multiple conditions that must all be satisfied for the selector to match. Effectively, all requirements within a single term are ANDed together. It’s possible that you may want to select on two different security groups that have unrelated requirements. In this case, you can specify multiple terms which will be ORed together to form your selection logic. The example below shows how this selection logic is fulfilled.

securityGroupSelectorTerms:
 # Select on any security group that has both the "karpenter.sh/discovery: ${CLUSTER_NAME}" tag
 # AND the "environment: test" tag OR any security group with the "my-security-group" name
 # OR any security group with ID "sg-063d7acfb4b06c82c"
 - tags:
 karpenter.sh/discovery: "${CLUSTER_NAME}"
 environment: test
 - name: my-security-group
 - id: sg-063d7acfb4b06c82c

Tip

Security groups may be specified by any tag, including “Name”. Selecting tags using wildcards (*) is supported.

Note

To verify if this restriction affects you, run the following commands.

CLUSTER_VPC_ID="$(aws eks describe-cluster --name $CLUSTER_NAME --query cluster.resourcesVpcConfig.vpcId --output text)"

aws ec2 describe-security-groups --filters Name=vpc-id,Values=$CLUSTER_VPC_ID Name=tag-key,Values=kubernetes.io/cluster/$CLUSTER_NAME --query 'SecurityGroups[].[GroupName]' --output text

If multiple securityGroups are printed, you will need more specific securityGroupSelectorTerms. We generally recommend that you use the karpenter.sh/discovery: $CLUSTER_NAME tag selector instead.

Examples

Select all assigned to a cluster:

spec:
 securityGroupSelectorTerms:
 - tags:
 kubernetes.io/cluster/$CLUSTER_NAME: "owned"

Select all with a specified tag key:

spec:
 securityGroupSelectorTerms:
 - tags:
 MyTag: '*'

Select by name and tag (all criteria must match):

spec:
 securityGroupSelectorTerms:
 - name: my-security-group
 tags:
 MyTag: '*' # matches all resources with the tag

Select using multiple tag terms:

spec:
 securityGroupSelectorTerms:
 - tags:
 Name: "my-security-group-1"
 - tags:
 Name: "my-security-group-2"

Select by name using a wildcard:

spec:
 securityGroupSelectorTerms:
 - name: "*Public*"

Select using ids:

spec:
 securityGroupSelectorTerms:
 - id: "sg-063d7acfb4b06c82c"
 - id: "sg-06e0cf9c198874591"

spec.role

spec:
 role: "KarpenterNodeRole-$CLUSTER_NAME"

spec.instanceProfile

You can provision and assign a role to an IAM instance profile using CloudFormation or by using the aws iam create-instance-profile and aws iam add-role-to-instance-profile commands in the CLI.

Note

Warning

When using spec.instanceProfile, ensure you are using pre-provisioned instance profiles that you manage yourself.

spec.amiSelectorTerms

AMI Selector Terms are required and are used to configure AMIs for Karpenter to use. AMIs are discovered through alias, id, owner, name, and tags.

This selection logic is modeled as terms, where each term contains multiple conditions that must all be satisfied for the selector to match. Effectively, all requirements within a single term are ANDed together. It’s possible that you may want to select on two different AMIs that have unrelated requirements. In this case, you can specify multiple terms which will be ORed together to form your selection logic. The example below shows how this selection logic is fulfilled.

amiSelectorTerms:
 # Select on any AMI that has both the `karpenter.sh/discovery: ${CLUSTER_NAME}`
 # AND `environment: test` tags OR any AMI with the name `my-ami` OR an AMI with
 # ID `ami-123` OR an AMI with ID matching the value of my-custom-parameter
 - tags:
 karpenter.sh/discovery: "${CLUSTER_NAME}"
 environment: test
 - name: my-ami
 - id: ami-123
 - ssmParameter: my-custom-parameter # ssm parameter name or ARN
 # Select EKS optimized AL2023 AMIs with version `v20240807`. This term is mutually
 # exclusive and can't be specified with other terms.
 # - alias: al2023@v20240807

An alias term can be used to select EKS-optimized AMIs. An alias is formatted as family@version. Family can be one of the following values:

al2
al2023
bottlerocket
windows2019
windows2022
windows2025

alias: al2023@v20240703

Bottlerocket uses a semantic version for their releases. You can pin bottlerocket as follows:

alias: bottlerocket@v1.20.4

The Windows family does not support pinning, so only latest is supported.

The following commands can be used to determine the versions availble for an alias in your region:

export K8S_VERSION="1.35"
aws ssm get-parameters-by-path --path "/aws/service/eks/optimized-ami/$K8S_VERSION/amazon-linux-2023/" --recursive | jq -cr '.Parameters[].Name' | grep -v "recommended" | awk -F '/' '{print $10}' | sed -r 's/.*(v[[:digit:]]+)$/\1/' | sort | uniq

export K8S_VERSION="1.35"
aws ssm get-parameters-by-path --path "/aws/service/eks/optimized-ami/$K8S_VERSION/amazon-linux-2/" --recursive | jq -cr '.Parameters[].Name' | grep -v "recommended" | awk -F '/' '{print $8}' | sed -r 's/.*(v[[:digit:]]+)$/\1/' | sort | uniq

export K8S_VERSION="1.35"
aws ssm get-parameters-by-path --path "/aws/service/bottlerocket/aws-k8s-$K8S_VERSION" --recursive | jq -cr '.Parameters[].Name' | grep -v "latest" | awk -F '/' '{print $7}' | sort | uniq

Warning

Tip

AMIs may be specified by any AWS tag, including Name. Selecting by tag or by name using wildcards (*) is supported.

Note

If amiSelectorTerms match more than one AMI, Karpenter will automatically determine which AMI best fits the workloads on the launched worker node under the following constraints:

When launching nodes, Karpenter automatically determines which architecture a custom AMI is compatible with and will use images that match an instanceType’s requirements.
- Unless using an alias, Karpenter cannot detect requirements other than architecture. If you need to specify different AMIs for different kind of nodes (e.g. accelerated GPU AMIs), you should use a separate EC2NodeClass.
If multiple AMIs are found that can be used, Karpenter will choose the latest one.
If no AMIs are found that can be used, then no nodes will be provisioned.

Examples

Select by AMI family and version:

 amiSelectorTerms:
 - alias: al2023@v20240807

Select all with a specified tag:

 amiSelectorTerms:
 - tags:
 karpenter.sh/discovery/MyClusterName: '*'

Select by name:

 amiSelectorTerms:
 - name: my-ami

Select by Name tag:

 amiSelectorTerms:
 - tags:
 Name: my-ami

Select by name and owner:

 amiSelectorTerms:
 - name: my-ami
 owner: self
 - name: my-ami
 owner: "0123456789"

Select by name using a wildcard:

spec:
 amiSelectorTerms:
 - name: "*EKS*"

Select by all under an owner:

spec:
 amiSelectorTerms:
 - name: "*"
 owner: self

Specify using ids:

 amiSelectorTerms:
 - id: "ami-123"
 - id: "ami-456"

Specify using custom ssm parameter name or ARN:

 amiSelectorTerms:
 - ssmParameter: "my-custom-parameter"

Note

spec.capacityReservationSelectorTerms

Feature State: Beta

For more information on utilizing ODCRs with Karpenter, refer to the Utilizing ODCRs Task.

Note

Examples

Select the reservations with the given IDs:

spec:
 capacityReservationSelectorTerms:
 - id: cr-123
 - id: cr-456

Select the reservations by tags:

spec:
 capacityReservationSelectorTerms:
 # Select all capacity reservations which have both matching tags
 - tags:
 key1: foo
 key2: bar
 # Additionally, select all capacity reservations with the following matching tag
 - tags:
 key3: foobar

Select by tags and owner ID:

spec:
 # Select all capacity reservations with the matching tags which are also owned by
 # the specified account.
 capacityReservationSelectorTerms:
 - tags:
 key: foo
 ownerID: 012345678901

Select by instance match criteria:

spec:
 capacityReservationSelectorTerms:
 # Select all open capacity reservations
 - instanceMatchCriteria: open

Select by instance match criteria and tags:

spec:
 capacityReservationSelectorTerms:
 # Select targeted capacity reservations with matching tags
 - instanceMatchCriteria: targeted
 tags:
 key: foo

spec.tags

Karpenter adds tags to all resources it creates, including EC2 Instances, EBS volumes, and Launch Templates. The default set of tags are listed below.

Name: <node-name>
karpenter.sh/nodeclaim: <nodeclaim-name>
karpenter.sh/nodepool: <nodepool-name>
karpenter.k8s.aws/ec2nodeclass: <ec2nodeclass-name>
kubernetes.io/cluster/<cluster-name>: owned
eks:eks-cluster-name: <cluster-name>

Additional tags can be added in the tags section, which will be merged with the default tags specified above.

spec:
 tags:
 InternalAccountingTag: 1234
 dev.corp.net/app: Calculator
 dev.corp.net/team: MyTeam

Note

spec.metadataOptions

Control the exposure of Instance Metadata Service on EC2 Instances launched by this EC2NodeClass using a generated launch template.

Refer to recommended, security best practices for limiting exposure of Instance Metadata and User Data to pods.

If metadataOptions are omitted from this EC2NodeClass, the following default settings are applied:

spec:
 metadataOptions:
 httpEndpoint: enabled
 httpProtocolIPv6: disabled
 httpPutResponseHopLimit: 1
 httpTokens: required

spec.blockDeviceMappings

spec:
 blockDeviceMappings:
 - deviceName: /dev/xvda
 ebs:
 volumeSize: 100Gi
 volumeType: gp3
 iops: 10000
 encrypted: true
 kmsKeyID: "1234abcd-12ab-34cd-56ef-1234567890ab"
 deleteOnTermination: true
 throughput: 125
 snapshotID: snap-0123456789
 volumeInitializationRate: 100

The following blockDeviceMapping defaults are used for each AMIFamily if no blockDeviceMapping overrides are specified in the EC2NodeClass

AL2

spec:
 blockDeviceMappings:
 - deviceName: /dev/xvda
 ebs:
 volumeSize: 20Gi
 volumeType: gp3
 encrypted: true

AL2023

spec:
 blockDeviceMappings:
 - deviceName: /dev/xvda
 ebs:
 volumeSize: 20Gi
 volumeType: gp3
 encrypted: true

Bottlerocket

spec:
 blockDeviceMappings:
 # Root device
 - deviceName: /dev/xvda
 ebs:
 volumeSize: 4Gi
 volumeType: gp3
 encrypted: true
 # Data device: Container resources such as images and logs
 - deviceName: /dev/xvdb
 ebs:
 volumeSize: 20Gi
 volumeType: gp3
 encrypted: true

Windows2019/Windows2022/Windows2025

spec:
 blockDeviceMappings:
 - deviceName: /dev/sda1
 ebs:
 volumeSize: 50Gi
 volumeType: gp3
 encrypted: true

Custom

The Custom AMIFamily ships without any default blockDeviceMappings.

spec.instanceStorePolicy

The instanceStorePolicy field controls how instance-store volumes are handled. By default, Karpenter and Kubernetes will simply ignore them.

RAID0

If you intend to use these volumes for faster node ephemeral-storage, set instanceStorePolicy to RAID0:

spec:
 instanceStorePolicy: RAID0

Instructions for each AMI family are listed below:

AL2

AL2023

Bottlerocket

Note

This automatic disk configuration is only available on Bottlerocket v1.22.0 and later.

Others

For all other AMI families, you must configure the disks yourself. Check out the setup-local-disks script in amazon-eks-ami to see how this is done for AL2.

Tip

Since the Kubelet & Containerd will be using the instance-store filesystem, you may consider using a more minimal root volume size.

spec.userData

You can control the UserData that is applied to your worker nodes via this field. This allows you to run custom scripts or pass-through custom configuration to Karpenter instances on start-up.

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
 name: bottlerocket-example
spec:
 ...
 amiFamily: Bottlerocket
 userData: |
 [settings.kubernetes]
 "kube-api-qps" = 30
 "shutdown-grace-period" = "30s"
 "shutdown-grace-period-for-critical-pods" = "30s"
 [settings.kubernetes.eviction-hard]
 "memory.available" = "20%"

This example adds SSH keys to allow remote login to the node (replace my-authorized_keys with your public key file):

Note

Also, my-authorized_key key is the public key. See Retrieve the public key material.

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
 name: al2-example
spec:
 ...
 amiFamily: AL2
 userData: |
 #!/bin/bash
 mkdir -p ~ec2-user/.ssh/
 touch ~ec2-user/.ssh/authorized_keys
 cat >> ~ec2-user/.ssh/authorized_keys <<EOF
 {{ insertFile "../my-authorized_keys" | indent 4 }}
 EOF
 chmod -R go-w ~ec2-user/.ssh/authorized_keys
 chown -R ec2-user ~ec2-user/.ssh

Alternatively, you can save the key in your SSM Parameter Store and use the get-parameter command mentioned below to retrieve the key for authorized_keys.

aws ssm get-parameter --name "<parameter-name>" --region <region> --with-decryption --query "Parameter.Value" --output text > /home/ec2-user/.ssh/authorized_keys

For more examples on configuring fields for different AMI families, see the examples here.

AL2

Your UserData can be in the MIME multi part archive format.
Karpenter will transform your custom user-data as a MIME part, if necessary, and then merge a final MIME part to the end of your UserData parts which will bootstrap the worker node. Karpenter will have full control over all the parameters being passed to the bootstrap script.
- Karpenter will continue to set MaxPods, ClusterDNS and all other parameters defined in spec.kubeletConfiguration as before.

Consider the following example to understand how your custom UserData will be merged -

Passed-in UserData (bash)

#!/bin/bash
echo "Running custom user data script (bash)"

Merged UserData (bash)

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"

--//
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash
echo "Running custom user data script (bash)"

--//
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash -xe
exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1
/etc/eks/bootstrap.sh 'test-cluster' --apiserver-endpoint 'https://test-cluster' --b64-cluster-ca 'ca-bundle' \
--use-max-pods false \
--kubelet-extra-args '--node-labels=karpenter.sh/capacity-type=on-demand,karpenter.sh/nodepool=test --max-pods=110'
--//--

Passed-in UserData (MIME)

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="BOUNDARY"

--BOUNDARY
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash
echo "Running custom user data script (mime)"

--BOUNDARY--

Merged UserData (MIME)

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"

--//
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash
echo "Running custom user data script (mime)"

--//
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash -xe
exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1
/etc/eks/bootstrap.sh 'test-cluster' --apiserver-endpoint 'https://test-cluster' --b64-cluster-ca 'ca-bundle' \
--use-max-pods false \
--kubelet-extra-args '--node-labels=karpenter.sh/capacity-type=on-demand,karpenter.sh/nodepool=test --max-pods=110'
--//--

Tip

You can set additional kubelet configuration properties, unavailable through spec.kubelet, by updating the kubelet-config.json file:

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
 name: kubelet-config-example
spec:
 amiFamily: AL2
 userData: |
 #!/bin/bash
 echo "$(jq '.kubeAPIQPS=50' /etc/kubernetes/kubelet/kubelet-config.json)" > /etc/kubernetes/kubelet/kubelet-config.json

AL2023

Your UserData may be in one of three formats: a MIME multi part archive, a NodeConfig YAML / JSON string, or a shell script.
Karpenter will transform your custom UserData into a MIME part, if necessary, and then create a MIME multi-part archive. This archive will consist of a generated NodeConfig, containing Karpenter’s default values, followed by the transformed custom UserData. For more information on the NodeConfig spec, refer to the AL2023 EKS Optimized AMI docs.

Warning

Passed-in UserData (NodeConfig)

apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
 kubelet:
 config:
 maxPods: 42

Merged UserData (NodeConfig)

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"

--//
Content-Type: application/node.eks.aws

apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
 kubelet:
 config:
 maxPods: 42

--//
Content-Type: application/node.eks.aws

# Karpenter Generated NodeConfig
apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
 cluster:
 apiServerEndpoint: https://test-cluster
 certificateAuthority: cluster-ca
 cidr: 10.100.0.0/16
 name: test-cluster
 kubelet:
 config:
 clusterDNS:
 - 10.100.0.10
 maxPods: 118
 flags:
 - --node-labels="karpenter.sh/capacity-type=on-demand,karpenter.sh/nodepool=default"

--//--

Passed-in UserData (bash)

#!/bin/bash
echo "Hello, AL2023!"

Merged UserData (bash)

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"

--//
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash
echo "Hello, AL2023!"

--//
Content-Type: application/node.eks.aws

# Karpenter Generated NodeConfig
apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
 cluster:
 apiServerEndpoint: https://test-cluster
 certificateAuthority: cluster-ca
 cidr: 10.100.0.0/16
 name: test-cluster
 kubelet:
 config:
 clusterDNS:
 - 10.100.0.10
 maxPods: 118
 flags:
 - --node-labels="karpenter.sh/capacity-type=on-demand,karpenter.sh/nodepool=default"

--//--

Passed-in UserData (MIME)

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"

--//
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash
echo "Hello, AL2023!"

--//
Content-Type: application/node.eks.aws

apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
 kubelet:
 config:
 maxPods: 42
--//

Merged UserData (MIME)

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"

--//
Content-Type: application/node.eks.aws

apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
 kubelet:
 config:
 maxPods: 42
--//
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash
echo "Hello, AL2023!"

--//
Content-Type: application/node.eks.aws

# Karpenter Generated NodeConfig
apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
 cluster:
 apiServerEndpoint: https://test-cluster
 certificateAuthority: cluster-ca
 cidr: 10.100.0.0/16
 name: test-cluster
 kubelet:
 config:
 clusterDNS:
 - 10.100.0.10
 maxPods: 118
 flags:
 - --node-labels="karpenter.sh/capacity-type=on-demand,karpenter.sh/nodepool=default"

--//--

Bottlerocket

Your UserData must be valid TOML.
Unknown TOML fields will be ignored when the final merged UserData is generated by Karpenter.

Warning

Consider the following example to understand how your custom UserData settings will be merged in.

Passed-in UserData

[settings.kubernetes.eviction-hard]
"memory.available" = "12%"
[settings.kubernetes]
"unknown-setting" = "unknown"
[settings.kubernetes.node-labels]
'field.controlled.by/karpenter' = 'will-be-overridden'

Merged UserData

[settings]
[settings.kubernetes]
api-server = 'https://cluster'
cluster-certificate = 'ca-bundle'
cluster-name = 'cluster'

[settings.kubernetes.node-labels]
'karpenter.sh/capacity-type' = 'on-demand'
'karpenter.sh/nodepool' = 'default'

[settings.kubernetes.node-taints]

[settings.kubernetes.eviction-hard]
'memory.available' = '12%%'

Device ownership in Bottlerocket

Bottlerocket v1.30.0+ supports device ownership using the security context provided in the Kubernetes specfile. To enable this, you will need the following user-data configurations:

[settings]
[settings.kubernetes]
device-ownership-from-security-context = true

This setting helps you enable Neuron workloads on Bottlerocket instances. See Accelerators/GPU Resources for more details.

Windows2019/Windows2022/Windows2025

Your UserData must be specified as PowerShell commands.
The UserData specified will be prepended to a Karpenter managed section that will bootstrap the kubelet.
Karpenter will continue to set ClusterDNS and all other parameters defined in spec.kubeletConfiguration as before.

Consider the following example to understand how your custom UserData settings will be merged in.

Passed-in UserData

Write-Host "Running custom user data script"

Merged UserData

<powershell>
Write-Host "Running custom user data script"
[string]$EKSBootstrapScriptFile = "$env:ProgramFiles\Amazon\EKS\Start-EKSBootstrap.ps1"
& $EKSBootstrapScriptFile -EKSClusterName 'test-cluster' -APIServerEndpoint 'https://test-cluster' -Base64ClusterCA 'ca-bundle' -KubeletExtraArgs '--node-labels="karpenter.sh/capacity-type=spot,karpenter.sh/nodepool=windows2022" --max-pods=110' -DNSClusterIP '10.0.100.10'
</powershell>

Windows Support Notice

spec:
 userData: |
 $global:EKSCluster = Get-EKSCluster -Name my-cluster

Custom

No merging is performed, your UserData must perform all setup required of the node to allow it to join the cluster.
Custom UserData must meet the following requirements to work correctly with Karpenter:
- It must ensure the node is registered with the karpenter.sh/unregistered:NoExecute taint (via kubelet configuration field registerWithTaints)
- It must set kubelet config options to match those configured in spec.kubelet

spec.detailedMonitoring

spec:
 detailedMonitoring: true

spec.associatePublicIPAddress

Note

spec.ipPrefixCount

status.subnets

Examples

spec:
 subnetSelectorTerms:
 - tags:
 karpenter.sh/discovery: "${CLUSTER_NAME}"
status:
 subnets:
 - id: subnet-0a462d98193ff9fac
 zone: us-east-2b
 - id: subnet-0322dfafd76a609b6
 zone: us-east-2c
 - id: subnet-0727ef01daf4ac9fe
 zone: us-east-2b
 - id: subnet-00c99aeafe2a70304
 zone: us-east-2a
 - id: subnet-023b232fd5eb0028e
 zone: us-east-2c
 - id: subnet-03941e7ad6afeaa72
 zone: us-east-2a

status.securityGroups

Examples

spec:
 securityGroupSelectorTerms:
 - tags:
 karpenter.sh/discovery: "${CLUSTER_NAME}"
status:
 securityGroups:
 - id: sg-041513b454818610b
 name: ClusterSharedNodeSecurityGroup
 - id: sg-0286715698b894bca
 name: ControlPlaneSecurityGroup-1AQ073TSAAPW

status.amis

Examples

AMIs resolved with an AL2 alias:

spec:
 amiSelectorTerms:
 - alias: al2@v20240807
status:
 amis:
 - id: ami-03c3a3dcda64f5b75
 name: amazon-linux-2-gpu
 requirements:
 - key: kubernetes.io/arch
 operator: In
 values:
 - amd64
 - key: karpenter.k8s.aws/instance-gpu-count
 operator: Exists
 - id: ami-03c3a3dcda64f5b75
 name: amazon-linux-2-gpu
 requirements:
 - key: kubernetes.io/arch
 operator: In
 values:
 - amd64
 - key: karpenter.k8s.aws/instance-accelerator-count
 operator: Exists
 - id: ami-06afb2d101cc4b8bd
 name: amazon-linux-2-arm64
 requirements:
 - key: kubernetes.io/arch
 operator: In
 values:
 - arm64
 - key: karpenter.k8s.aws/instance-gpu-count
 operator: DoesNotExist
 - key: karpenter.k8s.aws/instance-accelerator-count
 operator: DoesNotExist
 - id: ami-0e28b76d768af234e
 name: amazon-linux-2
 requirements:
 - key: kubernetes.io/arch
 operator: In
 values:
 - amd64
 - key: karpenter.k8s.aws/instance-gpu-count
 operator: DoesNotExist
 - key: karpenter.k8s.aws/instance-accelerator-count
 operator: DoesNotExist

AMIs resolved from tags:

spec:
 amiSelectorTerms:
 - tags:
 karpenter.sh/discovery: "${CLUSTER_NAME}"
status:
 amis:
 - id: ami-01234567890123456
 name: custom-ami-amd64
 deprecated: true
 requirements:
 - key: kubernetes.io/arch
 operator: In
 values:
 - amd64
 - id: ami-01234567890123456
 name: custom-ami-arm64
 requirements:
 - key: kubernetes.io/arch
 operator: In
 values:
 - arm64

status.capacityReservations

status.capacityReservations contains the following information for each resolved capacity reservation:

Field	Example	Description
`availabilityZone`	`us-east-1a`	The availability zone the capacity reservation is available in
`id`	`cr-56fac701cc1951b03`	The ID of the capacity reservation
`instanceMatchCriteria`	`open`	The instanceMatchCriteria for the capacity reservation. Can be `open` or `targeted`.
`instanceType`	`m5.large`	The EC2 instance type of the capacity reservation
`ownerID`	`459763720645`	The account ID that owns the capacity reservation
`reservationType`	`default`	The type of the capacity reservation. Can be `default` or `capacity-block`.
`state`	`active`	The state of the capacity reservation. Can be `active` or `expiring`.
`interruptible`	`true` or `false`	Whether the capacity reservation is interruptible.

Examples

status:
 capacityReservations:
 - availabilityZone: us-west-2a
 id: cr-01234567890123456
 instanceMatchCriteria: targeted
 instanceType: g6.48xlarge
 interruptible: false
 ownerID: "012345678901"
 reservationType: capacity-block
 state: expiring
 - availabilityZone: us-west-2c
 id: cr-12345678901234567
 instanceMatchCriteria: open
 instanceType: g6.48xlarge
 interruptible: true
 ownerID: "98765432109"
 reservationType: default
 state: active

status.instanceProfile

status.instanceProfile contains the resolved instance profile generated by Karpenter from the spec.role

spec:
 role: "KarpenterNodeRole-${CLUSTER_NAME}"
status:
 instanceProfile: "${CLUSTER_NAME}-0123456778901234567789"

status.conditions

NodeClasses have the following status conditions:

Condition Type	Description
SubnetsReady	Subnets are discovered.
SecurityGroupsReady	Security Groups are discovered.
InstanceProfileReady	Instance Profile is discovered.
AMIsReady	AMIs are discovered.
Ready	Top level condition that indicates if the nodeClass is ready. If any of the underlying conditions is `False` then this condition is set to `False` and `Message` on the condition indicates the dependency that was not resolved.

If a NodeClass is not ready, NodePools that reference it through their nodeClassRef will not be considered for scheduling.

V1.8: NodeClasses

Mon, 01 Jan 0001 00:00:00 +0000

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
 name: default
spec:
 template:
 spec:
 nodeClassRef:
 group: karpenter.k8s.aws
 kind: EC2NodeClass
 name: default
---
apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
 name: default
spec:
 kubelet:
 podsPerCore: 2
 maxPods: 20
 systemReserved:
 cpu: 100m
 memory: 100Mi
 ephemeral-storage: 1Gi
 kubeReserved:
 cpu: 200m
 memory: 100Mi
 ephemeral-storage: 3Gi
 evictionHard:
 memory.available: 5%
 nodefs.available: 10%
 nodefs.inodesFree: 10%
 evictionSoft:
 memory.available: 500Mi
 nodefs.available: 15%
 nodefs.inodesFree: 15%
 evictionSoftGracePeriod:
 memory.available: 1m
 nodefs.available: 1m30s
 nodefs.inodesFree: 2m
 evictionMaxPodGracePeriod: 60
 imageGCHighThresholdPercent: 85
 imageGCLowThresholdPercent: 80
 cpuCFSQuota: true
 clusterDNS: ["10.0.1.100"]
 # Optional, dictates UserData generation and default block device mappings.
 # May be ommited when using an `alias` amiSelectorTerm, otherwise required.
 amiFamily: AL2

 # Required, discovers subnets to attach to instances
 # Each term in the array of subnetSelectorTerms is ORed together
 # Within a single term, all conditions are ANDed
 subnetSelectorTerms:
 # Select on any subnet that has the "karpenter.sh/discovery: ${CLUSTER_NAME}"
 # AND the "environment: test" tag OR any subnet with ID "subnet-09fa4a0a8f233a921"
 - tags:
 karpenter.sh/discovery: "${CLUSTER_NAME}"
 environment: test
 - id: subnet-09fa4a0a8f233a921

 # Required, discovers security groups to attach to instances
 # Each term in the array of securityGroupSelectorTerms is ORed together
 # Within a single term, all conditions are ANDed
 securityGroupSelectorTerms:
 # Select on any security group that has both the "karpenter.sh/discovery: ${CLUSTER_NAME}" tag
 # AND the "environment: test" tag OR any security group with the "my-security-group" name
 # OR any security group with ID "sg-063d7acfb4b06c82c"
 - tags:
 karpenter.sh/discovery: "${CLUSTER_NAME}"
 environment: test
 - name: my-security-group
 - id: sg-063d7acfb4b06c82c

 # Optional, IAM role to use for the node identity.
 # Must specify one of "role" or "instanceProfile" for Karpenter to launch nodes
 role: "KarpenterNodeRole-${CLUSTER_NAME}"

 # Optional, IAM instance profile to use for the node identity.
 # Must specify one of "role" or "instanceProfile" for Karpenter to launch nodes
 instanceProfile: "KarpenterNodeInstanceProfile-${CLUSTER_NAME}"

 # Each term in the array of amiSelectorTerms is ORed together
 # Within a single term, all conditions are ANDed
 amiSelectorTerms:
 # Select on any AMI that has both the `karpenter.sh/discovery: ${CLUSTER_NAME}`
 # AND `environment: test` tags OR any AMI with the name `my-ami` OR an AMI with
 # ID `ami-123` OR an AMI with ID matching the value of my-custom-parameter
 - tags:
 karpenter.sh/discovery: "${CLUSTER_NAME}"
 environment: test
 - name: my-ami
 - id: ami-123
 - ssmParameter: my-custom-parameter # ssm parameter name or ARN
 # Select EKS optimized AL2023 AMIs with version `v20240703`. This term is mutually
 # exclusive and can't be specified with other terms.
 # - alias: al2023@v20240703

 # Optional, each term in the array of capacityReservationSelectorTerms is ORed together.
 capacityReservationSelectorTerms:
 - tags:
 karpenter.sh/discovery: ${CLUSTER_NAME}
 - id: cr-123
 - instanceMatchCriteria: open

 # Optional, propagates tags to underlying EC2 resources
 tags:
 team: team-a
 app: team-a-app

 # Optional, configures IMDS for the instance
 metadataOptions:
 httpEndpoint: enabled
 httpProtocolIPv6: disabled
 httpPutResponseHopLimit: 1 # This is changed to disable IMDS access from containers not on the host network
 httpTokens: required

 # Optional, configures storage devices for the instance
 blockDeviceMappings:
 - deviceName: /dev/xvda
 ebs:
 volumeSize: 100Gi
 volumeType: gp3
 iops: 10000
 encrypted: true
 kmsKeyID: "1234abcd-12ab-34cd-56ef-1234567890ab"
 deleteOnTermination: true
 throughput: 125
 snapshotID: snap-0123456789
 volumeInitializationRate: 100

 # Optional, use instance-store volumes for node ephemeral-storage
 instanceStorePolicy: RAID0

 # Optional, overrides autogenerated userdata with a merge semantic
 userData: |
 echo "Hello world"

 # Optional, configures detailed monitoring for the instance
 detailedMonitoring: true

 # Optional, configures if the instance should be launched with an associated public IP address.
 # If not specified, the default value depends on the subnet's public IP auto-assign setting.
 associatePublicIPAddress: true
status:
 # Resolved subnets
 subnets:
 - id: subnet-0a462d98193ff9fac
 zone: us-east-2b
 - id: subnet-0322dfafd76a609b6
 zone: us-east-2c
 - id: subnet-0727ef01daf4ac9fe
 zone: us-east-2b
 - id: subnet-00c99aeafe2a70304
 zone: us-east-2a
 - id: subnet-023b232fd5eb0028e
 zone: us-east-2c
 - id: subnet-03941e7ad6afeaa72
 zone: us-east-2a

 # Resolved security groups
 securityGroups:
 - id: sg-041513b454818610b
 name: ClusterSharedNodeSecurityGroup
 - id: sg-0286715698b894bca
 name: ControlPlaneSecurityGroup-1AQ073TSAAPW

 # Resolved AMIs
 amis:
 - id: ami-01234567890123456
 name: custom-ami-amd64
 requirements:
 - key: kubernetes.io/arch
 operator: In
 values:
 - amd64
 - id: ami-01234567890123456
 name: custom-ami-arm64
 requirements:
 - key: kubernetes.io/arch
 operator: In
 values:
 - arm64

 # Capacity Reservations
 capacityReservations:
 - availabilityZone: us-west-2a
 id: cr-01234567890123456
 instanceMatchCriteria: targeted
 instanceType: g6.48xlarge
 ownerID: "012345678901"
 reservationType: capacity-block
 state: expiring
 - availabilityZone: us-west-2c
 id: cr-12345678901234567
 instanceMatchCriteria: open
 instanceType: g6.48xlarge
 ownerID: "98765432109"
 reservationType: default
 state: active

 # Generated instance profile name from "role"
 instanceProfile: "${CLUSTER_NAME}-0123456778901234567789"
 conditions:
 - lastTransitionTime: "2024-02-02T19:54:34Z"
 status: "True"
 type: InstanceProfileReady
 - lastTransitionTime: "2024-02-02T19:54:34Z"
 status: "True"
 type: SubnetsReady
 - lastTransitionTime: "2024-02-02T19:54:34Z"
 status: "True"
 type: SecurityGroupsReady
 - lastTransitionTime: "2024-02-02T19:54:34Z"
 status: "True"
 type: AMIsReady
 - lastTransitionTime: "2024-02-02T19:54:34Z"
 status: "True"
 type: Ready

Refer to the NodePool docs for settings applicable to all providers. To explore various EC2NodeClass configurations, refer to the examples provided in the Karpenter Github repository.

spec.kubelet

kubelet:
 podsPerCore: 2
 maxPods: 20
 systemReserved:
 cpu: 100m
 memory: 100Mi
 ephemeral-storage: 1Gi
 kubeReserved:
 cpu: 200m
 memory: 100Mi
 ephemeral-storage: 3Gi
 evictionHard:
 memory.available: 5%
 nodefs.available: 10%
 nodefs.inodesFree: 10%
 evictionSoft:
 memory.available: 500Mi
 nodefs.available: 15%
 nodefs.inodesFree: 15%
 evictionSoftGracePeriod:
 memory.available: 1m
 nodefs.available: 1m30s
 nodefs.inodesFree: 2m
 evictionMaxPodGracePeriod: 60
 imageGCHighThresholdPercent: 85
 imageGCLowThresholdPercent: 80
 cpuCFSQuota: true
 clusterDNS: ["10.0.1.100"]

Note

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
spec:
 amiSelectorTerms:
 - alias: al2023@v20240807
 kubelet:
 maxPods: 42
 userData: |
 apiVersion: node.eks.aws/v1alpha1
 kind: NodeConfig
 spec:
 kubelet:
 config:
 # Configured through UserData since unavailable in `spec.kubelet`
 registryPullQPS: 10

Note that when using the Custom AMIFamily you will need to specify fields both in spec.kubelet and spec.userData.

Pods Per Core

Note

Max Pods

Note

Windows Support Notice

Reserved Resources

Note

Eviction Thresholds

kubelet:
 evictionHard:
 memory.available: 500Mi
 nodefs.available: 10%
 nodefs.inodesFree: 10%
 imagefs.available: 5%
 imagefs.inodesFree: 5%
 pid.available: 7%
 evictionSoft:
 memory.available: 1Gi
 nodefs.available: 15%
 nodefs.inodesFree: 15%
 imagefs.available: 10%
 imagefs.inodesFree: 10%
 pid.available: 10%

Supported Eviction Signals

Eviction Signal	Description
memory.available	memory.available := node.status.capacity[memory] - node.stats.memory.workingSet
nodefs.available	nodefs.available := node.stats.fs.available
nodefs.inodesFree	nodefs.inodesFree := node.stats.fs.inodesFree
imagefs.available	imagefs.available := node.stats.runtime.imagefs.available
imagefs.inodesFree	imagefs.inodesFree := node.stats.runtime.imagefs.inodesFree
pid.available	pid.available := node.stats.rlimit.maxpid - node.stats.rlimit.curproc

For more information on eviction thresholds, view the Node-pressure Eviction section of the official Kubernetes docs.

Soft Eviction Grace Periods

kubelet:
 evictionSoftGracePeriod:
 memory.available: 1m
 nodefs.available: 1m30s
 nodefs.inodesFree: 2m
 imagefs.available: 1m30s
 imagefs.inodesFree: 2m
 pid.available: 2m
 evictionMaxPodGracePeriod: 60

Pod Density

Note

Windows Support Notice

It’s currently not possible to specify custom networking with Windows nodes.

spec.amiFamily

Ubuntu Support Dropped at v1

Ubuntu NodeClass Example:

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
spec:
 amiFamily: AL2
 amiSelectorTerms:
 - id: ami-placeholder
 blockDeviceMappings:
 - deviceName: '/dev/sda1'
 rootVolume: true
 ebs:
 encrypted: true
 volumeType: gp3
 volumeSize: 20Gi

AL2

AL2 support dropped at Kubernetes 1.33

Kubernetes version 1.32 is the last version for which Amazon EKS will release Amazon Linux 2 (AL2) AMIs.

Note

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"

--//
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash -xe
exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1
/etc/eks/bootstrap.sh 'test-cluster' --apiserver-endpoint 'https://test-cluster' --b64-cluster-ca 'ca-bundle' \
--dns-cluster-ip '10.100.0.10' \
--use-max-pods false \
--kubelet-extra-args '--node-labels=karpenter.sh/capacity-type=on-demand,karpenter.sh/nodepool=test --max-pods=110'
--//--

AL2023

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"

--//
Content-Type: application/node.eks.aws

# Karpenter Generated NodeConfig
apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
 cluster:
 name: test-cluster
 apiServerEndpoint: https://example.com
 certificateAuthority: ca-bundle
 cidr: 10.100.0.0/16
 kubelet:
 config:
 maxPods: 110
 flags:
 - --node-labels=karpenter.sh/capacity-type=on-demand,karpenter.sh/nodepool=test

--//--

Bottlerocket

[settings]
[settings.kubernetes]
api-server = 'https://test-cluster'
cluster-certificate = 'ca-bundle'
cluster-name = 'test-cluster'
cluster-dns-ip = '10.100.0.10'
max-pods = 110

[settings.kubernetes.node-labels]
'karpenter.sh/capacity-type' = 'on-demand'
'karpenter.sh/nodepool' = 'test'

Windows2019

<powershell>
[string]$EKSBootstrapScriptFile = "$env:ProgramFiles\Amazon\EKS\Start-EKSBootstrap.ps1"
& $EKSBootstrapScriptFile -EKSClusterName 'test-cluster' -APIServerEndpoint 'https://test-cluster' -Base64ClusterCA 'ca-bundle' -KubeletExtraArgs '--node-labels="karpenter.sh/capacity-type=on-demand,karpenter.sh/nodepool=test" --max-pods=110' -DNSClusterIP '10.100.0.10'
</powershell>

Windows2022

<powershell>
[string]$EKSBootstrapScriptFile = "$env:ProgramFiles\Amazon\EKS\Start-EKSBootstrap.ps1"
& $EKSBootstrapScriptFile -EKSClusterName 'test-cluster' -APIServerEndpoint 'https://test-cluster' -Base64ClusterCA 'ca-bundle' -KubeletExtraArgs '--node-labels="karpenter.sh/capacity-type=on-demand,karpenter.sh/nodepool=test" --max-pods=110' -DNSClusterIP '10.100.0.10'
</powershell>

Custom

spec.subnetSelectorTerms

subnetSelectorTerms:
 # Select on any subnet that has the "karpenter.sh/discovery: ${CLUSTER_NAME}"
 # AND the "environment: test" tag OR any subnet with ID "subnet-09fa4a0a8f233a921"
 - tags:
 karpenter.sh/discovery: "${CLUSTER_NAME}"
 environment: test
 - id: subnet-09fa4a0a8f233a921

Tip

Subnets may be specified by any tag, including Name. Selecting tag values using wildcards (*) is supported.

Examples

Select all with a specified tag key:

spec:
 subnetSelectorTerms:
 - tags:
 karpenter.sh/discovery/MyClusterName: '*'

Select by name and tag (all criteria must match):

spec:
 subnetSelectorTerms:
 - tags:
 Name: my-subnet
 MyTag: '' # matches all resources with the tag

Select using multiple tag terms:

spec:
 subnetSelectorTerms:
 - tags:
 Name: "my-subnet-1"
 - tags:
 Name: "my-subnet-2"

Select using wildcards:

spec:
 subnetSelectorTerms:
 - tags:
 Name: "*Public*"

Select using ids:

spec:
 subnetSelectorTerms:
 - id: "subnet-09fa4a0a8f233a921"
 - id: "subnet-0471ca205b8a129ae"

spec.securityGroupSelectorTerms

This selection logic is modeled as terms, where each term contains multiple conditions that must all be satisfied for the selector to match. Effectively, all requirements within a single term are ANDed together. It’s possible that you may want to select on two different security groups that have unrelated requirements. In this case, you can specify multiple terms which will be ORed together to form your selection logic. The example below shows how this selection logic is fulfilled.

securityGroupSelectorTerms:
 # Select on any security group that has both the "karpenter.sh/discovery: ${CLUSTER_NAME}" tag
 # AND the "environment: test" tag OR any security group with the "my-security-group" name
 # OR any security group with ID "sg-063d7acfb4b06c82c"
 - tags:
 karpenter.sh/discovery: "${CLUSTER_NAME}"
 environment: test
 - name: my-security-group
 - id: sg-063d7acfb4b06c82c

Tip

Security groups may be specified by any tag, including “Name”. Selecting tags using wildcards (*) is supported.

Note

To verify if this restriction affects you, run the following commands.

CLUSTER_VPC_ID="$(aws eks describe-cluster --name $CLUSTER_NAME --query cluster.resourcesVpcConfig.vpcId --output text)"

aws ec2 describe-security-groups --filters Name=vpc-id,Values=$CLUSTER_VPC_ID Name=tag-key,Values=kubernetes.io/cluster/$CLUSTER_NAME --query 'SecurityGroups[].[GroupName]' --output text

If multiple securityGroups are printed, you will need more specific securityGroupSelectorTerms. We generally recommend that you use the karpenter.sh/discovery: $CLUSTER_NAME tag selector instead.

Examples

Select all assigned to a cluster:

spec:
 securityGroupSelectorTerms:
 - tags:
 kubernetes.io/cluster/$CLUSTER_NAME: "owned"

Select all with a specified tag key:

spec:
 securityGroupSelectorTerms:
 - tags:
 MyTag: '*'

Select by name and tag (all criteria must match):

spec:
 securityGroupSelectorTerms:
 - name: my-security-group
 tags:
 MyTag: '*' # matches all resources with the tag

Select using multiple tag terms:

spec:
 securityGroupSelectorTerms:
 - tags:
 Name: "my-security-group-1"
 - tags:
 Name: "my-security-group-2"

Select by name using a wildcard:

spec:
 securityGroupSelectorTerms:
 - name: "*Public*"

Select using ids:

spec:
 securityGroupSelectorTerms:
 - id: "sg-063d7acfb4b06c82c"
 - id: "sg-06e0cf9c198874591"

spec.role

spec:
 role: "KarpenterNodeRole-$CLUSTER_NAME"

spec.instanceProfile

You can provision and assign a role to an IAM instance profile using CloudFormation or by using the aws iam create-instance-profile and aws iam add-role-to-instance-profile commands in the CLI.

Note

Warning

When using spec.instanceProfile, ensure you are using pre-provisioned instance profiles that you manage yourself.

spec.amiSelectorTerms

AMI Selector Terms are required and are used to configure AMIs for Karpenter to use. AMIs are discovered through alias, id, owner, name, and tags.

This selection logic is modeled as terms, where each term contains multiple conditions that must all be satisfied for the selector to match. Effectively, all requirements within a single term are ANDed together. It’s possible that you may want to select on two different AMIs that have unrelated requirements. In this case, you can specify multiple terms which will be ORed together to form your selection logic. The example below shows how this selection logic is fulfilled.

amiSelectorTerms:
 # Select on any AMI that has both the `karpenter.sh/discovery: ${CLUSTER_NAME}`
 # AND `environment: test` tags OR any AMI with the name `my-ami` OR an AMI with
 # ID `ami-123` OR an AMI with ID matching the value of my-custom-parameter
 - tags:
 karpenter.sh/discovery: "${CLUSTER_NAME}"
 environment: test
 - name: my-ami
 - id: ami-123
 - ssmParameter: my-custom-parameter # ssm parameter name or ARN
 # Select EKS optimized AL2023 AMIs with version `v20240807`. This term is mutually
 # exclusive and can't be specified with other terms.
 # - alias: al2023@v20240807

An alias term can be used to select EKS-optimized AMIs. An alias is formatted as family@version. Family can be one of the following values:

al2
al2023
bottlerocket
windows2019
windows2022

alias: al2023@v20240703

Bottlerocket uses a semantic version for their releases. You can pin bottlerocket as follows:

alias: bottlerocket@v1.20.4

The Windows family does not support pinning, so only latest is supported.

The following commands can be used to determine the versions availble for an alias in your region:

export K8S_VERSION="1.34"
aws ssm get-parameters-by-path --path "/aws/service/eks/optimized-ami/$K8S_VERSION/amazon-linux-2023/" --recursive | jq -cr '.Parameters[].Name' | grep -v "recommended" | awk -F '/' '{print $10}' | sed -r 's/.*(v[[:digit:]]+)$/\1/' | sort | uniq

export K8S_VERSION="1.34"
aws ssm get-parameters-by-path --path "/aws/service/eks/optimized-ami/$K8S_VERSION/amazon-linux-2/" --recursive | jq -cr '.Parameters[].Name' | grep -v "recommended" | awk -F '/' '{print $8}' | sed -r 's/.*(v[[:digit:]]+)$/\1/' | sort | uniq

export K8S_VERSION="1.34"
aws ssm get-parameters-by-path --path "/aws/service/bottlerocket/aws-k8s-$K8S_VERSION" --recursive | jq -cr '.Parameters[].Name' | grep -v "latest" | awk -F '/' '{print $7}' | sort | uniq

Warning

Tip

AMIs may be specified by any AWS tag, including Name. Selecting by tag or by name using wildcards (*) is supported.

Note

If amiSelectorTerms match more than one AMI, Karpenter will automatically determine which AMI best fits the workloads on the launched worker node under the following constraints:

When launching nodes, Karpenter automatically determines which architecture a custom AMI is compatible with and will use images that match an instanceType’s requirements.
- Unless using an alias, Karpenter cannot detect requirements other than architecture. If you need to specify different AMIs for different kind of nodes (e.g. accelerated GPU AMIs), you should use a separate EC2NodeClass.
If multiple AMIs are found that can be used, Karpenter will choose the latest one.
If no AMIs are found that can be used, then no nodes will be provisioned.

Examples

Select by AMI family and version:

 amiSelectorTerms:
 - alias: al2023@v20240807

Select all with a specified tag:

 amiSelectorTerms:
 - tags:
 karpenter.sh/discovery/MyClusterName: '*'

Select by name:

 amiSelectorTerms:
 - name: my-ami

Select by Name tag:

 amiSelectorTerms:
 - tags:
 Name: my-ami

Select by name and owner:

 amiSelectorTerms:
 - name: my-ami
 owner: self
 - name: my-ami
 owner: "0123456789"

Select by name using a wildcard:

spec:
 amiSelectorTerms:
 - name: "*EKS*"

Select by all under an owner:

spec:
 amiSelectorTerms:
 - name: "*"
 owner: self

Specify using ids:

 amiSelectorTerms:
 - id: "ami-123"
 - id: "ami-456"

Specify using custom ssm parameter name or ARN:

 amiSelectorTerms:
 - ssmParameter: "my-custom-parameter"

Note

spec.capacityReservationSelectorTerms

Feature State: Beta

For more information on utilizing ODCRs with Karpenter, refer to the Utilizing ODCRs Task.

Note

Examples

Select the reservations with the given IDs:

spec:
 capacityReservationSelectorTerms:
 - id: cr-123
 - id: cr-456

Select the reservations by tags:

spec:
 capacityReservationSelectorTerms:
 # Select all capacity reservations which have both matching tags
 - tags:
 key1: foo
 key2: bar
 # Additionally, select all capacity reservations with the following matching tag
 - tags:
 key3: foobar

Select by tags and owner ID:

spec:
 # Select all capacity reservations with the matching tags which are also owned by
 # the specified account.
 capacityReservationSelectorTerms:
 - tags:
 key: foo
 ownerID: 012345678901

Select by instance match criteria:

spec:
 capacityReservationSelectorTerms:
 # Select all open capacity reservations
 - instanceMatchCriteria: open

Select by instance match criteria and tags:

spec:
 capacityReservationSelectorTerms:
 # Select targeted capacity reservations with matching tags
 - instanceMatchCriteria: targeted
 tags:
 key: foo

spec.tags

Karpenter adds tags to all resources it creates, including EC2 Instances, EBS volumes, and Launch Templates. The default set of tags are listed below.

Name: <node-name>
karpenter.sh/nodeclaim: <nodeclaim-name>
karpenter.sh/nodepool: <nodepool-name>
karpenter.k8s.aws/ec2nodeclass: <ec2nodeclass-name>
kubernetes.io/cluster/<cluster-name>: owned
eks:eks-cluster-name: <cluster-name>

Additional tags can be added in the tags section, which will be merged with the default tags specified above.

spec:
 tags:
 InternalAccountingTag: 1234
 dev.corp.net/app: Calculator
 dev.corp.net/team: MyTeam

Note

spec.metadataOptions

Control the exposure of Instance Metadata Service on EC2 Instances launched by this EC2NodeClass using a generated launch template.

Refer to recommended, security best practices for limiting exposure of Instance Metadata and User Data to pods.

If metadataOptions are omitted from this EC2NodeClass, the following default settings are applied:

spec:
 metadataOptions:
 httpEndpoint: enabled
 httpProtocolIPv6: disabled
 httpPutResponseHopLimit: 1
 httpTokens: required

spec.blockDeviceMappings

spec:
 blockDeviceMappings:
 - deviceName: /dev/xvda
 ebs:
 volumeSize: 100Gi
 volumeType: gp3
 iops: 10000
 encrypted: true
 kmsKeyID: "1234abcd-12ab-34cd-56ef-1234567890ab"
 deleteOnTermination: true
 throughput: 125
 snapshotID: snap-0123456789
 volumeInitializationRate: 100

The following blockDeviceMapping defaults are used for each AMIFamily if no blockDeviceMapping overrides are specified in the EC2NodeClass

AL2

spec:
 blockDeviceMappings:
 - deviceName: /dev/xvda
 ebs:
 volumeSize: 20Gi
 volumeType: gp3
 encrypted: true

AL2023

spec:
 blockDeviceMappings:
 - deviceName: /dev/xvda
 ebs:
 volumeSize: 20Gi
 volumeType: gp3
 encrypted: true

Bottlerocket

spec:
 blockDeviceMappings:
 # Root device
 - deviceName: /dev/xvda
 ebs:
 volumeSize: 4Gi
 volumeType: gp3
 encrypted: true
 # Data device: Container resources such as images and logs
 - deviceName: /dev/xvdb
 ebs:
 volumeSize: 20Gi
 volumeType: gp3
 encrypted: true

Windows2019/Windows2022

spec:
 blockDeviceMappings:
 - deviceName: /dev/sda1
 ebs:
 volumeSize: 50Gi
 volumeType: gp3
 encrypted: true

Custom

The Custom AMIFamily ships without any default blockDeviceMappings.

spec.instanceStorePolicy

The instanceStorePolicy field controls how instance-store volumes are handled. By default, Karpenter and Kubernetes will simply ignore them.

RAID0

If you intend to use these volumes for faster node ephemeral-storage, set instanceStorePolicy to RAID0:

spec:
 instanceStorePolicy: RAID0

Instructions for each AMI family are listed below:

AL2

AL2023

Bottlerocket

Note

This automatic disk configuration is only available on Bottlerocket v1.22.0 and later.

Others

For all other AMI families, you must configure the disks yourself. Check out the setup-local-disks script in amazon-eks-ami to see how this is done for AL2.

Tip

Since the Kubelet & Containerd will be using the instance-store filesystem, you may consider using a more minimal root volume size.

spec.userData

You can control the UserData that is applied to your worker nodes via this field. This allows you to run custom scripts or pass-through custom configuration to Karpenter instances on start-up.

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
 name: bottlerocket-example
spec:
 ...
 amiFamily: Bottlerocket
 userData: |
 [settings.kubernetes]
 "kube-api-qps" = 30
 "shutdown-grace-period" = "30s"
 "shutdown-grace-period-for-critical-pods" = "30s"
 [settings.kubernetes.eviction-hard]
 "memory.available" = "20%"

This example adds SSH keys to allow remote login to the node (replace my-authorized_keys with your public key file):

Note

Also, my-authorized_key key is the public key. See Retrieve the public key material.

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
 name: al2-example
spec:
 ...
 amiFamily: AL2
 userData: |
 #!/bin/bash
 mkdir -p ~ec2-user/.ssh/
 touch ~ec2-user/.ssh/authorized_keys
 cat >> ~ec2-user/.ssh/authorized_keys <<EOF
 {{ insertFile "../my-authorized_keys" | indent 4 }}
 EOF
 chmod -R go-w ~ec2-user/.ssh/authorized_keys
 chown -R ec2-user ~ec2-user/.ssh

Alternatively, you can save the key in your SSM Parameter Store and use the get-parameter command mentioned below to retrieve the key for authorized_keys.

aws ssm get-parameter --name "<parameter-name>" --region <region> --with-decryption --query "Parameter.Value" --output text > /home/ec2-user/.ssh/authorized_keys

For more examples on configuring fields for different AMI families, see the examples here.

AL2

Your UserData can be in the MIME multi part archive format.
Karpenter will transform your custom user-data as a MIME part, if necessary, and then merge a final MIME part to the end of your UserData parts which will bootstrap the worker node. Karpenter will have full control over all the parameters being passed to the bootstrap script.
- Karpenter will continue to set MaxPods, ClusterDNS and all other parameters defined in spec.kubeletConfiguration as before.

Consider the following example to understand how your custom UserData will be merged -

Passed-in UserData (bash)

#!/bin/bash
echo "Running custom user data script (bash)"

Merged UserData (bash)

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"

--//
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash
echo "Running custom user data script (bash)"

--//
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash -xe
exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1
/etc/eks/bootstrap.sh 'test-cluster' --apiserver-endpoint 'https://test-cluster' --b64-cluster-ca 'ca-bundle' \
--use-max-pods false \
--kubelet-extra-args '--node-labels=karpenter.sh/capacity-type=on-demand,karpenter.sh/nodepool=test --max-pods=110'
--//--

Passed-in UserData (MIME)

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="BOUNDARY"

--BOUNDARY
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash
echo "Running custom user data script (mime)"

--BOUNDARY--

Merged UserData (MIME)

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"

--//
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash
echo "Running custom user data script (mime)"

--//
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash -xe
exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1
/etc/eks/bootstrap.sh 'test-cluster' --apiserver-endpoint 'https://test-cluster' --b64-cluster-ca 'ca-bundle' \
--use-max-pods false \
--kubelet-extra-args '--node-labels=karpenter.sh/capacity-type=on-demand,karpenter.sh/nodepool=test --max-pods=110'
--//--

Tip

You can set additional kubelet configuration properties, unavailable through spec.kubelet, by updating the kubelet-config.json file:

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
 name: kubelet-config-example
spec:
 amiFamily: AL2
 userData: |
 #!/bin/bash
 echo "$(jq '.kubeAPIQPS=50' /etc/kubernetes/kubelet/kubelet-config.json)" > /etc/kubernetes/kubelet/kubelet-config.json

AL2023

Your UserData may be in one of three formats: a MIME multi part archive, a NodeConfig YAML / JSON string, or a shell script.
Karpenter will transform your custom UserData into a MIME part, if necessary, and then create a MIME multi-part archive. This archive will consist of a generated NodeConfig, containing Karpenter’s default values, followed by the transformed custom UserData. For more information on the NodeConfig spec, refer to the AL2023 EKS Optimized AMI docs.

Warning

Passed-in UserData (NodeConfig)

apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
 kubelet:
 config:
 maxPods: 42

Merged UserData (NodeConfig)

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"

--//
Content-Type: application/node.eks.aws

apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
 kubelet:
 config:
 maxPods: 42

--//
Content-Type: application/node.eks.aws

# Karpenter Generated NodeConfig
apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
 cluster:
 apiServerEndpoint: https://test-cluster
 certificateAuthority: cluster-ca
 cidr: 10.100.0.0/16
 name: test-cluster
 kubelet:
 config:
 clusterDNS:
 - 10.100.0.10
 maxPods: 118
 flags:
 - --node-labels="karpenter.sh/capacity-type=on-demand,karpenter.sh/nodepool=default"

--//--

Passed-in UserData (bash)

#!/bin/bash
echo "Hello, AL2023!"

Merged UserData (bash)

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"

--//
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash
echo "Hello, AL2023!"

--//
Content-Type: application/node.eks.aws

# Karpenter Generated NodeConfig
apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
 cluster:
 apiServerEndpoint: https://test-cluster
 certificateAuthority: cluster-ca
 cidr: 10.100.0.0/16
 name: test-cluster
 kubelet:
 config:
 clusterDNS:
 - 10.100.0.10
 maxPods: 118
 flags:
 - --node-labels="karpenter.sh/capacity-type=on-demand,karpenter.sh/nodepool=default"

--//--

Passed-in UserData (MIME)

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"

--//
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash
echo "Hello, AL2023!"

--//
Content-Type: application/node.eks.aws

apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
 kubelet:
 config:
 maxPods: 42
--//

Merged UserData (MIME)

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"

--//
Content-Type: application/node.eks.aws

apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
 kubelet:
 config:
 maxPods: 42
--//
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash
echo "Hello, AL2023!"

--//
Content-Type: application/node.eks.aws

# Karpenter Generated NodeConfig
apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
 cluster:
 apiServerEndpoint: https://test-cluster
 certificateAuthority: cluster-ca
 cidr: 10.100.0.0/16
 name: test-cluster
 kubelet:
 config:
 clusterDNS:
 - 10.100.0.10
 maxPods: 118
 flags:
 - --node-labels="karpenter.sh/capacity-type=on-demand,karpenter.sh/nodepool=default"

--//--

Bottlerocket

Your UserData must be valid TOML.
Unknown TOML fields will be ignored when the final merged UserData is generated by Karpenter.

Warning

Consider the following example to understand how your custom UserData settings will be merged in.

Passed-in UserData

[settings.kubernetes.eviction-hard]
"memory.available" = "12%"
[settings.kubernetes]
"unknown-setting" = "unknown"
[settings.kubernetes.node-labels]
'field.controlled.by/karpenter' = 'will-be-overridden'

Merged UserData

[settings]
[settings.kubernetes]
api-server = 'https://cluster'
cluster-certificate = 'ca-bundle'
cluster-name = 'cluster'

[settings.kubernetes.node-labels]
'karpenter.sh/capacity-type' = 'on-demand'
'karpenter.sh/nodepool' = 'default'

[settings.kubernetes.node-taints]

[settings.kubernetes.eviction-hard]
'memory.available' = '12%%'

Device ownership in Bottlerocket

Bottlerocket v1.30.0+ supports device ownership using the security context provided in the Kubernetes specfile. To enable this, you will need the following user-data configurations:

[settings]
[settings.kubernetes]
device-ownership-from-security-context = true

This setting helps you enable Neuron workloads on Bottlerocket instances. See Accelerators/GPU Resources for more details.

Windows2019/Windows2022

Your UserData must be specified as PowerShell commands.
The UserData specified will be prepended to a Karpenter managed section that will bootstrap the kubelet.
Karpenter will continue to set ClusterDNS and all other parameters defined in spec.kubeletConfiguration as before.

Consider the following example to understand how your custom UserData settings will be merged in.

Passed-in UserData

Write-Host "Running custom user data script"

Merged UserData

<powershell>
Write-Host "Running custom user data script"
[string]$EKSBootstrapScriptFile = "$env:ProgramFiles\Amazon\EKS\Start-EKSBootstrap.ps1"
& $EKSBootstrapScriptFile -EKSClusterName 'test-cluster' -APIServerEndpoint 'https://test-cluster' -Base64ClusterCA 'ca-bundle' -KubeletExtraArgs '--node-labels="karpenter.sh/capacity-type=spot,karpenter.sh/nodepool=windows2022" --max-pods=110' -DNSClusterIP '10.0.100.10'
</powershell>

Windows Support Notice

spec:
 userData: |
 $global:EKSCluster = Get-EKSCluster -Name my-cluster

Custom

No merging is performed, your UserData must perform all setup required of the node to allow it to join the cluster.
Custom UserData must meet the following requirements to work correctly with Karpenter:
- It must ensure the node is registered with the karpenter.sh/unregistered:NoExecute taint (via kubelet configuration field registerWithTaints)
- It must set kubelet config options to match those configured in spec.kubelet

spec.detailedMonitoring

spec:
 detailedMonitoring: true

spec.associatePublicIPAddress

Note

spec.ipPrefixCount

status.subnets

Examples

spec:
 subnetSelectorTerms:
 - tags:
 karpenter.sh/discovery: "${CLUSTER_NAME}"
status:
 subnets:
 - id: subnet-0a462d98193ff9fac
 zone: us-east-2b
 - id: subnet-0322dfafd76a609b6
 zone: us-east-2c
 - id: subnet-0727ef01daf4ac9fe
 zone: us-east-2b
 - id: subnet-00c99aeafe2a70304
 zone: us-east-2a
 - id: subnet-023b232fd5eb0028e
 zone: us-east-2c
 - id: subnet-03941e7ad6afeaa72
 zone: us-east-2a

status.securityGroups

Examples

spec:
 securityGroupSelectorTerms:
 - tags:
 karpenter.sh/discovery: "${CLUSTER_NAME}"
status:
 securityGroups:
 - id: sg-041513b454818610b
 name: ClusterSharedNodeSecurityGroup
 - id: sg-0286715698b894bca
 name: ControlPlaneSecurityGroup-1AQ073TSAAPW

status.amis

Examples

AMIs resolved with an AL2 alias:

spec:
 amiSelectorTerms:
 - alias: al2@v20240807
status:
 amis:
 - id: ami-03c3a3dcda64f5b75
 name: amazon-linux-2-gpu
 requirements:
 - key: kubernetes.io/arch
 operator: In
 values:
 - amd64
 - key: karpenter.k8s.aws/instance-gpu-count
 operator: Exists
 - id: ami-03c3a3dcda64f5b75
 name: amazon-linux-2-gpu
 requirements:
 - key: kubernetes.io/arch
 operator: In
 values:
 - amd64
 - key: karpenter.k8s.aws/instance-accelerator-count
 operator: Exists
 - id: ami-06afb2d101cc4b8bd
 name: amazon-linux-2-arm64
 requirements:
 - key: kubernetes.io/arch
 operator: In
 values:
 - arm64
 - key: karpenter.k8s.aws/instance-gpu-count
 operator: DoesNotExist
 - key: karpenter.k8s.aws/instance-accelerator-count
 operator: DoesNotExist
 - id: ami-0e28b76d768af234e
 name: amazon-linux-2
 requirements:
 - key: kubernetes.io/arch
 operator: In
 values:
 - amd64
 - key: karpenter.k8s.aws/instance-gpu-count
 operator: DoesNotExist
 - key: karpenter.k8s.aws/instance-accelerator-count
 operator: DoesNotExist

AMIs resolved from tags:

spec:
 amiSelectorTerms:
 - tags:
 karpenter.sh/discovery: "${CLUSTER_NAME}"
status:
 amis:
 - id: ami-01234567890123456
 name: custom-ami-amd64
 deprecated: true
 requirements:
 - key: kubernetes.io/arch
 operator: In
 values:
 - amd64
 - id: ami-01234567890123456
 name: custom-ami-arm64
 requirements:
 - key: kubernetes.io/arch
 operator: In
 values:
 - arm64

status.capacityReservations

status.capacityReservations contains the following information for each resolved capacity reservation:

Field	Example	Description
`availabilityZone`	`us-east-1a`	The availability zone the capacity reservation is available in
`id`	`cr-56fac701cc1951b03`	The ID of the capacity reservation
`instanceMatchCriteria`	`open`	The instanceMatchCriteria for the capacity reservation. Can be `open` or `targeted`.
`instanceType`	`m5.large`	The EC2 instance type of the capacity reservation
`ownerID`	`459763720645`	The account ID that owns the capacity reservation
`reservationType`	`default`	The type of the capacity reservation. Can be `default` or `capacity-block`.
`state`	`active`	The state of the capacity reservation. Can be `active` or `expiring`.

Examples

status:
 capacityReservations:
 - availabilityZone: us-west-2a
 id: cr-01234567890123456
 instanceMatchCriteria: targeted
 instanceType: g6.48xlarge
 ownerID: "012345678901"
 reservationType: capacity-block
 state: expiring
 - availabilityZone: us-west-2c
 id: cr-12345678901234567
 instanceMatchCriteria: open
 instanceType: g6.48xlarge
 ownerID: "98765432109"
 reservationType: default
 state: active

status.instanceProfile

status.instanceProfile contains the resolved instance profile generated by Karpenter from the spec.role

spec:
 role: "KarpenterNodeRole-${CLUSTER_NAME}"
status:
 instanceProfile: "${CLUSTER_NAME}-0123456778901234567789"

status.conditions

NodeClasses have the following status conditions:

Condition Type	Description
SubnetsReady	Subnets are discovered.
SecurityGroupsReady	Security Groups are discovered.
InstanceProfileReady	Instance Profile is discovered.
AMIsReady	AMIs are discovered.
Ready	Top level condition that indicates if the nodeClass is ready. If any of the underlying conditions is `False` then this condition is set to `False` and `Message` on the condition indicates the dependency that was not resolved.

If a NodeClass is not ready, NodePools that reference it through their nodeClassRef will not be considered for scheduling.

V1.9: NodeClasses

Mon, 01 Jan 0001 00:00:00 +0000

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
 name: default
spec:
 template:
 spec:
 nodeClassRef:
 group: karpenter.k8s.aws
 kind: EC2NodeClass
 name: default
---
apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
 name: default
spec:
 kubelet:
 podsPerCore: 2
 maxPods: 20
 systemReserved:
 cpu: 100m
 memory: 100Mi
 ephemeral-storage: 1Gi
 kubeReserved:
 cpu: 200m
 memory: 100Mi
 ephemeral-storage: 3Gi
 evictionHard:
 memory.available: 5%
 nodefs.available: 10%
 nodefs.inodesFree: 10%
 evictionSoft:
 memory.available: 500Mi
 nodefs.available: 15%
 nodefs.inodesFree: 15%
 evictionSoftGracePeriod:
 memory.available: 1m
 nodefs.available: 1m30s
 nodefs.inodesFree: 2m
 evictionMaxPodGracePeriod: 60
 imageGCHighThresholdPercent: 85
 imageGCLowThresholdPercent: 80
 cpuCFSQuota: true
 clusterDNS: ["10.0.1.100"]
 # Optional, dictates UserData generation and default block device mappings.
 # May be ommited when using an `alias` amiSelectorTerm, otherwise required.
 amiFamily: AL2

 # Required, discovers subnets to attach to instances
 # Each term in the array of subnetSelectorTerms is ORed together
 # Within a single term, all conditions are ANDed
 subnetSelectorTerms:
 # Select on any subnet that has the "karpenter.sh/discovery: ${CLUSTER_NAME}"
 # AND the "environment: test" tag OR any subnet with ID "subnet-09fa4a0a8f233a921"
 - tags:
 karpenter.sh/discovery: "${CLUSTER_NAME}"
 environment: test
 - id: subnet-09fa4a0a8f233a921

 # Required, discovers security groups to attach to instances
 # Each term in the array of securityGroupSelectorTerms is ORed together
 # Within a single term, all conditions are ANDed
 securityGroupSelectorTerms:
 # Select on any security group that has both the "karpenter.sh/discovery: ${CLUSTER_NAME}" tag
 # AND the "environment: test" tag OR any security group with the "my-security-group" name
 # OR any security group with ID "sg-063d7acfb4b06c82c"
 - tags:
 karpenter.sh/discovery: "${CLUSTER_NAME}"
 environment: test
 - name: my-security-group
 - id: sg-063d7acfb4b06c82c

 # Optional, IAM role to use for the node identity.
 # Must specify one of "role" or "instanceProfile" for Karpenter to launch nodes
 role: "KarpenterNodeRole-${CLUSTER_NAME}"

 # Optional, IAM instance profile to use for the node identity.
 # Must specify one of "role" or "instanceProfile" for Karpenter to launch nodes
 instanceProfile: "KarpenterNodeInstanceProfile-${CLUSTER_NAME}"

 # Each term in the array of amiSelectorTerms is ORed together
 # Within a single term, all conditions are ANDed
 amiSelectorTerms:
 # Select on any AMI that has both the `karpenter.sh/discovery: ${CLUSTER_NAME}`
 # AND `environment: test` tags OR any AMI with the name `my-ami` OR an AMI with
 # ID `ami-123` OR an AMI with ID matching the value of my-custom-parameter
 - tags:
 karpenter.sh/discovery: "${CLUSTER_NAME}"
 environment: test
 - name: my-ami
 - id: ami-123
 - ssmParameter: my-custom-parameter # ssm parameter name or ARN
 # Select EKS optimized AL2023 AMIs with version `v20240703`. This term is mutually
 # exclusive and can't be specified with other terms.
 # - alias: al2023@v20240703

 # Optional, each term in the array of capacityReservationSelectorTerms is ORed together.
 capacityReservationSelectorTerms:
 - tags:
 karpenter.sh/discovery: ${CLUSTER_NAME}
 - id: cr-123
 - instanceMatchCriteria: open

 # Optional, propagates tags to underlying EC2 resources
 tags:
 team: team-a
 app: team-a-app

 # Optional, configures IMDS for the instance
 metadataOptions:
 httpEndpoint: enabled
 httpProtocolIPv6: disabled
 httpPutResponseHopLimit: 1 # This is changed to disable IMDS access from containers not on the host network
 httpTokens: required

 # Optional, configures storage devices for the instance
 blockDeviceMappings:
 - deviceName: /dev/xvda
 ebs:
 volumeSize: 100Gi
 volumeType: gp3
 iops: 10000
 encrypted: true
 kmsKeyID: "1234abcd-12ab-34cd-56ef-1234567890ab"
 deleteOnTermination: true
 throughput: 125
 snapshotID: snap-0123456789
 volumeInitializationRate: 100

 # Optional, use instance-store volumes for node ephemeral-storage
 instanceStorePolicy: RAID0

 # Optional, overrides autogenerated userdata with a merge semantic
 userData: |
 echo "Hello world"

 # Optional, configures detailed monitoring for the instance
 detailedMonitoring: true

 # Optional, configures if the instance should be launched with an associated public IP address.
 # If not specified, the default value depends on the subnet's public IP auto-assign setting.
 associatePublicIPAddress: true
status:
 # Resolved subnets
 subnets:
 - id: subnet-0a462d98193ff9fac
 zone: us-east-2b
 - id: subnet-0322dfafd76a609b6
 zone: us-east-2c
 - id: subnet-0727ef01daf4ac9fe
 zone: us-east-2b
 - id: subnet-00c99aeafe2a70304
 zone: us-east-2a
 - id: subnet-023b232fd5eb0028e
 zone: us-east-2c
 - id: subnet-03941e7ad6afeaa72
 zone: us-east-2a

 # Resolved security groups
 securityGroups:
 - id: sg-041513b454818610b
 name: ClusterSharedNodeSecurityGroup
 - id: sg-0286715698b894bca
 name: ControlPlaneSecurityGroup-1AQ073TSAAPW

 # Resolved AMIs
 amis:
 - id: ami-01234567890123456
 name: custom-ami-amd64
 requirements:
 - key: kubernetes.io/arch
 operator: In
 values:
 - amd64
 - id: ami-01234567890123456
 name: custom-ami-arm64
 requirements:
 - key: kubernetes.io/arch
 operator: In
 values:
 - arm64

 # Capacity Reservations
 capacityReservations:
 - availabilityZone: us-west-2a
 id: cr-01234567890123456
 instanceMatchCriteria: targeted
 instanceType: g6.48xlarge
 ownerID: "012345678901"
 reservationType: capacity-block
 state: expiring
 - availabilityZone: us-west-2c
 id: cr-12345678901234567
 instanceMatchCriteria: open
 instanceType: g6.48xlarge
 ownerID: "98765432109"
 reservationType: default
 state: active

 # Generated instance profile name from "role"
 instanceProfile: "${CLUSTER_NAME}-0123456778901234567789"
 conditions:
 - lastTransitionTime: "2024-02-02T19:54:34Z"
 status: "True"
 type: InstanceProfileReady
 - lastTransitionTime: "2024-02-02T19:54:34Z"
 status: "True"
 type: SubnetsReady
 - lastTransitionTime: "2024-02-02T19:54:34Z"
 status: "True"
 type: SecurityGroupsReady
 - lastTransitionTime: "2024-02-02T19:54:34Z"
 status: "True"
 type: AMIsReady
 - lastTransitionTime: "2024-02-02T19:54:34Z"
 status: "True"
 type: Ready

Refer to the NodePool docs for settings applicable to all providers. To explore various EC2NodeClass configurations, refer to the examples provided in the Karpenter Github repository.

spec.kubelet

kubelet:
 podsPerCore: 2
 maxPods: 20
 systemReserved:
 cpu: 100m
 memory: 100Mi
 ephemeral-storage: 1Gi
 kubeReserved:
 cpu: 200m
 memory: 100Mi
 ephemeral-storage: 3Gi
 evictionHard:
 memory.available: 5%
 nodefs.available: 10%
 nodefs.inodesFree: 10%
 evictionSoft:
 memory.available: 500Mi
 nodefs.available: 15%
 nodefs.inodesFree: 15%
 evictionSoftGracePeriod:
 memory.available: 1m
 nodefs.available: 1m30s
 nodefs.inodesFree: 2m
 evictionMaxPodGracePeriod: 60
 imageGCHighThresholdPercent: 85
 imageGCLowThresholdPercent: 80
 cpuCFSQuota: true
 clusterDNS: ["10.0.1.100"]

Note

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
spec:
 amiSelectorTerms:
 - alias: al2023@v20240807
 kubelet:
 maxPods: 42
 userData: |
 apiVersion: node.eks.aws/v1alpha1
 kind: NodeConfig
 spec:
 kubelet:
 config:
 # Configured through UserData since unavailable in `spec.kubelet`
 registryPullQPS: 10

Note that when using the Custom AMIFamily you will need to specify fields both in spec.kubelet and spec.userData.

Pods Per Core

Note

Max Pods

Note

Windows Support Notice

Reserved Resources

Note

Eviction Thresholds

kubelet:
 evictionHard:
 memory.available: 500Mi
 nodefs.available: 10%
 nodefs.inodesFree: 10%
 imagefs.available: 5%
 imagefs.inodesFree: 5%
 pid.available: 7%
 evictionSoft:
 memory.available: 1Gi
 nodefs.available: 15%
 nodefs.inodesFree: 15%
 imagefs.available: 10%
 imagefs.inodesFree: 10%
 pid.available: 10%

Supported Eviction Signals

Eviction Signal	Description
memory.available	memory.available := node.status.capacity[memory] - node.stats.memory.workingSet
nodefs.available	nodefs.available := node.stats.fs.available
nodefs.inodesFree	nodefs.inodesFree := node.stats.fs.inodesFree
imagefs.available	imagefs.available := node.stats.runtime.imagefs.available
imagefs.inodesFree	imagefs.inodesFree := node.stats.runtime.imagefs.inodesFree
pid.available	pid.available := node.stats.rlimit.maxpid - node.stats.rlimit.curproc

For more information on eviction thresholds, view the Node-pressure Eviction section of the official Kubernetes docs.

Soft Eviction Grace Periods

kubelet:
 evictionSoftGracePeriod:
 memory.available: 1m
 nodefs.available: 1m30s
 nodefs.inodesFree: 2m
 imagefs.available: 1m30s
 imagefs.inodesFree: 2m
 pid.available: 2m
 evictionMaxPodGracePeriod: 60

Pod Density

Note

Windows Support Notice

It’s currently not possible to specify custom networking with Windows nodes.

spec.amiFamily

Ubuntu Support Dropped at v1

Ubuntu NodeClass Example:

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
spec:
 amiFamily: AL2
 amiSelectorTerms:
 - id: ami-placeholder
 blockDeviceMappings:
 - deviceName: '/dev/sda1'
 rootVolume: true
 ebs:
 encrypted: true
 volumeType: gp3
 volumeSize: 20Gi

AL2

AL2 support dropped at Kubernetes 1.33

Kubernetes version 1.32 is the last version for which Amazon EKS will release Amazon Linux 2 (AL2) AMIs.

Note

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"

--//
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash -xe
exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1
/etc/eks/bootstrap.sh 'test-cluster' --apiserver-endpoint 'https://test-cluster' --b64-cluster-ca 'ca-bundle' \
--dns-cluster-ip '10.100.0.10' \
--use-max-pods false \
--kubelet-extra-args '--node-labels=karpenter.sh/capacity-type=on-demand,karpenter.sh/nodepool=test --max-pods=110'
--//--

AL2023

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"

--//
Content-Type: application/node.eks.aws

# Karpenter Generated NodeConfig
apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
 cluster:
 name: test-cluster
 apiServerEndpoint: https://example.com
 certificateAuthority: ca-bundle
 cidr: 10.100.0.0/16
 kubelet:
 config:
 maxPods: 110
 flags:
 - --node-labels=karpenter.sh/capacity-type=on-demand,karpenter.sh/nodepool=test

--//--

Bottlerocket

[settings]
[settings.kubernetes]
api-server = 'https://test-cluster'
cluster-certificate = 'ca-bundle'
cluster-name = 'test-cluster'
cluster-dns-ip = '10.100.0.10'
max-pods = 110

[settings.kubernetes.node-labels]
'karpenter.sh/capacity-type' = 'on-demand'
'karpenter.sh/nodepool' = 'test'

Windows2019

<powershell>
[string]$EKSBootstrapScriptFile = "$env:ProgramFiles\Amazon\EKS\Start-EKSBootstrap.ps1"
& $EKSBootstrapScriptFile -EKSClusterName 'test-cluster' -APIServerEndpoint 'https://test-cluster' -Base64ClusterCA 'ca-bundle' -KubeletExtraArgs '--node-labels="karpenter.sh/capacity-type=on-demand,karpenter.sh/nodepool=test" --max-pods=110' -DNSClusterIP '10.100.0.10'
</powershell>

Windows2022

<powershell>
[string]$EKSBootstrapScriptFile = "$env:ProgramFiles\Amazon\EKS\Start-EKSBootstrap.ps1"
& $EKSBootstrapScriptFile -EKSClusterName 'test-cluster' -APIServerEndpoint 'https://test-cluster' -Base64ClusterCA 'ca-bundle' -KubeletExtraArgs '--node-labels="karpenter.sh/capacity-type=on-demand,karpenter.sh/nodepool=test" --max-pods=110' -DNSClusterIP '10.100.0.10'
</powershell>

Windows2025

<powershell>
[string]$EKSBootstrapScriptFile = "$env:ProgramFiles\Amazon\EKS\Start-EKSBootstrap.ps1"
& $EKSBootstrapScriptFile -EKSClusterName 'test-cluster' -APIServerEndpoint 'https://test-cluster' -Base64ClusterCA 'ca-bundle' -KubeletExtraArgs '--node-labels="karpenter.sh/capacity-type=on-demand,karpenter.sh/nodepool=test" --max-pods=110' -DNSClusterIP '10.100.0.10'
</powershell>

Custom

spec.subnetSelectorTerms

subnetSelectorTerms:
 # Select on any subnet that has the "karpenter.sh/discovery: ${CLUSTER_NAME}"
 # AND the "environment: test" tag OR any subnet with ID "subnet-09fa4a0a8f233a921"
 - tags:
 karpenter.sh/discovery: "${CLUSTER_NAME}"
 environment: test
 - id: subnet-09fa4a0a8f233a921

Tip

Subnets may be specified by any tag, including Name. Selecting tag values using wildcards (*) is supported.

Examples

Select all with a specified tag key:

spec:
 subnetSelectorTerms:
 - tags:
 karpenter.sh/discovery/MyClusterName: '*'

Select by name and tag (all criteria must match):

spec:
 subnetSelectorTerms:
 - tags:
 Name: my-subnet
 MyTag: '' # matches all resources with the tag

Select using multiple tag terms:

spec:
 subnetSelectorTerms:
 - tags:
 Name: "my-subnet-1"
 - tags:
 Name: "my-subnet-2"

Select using wildcards:

spec:
 subnetSelectorTerms:
 - tags:
 Name: "*Public*"

Select using ids:

spec:
 subnetSelectorTerms:
 - id: "subnet-09fa4a0a8f233a921"
 - id: "subnet-0471ca205b8a129ae"

spec.securityGroupSelectorTerms

This selection logic is modeled as terms, where each term contains multiple conditions that must all be satisfied for the selector to match. Effectively, all requirements within a single term are ANDed together. It’s possible that you may want to select on two different security groups that have unrelated requirements. In this case, you can specify multiple terms which will be ORed together to form your selection logic. The example below shows how this selection logic is fulfilled.

securityGroupSelectorTerms:
 # Select on any security group that has both the "karpenter.sh/discovery: ${CLUSTER_NAME}" tag
 # AND the "environment: test" tag OR any security group with the "my-security-group" name
 # OR any security group with ID "sg-063d7acfb4b06c82c"
 - tags:
 karpenter.sh/discovery: "${CLUSTER_NAME}"
 environment: test
 - name: my-security-group
 - id: sg-063d7acfb4b06c82c

Tip

Security groups may be specified by any tag, including “Name”. Selecting tags using wildcards (*) is supported.

Note

To verify if this restriction affects you, run the following commands.

CLUSTER_VPC_ID="$(aws eks describe-cluster --name $CLUSTER_NAME --query cluster.resourcesVpcConfig.vpcId --output text)"

aws ec2 describe-security-groups --filters Name=vpc-id,Values=$CLUSTER_VPC_ID Name=tag-key,Values=kubernetes.io/cluster/$CLUSTER_NAME --query 'SecurityGroups[].[GroupName]' --output text

If multiple securityGroups are printed, you will need more specific securityGroupSelectorTerms. We generally recommend that you use the karpenter.sh/discovery: $CLUSTER_NAME tag selector instead.

Examples

Select all assigned to a cluster:

spec:
 securityGroupSelectorTerms:
 - tags:
 kubernetes.io/cluster/$CLUSTER_NAME: "owned"

Select all with a specified tag key:

spec:
 securityGroupSelectorTerms:
 - tags:
 MyTag: '*'

Select by name and tag (all criteria must match):

spec:
 securityGroupSelectorTerms:
 - name: my-security-group
 tags:
 MyTag: '*' # matches all resources with the tag

Select using multiple tag terms:

spec:
 securityGroupSelectorTerms:
 - tags:
 Name: "my-security-group-1"
 - tags:
 Name: "my-security-group-2"

Select by name using a wildcard:

spec:
 securityGroupSelectorTerms:
 - name: "*Public*"

Select using ids:

spec:
 securityGroupSelectorTerms:
 - id: "sg-063d7acfb4b06c82c"
 - id: "sg-06e0cf9c198874591"

spec.role

spec:
 role: "KarpenterNodeRole-$CLUSTER_NAME"

spec.instanceProfile

You can provision and assign a role to an IAM instance profile using CloudFormation or by using the aws iam create-instance-profile and aws iam add-role-to-instance-profile commands in the CLI.

Note

Warning

When using spec.instanceProfile, ensure you are using pre-provisioned instance profiles that you manage yourself.

spec.amiSelectorTerms

AMI Selector Terms are required and are used to configure AMIs for Karpenter to use. AMIs are discovered through alias, id, owner, name, and tags.

This selection logic is modeled as terms, where each term contains multiple conditions that must all be satisfied for the selector to match. Effectively, all requirements within a single term are ANDed together. It’s possible that you may want to select on two different AMIs that have unrelated requirements. In this case, you can specify multiple terms which will be ORed together to form your selection logic. The example below shows how this selection logic is fulfilled.

amiSelectorTerms:
 # Select on any AMI that has both the `karpenter.sh/discovery: ${CLUSTER_NAME}`
 # AND `environment: test` tags OR any AMI with the name `my-ami` OR an AMI with
 # ID `ami-123` OR an AMI with ID matching the value of my-custom-parameter
 - tags:
 karpenter.sh/discovery: "${CLUSTER_NAME}"
 environment: test
 - name: my-ami
 - id: ami-123
 - ssmParameter: my-custom-parameter # ssm parameter name or ARN
 # Select EKS optimized AL2023 AMIs with version `v20240807`. This term is mutually
 # exclusive and can't be specified with other terms.
 # - alias: al2023@v20240807

An alias term can be used to select EKS-optimized AMIs. An alias is formatted as family@version. Family can be one of the following values:

al2
al2023
bottlerocket
windows2019
windows2022
windows2025

alias: al2023@v20240703

Bottlerocket uses a semantic version for their releases. You can pin bottlerocket as follows:

alias: bottlerocket@v1.20.4

The Windows family does not support pinning, so only latest is supported.

The following commands can be used to determine the versions availble for an alias in your region:

export K8S_VERSION="1.34"
aws ssm get-parameters-by-path --path "/aws/service/eks/optimized-ami/$K8S_VERSION/amazon-linux-2023/" --recursive | jq -cr '.Parameters[].Name' | grep -v "recommended" | awk -F '/' '{print $10}' | sed -r 's/.*(v[[:digit:]]+)$/\1/' | sort | uniq

export K8S_VERSION="1.34"
aws ssm get-parameters-by-path --path "/aws/service/eks/optimized-ami/$K8S_VERSION/amazon-linux-2/" --recursive | jq -cr '.Parameters[].Name' | grep -v "recommended" | awk -F '/' '{print $8}' | sed -r 's/.*(v[[:digit:]]+)$/\1/' | sort | uniq

export K8S_VERSION="1.34"
aws ssm get-parameters-by-path --path "/aws/service/bottlerocket/aws-k8s-$K8S_VERSION" --recursive | jq -cr '.Parameters[].Name' | grep -v "latest" | awk -F '/' '{print $7}' | sort | uniq

Warning

Tip

AMIs may be specified by any AWS tag, including Name. Selecting by tag or by name using wildcards (*) is supported.

Note

If amiSelectorTerms match more than one AMI, Karpenter will automatically determine which AMI best fits the workloads on the launched worker node under the following constraints:

When launching nodes, Karpenter automatically determines which architecture a custom AMI is compatible with and will use images that match an instanceType’s requirements.
- Unless using an alias, Karpenter cannot detect requirements other than architecture. If you need to specify different AMIs for different kind of nodes (e.g. accelerated GPU AMIs), you should use a separate EC2NodeClass.
If multiple AMIs are found that can be used, Karpenter will choose the latest one.
If no AMIs are found that can be used, then no nodes will be provisioned.

Examples

Select by AMI family and version:

 amiSelectorTerms:
 - alias: al2023@v20240807

Select all with a specified tag:

 amiSelectorTerms:
 - tags:
 karpenter.sh/discovery/MyClusterName: '*'

Select by name:

 amiSelectorTerms:
 - name: my-ami

Select by Name tag:

 amiSelectorTerms:
 - tags:
 Name: my-ami

Select by name and owner:

 amiSelectorTerms:
 - name: my-ami
 owner: self
 - name: my-ami
 owner: "0123456789"

Select by name using a wildcard:

spec:
 amiSelectorTerms:
 - name: "*EKS*"

Select by all under an owner:

spec:
 amiSelectorTerms:
 - name: "*"
 owner: self

Specify using ids:

 amiSelectorTerms:
 - id: "ami-123"
 - id: "ami-456"

Specify using custom ssm parameter name or ARN:

 amiSelectorTerms:
 - ssmParameter: "my-custom-parameter"

Note

spec.capacityReservationSelectorTerms

Feature State: Beta

For more information on utilizing ODCRs with Karpenter, refer to the Utilizing ODCRs Task.

Note

Examples

Select the reservations with the given IDs:

spec:
 capacityReservationSelectorTerms:
 - id: cr-123
 - id: cr-456

Select the reservations by tags:

spec:
 capacityReservationSelectorTerms:
 # Select all capacity reservations which have both matching tags
 - tags:
 key1: foo
 key2: bar
 # Additionally, select all capacity reservations with the following matching tag
 - tags:
 key3: foobar

Select by tags and owner ID:

spec:
 # Select all capacity reservations with the matching tags which are also owned by
 # the specified account.
 capacityReservationSelectorTerms:
 - tags:
 key: foo
 ownerID: 012345678901

Select by instance match criteria:

spec:
 capacityReservationSelectorTerms:
 # Select all open capacity reservations
 - instanceMatchCriteria: open

Select by instance match criteria and tags:

spec:
 capacityReservationSelectorTerms:
 # Select targeted capacity reservations with matching tags
 - instanceMatchCriteria: targeted
 tags:
 key: foo

spec.tags

Karpenter adds tags to all resources it creates, including EC2 Instances, EBS volumes, and Launch Templates. The default set of tags are listed below.

Name: <node-name>
karpenter.sh/nodeclaim: <nodeclaim-name>
karpenter.sh/nodepool: <nodepool-name>
karpenter.k8s.aws/ec2nodeclass: <ec2nodeclass-name>
kubernetes.io/cluster/<cluster-name>: owned
eks:eks-cluster-name: <cluster-name>

Additional tags can be added in the tags section, which will be merged with the default tags specified above.

spec:
 tags:
 InternalAccountingTag: 1234
 dev.corp.net/app: Calculator
 dev.corp.net/team: MyTeam

Note

spec.metadataOptions

Control the exposure of Instance Metadata Service on EC2 Instances launched by this EC2NodeClass using a generated launch template.

Refer to recommended, security best practices for limiting exposure of Instance Metadata and User Data to pods.

If metadataOptions are omitted from this EC2NodeClass, the following default settings are applied:

spec:
 metadataOptions:
 httpEndpoint: enabled
 httpProtocolIPv6: disabled
 httpPutResponseHopLimit: 1
 httpTokens: required

spec.blockDeviceMappings

spec:
 blockDeviceMappings:
 - deviceName: /dev/xvda
 ebs:
 volumeSize: 100Gi
 volumeType: gp3
 iops: 10000
 encrypted: true
 kmsKeyID: "1234abcd-12ab-34cd-56ef-1234567890ab"
 deleteOnTermination: true
 throughput: 125
 snapshotID: snap-0123456789
 volumeInitializationRate: 100

The following blockDeviceMapping defaults are used for each AMIFamily if no blockDeviceMapping overrides are specified in the EC2NodeClass

AL2

spec:
 blockDeviceMappings:
 - deviceName: /dev/xvda
 ebs:
 volumeSize: 20Gi
 volumeType: gp3
 encrypted: true

AL2023

spec:
 blockDeviceMappings:
 - deviceName: /dev/xvda
 ebs:
 volumeSize: 20Gi
 volumeType: gp3
 encrypted: true

Bottlerocket

spec:
 blockDeviceMappings:
 # Root device
 - deviceName: /dev/xvda
 ebs:
 volumeSize: 4Gi
 volumeType: gp3
 encrypted: true
 # Data device: Container resources such as images and logs
 - deviceName: /dev/xvdb
 ebs:
 volumeSize: 20Gi
 volumeType: gp3
 encrypted: true

Windows2019/Windows2022/Windows2025

spec:
 blockDeviceMappings:
 - deviceName: /dev/sda1
 ebs:
 volumeSize: 50Gi
 volumeType: gp3
 encrypted: true

Custom

The Custom AMIFamily ships without any default blockDeviceMappings.

spec.instanceStorePolicy

The instanceStorePolicy field controls how instance-store volumes are handled. By default, Karpenter and Kubernetes will simply ignore them.

RAID0

If you intend to use these volumes for faster node ephemeral-storage, set instanceStorePolicy to RAID0:

spec:
 instanceStorePolicy: RAID0

Instructions for each AMI family are listed below:

AL2

AL2023

Bottlerocket

Note

This automatic disk configuration is only available on Bottlerocket v1.22.0 and later.

Others

For all other AMI families, you must configure the disks yourself. Check out the setup-local-disks script in amazon-eks-ami to see how this is done for AL2.

Tip

Since the Kubelet & Containerd will be using the instance-store filesystem, you may consider using a more minimal root volume size.

spec.userData

You can control the UserData that is applied to your worker nodes via this field. This allows you to run custom scripts or pass-through custom configuration to Karpenter instances on start-up.

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
 name: bottlerocket-example
spec:
 ...
 amiFamily: Bottlerocket
 userData: |
 [settings.kubernetes]
 "kube-api-qps" = 30
 "shutdown-grace-period" = "30s"
 "shutdown-grace-period-for-critical-pods" = "30s"
 [settings.kubernetes.eviction-hard]
 "memory.available" = "20%"

This example adds SSH keys to allow remote login to the node (replace my-authorized_keys with your public key file):

Note

Also, my-authorized_key key is the public key. See Retrieve the public key material.

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
 name: al2-example
spec:
 ...
 amiFamily: AL2
 userData: |
 #!/bin/bash
 mkdir -p ~ec2-user/.ssh/
 touch ~ec2-user/.ssh/authorized_keys
 cat >> ~ec2-user/.ssh/authorized_keys <<EOF
 {{ insertFile "../my-authorized_keys" | indent 4 }}
 EOF
 chmod -R go-w ~ec2-user/.ssh/authorized_keys
 chown -R ec2-user ~ec2-user/.ssh

Alternatively, you can save the key in your SSM Parameter Store and use the get-parameter command mentioned below to retrieve the key for authorized_keys.

aws ssm get-parameter --name "<parameter-name>" --region <region> --with-decryption --query "Parameter.Value" --output text > /home/ec2-user/.ssh/authorized_keys

For more examples on configuring fields for different AMI families, see the examples here.

AL2

Your UserData can be in the MIME multi part archive format.
Karpenter will transform your custom user-data as a MIME part, if necessary, and then merge a final MIME part to the end of your UserData parts which will bootstrap the worker node. Karpenter will have full control over all the parameters being passed to the bootstrap script.
- Karpenter will continue to set MaxPods, ClusterDNS and all other parameters defined in spec.kubeletConfiguration as before.

Consider the following example to understand how your custom UserData will be merged -

Passed-in UserData (bash)

#!/bin/bash
echo "Running custom user data script (bash)"

Merged UserData (bash)

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"

--//
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash
echo "Running custom user data script (bash)"

--//
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash -xe
exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1
/etc/eks/bootstrap.sh 'test-cluster' --apiserver-endpoint 'https://test-cluster' --b64-cluster-ca 'ca-bundle' \
--use-max-pods false \
--kubelet-extra-args '--node-labels=karpenter.sh/capacity-type=on-demand,karpenter.sh/nodepool=test --max-pods=110'
--//--

Passed-in UserData (MIME)

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="BOUNDARY"

--BOUNDARY
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash
echo "Running custom user data script (mime)"

--BOUNDARY--

Merged UserData (MIME)

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"

--//
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash
echo "Running custom user data script (mime)"

--//
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash -xe
exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1
/etc/eks/bootstrap.sh 'test-cluster' --apiserver-endpoint 'https://test-cluster' --b64-cluster-ca 'ca-bundle' \
--use-max-pods false \
--kubelet-extra-args '--node-labels=karpenter.sh/capacity-type=on-demand,karpenter.sh/nodepool=test --max-pods=110'
--//--

Tip

You can set additional kubelet configuration properties, unavailable through spec.kubelet, by updating the kubelet-config.json file:

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
 name: kubelet-config-example
spec:
 amiFamily: AL2
 userData: |
 #!/bin/bash
 echo "$(jq '.kubeAPIQPS=50' /etc/kubernetes/kubelet/kubelet-config.json)" > /etc/kubernetes/kubelet/kubelet-config.json

AL2023

Your UserData may be in one of three formats: a MIME multi part archive, a NodeConfig YAML / JSON string, or a shell script.
Karpenter will transform your custom UserData into a MIME part, if necessary, and then create a MIME multi-part archive. This archive will consist of a generated NodeConfig, containing Karpenter’s default values, followed by the transformed custom UserData. For more information on the NodeConfig spec, refer to the AL2023 EKS Optimized AMI docs.

Warning

Passed-in UserData (NodeConfig)

apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
 kubelet:
 config:
 maxPods: 42

Merged UserData (NodeConfig)

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"

--//
Content-Type: application/node.eks.aws

apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
 kubelet:
 config:
 maxPods: 42

--//
Content-Type: application/node.eks.aws

# Karpenter Generated NodeConfig
apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
 cluster:
 apiServerEndpoint: https://test-cluster
 certificateAuthority: cluster-ca
 cidr: 10.100.0.0/16
 name: test-cluster
 kubelet:
 config:
 clusterDNS:
 - 10.100.0.10
 maxPods: 118
 flags:
 - --node-labels="karpenter.sh/capacity-type=on-demand,karpenter.sh/nodepool=default"

--//--

Passed-in UserData (bash)

#!/bin/bash
echo "Hello, AL2023!"

Merged UserData (bash)

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"

--//
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash
echo "Hello, AL2023!"

--//
Content-Type: application/node.eks.aws

# Karpenter Generated NodeConfig
apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
 cluster:
 apiServerEndpoint: https://test-cluster
 certificateAuthority: cluster-ca
 cidr: 10.100.0.0/16
 name: test-cluster
 kubelet:
 config:
 clusterDNS:
 - 10.100.0.10
 maxPods: 118
 flags:
 - --node-labels="karpenter.sh/capacity-type=on-demand,karpenter.sh/nodepool=default"

--//--

Passed-in UserData (MIME)

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"

--//
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash
echo "Hello, AL2023!"

--//
Content-Type: application/node.eks.aws

apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
 kubelet:
 config:
 maxPods: 42
--//

Merged UserData (MIME)

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"

--//
Content-Type: application/node.eks.aws

apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
 kubelet:
 config:
 maxPods: 42
--//
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash
echo "Hello, AL2023!"

--//
Content-Type: application/node.eks.aws

# Karpenter Generated NodeConfig
apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
 cluster:
 apiServerEndpoint: https://test-cluster
 certificateAuthority: cluster-ca
 cidr: 10.100.0.0/16
 name: test-cluster
 kubelet:
 config:
 clusterDNS:
 - 10.100.0.10
 maxPods: 118
 flags:
 - --node-labels="karpenter.sh/capacity-type=on-demand,karpenter.sh/nodepool=default"

--//--

Bottlerocket

Your UserData must be valid TOML.
Unknown TOML fields will be ignored when the final merged UserData is generated by Karpenter.

Warning

Consider the following example to understand how your custom UserData settings will be merged in.

Passed-in UserData

[settings.kubernetes.eviction-hard]
"memory.available" = "12%"
[settings.kubernetes]
"unknown-setting" = "unknown"
[settings.kubernetes.node-labels]
'field.controlled.by/karpenter' = 'will-be-overridden'

Merged UserData

[settings]
[settings.kubernetes]
api-server = 'https://cluster'
cluster-certificate = 'ca-bundle'
cluster-name = 'cluster'

[settings.kubernetes.node-labels]
'karpenter.sh/capacity-type' = 'on-demand'
'karpenter.sh/nodepool' = 'default'

[settings.kubernetes.node-taints]

[settings.kubernetes.eviction-hard]
'memory.available' = '12%%'

Device ownership in Bottlerocket

Bottlerocket v1.30.0+ supports device ownership using the security context provided in the Kubernetes specfile. To enable this, you will need the following user-data configurations:

[settings]
[settings.kubernetes]
device-ownership-from-security-context = true

This setting helps you enable Neuron workloads on Bottlerocket instances. See Accelerators/GPU Resources for more details.

Windows2019/Windows2022/Windows2025

Your UserData must be specified as PowerShell commands.
The UserData specified will be prepended to a Karpenter managed section that will bootstrap the kubelet.
Karpenter will continue to set ClusterDNS and all other parameters defined in spec.kubeletConfiguration as before.

Consider the following example to understand how your custom UserData settings will be merged in.

Passed-in UserData

Write-Host "Running custom user data script"

Merged UserData

<powershell>
Write-Host "Running custom user data script"
[string]$EKSBootstrapScriptFile = "$env:ProgramFiles\Amazon\EKS\Start-EKSBootstrap.ps1"
& $EKSBootstrapScriptFile -EKSClusterName 'test-cluster' -APIServerEndpoint 'https://test-cluster' -Base64ClusterCA 'ca-bundle' -KubeletExtraArgs '--node-labels="karpenter.sh/capacity-type=spot,karpenter.sh/nodepool=windows2022" --max-pods=110' -DNSClusterIP '10.0.100.10'
</powershell>

Windows Support Notice

spec:
 userData: |
 $global:EKSCluster = Get-EKSCluster -Name my-cluster

Custom

No merging is performed, your UserData must perform all setup required of the node to allow it to join the cluster.
Custom UserData must meet the following requirements to work correctly with Karpenter:
- It must ensure the node is registered with the karpenter.sh/unregistered:NoExecute taint (via kubelet configuration field registerWithTaints)
- It must set kubelet config options to match those configured in spec.kubelet

spec.detailedMonitoring

spec:
 detailedMonitoring: true

spec.associatePublicIPAddress

Note

spec.ipPrefixCount

status.subnets

Examples

spec:
 subnetSelectorTerms:
 - tags:
 karpenter.sh/discovery: "${CLUSTER_NAME}"
status:
 subnets:
 - id: subnet-0a462d98193ff9fac
 zone: us-east-2b
 - id: subnet-0322dfafd76a609b6
 zone: us-east-2c
 - id: subnet-0727ef01daf4ac9fe
 zone: us-east-2b
 - id: subnet-00c99aeafe2a70304
 zone: us-east-2a
 - id: subnet-023b232fd5eb0028e
 zone: us-east-2c
 - id: subnet-03941e7ad6afeaa72
 zone: us-east-2a

status.securityGroups

Examples

spec:
 securityGroupSelectorTerms:
 - tags:
 karpenter.sh/discovery: "${CLUSTER_NAME}"
status:
 securityGroups:
 - id: sg-041513b454818610b
 name: ClusterSharedNodeSecurityGroup
 - id: sg-0286715698b894bca
 name: ControlPlaneSecurityGroup-1AQ073TSAAPW

status.amis

Examples

AMIs resolved with an AL2 alias:

spec:
 amiSelectorTerms:
 - alias: al2@v20240807
status:
 amis:
 - id: ami-03c3a3dcda64f5b75
 name: amazon-linux-2-gpu
 requirements:
 - key: kubernetes.io/arch
 operator: In
 values:
 - amd64
 - key: karpenter.k8s.aws/instance-gpu-count
 operator: Exists
 - id: ami-03c3a3dcda64f5b75
 name: amazon-linux-2-gpu
 requirements:
 - key: kubernetes.io/arch
 operator: In
 values:
 - amd64
 - key: karpenter.k8s.aws/instance-accelerator-count
 operator: Exists
 - id: ami-06afb2d101cc4b8bd
 name: amazon-linux-2-arm64
 requirements:
 - key: kubernetes.io/arch
 operator: In
 values:
 - arm64
 - key: karpenter.k8s.aws/instance-gpu-count
 operator: DoesNotExist
 - key: karpenter.k8s.aws/instance-accelerator-count
 operator: DoesNotExist
 - id: ami-0e28b76d768af234e
 name: amazon-linux-2
 requirements:
 - key: kubernetes.io/arch
 operator: In
 values:
 - amd64
 - key: karpenter.k8s.aws/instance-gpu-count
 operator: DoesNotExist
 - key: karpenter.k8s.aws/instance-accelerator-count
 operator: DoesNotExist

AMIs resolved from tags:

spec:
 amiSelectorTerms:
 - tags:
 karpenter.sh/discovery: "${CLUSTER_NAME}"
status:
 amis:
 - id: ami-01234567890123456
 name: custom-ami-amd64
 deprecated: true
 requirements:
 - key: kubernetes.io/arch
 operator: In
 values:
 - amd64
 - id: ami-01234567890123456
 name: custom-ami-arm64
 requirements:
 - key: kubernetes.io/arch
 operator: In
 values:
 - arm64

status.capacityReservations

status.capacityReservations contains the following information for each resolved capacity reservation:

Field	Example	Description
`availabilityZone`	`us-east-1a`	The availability zone the capacity reservation is available in
`id`	`cr-56fac701cc1951b03`	The ID of the capacity reservation
`instanceMatchCriteria`	`open`	The instanceMatchCriteria for the capacity reservation. Can be `open` or `targeted`.
`instanceType`	`m5.large`	The EC2 instance type of the capacity reservation
`ownerID`	`459763720645`	The account ID that owns the capacity reservation
`reservationType`	`default`	The type of the capacity reservation. Can be `default` or `capacity-block`.
`state`	`active`	The state of the capacity reservation. Can be `active` or `expiring`.

Examples

status:
 capacityReservations:
 - availabilityZone: us-west-2a
 id: cr-01234567890123456
 instanceMatchCriteria: targeted
 instanceType: g6.48xlarge
 ownerID: "012345678901"
 reservationType: capacity-block
 state: expiring
 - availabilityZone: us-west-2c
 id: cr-12345678901234567
 instanceMatchCriteria: open
 instanceType: g6.48xlarge
 ownerID: "98765432109"
 reservationType: default
 state: active

status.instanceProfile

status.instanceProfile contains the resolved instance profile generated by Karpenter from the spec.role

spec:
 role: "KarpenterNodeRole-${CLUSTER_NAME}"
status:
 instanceProfile: "${CLUSTER_NAME}-0123456778901234567789"

status.conditions

NodeClasses have the following status conditions:

Condition Type	Description
SubnetsReady	Subnets are discovered.
SecurityGroupsReady	Security Groups are discovered.
InstanceProfileReady	Instance Profile is discovered.
AMIsReady	AMIs are discovered.
Ready	Top level condition that indicates if the nodeClass is ready. If any of the underlying conditions is `False` then this condition is set to `False` and `Message` on the condition indicates the dependency that was not resolved.

If a NodeClass is not ready, NodePools that reference it through their nodeClassRef will not be considered for scheduling.

Docs: CloudFormation

Mon, 01 Jan 0001 00:00:00 +0000

The Getting Started with Karpenter guide uses CloudFormation to bootstrap the cluster to enable Karpenter to create and manage nodes, as well as to allow Karpenter to respond to interruption events. This document describes the cloudformation.yaml file used in that guide. These descriptions should allow you to understand:

What Karpenter is authorized to do with your EKS cluster and AWS resources when using the cloudformation.yaml file
What permissions you need to set up if you are adding Karpenter to an existing cluster

Overview

To download a particular version of cloudformation.yaml, set the version and use curl to pull the file to your local system:

export KARPENTER_VERSION="1.10.0"
curl https://raw.githubusercontent.com/aws/karpenter-provider-aws/v"${KARPENTER_VERSION}"/website/content/en/preview/getting-started/getting-started-with-karpenter/cloudformation.yaml > cloudformation.yaml

Following some header information, the rest of the cloudformation.yaml file describes the resources that CloudFormation deploys. The sections of that file can be grouped together under the following general headings:

Node Authorization: Creates a NodeInstanceProfile, attaches a NodeRole to it, and connects it to an IAM Identity Mapping used to authorize nodes to the cluster. This defines the permissions each node managed by Karpenter has to access EC2 and other AWS resources. This doesn’t actually create the IAM Identity Mapping. That part is orchestrated by eksctl in the Getting Started guide.
Controller Authorization: Creates 5 Karpenter controller policies that are attached to the service account’s IAM role. Again, the actual service account creation (karpenter), that is combined with these policies, is orchestrated by eksctl in the Getting Started guide.
Interruption Handling: Allows the Karpenter controller to see and respond to interruptions that occur with the nodes that Karpenter is managing. See the Interruption section of the Disruption page for details.

A lot of the object naming that is done by cloudformation.yaml is based on the following:

Cluster name: With a username of bob the Getting Started Guide would name your cluster bob-karpenter-demo That name would then be appended to any name below where ${ClusterName} is included.
Partition: Any time an ARN is used, it includes the partition name to identify where the object is found. In most cases, that partition name is aws. However, it could also be aws-cn (for China Regions) or aws-us-gov (for AWS GovCloud US Regions).

Node Authorization

The following sections of the cloudformation.yaml file set up IAM permissions for Kubernetes nodes created by Karpenter. In particular, this involves setting up a node role that can be attached and passed to instance profiles that Karpenter generates at runtime:

KarpenterNodeRole

KarpenterNodeRole

This section of the template defines the IAM role attached to generated instance profiles. Given a cluster name of bob-karpenter-demo, this role would end up being named "KarpenterNodeRole-bob-karpenter-demo.

KarpenterNodeRole:
 Type: "AWS::IAM::Role"
 Properties:
 RoleName: !Sub "KarpenterNodeRole-${ClusterName}"
 Path: /
 AssumeRolePolicyDocument:
 Version: "2012-10-17"
 Statement:
 - Effect: Allow
 Principal:
 Service:
 !Sub "ec2.${AWS::URLSuffix}"
 Action:
 - "sts:AssumeRole"
 ManagedPolicyArns:
 - !Sub "arn:${AWS::Partition}:iam::aws:policy/AmazonEKS_CNI_Policy"
 - !Sub "arn:${AWS::Partition}:iam::aws:policy/AmazonEKSWorkerNodePolicy"
 - !Sub "arn:${AWS::Partition}:iam::aws:policy/AmazonEC2ContainerRegistryPullOnly"
 - !Sub "arn:${AWS::Partition}:iam::aws:policy/AmazonSSMManagedInstanceCore"

The role created here includes several AWS managed policies, which are designed to provide permissions for specific uses needed by the nodes to work with EC2 and other AWS resources. These include:

AmazonEKS_CNI_Policy: Provides the permissions that the Amazon VPC CNI Plugin needs to configure EKS worker nodes.
AmazonEKSWorkerNodePolicy: Lets Amazon EKS worker nodes connect to EKS Clusters.
AmazonEC2ContainerRegistryPullOnly: Allows pulling images from repositories in the Amazon EC2 Container Registry.
AmazonSSMManagedInstanceCore: Adds AWS Systems Manager service core functions for Amazon EC2.

If you were to use a node role from an existing cluster, you could skip this provisioning step and pass this node role to any EC2NodeClasses that you create. Additionally, you would ensure that the IAM Integration Policy has iam:PassRole permission to the role attached to the generated instance profiles.

Controller Authorization

This section sets the AWS permissions for the Karpenter Controller. When used in the Getting Started guide, eksctl uses these permissions to create a service account (karpenter) whose IAM role has all 5 controller policies attached.

The controller permissions are split across 5 managed IAM policies:

KarpenterControllerNodeLifecyclePolicy - EC2 instance and launch template lifecycle management
KarpenterControllerIAMIntegrationPolicy - IAM instance profile management
KarpenterControllerEKSIntegrationPolicy - EKS cluster discovery
KarpenterControllerInterruptionPolicy - SQS interruption queue access
KarpenterControllerResourceDiscoveryPolicy - Read-only resource discovery

Someone wanting to add Karpenter to an existing cluster, instead of using cloudformation.yaml, would need to create these IAM policies directly and assign them to the role leveraged by the service account using IRSA or EKS Pod Identity.

KarpenterControllerNodeLifecyclePolicy

The NodeLifecyclePolicy manages EC2 instance and launch template lifecycle operations. Given a cluster name of bob-karpenter-demo, this policy would be named: KarpenterControllerNodeLifecyclePolicy-bob-karpenter-demo

NodeLifecyclePolicy:
 Type: AWS::IAM::ManagedPolicy
 Properties:
 ManagedPolicyName: !Sub "KarpenterControllerNodeLifecyclePolicy-${ClusterName}"
 Path: /
 PolicyDocument: !Sub |
 {
 "Version": "2012-10-17",
 "Statement": [

AllowScopedEC2InstanceAccessActions

The AllowScopedEC2InstanceAccessActions statement ID (Sid) identifies a set of EC2 resources that are allowed to be accessed with RunInstances and CreateFleet actions. For RunInstances and CreateFleet actions, the Karpenter controller can read (but not create) image, snapshot, security-group, subnet and capacity-reservation EC2 resources, scoped for the particular AWS partition and region.

{
 "Sid": "AllowScopedEC2InstanceAccessActions",
 "Effect": "Allow",
 "Resource": [
 "arn:${AWS::Partition}:ec2:${AWS::Region}::image/*",
 "arn:${AWS::Partition}:ec2:${AWS::Region}::snapshot/*",
 "arn:${AWS::Partition}:ec2:${AWS::Region}:*:security-group/*",
 "arn:${AWS::Partition}:ec2:${AWS::Region}:*:subnet/*",
 "arn:${AWS::Partition}:ec2:${AWS::Region}:*:capacity-reservation/*"
 ],
 "Action": [
 "ec2:RunInstances",
 "ec2:CreateFleet"
 ]
}

AllowScopedEC2LaunchTemplateAccessActions

The AllowScopedEC2InstanceAccessActions statement ID (Sid) identifies launch templates that are allowed to be accessed with RunInstances and CreateFleet actions. For RunInstances and CreateFleet actions, the Karpenter controller can read (but not create) launch-template EC2 resources that have the kubernetes.io/cluster/${ClusterName} tag be set to owned and a karpenter.sh/nodepool tag, scoped for the particular AWS partition and region. This ensures that an instance launch can’t access launch templates that weren’t provisioned by Karpenter.

{
 "Sid": "AllowScopedEC2LaunchTemplateAccessActions",
 "Effect": "Allow",
 "Resource": "arn:${AWS::Partition}:ec2:${AWS::Region}:*:launch-template/*",
 "Action": [
 "ec2:RunInstances",
 "ec2:CreateFleet"
 ],
 "Condition": {
 "StringEquals": {
 "aws:ResourceTag/kubernetes.io/cluster/${ClusterName}": "owned"
 },
 "StringLike": {
 "aws:ResourceTag/karpenter.sh/nodepool": "*"
 }
 }
}

AllowScopedEC2InstanceActionsWithTags

The AllowScopedEC2InstanceActionsWithTags Sid allows the RunInstances, CreateFleet, and CreateLaunchTemplate actions requested by the Karpenter controller to create all fleet, instance, volume, network-interface, launch-template or spot-instances-request EC2 resources (for the partition and region). It also requires that the kubernetes.io/cluster/${ClusterName} tag be set to owned, aws:RequestTag/eks:eks-cluster-name be set to "${ClusterName}, and a karpenter.sh/nodepool tag be set to any value. This ensures that Karpenter is only allowed to create instances for a single EKS cluster.

{
 "Sid": "AllowScopedEC2InstanceActionsWithTags",
 "Effect": "Allow",
 "Resource": [
 "arn:${AWS::Partition}:ec2:${AWS::Region}:*:fleet/*",
 "arn:${AWS::Partition}:ec2:${AWS::Region}:*:instance/*",
 "arn:${AWS::Partition}:ec2:${AWS::Region}:*:volume/*",
 "arn:${AWS::Partition}:ec2:${AWS::Region}:*:network-interface/*",
 "arn:${AWS::Partition}:ec2:${AWS::Region}:*:launch-template/*",
 "arn:${AWS::Partition}:ec2:${AWS::Region}:*:spot-instances-request/*"
 ],
 "Action": [
 "ec2:RunInstances",
 "ec2:CreateFleet",
 "ec2:CreateLaunchTemplate"
 ],
 "Condition": {
 "StringEquals": {
 "aws:RequestTag/kubernetes.io/cluster/${ClusterName}": "owned"
 "aws:RequestTag/eks:eks-cluster-name": "${ClusterName}"
 },
 "StringLike": {
 "aws:RequestTag/karpenter.sh/nodepool": "*"
 }
 }
}

AllowScopedResourceCreationTagging

The AllowScopedResourceCreationTagging Sid allows EC2 CreateTags actions on fleet, instance, volume, network-interface, launch-template and spot-instances-request resources, While making RunInstance, CreateFleet, or CreateLaunchTemplate calls. Additionally, this ensures that resources can’t be tagged arbitrarily by Karpenter after they are created. Conditions that must be met include that aws:RequestTag/kubernetes.io/cluster/${ClusterName} be set to owned and aws:RequestTag/eks:eks-cluster-name be set to ${ClusterName}.

{
 "Sid": "AllowScopedResourceCreationTagging",
 "Effect": "Allow",
 "Resource": [
 "arn:${AWS::Partition}:ec2:${AWS::Region}:*:fleet/*",
 "arn:${AWS::Partition}:ec2:${AWS::Region}:*:instance/*",
 "arn:${AWS::Partition}:ec2:${AWS::Region}:*:volume/*",
 "arn:${AWS::Partition}:ec2:${AWS::Region}:*:network-interface/*",
 "arn:${AWS::Partition}:ec2:${AWS::Region}:*:launch-template/*",
 "arn:${AWS::Partition}:ec2:${AWS::Region}:*:spot-instances-request/*"
 ],
 "Action": "ec2:CreateTags",
 "Condition": {
 "StringEquals": {
 "aws:RequestTag/kubernetes.io/cluster/${ClusterName}": "owned",
 "aws:RequestTag/eks:eks-cluster-name": "${ClusterName}"
 "ec2:CreateAction": [
 "RunInstances",
 "CreateFleet",
 "CreateLaunchTemplate"
 ]
 },
 "StringLike": {
 "aws:RequestTag/karpenter.sh/nodepool": "*"
 }
 }
}

AllowScopedResourceTagging

The AllowScopedResourceTagging Sid allows EC2 CreateTags actions on all instances created by Karpenter after their creation. It enforces that Karpenter is only able to update the tags on cluster instances it is operating on through the kubernetes.io/cluster/${ClusterName}" and karpenter.sh/nodepool tags. Likewise, RequestTag/eks:eks-cluster-name must be set to ${ClusterName}, if it exists, and TagKeys must equal eks:eks-cluster-name, karpenter.sh/nodeclaim, and Name, for all values.

{
 "Sid": "AllowScopedResourceTagging",
 "Effect": "Allow",
 "Resource": "arn:${AWS::Partition}:ec2:${AWS::Region}:*:instance/*",
 "Action": "ec2:CreateTags",
 "Condition": {
 "StringEquals": {
 "aws:ResourceTag/kubernetes.io/cluster/${ClusterName}": "owned"
 },
 "StringLike": {
 "aws:ResourceTag/karpenter.sh/nodepool": "*"
 },
 "StringEqualsIfExists": {
 "aws:RequestTag/eks:eks-cluster-name": "${ClusterName}"
 },
 "ForAllValues:StringEquals": {
 "aws:TagKeys": [
 "eks:eks-cluster-name",
 "karpenter.sh/nodeclaim",
 "Name"
 ]
 }
 }
}

AllowScopedDeletion

The AllowScopedDeletion Sid allows TerminateInstances and DeleteLaunchTemplate actions to delete instance and launch-template resources, provided that karpenter.sh/nodepool and kubernetes.io/cluster/${ClusterName} tags are set. These tags must be present on all resources that Karpenter is going to delete. This ensures that Karpenter can only delete instances and launch templates that are associated with it.

{
 "Sid": "AllowScopedDeletion",
 "Effect": "Allow",
 "Resource": [
 "arn:${AWS::Partition}:ec2:${AWS::Region}:*:instance/*",
 "arn:${AWS::Partition}:ec2:${AWS::Region}:*:launch-template/*"
 ],
 "Action": [
 "ec2:TerminateInstances",
 "ec2:DeleteLaunchTemplate"
 ],
 "Condition": {
 "StringEquals": {
 "aws:ResourceTag/kubernetes.io/cluster/${ClusterName}": "owned"
 },
 "StringLike": {
 "aws:ResourceTag/karpenter.sh/nodepool": "*"
 }
 }
}

KarpenterControllerIAMIntegrationPolicy

The IAMIntegrationPolicy manages IAM instance profile operations. Given a cluster name of bob-karpenter-demo, this policy would be named: KarpenterControllerIAMIntegrationPolicy-bob-karpenter-demo

IAMIntegrationPolicy:
 Type: AWS::IAM::ManagedPolicy
 Properties:
 ManagedPolicyName: !Sub "KarpenterControllerIAMIntegrationPolicy-${ClusterName}"
 Path: /
 PolicyDocument: !Sub |
 {
 "Version": "2012-10-17",
 "Statement": [

AllowPassingInstanceRole

The AllowPassingInstanceRole Sid gives the Karpenter controller permission to pass (iam:PassRole) the node role (KarpenterNodeRole-${ClusterName}) to generated instance profiles. This gives EC2 permission explicit permission to use the KarpenterNodeRole-${ClusterName} when assigning permissions to generated instance profiles while launching nodes.

{
 "Sid": "AllowPassingInstanceRole",
 "Effect": "Allow",
 "Resource": "${KarpenterNodeRole.Arn}",
 "Action": "iam:PassRole",
 "Condition": {
 "StringEquals": {
 "iam:PassedToService": [
 "ec2.amazonaws.com",
 "ec2.amazonaws.com.cn"
 ]
 }
 }
}

AllowScopedInstanceProfileCreationActions

The AllowScopedInstanceProfileCreationActions Sid gives the Karpenter controller permission to create a new instance profile with iam:CreateInstanceProfile, provided that the request is made to a cluster with RequestTag kubernetes.io/cluster/${ClusterName} set to owned, the eks:eks-cluster-name set to ${ClusterName}, and topology.kubernetes.io/region set to the current region. Also, karpenter.k8s.aws/ec2nodeclass must be set to some value. This ensures that Karpenter can generate instance profiles on your behalf based on roles specified in your EC2NodeClasses that you use to configure Karpenter.

{
 "Sid": "AllowScopedInstanceProfileCreationActions",
 "Effect": "Allow",
 "Resource": "arn:${AWS::Partition}:iam::${AWS::AccountId}:instance-profile/*",
 "Action": [
 "iam:CreateInstanceProfile"
 ],
 "Condition": {
 "StringEquals": {
 "aws:RequestTag/kubernetes.io/cluster/${ClusterName}": "owned",
 "aws:RequestTag/eks:eks-cluster-name": "${ClusterName}",
 "aws:RequestTag/topology.kubernetes.io/region": "${AWS::Region}"
 },
 "StringLike": {
 "aws:RequestTag/karpenter.k8s.aws/ec2nodeclass": "*"
 }
 }
}

AllowScopedInstanceProfileTagActions

The AllowScopedInstanceProfileTagActions Sid gives the Karpenter controller permission to tag an instance profile with iam:TagInstanceProfile, provided that ResourceTag attributes kubernetes.io/cluster/${ClusterName} is set to owned and topology.kubernetes.io/region is set to the current region and RequestTag attributes kubernetes.io/cluster/${ClusterName} is set to owned, eks:eks-cluster-name is set to ${ClusterName}, and topology.kubernetes.io/region is set to the current region. Also, ResourceTag/karpenter.k8s.aws/ec2nodeclass and RequestTag/karpenter.k8s.aws/ec2nodeclass must be set to some value. This ensures that Karpenter is only able to act on instance profiles that it provisions for this cluster.

{
 "Sid": "AllowScopedInstanceProfileTagActions",
 "Effect": "Allow",
 "Resource": "arn:${AWS::Partition}:iam::${AWS::AccountId}:instance-profile/*",
 "Action": [
 "iam:TagInstanceProfile"
 ],
 "Condition": {
 "StringEquals": {
 "aws:ResourceTag/kubernetes.io/cluster/${ClusterName}": "owned",
 "aws:ResourceTag/topology.kubernetes.io/region": "${AWS::Region}",
 "aws:RequestTag/kubernetes.io/cluster/${ClusterName}": "owned",
 "aws:RequestTag/eks:eks-cluster-name": "${ClusterName}",
 "aws:RequestTag/topology.kubernetes.io/region": "${AWS::Region}"
 },
 "StringLike": {
 "aws:ResourceTag/karpenter.k8s.aws/ec2nodeclass": "*",
 "aws:RequestTag/karpenter.k8s.aws/ec2nodeclass": "*"
 }
 }
}

AllowScopedInstanceProfileActions

The AllowScopedInstanceProfileActions Sid gives the Karpenter controller permission to perform iam:AddRoleToInstanceProfile, iam:RemoveRoleFromInstanceProfile, and iam:DeleteInstanceProfile actions, provided that the request is made to a cluster with kubernetes.io/cluster/${ClusterName} set to owned and is made in the current region. Also, karpenter.k8s.aws/ec2nodeclass must be set to some value. This permission is further enforced by the iam:PassRole permission. If Karpenter attempts to add a role to an instance profile that it doesn’t have iam:PassRole permission on, that call will fail. Therefore, if you configure Karpenter to use a new role through the EC2NodeClass, ensure that you also specify that role within your iam:PassRole permission.

{
 "Sid": "AllowScopedInstanceProfileActions",
 "Effect": "Allow",
 "Resource": "arn:${AWS::Partition}:iam::${AWS::AccountId}:instance-profile/*",
 "Action": [
 "iam:AddRoleToInstanceProfile",
 "iam:RemoveRoleFromInstanceProfile",
 "iam:DeleteInstanceProfile"
 ],
 "Condition": {
 "StringEquals": {
 "aws:ResourceTag/kubernetes.io/cluster/${ClusterName}": "owned",
 "aws:ResourceTag/topology.kubernetes.io/region": "${AWS::Region}"
 },
 "StringLike": {
 "aws:ResourceTag/karpenter.k8s.aws/ec2nodeclass": "*"
 }
 }
}

KarpenterControllerEKSIntegrationPolicy

The EKSIntegrationPolicy manages EKS cluster discovery. Given a cluster name of bob-karpenter-demo, this policy would be named: KarpenterControllerEKSIntegrationPolicy-bob-karpenter-demo

EKSIntegrationPolicy:
 Type: AWS::IAM::ManagedPolicy
 Properties:
 ManagedPolicyName: !Sub "KarpenterControllerEKSIntegrationPolicy-${ClusterName}"
 Path: /
 PolicyDocument: !Sub |
 {
 "Version": "2012-10-17",
 "Statement": [

AllowAPIServerEndpointDiscovery

You can optionally allow the Karpenter controller to discover the Kubernetes cluster’s external API endpoint to enable EC2 nodes to successfully join the EKS cluster.

Note: If you are not using an EKS control plane, you will have to specify this endpoint explicitly. See the description of the aws.clusterEndpoint setting in the ConfigMap documentation for details.

The AllowAPIServerEndpointDiscovery Sid allows the Karpenter controller to get that information (eks:DescribeCluster) for the cluster (cluster/${ClusterName}).

{
 "Sid": "AllowAPIServerEndpointDiscovery",
 "Effect": "Allow",
 "Resource": "arn:${AWS::Partition}:eks:${AWS::Region}:${AWS::AccountId}:cluster/${ClusterName}",
 "Action": "eks:DescribeCluster"
}

KarpenterControllerInterruptionPolicy

The InterruptionPolicy manages access to the SQS interruption queue. Given a cluster name of bob-karpenter-demo, this policy would be named: KarpenterControllerInterruptionPolicy-bob-karpenter-demo

InterruptionPolicy:
 Type: AWS::IAM::ManagedPolicy
 Properties:
 ManagedPolicyName: !Sub "KarpenterControllerInterruptionPolicy-${ClusterName}"
 Path: /
 PolicyDocument: !Sub |
 {
 "Version": "2012-10-17",
 "Statement": [

AllowInterruptionQueueActions

Karpenter supports interruption queues, that you can create as described in the Interruption section of the Disruption page. This section of the cloudformation.yaml template can give Karpenter permission to access those queues by specifying the resource ARN. For the interruption queue you created (${KarpenterInterruptionQueue.Arn}), the AllowInterruptionQueueActions Sid lets the Karpenter controller have permission to delete messages (DeleteMessage), get queue URL (GetQueueUrl), and receive messages (ReceiveMessage).

{
 "Sid": "AllowInterruptionQueueActions",
 "Effect": "Allow",
 "Resource": "${KarpenterInterruptionQueue.Arn}",
 "Action": [
 "sqs:DeleteMessage",
 "sqs:GetQueueUrl",
 "sqs:ReceiveMessage"
 ]
}

KarpenterControllerResourceDiscoveryPolicy

The ResourceDiscoveryPolicy manages read-only resource discovery operations. Given a cluster name of bob-karpenter-demo, this policy would be named: KarpenterControllerResourceDiscoveryPolicy-bob-karpenter-demo

ResourceDiscoveryPolicy:
 Type: AWS::IAM::ManagedPolicy
 Properties:
 ManagedPolicyName: !Sub "KarpenterControllerResourceDiscoveryPolicy-${ClusterName}"
 Path: /
 PolicyDocument: !Sub |
 {
 "Version": "2012-10-17",
 "Statement": [

AllowRegionalReadActions

The AllowRegionalReadActions Sid allows DescribeAvailabilityZones, DescribeImages, DescribeInstances, DescribeInstanceTypeOfferings, DescribeInstanceTypes, DescribeLaunchTemplates, DescribeSecurityGroups, DescribeSpotPriceHistory, and DescribeSubnets actions for the current AWS region. This allows the Karpenter controller to do any of those read-only actions across all related resources for that AWS region.

{
 "Sid": "AllowRegionalReadActions",
 "Effect": "Allow",
 "Resource": "*",
 "Action": [
 "ec2:DescribeCapacityReservations",
 "ec2:DescribeImages",
 "ec2:DescribeInstances",
 "ec2:DescribeInstanceTypeOfferings",
 "ec2:DescribeInstanceTypes",
 "ec2:DescribeLaunchTemplates",
 "ec2:DescribeSecurityGroups",
 "ec2:DescribeSpotPriceHistory",
 "ec2:DescribeSubnets"
 ],
 "Condition": {
 "StringEquals": {
 "aws:RequestedRegion": "${AWS::Region}"
 }
 }
}

AllowSSMReadActions

The AllowSSMReadActions Sid allows the Karpenter controller to get SSM parameters (ssm:GetParameter) from the current region for SSM parameters generated by AWS services.

NOTE: If potentially sensitive information is stored in SSM parameters, you could consider restricting access to these messages further.

{
 "Sid": "AllowSSMReadActions",
 "Effect": "Allow",
 "Resource": "arn:${AWS::Partition}:ssm:${AWS::Region}::parameter/aws/service/*",
 "Action": "ssm:GetParameter"
}

AllowPricingReadActions

Because pricing information does not exist in every region at the moment, the AllowPricingReadActions Sid allows the Karpenter controller to get product pricing information (pricing:GetProducts) for all related resources across all regions.

{
 "Sid": "AllowPricingReadActions",
 "Effect": "Allow",
 "Resource": "*",
 "Action": "pricing:GetProducts"
}

AllowUnscopedInstanceProfileListAction

The AllowUnscopedInstanceProfileListAction Sid gives the Karpenter controller permission to perform iam:ListInstanceProfiles action to list instance profiles.

{
 "Sid": "AllowUnscopedInstanceProfileListAction",
 "Effect": "Allow",
 "Resource": "*",
 "Action": "iam:ListInstanceProfiles"
}

AllowInstanceProfileReadActions

The AllowInstanceProfileReadActions Sid gives the Karpenter controller permission to perform iam:GetInstanceProfile actions to retrieve information about a specified instance profile, including understanding if an instance profile has been provisioned for an EC2NodeClass or needs to be re-provisioned.

{
 "Sid": "AllowInstanceProfileReadActions",
 "Effect": "Allow",
 "Resource": "arn:${AWS::Partition}:iam::${AWS::AccountId}:instance-profile/*",
 "Action": "iam:GetInstanceProfile"
}

Interruption Handling

Settings in this section allow the Karpenter controller to stand-up an interruption queue to receive notification messages from other AWS services about the health and status of instances. For example, this interruption queue allows Karpenter to be aware of spot instance interruptions that are sent 2 minutes before spot instances are reclaimed by EC2. Adding this queue allows Karpenter to be proactive in migrating workloads to new nodes. See the Interruption section of the Disruption page for details.

Defining the KarpenterInterruptionQueuePolicy allows Karpenter to see and respond to the following:

AWS health events
Spot interruptions
Spot rebalance recommendations
Instance state changes

The resources defined in this section include:

KarpenterInterruptionQueue
KarpenterInterruptionQueuePolicy
ScheduledChangeRule
SpotInterruptionRule
RebalanceRule
InstanceStateChangeRule

KarpenterInterruptionQueue

The AWS::SQS::Queue resource is used to create an Amazon SQS standard queue. Properties of that resource set the QueueName to the name of your cluster, the time for which SQS retains each message (MessageRetentionPeriod) to 300 seconds, and enabling serverside-side encryption using SQS owned encryption keys (SqsManagedSseEnabled) to true. See SetQueueAttributes for descriptions of some of these attributes.

KarpenterInterruptionQueue:
 Type: AWS::SQS::Queue
 Properties:
 QueueName: !Sub "${ClusterName}"
 MessageRetentionPeriod: 300
 SqsManagedSseEnabled: true

KarpenterInterruptionQueuePolicy

The Karpenter interruption queue policy is created to allow AWS services that we want to receive instance notifications from to push notification messages to the queue. The AWS::SQS::QueuePolicy resource here applies EC2InterruptionPolicy to the KarpenterInterruptionQueue. The policy allows sqs:SendMessage actions to events.amazonaws.com and sqs.amazonaws.com services. It also allows the GetAtt function to get attributes from KarpenterInterruptionQueue.Arn. Additionally, it only allows access to the queue using encrypted connections over HTTPS (TLS) to adhere to Amazon SQS Security Best Practices.

KarpenterInterruptionQueuePolicy:
 Type: AWS::SQS::QueuePolicy
 Properties:
 Queues:
 - !Ref KarpenterInterruptionQueue
 PolicyDocument:
 Id: EC2InterruptionPolicy
 Statement:
 - Effect: Allow
 Principal:
 Service:
 - events.amazonaws.com
 - sqs.amazonaws.com
 Action: sqs:SendMessage
 Resource: !GetAtt KarpenterInterruptionQueue.Arn
 - Sid: DenyHTTP
 Effect: Deny
 Action: sqs:*
 Resource: !GetAtt KarpenterInterruptionQueue.Arn
 Condition:
 Bool:
 aws:SecureTransport: false
 Principal: "*"

Rules

This section allows Karpenter to gather AWS Health Events and direct them to a queue where they can be consumed by Karpenter. These rules include:

ScheduledChangeRule: The AWS::Events::Rule creates a rule where the EventPattern is set to send events from the aws.health source to KarpenterInterruptionQueue.

ScheduledChangeRule:
 Type: 'AWS::Events::Rule'
 Properties:
 EventPattern:
 source:
 - aws.health
 detail-type:
 - AWS Health Event
 Targets:
 - Id: KarpenterInterruptionQueueTarget
 Arn: !GetAtt KarpenterInterruptionQueue.Arn

SpotInterruptionRule: An EC2 Spot Instance Interruption warning tells you that AWS is about to reclaim a Spot instance you are using. This rule allows Karpenter to gather EC2 Spot Instance Interruption Warning events and direct them to a queue where they can be consumed by Karpenter. In particular, the AWS::Events::Rule here creates a rule where the EventPattern is set to send events from the aws.ec2 source to KarpenterInterruptionQueue.
```
SpotInterruptionRule:
 Type: 'AWS::Events::Rule'
 Properties:
 EventPattern:
 source:
 - aws.ec2
 detail-type:
 - EC2 Spot Instance Interruption Warning
 Targets:
 - Id: KarpenterInterruptionQueueTarget
 Arn: !GetAtt KarpenterInterruptionQueue.Arn
```
RebalanceRule: An EC2 Instance Rebalance Recommendation signal tells you that a Spot instance is at a heightened risk of being interrupted, allowing Karpenter to get new instances or simply rebalance workloads. This rule allows Karpenter to gather EC2 Instance Rebalance Recommendation signals and direct them to a queue where they can be consumed by Karpenter. In particular, the AWS::Events::Rule here creates a rule where the EventPattern is set to send events from the aws.ec2 source to KarpenterInterruptionQueue.
```
RebalanceRule:
 Type: 'AWS::Events::Rule'
 Properties:
 EventPattern:
 source:
 - aws.ec2
 detail-type:
 - EC2 Instance Rebalance Recommendation
 Targets:
 - Id: KarpenterInterruptionQueueTarget
 Arn: !GetAtt KarpenterInterruptionQueue.Arn
```
InstanceStateChangeRule: An EC2 Instance State-change Notification signal tells you that the state of an instance has changed to one of the following states: pending, running, stopping, stopped, shutting-down, or terminated. This rule allows Karpenter to gather EC2 Instance State-change signals and direct them to a queue where they can be consumed by Karpenter. In particular, the AWS::Events::Rule here creates a rule where the EventPattern is set to send events from the aws.ec2 source to KarpenterInterruptionQueue.
```
InstanceStateChangeRule:
 Type: 'AWS::Events::Rule'
 Properties:
 EventPattern:
 source:
 - aws.ec2
 detail-type:
 - EC2 Instance State-change Notification
 Targets:
 - Id: KarpenterInterruptionQueueTarget
 Arn: !GetAtt KarpenterInterruptionQueue.Arn
```
CapacityReservationInterruptionRule: An EC2 Capacity Reservation Interruption warning informs users that the capacity owner within an AWS Organization is reclaiming reserved capacity, triggering a notification for node reclamation. The AWS::Events::Rule here creates a rule to send events from the aws.ec2 source to KarpenterInterruptionQueue.
```
CapacityReservationInterruptionRule:
 Type: 'AWS::Events::Rule'
 Properties:
 EventPattern:
 source:
 - aws.ec2
 detail-type:
 - EC2 Capacity Reservation Instance Interruption Warning
 Targets:
 - Id: KarpenterInterruptionQueueTarget
 Arn: !GetAtt KarpenterInterruptionQueue.Arn
```