Recently, I started began upgrading our EKS clusters to Kubernetes version 1.30. With this version, any newly created managed node groups now default to using Amazon Linux 2023 (AL2023) as the node operating system. In contrast, Amazon Linux 2 (AL2) was the default for new node groups in earlier versions.
The Kubernetes release version 1.30 also named as Uwubernetes (released on April 2024) came with interesting changes and enhancements. Here I am highlighting those I think are the most important:
In this article, I won’t go into the details of the Kubernetes 1.30 release. Instead, my focus is to introduce and discuss the significant change in EKS (AWS Managed Kubernetes Service), where Amazon Linux 2023 (AL2023) has replaced Amazon Linux 2 (AL2) as the default operating system for managed node groups.
Note: If you prefer to continue using AL2, you can select it as the AMI type when creating a new node group.
What is Amazon Linux 2023 (AL2023)
From AWS website.
Amazon Linux 2023 (AL2023) is the next generation of Amazon Linux from Amazon Web Services (AWS). With AL2023, you can develop and run cloud and enterprise applications in a secure, stable, and high-performance runtime environment. Also you get an application environment that offers long-term support with access to the latest innovations in Linux. AL2023 is provided at no additional charge.
In general terms, this operating system comes with enhancements for running Linux on the cloud which make it more faster and secured. If you want to know more in details the performance and operational optimizations, you will find useful this link.
There is an extensive documentation as well about comparing AL2 and AL2023, check it out here.
Finally, if you need to know what are the packages and new packages with their version that comes on the new AMI based on AL2023, check this other link here.
There is a lot to say about the transition from AL2 to AL2023, but on this article I am going to focus in a specific change that can affect your Kubernetes workloads when performing this migration.
Containers and AL2023
Before, let me quickly explain what is the Instance Metadata Service (IMDS).
This is a feature that provides metadata about an EC2 instance to the instance itself. This metadata includes details such as the instance ID, AMI ID, security group, and other configuration information.
IMDS also provides temporary credentials for accessing AWS services securely without embedding credentials in the application. Applications running on the instance can access IMDS via HTTP requests to a special local endpoint, 169.254.169.254.
💔 The breaking change
AL2023 requires IMDSv2 by default. IMDSv2 has several benefits that help improve security posture. It uses a session-oriented authentication method that requires the creation of a secret token in a simple HTTP PUT request to start the session. A session's token can be valid for anywhere between 1 second and 6 hours.
For IMDSv2, the default hop count for managed node groups is set to 1. This means that containers won't have access to the node's credentials using IMDS.
Depending how you configured your workloads and in case they need access to the nodes’s credentials using IMDS then you will need to workaround this problem.
Following I show you different alternatives for solving this problem.
🚀 Alternatives to solve this problem
Use EKS Pod Identity
In my opinion, the best way is to configure your workloads in Kubernetes using EKS Pod Identity. Basically you associate an IAM role with a Kubernetes Service Account and then you configure your workloads (pods) to use the service account. More information about it here.
Use a Launch template with HttpPutResponseHopLimit
As mentioned before on this article, for IMDSv2 the the default hop count is set to 1. You can configure this hop limit to any value greater than 1 using a launch template for your EKS managed worker nodes on the Launch Template Metadata Options. For instance:
{ "HttpEndpoint" : String, "HttpProtocolIpv6" : String, "HttpPutResponseHopLimit" : 3, "HttpTokens" : String, "InstanceMetadataTags" : String }
Configure instance metadata options for new instances
Is it possible to set some default EC2 metadata settings at the AWS account level (per region) so any new EC2 instance will take the default settings. Keep in mind that you can override the default settings by specifying the setting at launch time of the EC2 instance.
Using the AWS CLI you can do this, for instance:
aws ec2 modify-instance-metadata-defaults \ --region us-east-1 \ --http-put-response-hop-limit 2
Using Terraform you can do it like this:
resource "aws_ec2_instance_metadata_defaults" "ec2_metadata_defaults" { http_put_response_hop_limit = 3 }
Change instance metadata options on existing instances
In case you need to change the metadata option to an existing running instance. You can do it with the AWS CLI and it won’t required the instance to restart or be replaced at all. You can perform this change for instance like this:
aws ec2 modify-instance-metadata-options --instance-id i-123456789 --http-put-response-hop-limit 3
Here are the alternatives available to address this situation. You might also be curious about which EC2 instances are still using IMDSv1 and whether they depend on this metadata version. Fortunately, AWS provides comprehensive documentation to guide you through this transition, which you can access here.
Resources
[1] Review release notes for Kubernetes versions on standard support
[2] What is Amazon Linux 2023?
[3] Performance and operational optimizations
[6] Learn how EKS Pod Identity grants pods access to AWS services
[7] EC2 LaunchTemplate MetadataOptions
[8] Configure instance metadata options for new instances
[9] Modify instance metadata options for existing instances
[10] Configure the Instance Metadata Service options
[11] Terraform Resource: aws_ec2_instance_metadata_defaults
[12] Terraform Resource: aws_launch_template
[13] Transition to using Instance Metadata Service Version 2