IMDSv2 Hop Limit
IMDS (Instance Metadata Service) is an API service on AWS EC2 instance which isn’t noticed by many people. This is completely normal because it is just how AWS builds it. As a user, we don’t necessarily need to know everything, but this is an important service, and I come to look into it deeper because of a recent issue in my project.
So we have recently upgraded our AWS SDK from version 3 to 4, and the test on our pipeline starts to fail due to some AWS credential problem. This is super weird because the EC2 instance which runs the test should be able to assume the EC2 instance role attached to it automatically. If we revert it back to v3, it works without any issue.
More investigation leads me to these documentations:
Support for Amazon EC2 IMDSv1
Support for Instance Metadata Service Version 1 (IMDSv1) has been removed. V4 of the SDK always uses Instance Metadata Service Version 2 (IMDSv2) when fetching credentials and other metadata from the IMDS. For more information about the IMDS, see Use the IMDS in the Amazon EC2 User Guide.
In a container environment, consider reconfiguration or increasing the hop limit to 2
The AWS SDKs use IMDSv2 calls by default. If the IMDSv2 call receives no response, some AWS SDKs retry the call and, if still unsuccessful, use IMDSv1. This can result in a delay, especially in a container environment. For those AWS SDKs that require IMDSv2, if the hop limit is 1 in a container environment, the call might not receive a response at all because going to the container is considered an additional network hop.
To mitigate these issues in a container environment, consider changing the configuration to pass settings (such as the AWS Region) directly to the container, or consider increasing the hop limit to 2. For information about the hop limit impact, see Add defense in depth against open firewalls, reverse proxies, and SSRF vulnerabilities with enhancements to the EC2 Instance Metadata Service. For information about changing the hop limit, see Change the PUT response hop limit.
They mention IMDSv2 and container environment, which looks right because our test is executed in a container on the pipeline. By increasing the hop limit the test passes. Basically, what AWS says is that from IMDSv2, a new authentication process is introduced to IMDS. To get instance metadata from IMDS, a PUT request for token needs to be made first, and to secure this process, AWS introduces a new metadata option called HttpPutResponseHopLimit. By default it is set to 1, and this makes sure only the host can get this token. If you are trying to call this PUT token API inside a container, then this hop limit needs to be increased to 2.
After this is fixed. I start to wonder how this mechanism works. How does AWS knows that this request is made from a container or host environment. To IMDS service, isn’t the caller all the same? While I dig deeper inside, I realise this is not controlled on the application level, but one level deeper on the network level.
Let’s read an excerpt of this documentation How Instance Metadata Service Version 2 works .
By default, the response to PUT requests has a response hop limit (time to live) of 1 at the IP protocol level. If you need a bigger hop limit, you can adjust it by using the modify-instance-metadata-options AWS CLI command. For example, you might need a bigger hop limit for backward compatibility with container services running on the instance. For more information, see Modify instance metadata options for existing instances.
From here we can see the hop limit is set as the TTL header of the IPv4 or IPv6 packet. Every time when this packet is handled by a layer 3 device, its value is decreased by 1. When the TTL becomes 0, the router will consider this packet as expired and drop it. By default, the container’s network mode is bridge, which means there is an extra layer 3 device in between. When AWS sets the hop limit for that token response to 1 by default, it enforces that the token can only be acquired in the host network, not any other network.
This packet header field is called Time to Live (TTL) in IPv4 and Hop Limit in IPv6. If you are hoping to understand more what is it and why it is designed, I find this 3-minute video helpful: