THE ISSUE
All prod application pods in the AWS EKS cluster were in Pending state while the system pods were up and running fine. As a result the whole application was down and cannot be used by the end customers
BACKGROUND
In the project we have a web application which is using quite a big number of microservices as a backend. We have a terraform script which provision the whole infrastructure to the AWS cloud using a lot of different web services and EKS kubernetes cluster in particular. The pods communicate between each other using various approaches like SQS or http rest requests.They also send requests and responses to the front end application and to the various databases, so they act as a middleware layer in the project architecture.
The EKS cluster itself has a self managed auto scaling group of nodes where the system pods are running and using Karpenter to manage the application pods and nodes. It manages scalability by increases/decreases or the number of nodes available into the nodepool based on the application workload traffic.
THE APPROACH
Using the “kubectl” tool from my AWS Workspace machine I logged in to the AWS prod account and to the kubernetes cluster as well. I listed all of the pods in all of the kubernetes namespaces and spotted that the system’s pods are ok while the application ones are not in healthy state. Open the logs for some of the failing ones and describe them to see the events into the description which can give me an idea what is going on and why they cannot start successfully. From the logs I saw that a pod cannot find and attach a security group with a given ID. From the pod’s descriptors I noticed that the application pods were trying to schedule using the default pod scheduler and not the Karpenter one. This immediately pointed me to the fact that there should be something wrong with Karpenter. I checked the Karpenter pods logs as well as the corresponding nodepools. In the node pool there are 3 different labels which group the application pods (pod labels match some of these 3 and are scheduled in such a type of node). The label groups (e.g nodepool workloads) are managed by corresponding “ec2nodeclass” object. In this case we have 3 corresponding node classes – one per a label. By describing them with kubectl I noticed that all of them were using configuration from dev env like security group id, ami image id, cluster naming with dev suffix, etc. Because of that as such objects with such ids don’t exist on prod, they cannot be found hence the pods cannot start.
THE SOLUTION
With kubectl I opened the ec2nodeclass descriptor for edit and started replacing everything which was dev env related with the corresponding data from prod like security group ids, ami images ids, various prefix/suffixes, etc. The activity was repeated for all of the three node classes for the corresponding nodepool labels. After saving the changes kubernetes automatically refreshes the nodepool and Karpenter and the pods became in healthy status automatically
RCA (Root Cause Analysis):
Background
We use a Jenkins pipeline to plan and apply terraform scripts hence to provision our infrastructure on the AWS cloud. The infrastructure project has different folders for the different stacks and one of them is for the kubernetes cluster. The pipeline is parameterized to select which env (dev, …, prod), which stack (folder), which GitHub branch to use or what targets. When an env is selected the pipeline automatically uses a corresponding kube.config profile. Like this the changes due to be applied on dev env will automatically use the right dev Kubernetes profile as well.
We have an option to replicate the process from our local machines even on prod. Using this approach it is a person’s responsibility to choose env and corresponding kubernetes profile. It is possible and nothing will flag it out if there is a mismatch between these two which can be problematic.
What’s happened
A devops person switched to prod aws account in order to apply some infrastructure changes on prod eks but forgot to switch the kuberntes profile as well. By applying these changes on prod and because the kube profile is pointing to the dev eks config, its configuration was promoted to prod as well
LESSONS LEARNED
Use only CI/CD automated pipelines (Jenkins / GitHub Actions) to provision infrastructure changes to environments
If using CI/CD process is not possible and an execution from local is required , create a new variable in the helm provider which is automatically populated based on the chosen AWS account. Like this the correct kubernetes profile will be used and there is no need for a person manually to unify both of them and to incidentally make a mistake like in this situation.
