Cloud / Amazon EKS Interview questions

Amazon EKS (Elastic Kubernetes Service) is a managed service that runs the Kubernetes control plane for you across multiple AWS Availability Zones, so you don't have to install, operate, or scale the control plane yourself.

AWS runs and patches the API server, etcd, and scheduler, and automatically replaces unhealthy control plane instances, while you focus on deploying workloads and managing worker nodes (EC2, Fargate, or hybrid nodes).

EKS is upstream-conformant Kubernetes, meaning existing manifests, Helm charts, and tools like kubectl work exactly as they would on any other certified Kubernetes distribution.

An EKS cluster has two logical halves: the control plane, fully managed by AWS, and the data plane, made up of the nodes that actually run your pods.

The control plane includes the API server, etcd, the scheduler, and controller manager, all running in an AWS-managed VPC. The data plane can be EC2 instances in managed or self-managed node groups, AWS Fargate for serverless pods, or hybrid/on-premises nodes connected via EKS Hybrid Nodes.

Networking is tied together by the Amazon VPC CNI plugin, which assigns pods real VPC IP addresses, and IAM integration connects Kubernetes RBAC to AWS permissions via IRSA or EKS Pod Identity.

The control plane is the brain of the cluster: it stores cluster state in etcd, exposes the Kubernetes API, and decides where pods should run. In EKS, AWS owns, patches, and scales this layer across at least two Availability Zones, and you never SSH into it.

Worker nodes are where your actual application containers execute. They run the kubelet, a container runtime, and the kube-proxy, and they register themselves with the control plane so the scheduler can place pods on them.

You are responsible for the worker node layer's capacity, patching (unless using managed node groups or Auto Mode), and security groups, while AWS is fully responsible for control plane availability and patching under the shared responsibility model.

flowchart LR
  A["Control Plane - AWS managed"] -->|schedules pods| B["Worker Node 1"]
  A -->|schedules pods| C["Worker Node 2"]
  B --> D["kubelet + container runtime"]
  C --> E["kubelet + container runtime"]

EKS supports three main ways to run compute for your pods. Managed node groups let AWS provision, tag, and roll out EC2 instances via Auto Scaling groups, handling draining and AMI updates for you.

Self-managed nodes are EC2 instances you launch and join to the cluster yourself, giving full control over the AMI and bootstrap script at the cost of more operational work.

Fargate profiles remove nodes entirely - AWS runs each pod in its own lightweight, right-sized compute environment with no EC2 instance to manage. More recently, EKS Auto Mode extends the managed model further, automatically provisioning, scaling, and upgrading both compute and core add-ons like the CNI and storage drivers.

A Fargate profile is a configuration object that tells EKS which pods should run on Fargate instead of on EC2 nodes, matched by Kubernetes namespace and optional label selectors.

When a pod matching a profile is scheduled, EKS provisions a dedicated, isolated compute environment sized to that pod's CPU and memory requests - there's no shared node, so pods don't compete for resources with unrelated workloads.

Each profile also specifies the subnets Fargate pods launch into and the pod execution role used to pull images and write logs. A common pattern is a profile scoped to the kube-system namespace for CoreDNS, alongside another for a specific application namespace.

eksctl create fargateprofile \
  --cluster my-cluster \
  --name fp-default \
  --namespace my-app \
  --labels billing=on-demand

eksctl is the official CLI for EKS that wraps CloudFormation to create the VPC, IAM roles, and control plane in a single command, instead of clicking through many console screens.

A minimal cluster can be created with one line specifying the cluster name, region, and Kubernetes version; eksctl then provisions a default VPC and a managed node group unless you pass a config file for more control.

For production, most teams use a YAML config file passed via -f so the entire cluster topology - VPC CIDR, node groups, add-ons, IRSA roles - is version-controlled and repeatable.

eksctl create cluster \
  --name demo-cluster \
  --region us-east-1 \
  --version 1.31 \
  --nodegroup-name standard-workers \
  --node-type t3.medium \
  --nodes 3

A kubeconfig file tells kubectl which cluster to talk to, what certificate authority to trust, and how to authenticate - it's the local file that turns a bare kubectl install into a working client for a specific cluster.

For EKS, you don't paste a static token into kubeconfig. Instead, the AWS CLI generates one dynamically each time via the aws eks update-kubeconfig command, which writes an entry that calls aws eks get-token (or the aws-iam-authenticator) behind the scenes using your current IAM credentials.

This means access is tied to whichever IAM identity is active when kubectl runs, and that identity must also be mapped to a Kubernetes RBAC identity via the aws-auth ConfigMap or EKS access entries.

aws eks update-kubeconfig \
  --name demo-cluster \
  --region us-east-1

IRSA lets a specific Kubernetes ServiceAccount assume an IAM role, so a pod can call AWS APIs (like S3 or DynamoDB) with narrowly scoped permissions instead of inheriting broad node-level IAM permissions shared by every pod on that node.

It works through an OIDC identity provider that EKS associates with your cluster; the ServiceAccount is annotated with the IAM role ARN, and AWS's webhook injects short-lived, auto-rotated credentials into the pod as a projected token.

AWS now also offers EKS Pod Identity as a simpler alternative that removes the OIDC provider setup and trust policy boilerplate, associating roles to service accounts directly through the EKS API, though IRSA remains widely used and fully supported.

The VPC CNI is the default networking plugin for EKS that assigns each pod a real, routable IP address from the VPC's CIDR range, rather than an overlay network address.

It works by attaching Elastic Network Interfaces (ENIs) to worker nodes and pre-allocating a pool of secondary IP addresses on each ENI, which the CNI's ipamd daemon hands out to pods as they're scheduled - keeping pod-to-pod and pod-to-AWS-service traffic on native VPC routing.

Because pod density is limited by how many ENIs and IPs an instance type supports, larger instances or features like prefix delegation are often needed to avoid running out of IPs on smaller nodes.

EKS add-ons are operational software components - like the VPC CNI, CoreDNS, kube-proxy, and the EBS/EFS CSI drivers - that AWS packages, publishes, and lets you manage through the EKS API instead of installing manually with kubectl or Helm.

Because AWS owns the add-on lifecycle, you can see available versions, apply security patches, and upgrade them with a single API call or console click, and EKS will warn you about version compatibility with your cluster's Kubernetes version.

Add-ons can be installed in two conflict modes: OVERWRITE, which replaces any existing self-managed version of that component, or PRESERVE, which keeps your existing configuration and just registers it for management.

aws eks create-addon \
  --cluster-name demo-cluster \
  --addon-name vpc-cni \
  --resolve-conflicts PRESERVE

EKS Anywhere is a deployment option that lets you run Kubernetes clusters on your own infrastructure - on-premises VMware, bare metal, or Nutanix - using the same open-source EKS Distro that powers EKS in AWS.

It provides a consistent cluster creation and lifecycle experience with the standard EKS tooling and conformance, letting teams keep clusters on-premises for latency, data-residency, or connectivity reasons while still using familiar upgrade and configuration patterns.

Unlike standard EKS, you operate the control plane yourself since there's no AWS-managed data center backing it; AWS instead offers optional paid support subscriptions for EKS Anywhere clusters.

EKS runs standard, open-source Kubernetes, giving you the full Kubernetes API, ecosystem (Helm, Operators, CRDs), and portability to other Kubernetes environments. ECS is AWS's own proprietary container orchestrator with a simpler, AWS-native API and no Kubernetes compatibility layer.

ECS tends to have a gentler learning curve and less operational overhead for teams fully committed to AWS, while EKS suits teams that want multi-cloud portability, need Kubernetes-specific tooling, or already have Kubernetes expertise.

EKS upgrades happen one minor version at a time - you can't skip, say, from 1.28 straight to 1.31 in a single step. You trigger a control plane upgrade via the console, CLI, or eksctl, and AWS updates the API server, etcd, and controllers with no downtime to the API, though brief connection drops can occur.

After the control plane is upgraded, you must separately upgrade the data plane: managed node groups can be upgraded in place (AWS drains and replaces nodes with new AMIs), while self-managed nodes need a manual rolling replacement.

Add-ons (VPC CNI, CoreDNS, kube-proxy) should be checked and updated for compatibility with the new version, since an outdated add-on can break networking or DNS after the jump.

eksctl upgrade cluster --name demo-cluster --version 1.32 --approve

The aws-auth ConfigMap, stored in the kube-system namespace, is the traditional bridge that maps IAM users and roles to Kubernetes usernames and groups, which is what lets an IAM identity actually authenticate and receive RBAC permissions inside the cluster.

Without an entry in this ConfigMap (or the newer EKS access entries API), an IAM principal can generate a valid token but the API server will reject it as an unknown user - cluster creation only automatically grants access to the identity that created the cluster.

AWS now recommends the EKS access entries API for managing this mapping instead of hand-editing the ConfigMap, since it validates IAM ARNs up front and avoids the classic "typo locks everyone out" failure mode.

By default, every EKS cluster exposes a public endpoint reachable from the internet, secured by IAM authentication and Kubernetes RBAC rather than a public/no-auth model.

You can also enable a private endpoint that resolves inside the cluster's VPC via a Route 53 private hosted zone, letting nodes and in-VPC clients reach the API without crossing the internet.

These aren't mutually exclusive: clusters can run public-only, private-only, or both public and private simultaneously, and public access can be restricted to specific CIDR ranges for an extra layer of network-level control.

1. Public endpoint (internet-reachable, IAM + RBAC secured)
2. Private endpoint (VPC-internal, resolved via Route 53)
3. Public endpoint restricted to allow-listed CIDR blocks

These three options trade off control against operational effort. Managed node groups give AWS ownership of the underlying Auto Scaling group, AMI rollout, and graceful node draining, while still letting you customize instance types, labels, and taints.

Self-managed nodes give you full control over the AMI, bootstrap script, and lifecycle hooks - useful for custom AMIs or non-standard bootstrapping - but you own patching, scaling, and draining yourself.

Fargate removes the node concept entirely: each pod is its own micro-VM-backed environment with no shared kernel or capacity planning, ideal for bursty or unpredictable workloads, though it doesn't support DaemonSets, privileged containers, or certain host-level networking features.

The AWS Load Balancer Controller watches for Kubernetes Ingress and Service objects and translates them into real AWS Application Load Balancers (ALB) or Network Load Balancers (NLB), rather than relying on the older in-tree cloud provider logic.

For an Ingress resource, it provisions an ALB and configures listener rules, target groups, and health checks based on annotations you set (like alb.ingress.kubernetes.io/scheme), and it can target pods directly via IP mode instead of routing through NodePorts.

For a Service of type LoadBalancer, it can provision an NLB, which is preferred for TCP-level traffic, non-HTTP protocols, or when preserving the client's source IP matters.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app-ingress
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip

A Service gives a stable network identity and load-balances traffic to a set of pods based on label selectors, operating at Layer 4 in its ClusterIP and NodePort forms.

An Ingress sits a layer above that: it's an HTTP/HTTPS routing rule set (host- and path-based routing, TLS termination) that's implemented by a controller - in EKS, typically the AWS Load Balancer Controller - which then provisions the actual ALB.

In practice, you often use both together: Services provide stable internal endpoints for pods, and a single Ingress fronts multiple Services under one ALB, avoiding the cost and management overhead of a separate load balancer per Service.

The Amazon EBS CSI driver is the add-on that lets Kubernetes dynamically create and attach EBS volumes when a pod requests persistent storage, replacing the deprecated in-tree EBS provisioner.

You define a StorageClass pointing at the ebs.csi.aws.com provisioner, then a PersistentVolumeClaim referencing that class; the driver's controller then calls the EC2 API to create a matching volume and attaches it to the node hosting the pod.

Because EBS volumes are zonal, the pod and its volume must land in the same Availability Zone - the CSI driver handles this through topology-aware provisioning, but it does mean an EBS-backed pod can't simply reschedule to any node across AZs.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ebs-gp3
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
volumeBindingMode: WaitForFirstConsumer

Cluster Autoscaler works by scaling existing EC2 Auto Scaling groups up or down based on pending, unschedulable pods - it's constrained to the instance types and sizes predefined in whichever ASGs you've set up.

Karpenter takes a more direct approach: it watches for unschedulable pods and calls the EC2 API itself to launch precisely the instance types, sizes, and Availability Zones needed, without going through an Auto Scaling group at all, which usually means faster provisioning and tighter bin-packing.

Karpenter also natively supports flexible, cost-aware selection across many instance families and Spot capacity pools in a single NodePool definition, whereas achieving similar flexibility with Cluster Autoscaler generally requires managing several separate node groups.

Security Groups for Pods let you attach an EC2 security group directly to individual pods, rather than only to the node they run on, giving fine-grained network control at the pod level using native AWS security group rules.

It requires the VPC CNI's ENABLE_POD_ENI setting and works by attaching a "branch" ENI to the pod via the vpc-resource-controller, so the pod effectively gets its own network interface with its own security group, separate from the node's.

This is commonly used when a workload needs access to a resource (like an RDS database) that's locked down to a specific security group, without opening that access to every pod on the same node.

apiVersion: vpcresources.k8s.aws/v1beta1
kind: SecurityGroupPolicy
metadata:
  name: db-access
spec:
  podSelector:
    matchLabels: {role: db-client}
  securityGroups:
    groupIds: ["sg-0123456789abcdef0"]

Static access keys baked into a pod (as environment variables or mounted secrets) never expire on their own, are easy to leak through logs or misconfigured Secrets, and typically grant the same broad permissions to every pod that uses them.

IRSA instead issues short-lived, automatically rotated session tokens scoped to exactly the IAM role attached to that pod's ServiceAccount, so a compromised pod only exposes the narrow permissions it was actually granted, not a long-lived key with account-wide reach.

It also removes an entire class of operational risk: there's no credential to rotate manually, no risk of a key being copy-pasted into the wrong manifest, and access can be revoked instantly just by updating the IAM trust policy or role permissions.

Start by disabling the public endpoint or restricting it to specific CIDR ranges, and enabling the private endpoint so the API server resolves to a private IP inside the cluster's VPC via Route 53.

Because a fully private cluster's API server is unreachable from outside the VPC, you'll need a bastion host, VPN, AWS Client VPN, or Direct Connect to reach it for kubectl access, and worker nodes must be able to resolve the private endpoint via DNS.

You also need VPC endpoints (PrivateLink) for services like ECR, S3, and STS so that nodes without internet access can still pull images and authenticate, since a private-only cluster typically also means private subnets with no NAT gateway route to those services.

aws eks update-cluster-config \
  --name demo-cluster \
  --resources-vpc-config endpointPublicAccess=false,endpointPrivateAccess=true

Spreading nodes across at least two or three AZs protects workloads from a single data center failure - if one AZ has a power or network event, pods can still be rescheduled onto healthy nodes in the surviving AZs.

It also plays well with EBS's zonal nature and Kubernetes' built-in topology spread constraints and pod anti-affinity rules, which can actively enforce spreading replicas of the same Deployment across different zones rather than clustering them together by chance.

A common mistake is defining a node group per AZ but leaving replica counts and affinity rules unset, which lets the scheduler pack all replicas into a single zone anyway - true resilience requires setting topologySpreadConstraints explicitly, not just having multi-AZ node capacity available.

EKS control plane components - API server, audit, authenticator, controller manager, and scheduler logs - are off by default and must be explicitly enabled per log type, since each one adds CloudWatch Logs cost and volume.

Once enabled through the console, CLI, or Infrastructure-as-Code, logs stream into a CloudWatch Logs group named after the cluster, where you can build metric filters and alarms - for example, alerting on repeated 403s in the audit log, which often signals an RBAC misconfiguration.

The audit log type is particularly valuable for security reviews since it records every request made to the API server, including who made it and what resource was affected, useful for tracing unauthorized access attempts after the fact.

aws eks update-cluster-config \
  --name demo-cluster \
  --logging '{"clusterLogging":[{"types":["api","audit","authenticator"],"enabled":true}]}'

Prometheus is typically deployed into the cluster via the kube-prometheus-stack Helm chart, which also installs node-exporter (host-level metrics) and kube-state-metrics (Kubernetes object state) alongside Prometheus itself.

Prometheus scrapes metrics endpoints across the cluster on a pull basis and stores them as time series, while Grafana connects to Prometheus as a data source to render dashboards - CPU/memory per namespace, pod restart counts, node pressure, and more.

For a managed alternative, Amazon Managed Service for Prometheus and Amazon Managed Grafana offload storage and dashboard hosting to AWS, removing the operational burden of scaling Prometheus's own storage as cluster metric volume grows.

helm install kube-prom prometheus-community/kube-prometheus-stack \
  --namespace monitoring --create-namespace

By default, Kubernetes Secrets are only base64-encoded, not encrypted, in etcd - EKS lets you add envelope encryption using a customer-managed AWS KMS key, so Secret data is encrypted at rest with a key you control and can rotate or revoke.

For secrets that live outside the cluster (database credentials, API keys), the AWS Secrets and Configuration Provider (ASCP) for the Secrets Store CSI Driver mounts values from AWS Secrets Manager or Parameter Store directly as files in the pod, avoiding storing them as native Kubernetes Secrets at all.

Combining KMS envelope encryption for what must exist as Secrets with the CSI driver for externally-managed values gives layered protection, and access to both should be scoped tightly via IAM and Kubernetes RBAC.

A Role grants permissions scoped to a single namespace - for example, allowing a user to list pods only within the billing namespace - and is bound to subjects via a RoleBinding.

A ClusterRole defines the same kind of rule set but can be applied cluster-wide, or reused across multiple namespaces, and is also required for permissions on non-namespaced resources like nodes or PersistentVolumes.

Critically, a ClusterRole can still be scoped to one namespace by pairing it with a RoleBinding instead of a ClusterRoleBinding - this pattern is common for reusing one permission template (like "view-only") across many namespaces without duplicating the rule definitions.

A taint applied to a node group repels pods by default - only pods with a matching toleration are allowed to schedule there, which is useful for reserving specialized nodes (like GPU instances) for specific workloads.

In EKS, you set taints directly on a managed node group's configuration so every node it launches carries the taint automatically, rather than tainting nodes manually one by one after they join.

Combine taints with node labels and a matching nodeSelector or affinity rule on the pod spec so the workload is both allowed onto the tainted node and actively steered there, since a toleration alone only permits scheduling - it doesn't force it.

tolerations:
- key: "workload-type"
  operator: "Equal"
  value: "gpu"
  effect: "NoSchedule"
nodeSelector:
  workload-type: gpu

Spot Instances can cost up to 90% less than On-Demand for the same instance type, making them attractive for stateless, fault-tolerant, or batch workloads where occasional interruption is acceptable in exchange for large savings.

AWS can reclaim a Spot instance with roughly a two-minute warning, so workloads need to tolerate sudden termination - Kubernetes handles this gracefully if you deploy the AWS Node Termination Handler or use Karpenter, which drains pods proactively on that interruption notice.

A common pattern mixes a small baseline of On-Demand nodes for critical, always-available capacity with a larger pool of Spot nodes across diverse instance types for elastic, cost-efficient scaling - diversifying instance types reduces the chance of many nodes being reclaimed at once.

Fargate fits best when workloads are bursty or unpredictable and you'd rather pay per-pod than keep spare EC2 capacity idle waiting for traffic spikes - there's no node-level capacity planning at all.

It also suits multi-tenant setups where strict pod-level isolation matters, since each Fargate pod runs in its own dedicated micro-VM with no shared kernel, reducing the blast radius of a container escape compared to pods sharing a node.

EC2-based node groups remain the better choice when you need DaemonSets, privileged containers, GPU workloads, very high pod density for cost efficiency, or fine-grained control over the host OS and kernel parameters - all of which Fargate does not support.

Cross-account access starts with an IAM role in the cluster's own account that a principal in another account is allowed to assume, typically via an IAM role trust policy naming the other account (or a specific role ARN) as a trusted principal.

That assumed role then needs an entry in the cluster's access configuration - either the aws-auth ConfigMap or, preferably, an EKS access entry - mapping it to appropriate Kubernetes RBAC permissions, since assuming the IAM role alone doesn't grant any Kubernetes-level access.

A common real-world case is a central platform team's account needing read access to application clusters owned by other teams' accounts; scoping the assumed role's RBAC binding to a read-only ClusterRole keeps that cross-account access auditable and minimal.

A service mesh injects a sidecar proxy (Envoy, in both Istio and the now-deprecated AWS App Mesh) next to every pod, intercepting all inbound and outbound traffic so networking concerns are handled outside the application code.

This unlocks mutual TLS between services without changing application code, fine-grained traffic shifting for canary or blue/green deployments, automatic retries and circuit breaking, and detailed per-service traffic metrics and distributed tracing.

The trade-off is added complexity and latency from the extra network hop through each sidecar, plus real operational overhead in managing the mesh's own control plane - most teams adopt a mesh only once they have enough services that manual traffic and security policy management has become genuinely painful.

Helm packages a set of Kubernetes manifests as a versioned "chart" with configurable values, letting you template things like replica counts, image tags, and resource limits instead of hand-editing raw YAML for every environment.

You install a chart into an EKS cluster with helm install, pointing at a values file for environment-specific overrides (dev vs. prod), and Helm tracks the resulting release as a unit so you can upgrade, roll back, or uninstall it as a whole rather than tracking individual manifests.

For AWS-native components specifically, Helm is also how you typically install the AWS Load Balancer Controller, Cluster Autoscaler, Karpenter, and the kube-prometheus-stack, since AWS and the community publish official charts for each.

helm upgrade --install my-app ./charts/my-app \
  -f values-prod.yaml \
  --namespace production --create-namespace

In GitOps, a Git repository is the single source of truth for what should be running in the cluster - instead of running kubectl apply manually, ArgoCD continuously compares the live cluster state against the manifests in Git and reconciles any drift.

An Application resource in ArgoCD points at a Git repo path (raw YAML, Kustomize, or a Helm chart) and a target cluster/namespace; ArgoCD's controller polls or receives webhooks on changes, then either auto-syncs or waits for manual approval depending on policy.

This gives EKS clusters a full audit trail (every change is a Git commit), easy rollback (revert the commit), and consistent multi-cluster deployment, since the same Git repo can drive several EKS clusters across environments or regions.

sequenceDiagram
  participant Dev
  participant Git
  participant ArgoCD
  participant EKS
  Dev->>Git: push manifest change
  ArgoCD->>Git: detect diff
  ArgoCD->>EKS: apply reconciled state

An EKS upgrade moves through distinct phases rather than happening as one atomic action. First, AWS upgrades the control plane's API server, etcd, scheduler, and controller manager one minor version at a time, provisioning new control plane instances and cutting traffic over with no API downtime.

Deprecated API versions are the biggest risk at this stage - if your manifests or Helm charts reference an API removed in the target version, applying them will fail post-upgrade, so checking deprecated API usage beforehand (via tools like Pluto) is standard practice.

Next comes the data plane: managed node groups can be upgraded in place, where AWS launches new nodes on the updated AMI, cordons and drains old nodes respecting PodDisruptionBudgets, then terminates them - self-managed nodes and Karpenter-provisioned nodes require you to trigger this rollover yourself.

Finally, add-ons (VPC CNI, CoreDNS, kube-proxy, EBS CSI driver) need their own compatibility check and upgrade, since running an add-on version too old for the new control plane version is a common source of post-upgrade networking or DNS failures.

flowchart TD
  A["Check deprecated APIs"] --> B["Upgrade control plane"]
  B --> C["Upgrade/roll data plane nodes"]
  C --> D["Upgrade add-ons for compatibility"]
  D --> E["Validate workloads"]

AWS runs multiple API server replicas and an etcd cluster spread across at least two, typically three, Availability Zones for every EKS cluster, so a single AZ failure doesn't take down API access or lose committed cluster state.

etcd itself uses the Raft consensus protocol, requiring a majority (quorum) of its members to agree before committing a write - spreading members across AZs means the cluster tolerates the loss of a whole AZ's etcd members while still maintaining quorum with the rest.

A network load balancer in front of the API server replicas handles failover transparently to clients, and AWS continuously monitors control plane health, automatically replacing any unhealthy component without any customer action or visible downtime for well-behaved clients that retry transient errors.

Start with kubectl describe node <name> to read the node's Conditions section - a NetworkUnavailable or MemoryPressure condition points to a very different root cause than a kubelet that's simply stopped reporting.

If the condition suggests networking, check whether the VPC CNI's aws-node pod on that node is running and check its logs; a common cause is the node running out of available IP addresses to assign to pods, which surfaces as CNI errors rather than an obvious node fault.

If the kubelet itself seems unresponsive, connect via SSM Session Manager (avoiding the need for SSH/bastion setup) and check systemctl status kubelet and its journal logs - frequent culprits include disk pressure from log/image buildup, an expired or misconfigured node IAM role losing API server trust, or security group rules blocking the node-to-control-plane path.

If none of that resolves it, cordon and drain the node, then let the node group (or Karpenter) terminate and replace it - for managed node groups this is often faster than continuing to debug a single bad instance.

CrashLoopBackOff means the container inside the pod is starting, exiting, and Kubernetes is repeatedly restarting it with an increasing backoff delay (10s, 20s, 40s, up to a cap) between attempts - it's a symptom, not a root cause by itself.

The kubelet on the node runs the restart loop locally: it invokes the container runtime to start the container, watches the exit code, and if the exit wasn't clean (non-zero, or an OOM kill), it waits the current backoff interval before trying again and reports the pod's state back to the API server.

kubectl logs <pod> --previous is the critical first command, since it retrieves logs from the last failed attempt rather than the pod's rapidly restarting current instance, and kubectl describe pod reveals whether the last termination reason was OOMKilled, a failed liveness probe, or an application-level panic/exit.

Common root causes on EKS specifically include missing IRSA/Pod Identity permissions causing the app to fail on startup when it can't reach an AWS API, a misconfigured ConfigMap or Secret mount, or resource limits set too low for the container's actual memory usage.

sequenceDiagram
  participant Kubelet
  participant Runtime
  participant API as API Server
  Kubelet->>Runtime: start container
  Runtime-->>Kubelet: exits non-zero
  Kubelet->>API: report CrashLoopBackOff
  Kubelet->>Runtime: retry after backoff delay

When a node's attached ENIs have no more secondary IP addresses to assign, new pods scheduled to that node stay stuck in ContainerCreating, and the CNI logs an error indicating it couldn't allocate an IP - the pod isn't rejected by the scheduler, it's stuck at the network setup step.

The ipamd daemon on the node normally pre-allocates a warm pool of extra IPs ahead of demand, but if the instance type's maximum ENI/IP capacity is fully exhausted, ipamd can't attach another ENI or secondary IP no matter how much warm-pool tuning you do.

The practical fixes are increasing the instance size (larger instances support more ENIs/IPs), enabling prefix delegation so each ENI slot gets a /28 block of IPs instead of one IP at a time - dramatically raising pod density per node - or reducing pod density expectations by adjusting max-pods per node type.

Cluster Autoscaler polls the scheduler's state every ~10 seconds, looking for pods in Pending status that failed to schedule due to insufficient resources - that's its sole scale-up trigger, not raw CPU/memory utilization.

For scale-up, it simulates whether adding a node from each configured Auto Scaling group would let the pending pod schedule, using the same predicates the real scheduler would apply, then picks a group and calls the EC2 Auto Scaling API to increase its desired capacity.

For scale-down, it looks for nodes whose utilization has stayed below a threshold (default 50%) for a set period, checks that every pod on the node could be rescheduled elsewhere without violating PodDisruptionBudgets, node affinity, or other constraints, then cordons, drains, and terminates it.

Because it works at the Auto Scaling group level, it has no visibility into aggregate cross-group instance flexibility the way Karpenter does - it can only grow or shrink the specific groups it's been given.

Karpenter reads the aggregate resource requests (CPU, memory, GPU) and scheduling constraints (node affinity, taints/tolerations, topology spread) of all currently unschedulable pods together, rather than reacting to one pod at a time.

It then consults the constraints defined in your NodePool and EC2NodeClass - allowed instance families, architectures, capacity types (Spot vs On-Demand), and zones - and computes the set of EC2 instance types that satisfy every pending pod's requirements at once.

From that candidate set, Karpenter's provisioning algorithm favors the option that best "bin-packs" the pending pods (minimizing wasted capacity and the number of new nodes needed) while respecting your price and diversity preferences, then calls the EC2 API directly to launch it - no Auto Scaling group involved.

flowchart LR
  A["Unschedulable pods"] --> B["Aggregate requests + constraints"]
  B --> C["Match against NodePool / EC2NodeClass"]
  C --> D["Select best bin-packing instance type"]
  D --> E["Launch via EC2 API"]

Transit Gateway acts as a central hub that VPCs attach to, letting an EKS cluster's VPC route traffic to other VPCs (shared services, other application VPCs, or on-premises networks via a Direct Connect/VPN gateway attachment) without a full mesh of individual VPC peering connections.

You attach the cluster's VPC and each target VPC to the same Transit Gateway, then add routes in each VPC's route tables pointing traffic destined for the other VPC's CIDR at the Transit Gateway attachment, and configure the Transit Gateway's own route table to direct traffic correctly between attachments.

Security groups still apply end-to-end even through the Transit Gateway, so pods reaching resources in another VPC (like a shared RDS instance) need both the routing path and the security group rules on the target resource to explicitly allow traffic from the EKS VPC's CIDR.

Pod Security Policies (PSP) were the original Kubernetes admission mechanism for restricting risky pod configurations (privileged containers, host networking, etc.), but PSP was deprecated in Kubernetes 1.21 and removed entirely in 1.25 - EKS clusters on 1.25+ simply don't have it available.

Pod Security Standards (PSS) is the replacement model: it defines three built-in policy levels - privileged, baseline, and restricted - enforced through the built-in Pod Security Admission controller using simple namespace labels, with no separate policy objects or RBAC bindings to manage.

Because PSS is less flexible than PSP for highly custom rules, many EKS teams pair it with a policy engine like OPA Gatekeeper or Kyverno for anything beyond the three standard levels, using PSS for baseline hygiene and the policy engine for organization-specific rules.

The default VPC CNI historically didn't enforce Kubernetes NetworkPolicy objects on its own, so Calico (or the VPC CNI's newer built-in network policy support) is layered in specifically to evaluate and enforce those policies at the pod's network interface.

Calico installs as a DaemonSet alongside the VPC CNI - it doesn't replace pod IP addressing, it adds an eBPF or iptables-based policy engine that intercepts traffic and checks it against your NetworkPolicy rules before allowing or dropping packets.

A typical hardening pattern starts with a default-deny NetworkPolicy per namespace, then explicit allow rules opening only the specific ingress/egress paths each workload actually needs - for example, only permitting a frontend namespace to reach a backend namespace on its service port.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: production
spec:
  podSelector: {}
  policyTypes: ["Ingress", "Egress"]

Right-sizing is usually the biggest lever: setting realistic CPU/memory requests (rather than copy-pasted defaults) lets the scheduler and autoscaler pack pods efficiently instead of over-provisioning nodes for padding no workload actually uses.

On the compute side, blending Spot Instances for fault-tolerant workloads with a smaller On-Demand or Savings Plan baseline for critical services captures large discounts without sacrificing availability, and Karpenter's consolidation feature actively repacks pods onto fewer, cheaper nodes as workloads change.

Fargate can reduce cost for bursty or low-traffic namespaces where idle EC2 capacity would otherwise sit unused, while EKS Auto Mode reduces the operational cost of managing scaling and upgrades, even if its per-resource pricing carries a premium over self-managed EC2.

Finally, EKS control plane cost itself is a small flat hourly fee per cluster - the real savings almost always come from data-plane efficiency, so consolidating multiple small clusters isn't usually where the biggest wins are found compared to fixing node and workload sizing.

When a user or controller submits a resource (say, a Deployment) via kubectl apply, the API server first runs authentication and RBAC authorization, then - before persisting the object - calls out to any registered validating admission webhooks, including Gatekeeper's.

Gatekeeper's webhook receives the incoming object as an AdmissionReview request and evaluates it against the Rego policies defined in its installed ConstraintTemplates and their corresponding Constraint instances - for example, a constraint requiring every pod to set resource limits.

If the object violates a constraint set to enforcementAction: deny, Gatekeeper returns a rejection in the AdmissionReview response, the API server blocks the write, and the user sees an error naming the violated policy; constraints set to dryrun instead just log the violation without blocking.

Because this happens synchronously in the request path, a slow or unavailable Gatekeeper webhook can delay or block all matching API writes cluster-wide - which is why Gatekeeper deployments should run with multiple replicas and a sensible failurePolicy setting.

sequenceDiagram
  participant User
  participant API as API Server
  participant GK as Gatekeeper Webhook
  User->>API: kubectl apply
  API->>GK: AdmissionReview request
  GK->>GK: evaluate Rego constraints
  GK-->>API: allow or deny
  API-->>User: persist or reject

Individual EKS clusters have documented soft limits - by default up to 1,000 nodes and around 4,000 pods per cluster in many configurations - though these are AWS service quotas that can often be raised via a support request rather than hard architectural ceilings.

The more practical bottleneck is usually the API server and etcd's ability to handle request and watch load as object counts grow; large numbers of ConfigMaps, Secrets, or extremely chatty controllers can degrade API latency well before you hit the documented node/pod maximums.

Common workarounds include splitting workloads across multiple clusters (by team, environment, or region) once a single cluster's control plane shows sustained API latency, using prefix delegation and larger instances to reduce total node count for a given pod count, and auditing controllers/operators for excessive watch or list calls that add unnecessary API load.

For very large fleets, some organizations adopt a "cluster-per-team" or "cell-based" model specifically to keep each individual cluster's object count and blast radius manageable, rather than pushing a single cluster to its absolute ceiling.

Because an EKS control plane is regional, true multi-region resilience means running independent clusters in each target region, not stretching one cluster across regions - there's no built-in cross-region control plane failover.

Application manifests and Helm releases are usually kept in sync across regions with the same GitOps repository (ArgoCD or Flux) targeting multiple clusters, so both regions run an identical, continuously reconciled configuration rather than drifting apart between failover events.

Stateful data is the harder problem: databases typically need cross-region replication (RDS cross-region read replicas, DynamoDB Global Tables, or application-level replication) set up independently of Kubernetes, since Kubernetes itself has no mechanism for replicating persistent volume data across regions.

Traffic failover is handled above the cluster layer entirely, usually via Route 53 health checks and failover routing policies, or Global Accelerator, that detect a region's unhealthy endpoints and redirect client traffic to the healthy region's load balancer.

The control plane is part of AWS's managed service boundary - AWS owns the underlying infrastructure running the API server and etcd, patches it, and is contractually responsible for its availability and security under the shared responsibility model, which requires it to stay outside customer access entirely.

Allowing direct SSH would break that model: a customer-side change to etcd or the API server binary could destabilize the control plane in ways AWS could no longer guarantee or safely support, and it would undermine the isolation that lets AWS patch and upgrade the plane without customer coordination.

Instead, every supported interaction with the control plane goes through the Kubernetes API itself (via kubectl, IAM-authenticated tokens, and RBAC) or the EKS control-plane-level API (cluster config updates, logging, endpoint access) - which is sufficient for essentially all legitimate operational needs, from debugging to configuration changes, without ever needing host-level access.