Prev Next

Cloud / Rancher Kubernetes Management Interview questions

Last updated

1. What is Rancher? 2. What are the main components of Rancher architecture? 3. What is the Rancher local cluster? 4. What are downstream clusters in Rancher? 5. What is RKE2? 6. What is K3s? 7. What is the cattle-cluster-agent? 8. What is the purpose of cert-manager when installing Rancher? 9. How do you install Rancher on Kubernetes using Helm? 10. What are the types of cluster provisioning in Rancher? 11. What is an imported cluster in Rancher? 12. What is a Rancher project? 13. What are the types of roles in Rancher RBAC? 14. List the authentication providers Rancher supports? 15. What is Fleet in Rancher? 16. What is the Rancher Apps and Marketplace? 17. What is the Rancher Backup operator? 18. What is the Rancher server-url setting? 19. How do you use the Rancher CLI? 20. What are cloud credentials and node drivers in Rancher? 21. What is the Authorized Cluster Endpoint in Rancher? 22. What is the difference between RKE1 and RKE2? 23. What is the difference between a Rancher project and a Kubernetes namespace? 24. How does Rancher communicate with downstream clusters? 25. Why does Rancher require the Kubernetes API Aggregation Layer? 26. When would you choose Let's Encrypt over Rancher-generated certificates? 27. Why should you avoid running workloads on the Rancher local cluster? 28. What happens when the Rancher server becomes unavailable? 29. How does Rancher RBAC map to Kubernetes RBAC? 30. What is the difference between a GitRepo and a Bundle in Fleet? 31. How do you target specific clusters with Fleet? 32. How do you back up and restore Rancher? 33. What is the difference between Rancher-provisioned and hosted Kubernetes clusters? 34. How do you use Terraform with Rancher? 35. How do you run Rancher in an air-gapped environment? 36. How do you upgrade Rancher in a high-availability setup? 37. What happens when you upgrade Rancher with RKE1 clusters present? 38. How do you migrate RKE1 clusters to RKE2? 39. How do you take and restore etcd snapshots for an RKE2 cluster in Rancher? 40. How do you harden a Rancher installation? 41. What is the Rancher CIS Benchmark scan and how does it work? 42. Explain the lifecycle of provisioning an RKE2 cluster in Rancher? 43. Explain the execution flow of a Fleet deployment from Git to cluster? 44. Explain the execution flow of a kubectl request through Rancher? 45. How do you troubleshoot a downstream cluster stuck in Unavailable? 46. How do you troubleshoot a cluster stuck waiting for nodes during provisioning? 47. How do you troubleshoot cattle-cluster-agent certificate errors? 48. How does Rancher Turtles integrate Cluster API? 49. How can you optimize Rancher server for many downstream clusters? 50. How do you upgrade a Rancher-managed RKE2 cluster safely?

1. What is Rancher?

Rancher is an open-source platform for managing many Kubernetes clusters from one place. It does not replace Kubernetes. It sits on top of your clusters, wherever they run, and gives you a single UI, API, and authentication layer for all of them.

With Rancher you can provision RKE2 or K3s clusters, import existing ones, and manage hosted clusters such as EKS, AKS, and GKE. It adds central RBAC, a Helm-based app catalog, GitOps delivery through Fleet, and add-ons such as monitoring and backup.

Rancher itself runs as a workload on a Kubernetes cluster and is normally installed with Helm.

Take quiz
How does Rancher relate to the Kubernetes clusters it manages?
It replaces the kube-apiserver in every cluster
It only builds container images for those clusters
It is a container runtime that replaces containerd
It sits above them and manages clusters built on Kubernetes distributions
Where does the Rancher server normally run?
Inside the kubelet of every downstream node
As a Helm-installed workload on a Kubernetes cluster
Only as a SaaS service hosted by Kubernetes upstream
As a systemd service on a hypervisor with no Kubernetes

2. What are the main components of Rancher architecture?

A Rancher setup has a management plane (the Rancher server) and agents that run in every managed cluster. The server never needs to be inside the clusters it manages.

Component Where it runs Role
Rancher server cattle-system namespace of the local cluster API, UI, authentication proxy, cluster controllers
rancher-webhook Local and downstream clusters Validates and mutates Rancher resources such as roles and projects
cattle-cluster-agent cattle-system of each downstream cluster Opens the tunnel to the server and runs cluster-level controllers
cattle-node-agent DaemonSet on downstream nodes Node-level operations; fallback connection if the cluster agent is down
fleet-controller / fleet-agent Local cluster / downstream clusters GitOps deployment
rancher-system-agent Nodes provisioned by Rancher (RKE2/K3s) Applies provisioning plans to the node

Rancher keeps its own state as Kubernetes custom resources in the local cluster's etcd, so there is no separate database to run.

Take quiz
Which component carries the connection between a downstream cluster and the Rancher server?
cattle-cluster-agent
kube-proxy
fleet-controller
rancher-webhook
Where does Rancher store its own configuration and state?
In a ConfigMap on each downstream node
In files under /etc/rancher on the load balancer
As custom resources in the local cluster's etcd
In an external MySQL database that must be installed first

3. What is the Rancher local cluster?

The local cluster is the Kubernetes cluster that Rancher is installed on. It shows up in the UI under the name local and is created automatically when Rancher starts.

It holds everything Rancher needs to function: the Rancher deployment, the custom resources for users, clusters, projects and roles, Fleet, and the webhook. Because all of that lives in its etcd, the local cluster is effectively the control plane for every other cluster you manage.

You can view and manage it from the UI like any other cluster, but it should be treated as infrastructure, not as a place to run applications.

Take quiz
What is the name of the cluster Rancher is installed on, as shown in the UI?
cattle-system
local
rancher-main
upstream-default
Why is the local cluster so important operationally?
It stores the container images of every cluster
It provides the container network for downstream nodes
It runs every application pod from all downstream clusters
Its etcd stores the Rancher objects that manage all downstream clusters

4. What are downstream clusters in Rancher?

Downstream clusters are the Kubernetes clusters that a Rancher server manages. They run your actual workloads, while the Rancher server stays on the local cluster.

A downstream cluster can be created by Rancher (RKE2 or K3s on cloud VMs, vSphere, or existing machines), hosted by a cloud provider (EKS, AKS, GKE), or imported after being built elsewhere. Whatever the origin, Rancher deploys a cattle-cluster-agent into it, and the agent dials back to the server.

If the Rancher server goes offline, downstream clusters and their workloads keep running.

Take quiz
How does a downstream cluster become reachable from the Rancher server?
Rancher installs a second kube-apiserver on its own node
Each pod registers itself with the Rancher ingress
A cattle-cluster-agent inside it connects back to Rancher
Rancher mounts the downstream etcd directly
Which of these can be a downstream cluster?
An imported cluster that was built outside Rancher
Only clusters that run on the same nodes as Rancher
Only clusters created by the rke command line tool
Only clusters that use Docker as the runtime

5. What is RKE2?

RKE2 (also called RKE Government) is SUSE's CNCF-certified Kubernetes distribution and the recommended way to build clusters with Rancher. It ships as a single binary and runs as systemd services, rke2-server and rke2-agent.

The control plane components run as static pods managed by the kubelet, and the container runtime is containerd, so Docker is not required. It is designed to be secure by default, with CIS hardening in mind and a FIPS 140-2 compliant build available. The default CNI is Canal, and Calico, Cilium, and Multus can be selected.

curl -sfL https://get.rke2.io | sh -
systemctl enable --now rke2-server.service

Settings live in /etc/rancher/rke2/config.yaml.

Take quiz
How do RKE2 control plane components run?
As Windows services
As Deployments managed by Rancher on the local cluster
As Docker containers started by the rke binary
As static pods managed by the kubelet
Which file holds RKE2 node configuration?
/opt/rke2/settings.ini
/etc/rancher/rke2/config.yaml
/etc/kubernetes/rke2.conf
/var/lib/rancher/rke2.json
Which container runtime does RKE2 use?
CRI-O only
Docker Engine
rkt
containerd

6. What is K3s?

K3s is a lightweight, CNCF-certified Kubernetes distribution packaged as a single small binary. It targets edge, IoT, CI, and small-footprint environments, but it is also production-grade and works fine as the base cluster for Rancher.

Storage defaults differ from RKE2. A single server uses SQLite, while HA setups use embedded etcd or an external SQL database. K3s also bundles Traefik as ingress, a service load balancer, and a local-path storage provisioner, all of which can be disabled.

curl -sfL https://get.k3s.io | sh -

Rancher can provision K3s clusters the same way it provisions RKE2.

Take quiz
What does a single-server K3s install use for its datastore by default?
SQLite
A hosted etcd service
Embedded Cassandra
MongoDB
What is K3s mainly optimized for?
Replacing Helm as the package manager
Running only on Windows hosts
A small footprint, such as edge and resource-limited nodes
Providing a managed control plane in the cloud

7. What is the cattle-cluster-agent?

The cattle-cluster-agent is a Deployment in the cattle-system namespace of every downstream cluster. It makes an outbound WebSocket connection to the Rancher server using the server-url and keeps that tunnel open.

Through the tunnel, Rancher reaches the cluster's API server without any inbound firewall rule on the downstream side. The agent also runs controllers that sync cluster state back to Rancher, and it impersonates the logged-in user when Rancher proxies a request.

If the cluster agent cannot connect, the cattle-node-agent running on the nodes can serve as a fallback tunnel.

Take quiz
In which direction is the cattle-cluster-agent connection established?
Through a VPN that Rancher creates automatically
Outbound from the downstream cluster to the Rancher server
Inbound from the Rancher server to port 6443 only
From the user's browser straight to the node
Which agent can act as a fallback if the cluster agent is unavailable?
rancher-webhook
kube-api-auth
fleet-agent
cattle-node-agent

8. What is the purpose of cert-manager when installing Rancher?

cert-manager issues and renews the TLS certificate for the Rancher ingress. It is needed when the Helm chart option ingress.tls.source is rancher (the default, using a Rancher-generated CA) or letsEncrypt.

If you bring your own certificate with ingress.tls.source=secret, cert-manager is not required. When it is used, install it before Rancher, along with its CRDs, and use a version listed as compatible with your Rancher release.

helm install cert-manager jetstack/cert-manager \
  --namespace cert-manager --create-namespace \
  --set crds.enabled=true

Take quiz
When can you skip installing cert-manager for Rancher?
When you use Rancher on more than three nodes
When the cluster runs K3s instead of RKE2
When you supply your own certificate as a TLS secret
Never, Rancher refuses to start without it
Which Helm value selects where the Rancher TLS certificate comes from?
ingress.tls.source
hostname.certSource
tls.provider.name
rancher.certMode

9. How do you install Rancher on Kubernetes using Helm?

Add the Rancher chart repository, create the cattle-system namespace, install cert-manager if you need it, then install the chart with the hostname that users and agents will use.

helm repo add rancher-latest https://releases.rancher.com/server-charts/latest
kubectl create namespace cattle-system
helm install rancher rancher-latest/rancher \
  --namespace cattle-system \
  --set hostname=rancher.example.com \
  --set bootstrapPassword=ChangeMe123 \
  --set replicas=3
kubectl -n cattle-system rollout status deploy/rancher

The hostname must resolve to your load balancer or ingress. On first login you set the admin password and confirm the server URL. The stable repo is the better pick for production, and the SUSE Rancher Prime edition uses its own repo.

Take quiz
Which Helm value tells Rancher the DNS name clients will use?
ingress.domain
rancher.dnsLabel
serverName
hostname
Into which namespace is Rancher installed by the standard Helm procedure?
rancher-server
cattle-system
fleet-default
kube-system

10. What are the types of cluster provisioning in Rancher?

Rancher supports four ways to bring a cluster under management. They differ in how much of the cluster lifecycle Rancher owns.

Type What Rancher does Example
Node driver (infrastructure provider) Creates the VMs and installs RKE2 or K3s on them Machine pools on EC2, Azure, vSphere, Harvester
Custom cluster Installs RKE2 or K3s on machines you already have, via a registration command Bare-metal servers or pre-built VMs
Hosted Kubernetes Calls the cloud API through an operator; the provider runs the control plane EKS, AKS, GKE
Imported Only installs the agent; lifecycle stays with whoever built it A kubeadm-built or other conformant cluster

Pick node drivers when you want full automation and scaling, and import when the cluster already exists.

Take quiz
Which provisioning type installs RKE2 on servers you already own using a registration command?
Custom cluster
Node driver machine pool
Hosted Kubernetes
Imported cluster
Which type leaves cluster lifecycle management mostly with the original builder?
Machine pool cluster
Custom cluster with roles
Imported cluster
Node driver cluster

11. What is an imported cluster in Rancher?

An imported cluster is an existing Kubernetes cluster that you register with Rancher instead of creating through it. Rancher gives you a registration command, which is a kubectl apply of a manifest that deploys the cattle-cluster-agent.

Once the agent connects, you get the usual benefits: single sign-on, RBAC, projects, monitoring, the app catalog, and Fleet. Rancher does not own the node lifecycle, so you cannot scale nodes or change the Kubernetes version from Rancher in general. Imported RKE2 and K3s clusters get some extra upgrade support.

Removing the cluster from Rancher does not delete it, but you should clean up the leftover agents.

Take quiz
What does the registration command for an imported cluster install?
A replacement kube-scheduler
The cattle-cluster-agent and related objects
Rancher's own server replicas
A new etcd cluster on the nodes
What happens to an imported cluster when you remove it from Rancher?
Rancher deletes all its nodes
It is migrated to the local cluster
Its etcd data is wiped
The cluster keeps running; Rancher just stops managing it

12. What is a Rancher project?

A project is a Rancher concept that groups several namespaces inside one cluster so you can manage them together. It is not a native Kubernetes resource. Rancher tracks membership through a label and annotation on the namespace (field.cattle.io/projectId).

Projects exist to apply things once: role bindings for a team, resource quotas, and default container limits. Every cluster starts with two projects, Default and System.

A namespace can belong to only one project at a time, and a namespace with no project is allowed but gets no project-level permissions.

Take quiz
What is the main purpose of a Rancher project?
To replace namespaces entirely
To define the container runtime for a node pool
To group namespaces in a cluster for shared access control and quotas
To group several clusters under one billing account
How many projects can one namespace belong to?
One at a time
Any number
Up to three
None, a namespace is never in a project

13. What are the types of roles in Rancher RBAC?

Rancher layers three scopes of roles on top of Kubernetes RBAC. Each one is bound to a user or group at a different level.

Scope Built-in examples Applies to
Global Administrator, Standard User, User-Base The whole Rancher installation
Cluster Cluster Owner, Cluster Member One cluster and everything in it
Project Project Owner, Project Member, Read-Only The namespaces in one project

You can also define custom roles at each scope. Permissions flow downward, so a Cluster Owner has rights in every project of that cluster, while a Project Member has none outside the project.

Take quiz
Which role scope applies to just the namespaces of one project?
Global role
Node role
Cluster role
Project role
What does a Cluster Owner automatically get?
Access to the local cluster
Access to every project in that cluster
Administrator rights on all clusters
Only read access to the cluster

14. List the authentication providers Rancher supports?

Rancher always has local authentication for built-in accounts, and it can delegate login to an external identity provider.

  • Directory services: Active Directory, OpenLDAP, FreeIPA
  • OAuth / social: GitHub, Google
  • SAML: Okta, PingIdentity, Shibboleth, Keycloak, ADFS, Microsoft Entra ID (Azure AD)
  • OIDC: Keycloak (OIDC), Amazon Cognito, generic OIDC providers

The exact list changes between releases, so check the docs for your version. In practice, teams configure one external provider, map its groups to Rancher roles, and keep a break-glass local admin.

Take quiz
Which of these is a directory service Rancher can authenticate against?
OpenLDAP
etcd
Prometheus
Harbor
Why map external groups to Rancher roles instead of individual users?
Rancher cannot bind roles to individual users
Groups bypass cluster RBAC entirely
Access follows group membership in the identity provider
It removes the need for a local admin forever

15. What is Fleet in Rancher?

Fleet is the GitOps continuous delivery engine built into Rancher. It watches Git repositories and deploys their contents (plain YAML, Helm charts, or Kustomize) to many clusters at once.

A GitRepo resource points Fleet at a repo. Fleet turns the content into Bundles and rolls them out to target clusters through a fleet-agent running in each one. Two workspaces exist by default: fleet-default for downstream clusters and fleet-local for the Rancher local cluster.

Its main design goal is scale, meaning thousands of clusters from a few Git repos.

Take quiz
Which resource tells Fleet which Git repository to deploy?
HelmRelease
GitRepo
ApplicationSet
KustomizationSource
Which Fleet workspace is used for downstream clusters by default?
fleet-downstream
fleet-local
cattle-fleet-system
fleet-default

16. What is the Rancher Apps and Marketplace?

Apps & Marketplace is the Helm chart catalog in the Rancher UI. You pick a cluster, browse charts from configured repositories, fill in values, and install.

By default it includes Rancher's own charts such as monitoring, logging, backup, and CIS scanning, plus partner charts. You can add your own Helm or Git-based repositories, including private ones. Rancher charts often show a major version starting with 100 (for example 100.0.0+up2.1.0); the upstream version follows the +up suffix.

Installs are standard Helm releases, so helm list on the cluster sees them.

Take quiz
What does the +up part of a chart version like 100.0.0+up2.1.0 indicate?
The number of replicas installed
The Kubernetes version it needs
The upstream project's version
The Rancher server patch level
What is installed when you deploy an app from the Rancher marketplace?
A regular Helm release in the chosen cluster
A Fleet GitRepo in the local cluster only
A Docker Compose project
A VM image on the cloud provider

17. What is the Rancher Backup operator?

The Backup operator (the rancher-backup chart) saves the Kubernetes resources that make up a Rancher installation, so you can restore Rancher after a failure or move it to a new cluster. It works with two custom resources, Backup and Restore.

Backups go to a persistent volume or an S3-compatible bucket and can be scheduled. They cover Rancher's own objects (users, clusters, roles, settings, and so on), not your downstream workloads and not the downstream etcd.

apiVersion: resources.cattle.io/v1
kind: Backup
metadata:
  name: rancher-nightly
spec:
  resourceSetName: rancher-resource-set
  schedule: "0 2 * * *"
  retentionCount: 7

Take quiz
What does the Rancher Backup operator capture?
Every application PV in downstream clusters
Downstream etcd snapshots
Container images from the registry
Rancher's own resources such as users, clusters and roles
Which custom resource triggers a restore?
RancherRecovery
Restore
Rollback
BackupRun

18. What is the Rancher server-url setting?

server-url is a global setting holding the URL, such as https://rancher.example.com, that downstream agents use to reach Rancher. It is embedded in registration commands, agent manifests, and generated kubeconfigs.

You confirm it on first login, so it must be a stable DNS name reachable from every downstream node, not a temporary IP. Changing it later breaks the connection of existing agents until they are redeployed with the new value, so plan it before registering clusters.

A common cause of clusters stuck in Unavailable is a server-url that downstream nodes cannot resolve or reach.

Take quiz
Who consumes the server-url value?
Downstream agents that connect back to Rancher
The kube-scheduler of the local cluster
Only the Rancher UI logo
The container image registry
Why avoid changing server-url after clusters are registered?
Rancher deletes all projects
It forces an RKE2 upgrade
Existing agents still point at the old URL until redeployed
It resets every user's password

19. How do you use the Rancher CLI?

The Rancher CLI is a binary named rancher that talks to the Rancher API. You log in once with an API bearer token created under Account & API Keys, then work with clusters and projects from the terminal.

rancher login https://rancher.example.com --token token-abcde:xxxxxxxx
rancher clusters ls
rancher context switch
rancher kubectl get pods -A

rancher kubectl runs kubectl against the cluster in your current context, using Rancher authentication. The CLI is handy for scripts, but for full automation most teams prefer the Rancher API or Terraform.

Take quiz
What does the rancher kubectl command do?
Creates a new Rancher server
Runs kubectl against the cluster in your current Rancher context
Installs kubectl on every node
Uploads a kubeconfig to GitHub
What do you need to run rancher login?
An SSH key for the local cluster nodes
The etcd snapshot name
A Helm chart name
The server URL and an API bearer token

20. What are cloud credentials and node drivers in Rancher?

Cloud credentials are secrets (API keys, subscription IDs, vCenter logins) that Rancher stores so it can call an infrastructure provider on your behalf. They are reusable across clusters and permission-controlled.

Node drivers are the plugins that know how to create a VM at a specific provider, such as Amazon EC2, Azure, DigitalOcean, Linode, vSphere, or Harvester. Each machine pool uses a driver, a credential, and a machine configuration (instance type, image, network, disk).

Drivers can be activated or deactivated in the cluster management section, which is useful for limiting where people can build clusters.

Take quiz
What do cloud credentials give Rancher?
A copy of the Git repo
A TLS certificate for the ingress
Authorized API access to an infrastructure provider
Direct root access to downstream pods
What does a node driver do?
Creates machines at a specific infrastructure provider
Drains nodes before an upgrade
Converts Helm charts into Fleet bundles
Encrypts etcd snapshots

21. What is the Authorized Cluster Endpoint in Rancher?

The Authorized Cluster Endpoint (ACE) lets kubectl talk directly to a downstream cluster's API server instead of going through the Rancher server proxy. The kubeconfig downloaded from Rancher then contains an extra context pointing at the cluster itself.

It helps in two cases. You avoid the extra hop through Rancher for latency-sensitive or heavy API use, and kubectl keeps working if the Rancher server is unreachable. Authentication still uses Rancher credentials, validated by the kube-api-auth webhook on the control plane.

ACE is available for RKE2 and K3s clusters, and you can point it at an FQDN or load balancer for the API server.

Take quiz
What is a key benefit of the Authorized Cluster Endpoint?
It removes the need for any authentication
It stores workloads on the Rancher node
It replaces the cattle-cluster-agent
kubectl can reach the cluster API directly, even if Rancher is down
Which component validates Rancher credentials on the downstream control plane for ACE?
fleet-agent
kube-api-auth
rancher-webhook
cattle-node-agent

22. What is the difference between RKE1 and RKE2?

Both are Rancher-built Kubernetes distributions, but they are architecturally different, and RKE1 is end of life (July 31, 2025). Rancher 2.12 and later no longer provision or manage downstream RKE1 clusters.

Aspect RKE1 RKE2
Container runtime Docker containerd
Control plane Docker containers created by the rke tool Static pods managed by kubelet
Install method rke CLI or Rancher over SSH Single binary with systemd services
Security posture Hardening applied manually Secure defaults, CIS-aligned, FIPS build available
Status End of life Actively supported, recommended

RKE2 is also the base for the Rancher v2 provisioning framework, which is why new clusters should start there.

Take quiz
Which container runtime does RKE1 depend on?
Docker
Podman
CRI-O
containerd only
What is the support status of RKE1 as of Rancher 2.12 and later?
Supported only on Windows nodes
Replaced by K3s with no migration needed
Not provisioned or managed by Rancher; RKE1 reached end of life
Fully supported with new features

23. What is the difference between a Rancher project and a Kubernetes namespace?

A namespace is a native Kubernetes object that scopes resource names and can carry quotas and RBAC. A project is a Rancher-only grouping of namespaces inside one cluster.

Namespace Project
Native Kubernetes object Rancher custom resource and label
Isolates object names Groups namespaces for shared policy
Quota set per namespace Quota set once, split across member namespaces
RBAC bound with RoleBinding RBAC bound once, Rancher creates RoleBindings in each member namespace
Exists with or without Rancher Exists only where Rancher manages the cluster

Use projects when one team owns several namespaces. If you remove Rancher, the namespaces and their bindings stay, but the project grouping is lost.

Take quiz
How is a project-level role applied inside the cluster?
It creates a node taint
Rancher creates RoleBindings in each namespace of the project
It writes a ClusterRole for every pod
It edits the kube-apiserver flags
Which statement is true about projects?
They replace namespaces
They are built into every kubeadm cluster
They contain multiple clusters
They are a Rancher concept that groups namespaces

24. How does Rancher communicate with downstream clusters?

Communication is agent-initiated and outbound. The cattle-cluster-agent in each downstream cluster opens a WebSocket tunnel to the Rancher server, so the downstream side needs no inbound rule for Rancher.

When a user runs kubectl or uses the UI, the Rancher server authenticates the request, sends it through the tunnel, and the agent forwards it to the kube-apiserver while impersonating the user. For clusters Rancher provisions with RKE2 or K3s, the rancher-system-agent on each node also contacts Rancher to fetch plan secrets with install and configuration instructions.

Rancher itself only needs outbound access to cloud provider APIs when it creates machines. So the firewall rule that matters is downstream nodes to Rancher on 443.

Take quiz
Which network rule is essential for a downstream cluster to stay connected?
Each worker must expose port 2379 publicly
Rancher must reach every pod IP directly
Downstream nodes must be able to reach the Rancher server on 443
The control plane must accept SSH from the internet
What does the cluster agent do before forwarding a proxied request to the API server?
Impersonates the Rancher user
Signs the request with the etcd key
Rewrites the pod specs
Strips the Authorization header and uses anonymous access

25. Why does Rancher require the Kubernetes API Aggregation Layer?

Recent Rancher releases extend the Kubernetes API by registering their own extension API server. That server is hooked into the cluster through the API Aggregation Layer, so the kube-apiserver can forward requests for Rancher's added API groups to it.

If the aggregation layer is not enabled on the cluster where Rancher runs (the local cluster), those APIs cannot be registered and Rancher will not work correctly. This is a requirement on the local cluster only, not on downstream clusters.

RKE2, K3s, and the managed Kubernetes services (EKS, AKS, GKE) enable the layer by default. It mostly bites people on hand-built clusters where the aggregator flags and front-proxy certificates were not configured.

Take quiz
On which cluster must the API Aggregation Layer be enabled for Rancher?
Every downstream cluster
Only on the cluster running cert-manager
Only on clusters running Windows nodes
The cluster where Rancher is installed
Which clusters are most likely to be missing aggregation layer support?
Default K3s installations
Hand-built clusters where the aggregator was never configured
Default RKE2 installations
Managed EKS clusters

26. When would you choose Let's Encrypt over Rancher-generated certificates?

Choose Let's Encrypt when the Rancher hostname is publicly resolvable and reachable on port 80, because cert-manager proves ownership with the HTTP-01 challenge. Browsers and agents trust the certificate by default, so you avoid distributing a private CA.

ingress.tls.source Best fit Watch out for
rancher (default) Labs and internal setups Self-signed CA; browsers warn; agents verify by CA checksum
letsEncrypt Public Rancher with open port 80 Not usable air-gapped or on private DNS; needs email setting
secret Corporate PKI, air-gapped You rotate certs yourself; set privateCA=true for a private CA

Switching the source later changes the CA that agents see, so test it before production.

Take quiz
What does cert-manager need in order to issue a Let's Encrypt certificate for Rancher?
A hostname reachable from the internet on port 80 for HTTP-01
Access to the downstream etcd
Docker installed on each node
A private CA stored on the load balancer
Which TLS source fits an air-gapped network with a corporate CA?
letsEncrypt
none, Rancher runs without TLS
secret
rancher with ACME

27. Why should you avoid running workloads on the Rancher local cluster?

The local cluster is the control plane for every cluster you manage, so its stability and security have a much larger blast radius than a normal cluster. Mixing application workloads into it puts that at risk.

  • Resource contention: a noisy pod can starve the Rancher deployment or etcd, slowing the UI and agents for all clusters.
  • Security: anyone with a foothold in the local cluster is one step from every managed cluster.
  • Maintenance: upgrading the local cluster or Rancher becomes an application outage as well.
  • Backup scope: the Backup operator targets Rancher objects, not app data.

Run Rancher on a small dedicated cluster of three nodes and put applications in downstream clusters.

Take quiz
Which risk is the strongest argument against app workloads on the local cluster?
It forces Docker to be installed
A compromise or overload affects management of every cluster
It prevents RKE2 from using containerd
It blocks Helm from working
What is a typical recommendation for hosting Rancher?
A single laptop-class node
A cluster with no persistent storage ever
The same cluster as production databases
A small dedicated cluster of three nodes

28. What happens when the Rancher server becomes unavailable?

Downstream clusters keep running. Pods, Services, ingress, and the kubelet-to-API-server traffic inside each cluster do not depend on Rancher.

What you lose is management. The UI, API, and the Rancher authentication proxy stop working, so kubectl through Rancher fails. Provisioning, scaling, upgrades, and Fleet updates pause until Rancher returns, and the agents keep retrying their tunnel.

If the Authorized Cluster Endpoint is enabled, kubectl can still talk straight to the downstream API server, and tokens keep validating because the cluster stores copies of them. Machine replacement from machine pools will not happen during the outage.

Take quiz
What happens to running pods in a downstream cluster when Rancher goes offline?
They stop receiving traffic until Rancher returns
They move to the local cluster
They keep running
They are evicted after 5 minutes
Which access path can still work while Rancher is down?
kubectl through the Authorized Cluster Endpoint
Rancher CLI kubectl via the proxy
The Rancher UI cluster explorer
Fleet Git sync from the local cluster

29. How does Rancher RBAC map to Kubernetes RBAC?

Rancher stores permissions as its own objects, RoleTemplates and the bindings that attach them to users, then its controllers translate them into native Kubernetes roles and bindings in the target clusters.

Rancher object What Rancher creates
Global role + GlobalRoleBinding ClusterRoles and bindings on the local cluster; can grant cluster access
ClusterRoleTemplateBinding (CRTB) ClusterRoleBinding in the downstream cluster
ProjectRoleTemplateBinding (PRTB) RoleBindings in each namespace of the project

User identities are not Kubernetes users. Rancher authenticates the user, then the cluster agent impersonates a user-xxxxx identity with the matching group, and the API server enforces the generated bindings. Debugging an access issue usually means checking the binding objects in the local cluster, then the generated ones downstream.

Take quiz
Which Rancher object produces RoleBindings in the namespaces of a project?
GlobalRoleBinding
ClusterRoleTemplateBinding
NodeTemplate
ProjectRoleTemplateBinding
How does the kube-apiserver learn who the Rancher user is?
The kubelet issues the token
The cluster agent impersonates the user
Users log in with local Linux accounts
Rancher copies the user's password to each cluster

30. What is the difference between a GitRepo and a Bundle in Fleet?

A GitRepo is what you create. It tells Fleet where the source lives and where to send it. A Bundle is what Fleet generates, a packaged set of resources that can be deployed.

GitRepo Bundle
Points to a Git URL, branch, and paths Holds the rendered manifests or Helm chart
Created by users Created by Fleet, one per detected path
Defines targets and credentials Carries targets and per-cluster overrides from fleet.yaml
Lives in the workspace, e.g. fleet-default Produces a BundleDeployment per matching cluster

You can also create Bundles without Git by running fleet apply and applying the generated YAML.

Take quiz
Who normally creates Bundle objects in a GitOps setup?
Fleet, from the paths in a GitRepo
The kubelet on each node
cert-manager
The user, by hand, for every cluster
What does Fleet create for each cluster that matches a Bundle's targets?
A Namespace
A NodePool
A BundleDeployment
A RoleTemplate

31. How do you target specific clusters with Fleet?

Targets are set in the GitRepo under spec.targets. You can match clusters by clusterSelector labels, clusterGroup, or clusterName. If no target matches, nothing is deployed.

apiVersion: fleet.cattle.io/v1alpha1
kind: GitRepo
metadata:
  name: apps
  namespace: fleet-default
spec:
  repo: https://github.com/example/fleet-apps
  branch: main
  paths: [guestbook]
  targets:
  - name: prod
    clusterSelector:
      matchLabels:
        env: prod
  - name: dev
    clusterSelector:
      matchLabels:
        env: dev

Labels go on the Fleet cluster objects, which you edit in Continuous Delivery. For per-environment differences, use targetCustomizations in the bundle's fleet.yaml to change Helm values.

Take quiz
What happens in Fleet if a GitRepo has no matching targets?
It deploys to every cluster
Nothing is deployed
It fails the Rancher upgrade
It deploys only to the local cluster
Where do you add the env: prod label that the clusterSelector matches?
On the Git commit message
On the container image
On the Rancher server Deployment
On the Fleet cluster object

32. How do you back up and restore Rancher?

Use the Backup operator. Take a backup, keep it off the cluster, and restore into a fresh cluster if the original is lost.

  1. Install the rancher-backup-crd and rancher-backup charts on the local cluster.
  2. Create a Backup resource with an S3 or PV storage location and a schedule.
  3. Also take an etcd snapshot of the local cluster as a second safety net.
  4. To restore, build a new cluster, install the same two charts, and create a Restore that names the backup file.
  5. Install Rancher with the same hostname and version, then repoint DNS.
apiVersion: resources.cattle.io/v1
kind: Restore
metadata:
  name: restore-rancher
spec:
  backupFilename: rancher-nightly-2026-11-19T02-00-00Z.tar.gz
  prune: false

Use prune: false when restoring into a new cluster, and restore to the same Rancher version that took the backup. Downstream clusters reconnect once DNS points at the restored Rancher.

Take quiz
What is the purpose of prune: false in a Restore into a new cluster?
It skips restoring users
It restores only the etcd snapshot
It stops the restore from deleting resources not in the backup
It disables the TLS secret
In what order do you restore Rancher onto a new cluster?
Install backup charts, restore, then install Rancher
Install Rancher first and restore later
Restore Fleet, then delete the backup charts
Restore only after upgrading to a newer Rancher

33. What is the difference between Rancher-provisioned and hosted Kubernetes clusters?

The difference is who runs the control plane. With provisioned RKE2 or K3s, Rancher manages the nodes and the Kubernetes lifecycle. With hosted clusters, the cloud provider does.

Aspect Rancher-provisioned Hosted (EKS, AKS, GKE)
Control plane and etcd Run on your nodes, managed by Rancher Run by the cloud provider
Upgrades Rancher drives them via system-agent Provider API, triggered through an operator
Etcd snapshots Available in Rancher Not exposed; provider handles resilience
Node scaling Machine pools in Rancher Provider node groups or pools
Rancher component Provisioning framework eks-operator, aks-operator, gke-operator

In both cases, the cattle-cluster-agent gives you the same RBAC, projects, and Fleet experience.

Take quiz
Who operates the etcd and control plane of an EKS cluster managed through Rancher?
Rancher on your nodes
The cattle-node-agent
Fleet
The cloud provider
Which Rancher component calls cloud APIs for a hosted EKS cluster?
rancher-backup
eks-operator
kube-api-auth
rancher-system-agent

34. How do you use Terraform with Rancher?

Use the rancher2 provider. It manages Rancher objects such as cloud credentials, machine configs, clusters, projects, and role bindings, so you can describe clusters as code.

provider "rancher2" {
  api_url   = "https://rancher.example.com"
  token_key = var.rancher_token
}

resource "rancher2_cluster_v2" "prod" {
  name               = "prod-rke2"
  kubernetes_version = "v1.33.5+rke2r1"
  rke_config {
    machine_pools {
      name                         = "all"
      cloud_credential_secret_name = rancher2_cloud_credential.aws.id
      control_plane_role           = true
      etcd_role                    = true
      worker_role                  = true
      quantity                     = 3
      machine_config {
        kind = rancher2_machine_config_v2.aws.kind
        name = rancher2_machine_config_v2.aws.name
      }
    }
  }
}

The v2 resource, rancher2_cluster_v2, is for RKE2 and K3s. For the first install, rancher2_bootstrap sets the admin password and returns a token for the rest of the run.

Take quiz
Which Terraform resource creates an RKE2 or K3s cluster with the v2 provisioning framework?
rancher2_cluster_v2
rancher2_node_template
rancher2_rke_cluster
rancher2_helm_release
What does rancher2_bootstrap handle?
Backing up etcd
Creating Fleet bundles
First-time admin password and token setup
Draining nodes before an upgrade

35. How do you run Rancher in an air-gapped environment?

An air-gapped install needs every image and chart available from a private registry you control, because nothing can be pulled from the internet.

  1. Download rancher-images.txt for your version and mirror the images with the save and load scripts.
  2. Mirror cert-manager images and chart, and the RKE2 or K3s air-gap artifacts.
  3. Install Rancher with --set systemDefaultRegistry=registry.example.com and --set useBundledSystemChart=true.
  4. If the registry uses a private CA, add --set privateCA=true and provide the CA bundle.
  5. Configure downstream nodes with a registries.yaml and the same system default registry.

useBundledSystemChart=true makes Rancher use charts packaged with it instead of fetching them from GitHub, which would fail offline.

Take quiz
Which Helm value points Rancher's system images to your private registry?
hostname
systemDefaultRegistry
privateImageHost
ingress.tls.source
Why set useBundledSystemChart=true in an air-gapped install?
To skip TLS verification entirely
To enable Docker as the runtime
To disable cert-manager
So Rancher does not try to download system charts from the internet

36. How do you upgrade Rancher in a high-availability setup?

Upgrade Rancher with helm upgrade after confirming compatibility and taking a backup. Never skip the backup, because rolling back Rancher means restoring one.

  1. Check the support matrix for the target Rancher version, the local cluster's Kubernetes version, and cert-manager.
  2. Take a Backup operator backup and an etcd snapshot.
  3. Update the repo and save the current values: helm get values rancher -n cattle-system -o yaml > values.yaml.
  4. Upgrade with the same values file and the new chart version.
  5. Watch the rollout and then confirm downstream clusters reconnect.
helm repo update
helm upgrade rancher rancher-latest/rancher \
  --namespace cattle-system -f values.yaml --version 2.13.3
kubectl -n cattle-system rollout status deploy/rancher

From Rancher 2.12 onward, use Helm 3.18 or newer. Upgrade one minor version at a time.

Take quiz
What does a Rancher rollback require?
Deleting the cattle-system namespace
Running helm rollback and nothing else
Restoring from a backup taken before the upgrade
Downgrading each downstream cluster first
Why save the Helm values before upgrading?
To reuse the same configuration, such as hostname and TLS source
Helm cannot read the cluster otherwise
To rotate the admin password
To disable the webhook

37. What happens when you upgrade Rancher with RKE1 clusters present?

Recent Rancher versions refuse. Rancher includes a pre-upgrade validation that fails and lists any RKE1 resources it finds, because Rancher 2.12 and later no longer provision or manage downstream RKE1 clusters.

The clusters themselves do not stop. They keep running as ordinary Docker-based clusters, but Rancher can no longer upgrade, scale, or reconfigure them, and security fixes for RKE1 are over.

So the order of work is: inventory RKE1 clusters, replatform them to RKE2, remove the old RKE1 objects from Rancher, then upgrade. If you cannot finish migration in time, stay on a Rancher version that still supports RKE1 and treat that as temporary.

Take quiz
What does the pre-upgrade check do when RKE1 resources exist?
Silently converts them to RKE2
Skips them and continues
Deletes them from the local cluster
Fails the upgrade and lists the RKE1 resources
What should you do before upgrading to Rancher 2.12 or later?
Turn off the cluster agent
Replatform RKE1 clusters to RKE2
Downgrade cert-manager
Install Docker on every node again

38. How do you migrate RKE1 clusters to RKE2?

There is no in-place upgrade from RKE1 to RKE2. You build a new RKE2 cluster, move workloads, and retire the old one.

  1. Inventory what runs on the RKE1 cluster: manifests, Helm releases, PVCs, secrets, ingress, and custom CNI settings.
  2. Create the RKE2 cluster in Rancher with machine pools or a custom cluster.
  3. Redeploy applications through GitOps or Helm. Move data with Velero or storage snapshots.
  4. Test, then shift traffic with DNS or the load balancer.
  5. Remove the RKE1 cluster and its Rancher objects.

Expect differences. Docker becomes containerd, so anything depending on the Docker socket breaks. Pod Security Policies are gone, and ingress controller defaults and storage classes may differ. Run both clusters side by side until you have verified the new one.

Take quiz
Why can't an RKE1 cluster be upgraded in place to RKE2?
They use different runtimes and architecture, so you replatform instead
RKE2 only supports one node
RKE1 cannot use etcd
Rancher disables upgrades on Fridays
Which workload dependency typically breaks when moving from RKE1 to RKE2?
Deployments with probes
Pods using Services
Anything that mounts the Docker socket
Pods using ConfigMaps

39. How do you take and restore etcd snapshots for an RKE2 cluster in Rancher?

RKE2 snapshots etcd automatically, by default every 12 hours with the last five kept, stored locally under /var/lib/rancher/rke2/server/db/snapshots. In Rancher you can change the schedule and retention and send snapshots to S3 in the cluster's etcd settings.

To snapshot on demand, open the cluster's Snapshots tab and take one before risky changes. To restore, select the snapshot and choose what to restore: etcd only, etcd plus the Kubernetes version, or etcd plus the full cluster configuration.

# on a server node, outside Rancher
rke2 etcd-snapshot save --name pre-upgrade
rke2 server --cluster-reset \
  --cluster-reset-restore-path=/var/lib/rancher/rke2/server/db/snapshots/pre-upgrade

A restore replaces cluster state with the snapshot, so anything created since is lost. Store a copy off the node.

Take quiz
What are the default RKE2 etcd snapshot settings?
Only on demand, none kept
Every 12 hours, keeping the last five
Hourly, keeping 100
Daily, kept forever
What is the effect of restoring an etcd snapshot?
Only secrets are reverted
Container images are rolled back
Nodes are recreated from cloud images
The cluster state returns to the point of the snapshot

40. How do you harden a Rancher installation?

Hardening Rancher means protecting the local cluster, the entry point, and who can log in. Start with the SUSE Rancher hardening guide for your version and work through these areas.

  • Local cluster: dedicated RKE2 or K3s cluster, hardened to CIS, with workloads kept off it.
  • Entry point: trusted TLS certificate, restrict who can reach 443, and make sure the load balancer supports WebSockets.
  • Identity: connect an external provider with MFA at the IdP, trim local accounts, and apply least-privilege roles.
  • Tokens: shorten lifetime with auth-token-max-ttl-minutes and delete stale API keys.
  • Visibility: turn on API audit logging, for example --set auditLog.enabled=true in Helm.
  • Patching: apply patch releases quickly, since they fix privilege escalation and proxy issues.

Run the CIS scan regularly on every cluster, not only the local one.

Take quiz
Which Helm option enables Rancher API audit logging?
hostname.audit=on
rancher.audit=none
auditLog.enabled=true
tls.audit.mode=strict
Which setting limits how long API tokens remain valid?
auth-token-max-ttl-minutes
system-default-registry
server-url
cacerts

41. What is the Rancher CIS Benchmark scan and how does it work?

The CIS Benchmark feature (the rancher-cis-benchmark chart) checks a cluster's configuration against the CIS Kubernetes Benchmark and reports pass, fail, and warn results. It uses kube-bench underneath.

You install the chart from Apps, choose a ClusterScanProfile that matches your distribution and benchmark version (for example an RKE2 hardened or permissive profile, or ones for K3s, EKS, GKE, and AKS), and start a ClusterScan. Results appear as a ClusterScanReport that you can download.

Scans can be scheduled and can alert on failures. They only report. Fixing a failed check, such as an etcd file permission or API server flag, is still your job.

Take quiz
Which tool does the Rancher CIS Benchmark scan use underneath?
Falco
Nmap
Trivy
kube-bench
What does a ClusterScan do to failed checks?
Rolls back the Kubernetes version
Reports them without changing the cluster
Fixes them automatically
Deletes the failing pods

42. Explain the lifecycle of provisioning an RKE2 cluster in Rancher?

When you create an RKE2 cluster from a node driver, Rancher turns your form into Kubernetes objects and lets controllers do the rest. The steps below follow one machine pool.

flowchart TD
A["User creates cluster in UI or Terraform"] --> B["provisioning.cattle.io Cluster created"]
B --> C["Rancher creates Cluster API style Machine objects"]
C --> D["Provision job runs rancher-machine with cloud credential"]
D --> E["VM boots and rancher-system-agent installs"]
E --> F["Rancher writes plan secret for the machine"]
F --> G["system-agent installs and starts RKE2"]
G --> H["First control plane node initializes etcd"]
H --> I["Other nodes join using join URL and token"]
I --> J["cattle-cluster-agent deployed"]
J --> K["Cluster becomes Active"]
  1. Declare: the Cluster object lists machine pools, roles, and the Kubernetes version.
  2. Create machines: a provisioning job in fleet-default calls the node driver to create each VM.
  3. Register: the VM runs the system agent, which connects to Rancher.
  4. Plan: Rancher writes a plan secret per machine with the files and commands to run. The agent executes it and reports back.
  5. Join: the first control plane node starts etcd, and the rest join it. Workers join after the control plane is ready.
  6. Finish: the cluster agent and Fleet agent are deployed, and the cluster turns Active.

Because every step is an object with conditions, a stuck cluster shows exactly which step is waiting.

Take quiz
What tells the rancher-system-agent what to install and configure on a node?
A plan secret written by Rancher
A cloud-init script stored in Git
A DaemonSet YAML in kube-system
An environment variable on the load balancer
Which node starts etcd and initializes the cluster in a new RKE2 cluster?
Any worker node
The cloud provider's load balancer
The first control plane node
The Rancher local cluster node
Where does the provisioning job that creates machines run?
In the fleet-default namespace on the local cluster
On the user's laptop
On each downstream worker
In kube-system on the new cluster

43. Explain the execution flow of a Fleet deployment from Git to cluster?

Fleet separates what to deploy (the Git content) from where it goes (the target clusters), and moves it with pull-based agents.

sequenceDiagram
participant G as Git repo
participant J as Fleet git job
participant C as fleet-controller
participant A as fleet-agent downstream
G->>J: poll or webhook detects new commit
J->>C: fleet apply creates Bundle
C->>C: match targets, create BundleDeployment per cluster
A->>C: watch BundleDeployment
A->>A: render and install as Helm release
A-->>C: report status and drift
  1. The GitRepo is polled (15 seconds by default) or triggered by a webhook.
  2. A job clones the repo and runs fleet apply, which creates a Bundle per path.
  3. The controller evaluates targets and creates a BundleDeployment for each matching cluster, applying overrides.
  4. Each fleet-agent pulls its BundleDeployment, installs the content as a Helm release, and reports status.
  5. If someone changes a deployed object by hand, the agent flags drift and can correct it.

Agents connect outward to the management cluster, so downstream clusters do not need an inbound path.

Take quiz
How does the fleet-agent get its work?
It polls Docker Hub for new tags
It watches BundleDeployments from the management cluster
It receives a push over SSH from GitHub
It reads a ConfigMap on its own node
What does Fleet create after a Bundle's targets are matched?
One GitRepo per pod
One Namespace per commit
One Helm repo per node
One BundleDeployment per matching cluster

44. Explain the execution flow of a kubectl request through Rancher?

When you use a Rancher-generated kubeconfig, kubectl sends the request to the Rancher server rather than to the cluster, and Rancher relays it.

sequenceDiagram
participant U as kubectl
participant R as Rancher server
participant A as cattle-cluster-agent
participant K as kube-apiserver
U->>R: GET /k8s/clusters/c-abc123 with bearer token
R->>R: authenticate token and resolve user and groups
R->>A: forward through existing WebSocket tunnel
A->>K: same request with Impersonate-User and Impersonate-Group
K->>K: evaluate RBAC bindings
K-->>A: response
A-->>R: response
R-->>U: response
  1. The kubeconfig server URL looks like https://rancher.example.com/k8s/clusters/<cluster-id>.
  2. Rancher validates the token and works out the user and groups.
  3. The request travels down the agent's tunnel.
  4. The agent adds impersonation headers, so the API server evaluates the user's permissions, not the agent's.
  5. The response returns along the same path.

This is why Rancher RBAC works on clusters it did not build, and why the Authorized Cluster Endpoint exists as a shortcut around this hop.

Take quiz
Which headers let the downstream API server evaluate the real user's permissions?
X-Rancher-Admin and X-Cluster-Id
Cache-Control and ETag
Impersonate-User and Impersonate-Group
X-Forwarded-For and Host
What does the kubeconfig server URL point to when using the Rancher proxy?
The Rancher server under /k8s/clusters/<cluster-id>
The downstream node IP on port 22
The container registry
The etcd client port

45. How do you troubleshoot a downstream cluster stuck in Unavailable?

Unavailable means Rancher lost contact with the cluster. Work from the agent outward, because most causes are connectivity or certificate problems, not Kubernetes itself.

  1. Check the cluster's conditions in the UI or with kubectl get clusters.management.cattle.io on the local cluster.
  2. Use the direct kubeconfig on a downstream control plane node and confirm the API server is up: kubectl get nodes.
  3. Check the agent: kubectl -n cattle-system get pods and kubectl -n cattle-system logs -l app=cattle-cluster-agent.
  4. Test the path from a downstream node: curl -sk https://rancher.example.com/ping should return pong.
  5. Compare the agent's CA checksum with the server's if logs show TLS errors.
Symptom in agent logs Likely cause
Cannot resolve host DNS for server-url missing on downstream nodes
Timeout or connection refused Firewall or proxy blocking 443, or load balancer idle timeout
x509 or checksum mismatch Server certificate or CA changed
Pod Pending or OOMKilled Not enough node resources for the agent

Also check clock drift and expired RKE2 certificates on long-lived clusters.

Take quiz
What response should https:// /ping return when Rancher is healthy?
A JSON list of clusters
An empty body with status 204
200 OK with an HTML login page only
pong
Agent logs show 'no such host' for the Rancher URL. What is the most likely cause?
etcd needs defragmenting
DNS for the server-url does not resolve from downstream nodes
The cluster has too many namespaces
The Fleet repo is empty

46. How do you troubleshoot a cluster stuck waiting for nodes during provisioning?

Find out which stage failed: the machine was never created, the machine exists but the agent cannot register, or RKE2 starts but cannot form the cluster.

  1. List machines and plans on the local cluster, for example kubectl -n fleet-default get machines.cluster.x-k8s.io, and read the conditions on the cluster object.
  2. If no VM appeared, read the logs of the machine provisioning job pod in fleet-default. Bad cloud credentials, missing quotas, and wrong subnets show up here.
  3. If the VM exists, SSH in and run journalctl -u rancher-system-agent -f. A failure to reach Rancher on 443 or a bad server-url appears here.
  4. If the agent ran, check journalctl -u rke2-server -f for etcd, port, or join errors.
  5. For custom clusters, ensure the node is clean. Leftover /etc/rancher and /var/lib/rancher from an old cluster will cause joins to fail.

Also verify unique hostnames, synchronized time, and that required ports are open between nodes. Re-running the registration command on a half-configured node usually makes things worse, so clean it first.

Take quiz
Which log helps when the VM exists but never registers with Rancher?
rancher-system-agent journal on the node
kube-proxy log on the local cluster
Grafana dashboard for etcd
Harbor audit log
Why should you wipe /etc/rancher and /var/lib/rancher before reusing a node in a custom cluster?
It resets the cloud credential
Rancher refuses nodes with any files
Leftover state from an old cluster breaks joining
It speeds up image pulls

47. How do you troubleshoot cattle-cluster-agent certificate errors?

These errors mean the agent does not trust what Rancher presents. The usual trigger is a change in the Rancher certificate or CA after the agent was deployed, such as moving from the Rancher-generated CA to Let's Encrypt or a private CA.

Typical log messages are x509: certificate signed by unknown authority or a checksum error from the agent, since it stores a CA checksum in CATTLE_CA_CHECKSUM.

  1. Read the agent's value: kubectl -n cattle-system get deploy cattle-cluster-agent -o yaml | grep CATTLE_CA_CHECKSUM.
  2. Compute the server's: curl -sk https://rancher.example.com/v3/settings/cacerts | jq -r .value | sha256sum.
  3. If they differ, confirm the cacerts setting and the server-url match the certificate you actually serve.
  4. Redeploy the agent. Provisioned clusters get updated by Rancher. For imported clusters, re-apply the registration manifest.
  5. Check that intermediate certificates are in the served chain, since a missing one causes the same error.

Prevent repeat incidents by deciding the TLS source before registering clusters, and test a certificate change on a non-production Rancher first.

Take quiz
Which environment variable stores the CA checksum the agent expects?
RANCHER_TLS_PIN
CATTLE_CA_CHECKSUM
CATTLE_TOKEN_HASH
KUBE_CA_SHA
What commonly causes a checksum mismatch on a running agent?
The node has too many pods
The Helm client is outdated
The cluster uses containerd
The Rancher certificate or CA changed after the agent deployed

48. How does Rancher Turtles integrate Cluster API?

Rancher Turtles connects Cluster API (CAPI) to Rancher. CAPI lets you declare clusters as Kubernetes objects, and Turtles makes sure those clusters appear in Rancher without extra manual steps.

It installs CAPI core and infrastructure providers through the Cluster API Operator, using a CAPIProvider resource per provider. When a CAPI Cluster is created, Turtles imports it into Rancher automatically, so it gets the agent, RBAC, Fleet, and the rest of the Rancher feature set. The Prime edition also offers certified providers through the Apps catalog.

Older setups required disabling Rancher's embedded Cluster API feature before installing the Turtles chart, and packaging has changed across releases, so follow the guide for your Rancher version. Pair it with Fleet to keep CAPI cluster definitions in Git.

Take quiz
What does Rancher Turtles do automatically for a CAPI-created cluster?
Replaces its etcd
Deletes its nodes
Imports it into Rancher
Converts it to RKE1
Which resource defines a CAPI provider to be installed through Turtles?
CAPIProvider
ProviderBundle
ClusterDriver
MachineTemplatePool

49. How can you optimize Rancher server for many downstream clusters?

As the cluster count grows, load on the local cluster comes from agents holding connections, controllers watching resources, and everything stored in etcd. Tuning means giving it the right base and reducing avoidable churn.

  • Right-size the local cluster: three nodes sized to the documented tier for your cluster and node counts, and fast local SSD or NVMe for etcd.
  • Run three Rancher replicas behind a load balancer that supports WebSockets and has a long idle timeout.
  • Keep etcd lean: watch its database size, and clear expired tokens by setting a token TTL.
  • Move heavy kubectl traffic off the proxy with the Authorized Cluster Endpoint.
  • Be selective with Fleet: avoid enormous bundles and raise the polling interval or use webhooks if many repos poll often.
  • Keep data downstream: run Prometheus in each cluster rather than pulling metrics into Rancher.
  • Stay current: scale fixes arrive in patch releases.

Load test with a realistic number of agents before adding the next batch of clusters, and watch API server latency and etcd metrics on the local cluster.

Take quiz
Which load balancer feature is essential for Rancher with many agents?
UDP multicast routing
Disabling TLS
Layer 2 bridging
WebSocket support with a long idle timeout
How does the Authorized Cluster Endpoint reduce load on Rancher?
It compresses etcd
kubectl traffic bypasses the Rancher proxy
It stores the logs in Rancher
It disables the agent tunnel

50. How do you upgrade a Rancher-managed RKE2 cluster safely?

In Rancher, you edit the cluster and pick a newer Kubernetes version. Rancher then upgrades nodes through system-agent plans, control plane nodes first, then workers, and replaces components one batch at a time.

  1. Check the Rancher support matrix and the Kubernetes release notes for removed APIs. Scan your manifests for deprecated ones.
  2. Take an etcd snapshot, so you can restore the cluster configuration and version.
  3. Review PodDisruptionBudgets and replica counts, since drained nodes must be able to evict pods.
  4. Edit the cluster and set the target version. Upgrade by one minor version at a time.
  5. Tune control plane concurrency and worker concurrency (both default to 1), plus the drain options, such as drain timeout and ignoring DaemonSets.
  6. Watch node status and workload health, and verify add-ons like the CNI and ingress afterward.

If it goes wrong, use the snapshot's Kubernetes version and cluster config restore option. A slower concurrency setting costs time but limits risk, which is usually the right trade for production.

Take quiz
In which order does Rancher upgrade nodes of an RKE2 cluster?
Control plane nodes first, then workers
All workers first, then etcd
All nodes at once
Only nodes labelled upgrade=true
Which setting limits how many worker nodes are upgraded at the same time?
Server URL
Project quota
Worker concurrency
Snapshot retention
«
»

Comments & Discussions