Kubernetes Architecture — Control Plane, Nodes, and How kubectl apply Really Works

The Problem

People learn kubectl apply long before they understand what happens after they press Enter. So when a Pod stays Pending, or a node goes NotReady, they have no mental map to debug from. Kubernetes is not magic — it is a handful of components that watch a shared database and reconcile reality toward your desired state. Once you can name those components and trace a request through them, troubleshooting stops being guesswork.

A Kubernetes cluster is split into two planes: the control plane that makes global decisions, and the worker nodes that actually run your containers. Every component communicates through one hub — the API server — and the desired state lives in one place — etcd.

The Control Plane

The control plane is the brain. It decides what should run where and continuously works to make the cluster match your declared intent.

kube-apiserver

The API server is the only component that talks to etcd, and the only thing anything else talks to. kubectl, the scheduler, the kubelets, and the controllers all go through it. It handles authentication, authorization, admission control, and validation before any change is persisted. If the API server is down, the cluster keeps running existing Pods but you cannot make changes.

etcd

etcd is a distributed key-value store and the single source of truth for the entire cluster. Every object — Pods, Services, ConfigMaps, Secrets — is stored here. Lose etcd without a backup and you lose the cluster's state. Back it up.

kube-scheduler

The scheduler watches for Pods that have no node assigned and picks the best node for each one. It scores nodes based on resource requests, affinity and anti-affinity rules, taints and tolerations, and spreading constraints. It does not start the Pod — it only writes the chosen node name back onto the Pod object.

kube-controller-manager

This runs the control loops that reconcile desired versus actual state: the Deployment controller, ReplicaSet controller, Node controller, Job controller, and more. Each loop watches objects and nudges reality toward spec — for example, creating a replacement Pod when one dies.

cloud-controller-manager

On a managed or cloud cluster, this component talks to the cloud provider's API to provision load balancers, attach volumes, and manage node lifecycle. It keeps cloud-specific logic out of the core controllers.

The Node Components

Every worker node runs three things that turn scheduling decisions into running containers.

The kubelet is the workhorse: it reports node and Pod status back to the API server, runs the probes, and restarts containers according to the restart policy.

How a kubectl apply Flows Through the Cluster

This is the sequence worth memorizing. When you run:

kubectl apply -f deployment.yaml

kubectl → API server. kubectl sends the manifest to the API server over HTTPS. The server authenticates you, runs authorization (RBAC) and admission controllers, validates the object, and writes it to etcd. At this point the object exists but nothing is running yet.

Controller reconciles. The Deployment controller (in controller-manager) sees the new Deployment, creates a ReplicaSet, and the ReplicaSet controller creates the required number of Pod objects — still unscheduled.

Scheduler assigns a node. The scheduler notices Pods with no nodeName, scores the nodes, and writes the chosen node back onto each Pod via the API server.

kubelet starts the containers. The kubelet on the chosen node sees a Pod bound to it, tells the container runtime to pull images and start containers, then runs the probes and reports status.

kube-proxy wires networking. If a Service targets those Pods, kube-proxy updates routing rules so traffic reaches them.

Notice that no component commands another directly. Each one watches the API server and reacts. This watch-and-reconcile pattern is the core design idea of Kubernetes.

Verifying the Architecture Yourself

You can see most of these components from the command line:

# Control plane pods (on kubeadm-style clusters)
kubectl get pods -n kube-system

# Node agents and their status
kubectl get nodes -o wide

# Watch the scheduling + startup flow live
kubectl get events --sort-by=.lastTimestamp

# Inspect what the scheduler decided for a pod
kubectl describe pod <pod-name>

Common Mistakes to Avoid

• Not backing up etcd. It holds the entire cluster state. No etcd backup means no disaster recovery.
• Assuming the scheduler starts Pods. It only assigns a node. The kubelet starts containers. A Pending Pod is usually a scheduling problem (no node fits), not a kubelet problem.
• Treating a NotReady node as an app bug. That is a node/kubelet issue — check the kubelet, runtime, and network on that node first.
• Running workloads on the control plane in production. Keep the control plane isolated so a noisy app can't starve the API server or etcd.
• Forgetting that everything goes through the API server. If the API server is overloaded, every component slows down. Watch its latency and request rates.

FAQ

What is the difference between the control plane and a worker node?

The control plane makes global decisions — scheduling, reconciliation, and storing cluster state in etcd — while worker nodes run your actual application containers via the kubelet and container runtime. In managed services like EKS or GKE, the cloud provider operates the control plane for you and you only manage the nodes.

Does the scheduler run my Pods?

No. The scheduler only decides which node a Pod should run on and records that choice. The kubelet on the chosen node is what actually tells the container runtime to pull images and start containers.

What happens if etcd is lost?

If etcd is lost and you have no backup, the cluster's entire declared state is gone — all object definitions disappear. Running containers may continue for a while, but you cannot recover or manage the cluster. This is why regular, tested etcd backups are non-negotiable in production.

Why does everything go through the API server?

Centralizing all communication through the API server gives Kubernetes one place to enforce authentication, authorization, validation, and admission control, and one consistent view of state. Components stay decoupled because they watch the API server rather than calling each other directly.