Abstract
These are my notes from reading Kubernetes Up & Running by Kelsey Hightower, Brendan Burns, and Joe Beda. Kelsey Hightower is a Staff Developer Advocate for the Google Cloud Platform. Brendan Burns is a Distinguished Engineer in Microsoft Azure and cofounded the Kubernetes project at Google. Joe Beda is the CTO of Heptio and cofounded the Kubernetes project, as well as Google Compute Engine.
This is a phenomenal book that covers both the whys and hows of Kubernetes. I read the 1st edition, but a 2nd edition is coming out soon. I’m using this as study material for my CKAD and CKA certifications.
Bonus material: Kubernetes is commonly stylized as k8s. This is because the first letter of Kubernetes is k, the last letter is s, and there are 8 letters in-between the first and last.
Chapter 1: Introduction
Kubernetes delivers 4 main benefits: velocity, scaling (of both software and teams), abstraction of infrastructure, and efficiency.
Velocity is achieved using 3 core concepts: immutability, declarative configuration, and online self-healing systems. Immutability is the practice of replacing the current image with a brand new one, rather than updating it incrementally. The advantage of this approach is that there is a record of the delta between the images that can be used to troubleshoot any errors. Furthermore, you can rollback to the previous image if the new one doesn’t work - this is much more difficult if you are applying incremental changes to the same image. This ties directly into the concept of declarative configuration. Kubernetes itself manages changes to its state in a declarative fashion - each new desired state is declared concretely. Changes are not implied imperatively. Finally, Kubernetes manages this declared state continuously. If the actual state is different from the desired state, k8s will take action to create the desired state. All of these concepts add up to less time spent on operations and debugging, and more time developing new features.
Kubernetes achieves scaling for both applications and teams via decoupled architecture. By isolating all components of a distributed system via APIs and load balancers, each system may scale independently. APIs result in a crisp surface area between components. Teams can stay relatively small (and therefore agile) and have ownership of their microservice without having to coordinate with other teams - everyone can refer to the API schema. Additionally, clusters are able to add and remove machines very easily. This allows for low lead times on additional compute substrate, allowing applications to scale as needed, but it also means that a small operations team can support many clusters and many more development teams. This idea of fungible machines within a cluster ties directly into abstracted infrastructure.
Because k8s treats all machines as fungible, it allows for portability of Kubernetes APIs between IaaS providers (i.e., public cloud) and data centers. There are specific “gotchas” if you are using vendor managed services (e.g., AWS DynamoDB), but for the most part, Kubernetes is able to abstract components like load balancers and storage within different clouds. Bonus material: although not in this book, GCP Anthos takes this idea one step further. Anthos provides a single pane of glass to manage a Kubernetes cluster or clusters that can span multiple clouds as well as on-prem. This gives companies a highly abstracted viewpoint for application development and operations. It also gives the business a clear road map for cloud migration, multi-cloud HA/DR capabilities, and more.
Lastly, k8s achieves efficiency by colocating applications on the same machine without impacting the applications themselves. This means that more work can be done by fewer machines. This results in direct economic benefits, but also allows for new development methodologies. Rather than scaling testing environments at the VM level, a single cluster can support the entire testing environment. This makes it possible to test every single commit or pull request, every time, throughout your entire stack. This efficiency and thoroughness feeds directly back into developer productivity and velocity.
Chapter 2: Creating and Running Containers
The Docker image format for containers is the de facto standard. It is composed of a series of root filesystem layers, one on top of another. Each layer adds, removes, or modifies the preceding layer in the filesystem. In practice, it looks something like this:
- Container A: base operating system only
- Container B: build upon #A, by adding Ruby v2.1.10
- Container D: build upon #B by adding Rails v4.2.6
- Container C: build upon #A, by adding Golang v1.6
- Container B: build upon #A, by adding Ruby v2.1.10
The layering of these filesystems can result in extensive directed acyclic graphs that encompass a multitude of container images. It is worth noting that there is a counter-intuitive problem regarding the layering of filesystems. Deleting a file in a newer layer does not remove it from the preceding layers. This means that passwords/secrets should never be baked into images because they will still exist and be accessible to anyone with the right tools. Piggybacking on this idea, it’s important to order your layers according to which layers are most likely to change; these layers should be “at the top” or the newest of all layers. In the example above, if Container A were to change often, Containers B, C, and D would each need to be re-built to incorporate the changes. Beyond this, it is a best practice to keep your application containers as lean as possible. The smaller the container binary, the more efficiently they can be allocated on compute.
Docker provides the ability to limit the amount of resources used by a container by exposing the underlying cgroup technology provided by the Linux kernel. Flags like --memory, --memory-swap, and --cpu-shares accomplish this. Setting these restrictions is important to ensure that colocated applications have fair and predictable access to compute resources.
Whenever you build a new Docker image, it remains on your computer until explicitly deleted - even if you create a new image with the exact same tag/name. Use $ docker images to list what is currently on your computer and remove what you’re no longer using. Bonus material: The book lists a deprecated garbage collector as a tool to stay on top of this. You can use $ docker system prune instead. Set this up as a cron job to be extra clean (and fancy).
Chapter 3: Deploying a Kubernetes Cluster
At this point, the book describes multiple ways to get a Kubernetes cluster up and running (yuck yuck yuck) so that we can begin to interact with the k8s API. Most of these options are via public cloud. The book references services for both Azure and Amazon Web Services that have since been replaced by Kubernetes-as-a-Service offerings, similar to Google Kubernetes Engine (GKE). I’ve provided links to the new services. It is also worth noting that some of the commands listed in the book have changed as services have evolved since the time of publishing. Reference the documentation for your chosen service/tool to get your k8s cluster up and running.
kubectl is the official Kubernetes CLI tool and provides all of the functionality needed to interact with your clusters APIs. It is distinctly different from kubeadm. Using commands like $ kubectl describe nodes, you can see detailed information about all of the components of your cluster and what is running on them. The components that comprise Kubernetes are actually deployed using Kubernetes itself. These core components are located in the kube-system namespace, where a namespace is an entity for organizing and isolating Kubernetes resources (kind of like a folder in a filesystem.)
kube-proxy is responsible for routing network traffic to load-balanced services within the cluster. It has to be running on every node in order to function properly. Kubernetes also runs a DNS server, which provides naming and discovery for all of the services defined in the cluster. The DNS server will run several duplicates of itself depending on the size of the cluster. These replicas are managed by a kube-dns deployment and there is a separate kube-dns Service that handles load balancing for the DNS server.
Finally, there is a Kubernetes dashboard UI. Similar to kube-dns, there is a Deployment (this time only a single replica) to manage reliability for the dashboard as well as a Service to manage load balancing. Both run under the name kubernetes-dashboard. Bonus material: The dashboard is not deployed by default. You can deploy it using $ kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-beta4/aio/deploy/recommended.yaml. Most public cloud KaaS offerings will remove the need for the dashboard by offering similar information about your cluster through the web console.
Chapter 4: Common kubectl Commands
The following are kubectl commands that apply to all Kubernetes objects.
Namespaces
By default, kubectl interacts with objects in the default namespace. If you need access to a different namespace (like kube-system) you need to pass the --namespace flag.
Contexts
You can use contexts to rewrite default configurations. For example, you can the default namespace or even the k8s cluster that you are managing. Contexts are usually stored in a configuration file at $HOME/.kube/config. By defining and using different contexts, you can save yourself from having to explicitly type a lot of parameters and values.
Viewing Kubernetes API Objects
Everything within Kubernetes is defined by a RESTful schema. Each k8s object exists at a unique HTTP path (e.g., https://galens-cluster/api/v1/namespaces/default/pods/galens-pod) and kubectl works by sending HTTP requests to those endpoints. In RESTful fashion, the most basic command for viewing objects is get. You can use this to get all of a resource type or a specific object. You can add the -o wide flag to have the output include more information on longer lines, as well as -o json and -o yaml. A particularly useful flag is --no-headers. This skips the header at the beginning of a human-readable table and makes it very easy to pipe the output into other Unix commands. It is also very common to need to retrieve a specific field from the object. kubectl uses the JSONPath query language to select fields from the returned object. For example, $ kubectl get pods galens-pod -o jsonpath --template={.status.podIP} would return the IP address of the Pod. Finally, $ kubectl describe <resource-name> <obj-name> provides rich, multiline, human-readable information about an object, as well as any other related objects and events.
Creating, Updating, and Destroying Kubernetes Objects
Most of the time, k8s objects are created from a template file (YAML or JSON) using $ kubectl apply -f galens-object.yaml. Similarly, any changes to the object’s template are applied using the same command. You can use $ kubectl edit <resource-name> <obj-name> to modify objects inplace. You can use $ kubectl delete -f galens-object.yaml or $ kubectl delete <resource-name> <obj-name>, but Kubernetes will not ask for confirmation before deleting the resource - be careful!
Labeling and Annotating Objects
Labels and annotations are key-value pairs that can be used in various ways within Kubernetes. They can be applied using $ kubectl label and $ kubectl annotate respectively. Kubernetes will not allow you to overwrite an existing label by default. To do that, you need to use the --overwrite flag.
Debugging Commands You can get the logs from a pod by running $ kubectl logs <pod-name> and add the -c flag to specify a specific container if there are multiple in the pod. By default, logs will return the current logs and exit. You can use -f(follow) to continuously stream the container logs. You are also able to get an active shell within a container using $ kubectl exex -it <pod-name> -- bash. Lastly, use $ kubectl cp <pod-name>:/path/to/remote/file /path/to/local/file to copy files back and forth between your local machine and the container.
Help As always - --help is here to make everything possible!