Dealing with Rancher Snapshots

A bare kubernetes cluster managed with rancher is hosted and provided by our computing center (GWDG). They offer great support when we have to scale the nodes, but after all some parts of the maintenance is our business. Setting the kubernetes version from a provided list or managing ingress controller or certbot are those tasks. Before applying any changes to the cluster, we create a snapshot of all kubernetes resources. This is quite easy with the Rancher UI. Afterwards the UI allows for either restoring or deleting a snapshot. To get a single resource out of the snapshot is undocumented so far, so here we go:

tl;dr

Get access to an etcd node, find the zipped snapshot file and copy with kubectl cp to your local machine. Extract the binary data you find in the zip container with etcd-provided tool called auger.

Prerequisits

Of course you need administrative access to a kubernetes cluster utilizing etcd. This tutorial assumes a local machine running some Unix-like system.

Getting the snapshot

I would love to have a download button in the web interface, but unfortunately we have to use the universal pliers and interact directly on an etcd node. We can check for their names using

kubectl get nodes

NAME          STATUS   ROLES               AGE    VERSION
sub-dev-em4   Ready    controlplane,etcd   242d   v1.26.15
sub-dev-em5   Ready    controlplane,etcd   242d   v1.26.15
sub-dev-em6   Ready    controlplane,etcd   241d   v1.26.15
sub-dev-w1    Ready    worker              238d   v1.26.15
[…]

We are using sub-dev-em4 here since it is hosting etcd, but one of the other etcd nodes should also contain the snapshots. With a debug container on installed on this host, we can search for and copy the snapshot file.

kubectl debug node/sub-dev-em4 -it --image=fedora

This will create a debug containe on the node and provide an interactive terminal. So the following command is executed within this container.

ls -alh /host/opt/rke/etcd-snapshots

This will return the list of available snapshots with timestamps, so we can identify the one created last (or before the maintenance operation). To copy files from running containers kubectl provides the cp command in a way that is commonly known. Utilizing this we can copy over the remote snapshot.

kubectl cp node-debugger-sub-dev-em4-482bl:/host/opt/rke/etcd-snapshots/c-vvhxx-ml-tq6jh_2024-11-06T14\:21\:09Z.zip /tmp/rancher-snapshot.zip &&
cd $(mktemp -d) &&
unzip /tmp/rancher-snapshot.zip &&
ls -alh backup/
export ETCD_SNAPSHOT_PATH=$PWD/$(ls backup/)

At this point, we should remind ourselves that a kubernetes resource might contain sensitive information. Please ensure that you delete the snapshot from your local machine as soon you got the required information.

Inspect the Snapshot with Auger

The etcd team provides a tool for extracting information from the binary file listed with the last command. In ancient times they stored plain text files, but migrated to binary for performance reasons years ago. That is why we need a special tool for reading the data: auger. We should follow the installation instructions from their README.md.

Finally, auger can read all the data or a specific resource:

build/auger analyze -f $ETCD_SNAPSHOT_PATH
build/auger extract -f $ETCD_SNAPSHOT_PATH -k /registry/configmaps/myapp/myconfigmap -o yaml

Make auger

On my machine, I had to adjust the Makefile for creating the executable binary in the following way.

diff --git a/Makefile b/Makefile
index 1ba26e4..dbd7900 100644
--- a/Makefile
+++ b/Makefile
@@ -64,7 +64,7 @@ release:
 # Build used inside docker by 'release'
 release-docker-build:
        export GOPATH=/go
-       GOOS=$(GOOS) GOARCH=$(GOARCH) CGO_ENABLED=$(CGO_ENABLED) GO111MODULE=on go build
+       GOOS=$(GOOS) GOARCH=$(GOARCH) CGO_ENABLED=$(CGO_ENABLED) GO111MODULE=on go build -buildvcs=false