Delete stale feature branches in a Kubernetes cluster

Published in

ITNEXT

7 min readJul 1, 2020

Feature branch (aka deploy preview, preview app) means that a pull request is deployed as a separate instance of your application. It allows one to prevent errors and bugs as other programmers or product managers can check a feature. This article presents an approach to delete feature branches' resources in a Kubernetes cluster after its pull request is already merged to production.

Feature Branch

One way to create a feature branch in a Kubernetes cluster is to use namespaces to separate production deployment from other deployments. Production configurations may look similar to:

apiVersion: v1
kind: Namespace
metadata:
name: medium-back-end
...apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: medium-back-end
spec:
  replicas: 3
...

Otherwise, feature branches always have a different namespace. Such as -pr- prefix or postfix in its name. One example is illustrated below:

kind: Namespace
apiVersion: v1
metadata:
  name: medium-back-end-pr-17
...kind: Deployment
apiVersion: apps/v1
metadata:
  namespace: medium-back-end-pr-17
spec:
  replicas: 1
...

So, I have created the Kubernetes operator (an application that has access to a cluster’s resources). You can check it out on Github. It deletes namespaces associated with stale feature branches. In Kubernetes, if a namespace is deleted, all other resources in this namespace are automatically deleted too.

$ kubectl get pods --all-namespaces | grep -e "-pr-"
NAMESPACE              ... AGE
medium-back-end-pr-264 ... 4d8h
medium-back-end-pr-265 ... 5d7h

You can find more information about the implementation of feature branches using namespaces here and here.

Motivation

To understand the motivation of the project, let’s check common continuous integration for a pull request and its lifecycle:

A new commit is pushed to a branch.
Code style and tests are passed.
A feature branch’s configurations are applied (for instance, a pull request’s identifier is added to a ready-to-use template).
Configurations are deployed to a cluster with kubectl apply .
The branch is merged into a production branch (for instance, master).

One important thing is that a good lifecycle will delete all existing feature branch resources for a particular commit before applying configurations to the new commit. It’s needed to ensure that each commit’s deployment is done from a clear state.

The point is: when a pull request is merged into a production branch its Kubernetes resources still live in a cluster. This happens because, after the merge, the only production branch will be built on continuous integration (that has no scenario for deletion of stale feature branches). Ok, how do we handle this?

Approaches

These are three ways to delete a feature branch’s resources after its branch is merged into a production branch. Neither this project nor these three methods are ideal. Each of you can choose whichever approach is most appropriate for your case.

On each production branch build, detect which branch was merged last and delete.

This can be done only by fetching commits history. In this case, a commit should contain an identifier of its pull request (a number or branch name).
Sometimes a production branch’s builds fail on the stage you do not want to rebuild. For instance, you have the following stages: download a project, run the project, run tests, make a release, send a notification, delete stale feature branches. If the build fails on the send a notification stage, you will unlikely be able to rebuild it easily.
Without any context, the stage to delete stale feature branches is not obvious for newcomers.

2. Integrate a webhook to your continuous integration system (example).

This may not fit your development principle. For instance, Jenkins supports only one type of pipeline where you can have your pipeline’s configuration file uploaded to a source code server. So, to create a webhook, you will need separate scripts that process the webhook’s data which will not be uploaded to the source code server and should be maintained in a user interface.

3. Create your own Cronjob resource in a Kubernetes cluster.

This also requires development and maintenance, especially when moving this from one company to another.
Furthermore, this project works with nearly the same principle as the Cronjob resource, so you lose nothing while reusing.

Installation

Apply the latest release configurations with the command below. This will create the StaleFeatureBranch resource, install the operator into stale-feature-branch-operator namespace, create a service account, and necessary RBAC roles.

$ kubectl apply -f \ https://raw.githubusercontent.com/dmytrostriletskyi/stale-feature-branch-operator/master/configs/production.yml

Usage

After applying the installation instructions above, create and apply a configuration file with feature-branch.dmytrostriletskyi.com/v1 as apiVersion and StaleFeatureBranch as kind:

apiVersion: feature-branch.dmytrostriletskyi.com/v1
kind: StaleFeatureBranch
metadata:
  name: stale-feature-branch
spec:
  namespaceSubstring: -pr-
  afterDaysWithoutDeploy: 3

Choose any metadata’s name for the resource and dive into specifications:

namespaceSubstring is needed to get all feature branches' namespaces. For instance, the example above will grab medium-back-end-pr-17 and medium-back-end-pr-33 if there are namespaces medium-back-end, medium-front-end, medium-back-end-pr-17, medium-back-end-pr-33 in a cluster as the -pr- substring occurs there.
afterDaysWithoutDeploy is needed to delete only old namespaces. If you set 3 days there, namespaces created 1 day or 2 days ago will not be deleted, but namespaces created 3 days, 1 hour or 4 days ago will be deleted.

It processes feature branches’ namespaces every 30 minutes by default. The last available parameter in specifications is checkEveryMinutes. You can configure a frequency of the processes in minutes if the default value doesn't fit your needs.

How It Works Under the Hood

This guideline shows how the deletion of stale feature branches works under the hood. You should not reproduce the instructions below for a production cluster as it’s just a detailed example to understand the behavior of the operator. For this chapter, you will run a testing Kubernetes cluster on your personal computer.

Requirements are:

Docker. Virtualization to run the software in packages called containers.
Minikube. Runs a single-node Kubernetes cluster in a virtual machine (or Docker) on your personal computer.
kubectl. Command-line interface to access Kubernetes cluster.

Start Kubernetes cluster on your personal computer with the following command:

$ minikube start --vm-driver=docker
minikube v1.11.0 on Darwin 10.15.5
Using the docker driver based on existing profile.
Starting control plane node minikube in cluster minikube.

After, choose the created cluster as the main one for kubectl. It's needed for cases in which you work with many clusters from a single computer:

$ kubectl config use-context minikube
Switched to context "minikube".

We will apply production configurations. But as these are production configurations, they will expect old namespaces present in your cluster. Our cluster is fresh, and no old resources are present there. As you do not have them, the operator allows you to specify the debug parameter. If the debug is enabled, all namespaces will be deleted without checking for an oldness:

Copy the production configurations to your personal computer:

$ curl https://raw.githubusercontent.com/dmytrostriletskyi/stale-feature-branch-operator/master/configs/production.yml > stale-feature-branch-production-configs.yml

Enable debug by changing the setting. For Linux it's:

$ sed -i 's|false|true|g' stale-feature-branch-production-configs.yml

For macOS it's:

$ sed -i "" 's|false|true|g' stale-feature-branch-production-configs.yml

Apply the changed production configurations:

$ kubectl apply -f stale-feature-branch-production-configs.yml

Fetch all resources (kinds we can use) in Kubernetes cluster. You will see StaleFeatureBranch resource available to be created:

$ kubectl api-resources | grep stalefeaturebranches                 NAME                 ... KIND
stalefeaturebranches ... StaleFeatureBranch

Fetch pods in stale-feature-branch-operator namespace. You will see an operator that listens for new StaleFeatureBranch resources running there:

$ kubectl get pods --namespace stale-feature-branch-operator
NAME                                           ... STATUS  ... AGE stale-feature-branch-operator-6bfbfd4df8-m7sch ... Running ... 38s

Fetch the operator’s logs to ensure it’s running:

$ kubectl logs stale-feature-branch-operator-6bfbfd4df8-m7sch -n stale-feature-branch-operator... "msg":"Operator Version: 0.0.1"}
... "msg":"Starting EventSource", ... , "source":"kind source: /"}
... "msg":"Starting Controller", ...}
... "msg":"Starting workers", ..., "worker count":1}

Install fixture (ready-to-use configurations to simulate Kubernetes cluster resources) for StaleFeatureBranch :

$ kubectl apply -f https://raw.githubusercontent.com/dmytrostriletskyi/stale-feature-branch-operator/master/fixtures/stale-feature-branch.yml

The fixture will check for namespaces that contain -pr-1 in their names once a minute:

apiVersion: feature-branch.dmytrostriletskyi.com/v1
kind: StaleFeatureBranch
metadata:
  name: stale-feature-branch
spec:
  namespaceSubstring: -pr-
  afterDaysWithoutDeploy: 1 
  checkEveryMinutes: 1

The operator accepted the created resource and is ready to delete namespaces that match with the configurations above:

$ kubectl logs stale-feature-branch-operator-6bfbfd4df8-m7sch -n stale-feature-branch-operator... "msg":"Stale feature branch is being processing.",
"namespaceSubstring":"-pr","afterDaysWithoutDeploy":1,
"checkEveryMinutes":1,"isDebug":"true"}

Create fixtures that contain two namespaces project-pr-1 and project-pr-2 with many other resources as well (deployment, service, secrets, etc.):

$ kubectl apply -f https://raw.githubusercontent.com/dmytrostriletskyi/stale-feature-branch-operator/master/fixtures/first-feature-branch.yml -f https://raw.githubusercontent.com/dmytrostriletskyi/stale-feature-branch-operator/master/fixtures/second-feature-branch.yml...
namespace/project-pr-1 created
deployment.apps/project-pr-1 created
service/project-pr-1 created horizontalpodautoscaler.autoscaling/project-pr-1 created secret/project-pr-1 created
configmap/project-pr-1 created
ingress.extensions/project-pr-1 created
namespace/project-pr-2 created
deployment.apps/project-pr-2
created service/project-pr-2 created horizontalpodautoscaler.autoscaling/project-pr-2 created secret/project-pr-2 created
configmap/project-pr-2 created
ingress.extensions/project-pr-2 created

You can check their existence with the following command:

$ kubectl get namespace,pods,deployment,service,horizontalpodautoscaler,configmap,ingress -n project-pr-1 && kubectl get namespace,pods,deployment,service,horizontalpodautoscaler,configmap,ingress -n project-pr-2...
NAME                              ... READY ... STATUS  ... AGE
pod/project-pr-1-848d5fdff6-rpmzw ... 1/1   ... Running ... 67sNAME                         ... READY ... AVAILABLE ... AGE
deployment.apps/project-pr-1 ... 1/1   ... 1         ... 67s
...

As mentioned above, when debug is enabled, all namespaces will be deleted without checking for oldness. Check the operator’s logs:

$ kubectl logs stale-feature-branch-operator-6bfbfd4df8-m7sch -n stale-feature-branch-operator... "msg":"Namespace should be deleted due to debug mode is enabled.","namespaceName":"project-pr-1"}
... "msg":"Namespace is being processing.","namespaceName":"project-pr-1"}
... "msg":"Namespace has been deleted.","namespaceName":"project-pr-1"}
... "msg":"Namespace should be deleted due to debug mode is enabled.","namespaceName":"project-pr-2"}
... "msg":"Namespace is being processing.","namespaceName":"project-pr-2"}
... "msg":"Namespace has been deleted.","namespaceName":"project-pr-2"}

If you check resources again, the output would be Terminating or empty;

$ kubectl get namespace,pods,deployment,service,horizontalpodautoscaler,configmap,ingress -n project-pr-1 && kubectl get namespace,pods,deployment,service,horizontalpodautoscaler,configmap,ingress -n project-pr-2
...

You can go through the process of the creation of resources again. In the end, in a minute or less, resources will once again be deleted.

Thanks for your attention! And again, the Kubernetes operator on Github. :)