Kubernetes makes Feature Flags Incredibly Simple.
Update
Also take a look at sunng’s library Stavka which can read and watch Kubernetes configMaps, and via the same mechanism, could trivially watch a pods labels and annotations, as described in this article.
Introduction
Feature flags are used to change runtime behavior of a program without restarting it. While they are essential in a cloud native environment, the must be used judiciously. In the past, they could be fairly tricky to implement across an organization’s microservices, but Kubernetes has made them trivial to implement. Here we’re going to implement them via labels and annotations, but you can also implement them by connecting to the Kubernetes API directly.
In Kubernetes, labels are part of a resources’ identity, and can be used via selectors to include/exclude particular resources. Annotations are similar, but do not participate in a resources’ identity, and cannot be used to select resources. Annotations are frequently used to store data about a resource.
Labels are specified in our yaml at spec.template.metadata.labels, and annotations right next door at spec.template.metadata.annotations. They will be available to our Clojure code at /etc/podinfo/labels and /etc/podinfo/annotations respectively.
Sample use cases
- Turn on/off a repl in a specific instance.
- Turn on/off profiling of a specific instance.
- Change the logging level in production, to capture detailed logs during a specific event.
- Change caching strategy at runtime.
- Change timeouts in production.
- Toggle spec verification.
Wrangling labels and annotations from the shell.
# Add a label
$ kubectl label pod my-pod-name a-label=foo
# Show labels
$ kubectl get pods --show-labels
# If you only want to show specific labels, use -L=<label1>,<label2>
# Update a label
$ kubectl label pod my-pod-name a-label=bar --override
# Delete a label
$ kubectl label pod my-pod-name a-label-
# Add an annotation
$ kubectl annotatate pod my-pod-name an-annotation=foo
# Show annotations
$ kubectl describe pod my-pod-name
# Update an annotation
$ kubectl annotation pod my-pod-name an-annotation=foo --override
# Delete an annotation
$ kubectl annotation pod my-pod-name an-annotation-
We’ll use the Kubernetes downward-api to expose labels and annotations directly to our application. We’ll end up with two files (“labels” and “annotations”) in /etc/podinfo.
First we add the downward api to spec.volumes. Note that we’re adding both labels and annotations into the same volume. You can also expose certain container fields as a file consisting of a single value, see here for more info.
volumes:
- name: podinfo
downwardAPI:
items:
- path: "labels"
fieldRef:
fieldPath: metadata.labels
- path: "annotations"
fieldRef:
fieldPath: metadata.annotations
Now we’re going to specify where we mount it into the container. We’ll add the following to spec.template.volumeMounts.
volumeMounts:
- name: podinfo
mountPath: /etc/podinfo
readOnly: false
Our deps are simple, and are saved as deps.edn locally.
{deps {juxt/dirwatch {:mvn/version "0.2.3"}}}
Note: For juxt/dirwatch, use version 0.2.3, as there is a bug in 0.2.4 that stops it from working.
Here’s the script we’re going to run. (It’s saved locally as script.clj)
(require '[juxt.dirwatch :refer (watch-dir)])
(require '[clojure.java.io :as io])
(require '[clojure.edn :as edn])
(def annotations (atom {}))
;; This is just so we can see changes reflected in the log.
(add-watch annotations :change
(fn [ctx k old-value new-value]
(when (not= old-value new-value)
(println new-value))))
(defn load-props
"The files produced by the downward api are close enough to property
files that we'll use the built in property reader to parse them."
[file-handle]
(with-open [^java.io.Reader reader (io/reader file-handle)]
(let [props (java.util.Properties.)]
(.load props reader)
(->> (for [[k v] props]
[(keyword k) (edn/read-string v)])
(into {} )))))
(defn downward-api-watcher
"When a file event comes in, reload the atom. Note that we don't use
the given file handle, as it will point to the temporary file."
[{:keys [file, count, action]}]
(let [fname (.getName file)]
;; Race condition?
(when (= fname "annotations")
(reset! annotations (load-props (io/file "/etc/podinfo/annotations"))))))
(watch-dir downward-api-watcher
(io/file "/etc/podinfo/"))
We install it into our cluster via:
kubectl run -it --restart=Never \
--image=clojure:openjdk-11-tools-deps \
--dry-run \
-oyaml \
--rm=true \
--command -- "clojure" "-Sdeps" "$(cat deps.edn)" "-e" "$(cat script.clj)"
;; Example events
{:file #object[java.io.File 0x6c025238 /etc/podinfo/..data/labels], :count 1, :action :create}
{:file #object[java.io.File 0x26b1ce79 /etc/podinfo/..data/annotations], :count 1, :action :create}
You can test it as following.
$ kubectl apply -f feature-flag.yaml
deployment.extensions "feature-flag" created
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
feature-flag-78db4f4694-cxmrp 1/1 Running 0 6s
$ kubectl annotate pod feature-flag-78db4f4694-cxmrp foo=bar
pod "feature-flag-78db4f4694-cxmrp" annotated
Here is a yaml listing, with some cut down Clojure code, if you want to test it simply as a single resource.
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
creationTimestamp: null
labels:
run: feature-flag
name: feature-flag
spec:
replicas: 1
selector:
matchLabels:
run: feature-flag
strategy: {}
template:
metadata:
creationTimestamp: null
labels:
run: feature-flag
annotations:
- cloudnativeclojure.org/my_cool_feature="off"
spec:
containers:
- command:
- clojure
- -Sdeps
- '{deps {juxt/dirwatch {:mvn/version "0.2.3"}}}'
- -e
- |-
(require '[juxt.dirwatch :refer (watch-dir)])
(watch-dir println (clojure.java.io/file "/etc/podinfo/"))
image: clojure:openjdk-11-tools-deps
name: feature-flag
resources: {}
volumeMounts:
- name: podinfo
mountPath: /etc/podinfo
readOnly: false
volumes:
- name: podinfo
downwardAPI:
items:
- path: "labels"
fieldRef:
fieldPath: metadata.labels
- path: "annotations"
fieldRef:
fieldPath: metadata.annotations
status: {}
Thanks to Jonathan Morris, Mia Tyzzer, and Arthur Ulfeldt for reading early drafts of this.