Jekyll2019-03-05T18:52:32+00:00https://cloudnativeclojure.org/feed.xmlCloud Native ClojureWhere Clojure meets KubernetesKubernetes makes Feature Flags Incredibly Simple.2019-03-03T09:01:01+00:002019-03-03T09:01:01+00:00https://cloudnativeclojure.org/feature-flags/annotations/2019/03/03/feature-flags<h3 id="update">Update</h3>
<p><a href="https://www.reddit.com/r/Clojure/comments/axc5gu/easy_feature_flags_with_clojure_and_kubernetes/">Discussion on Reddit/Clojure</a></p>
<p>Also take a look at sunng’s library
<a href="https://github.com/sunng87/stavka">Stavka</a> which can read and watch
Kubernetes configMaps, and via the same mechanism, could trivially
watch a pods labels and annotations, as described in this article.</p>
<h3 id="introduction">Introduction</h3>
<p>Feature flags are used to change runtime behavior of a program without
restarting it. While they are essential in a cloud native environment,
the must be used judiciously. In the past, they could be fairly
tricky to implement across an organization’s microservices, but
Kubernetes has made them trivial to implement. Here we’re going to
implement them via labels and annotations, but you can also implement
them by connecting to the Kubernetes API directly.</p>
<p>In Kubernetes, labels are part of a resources’ identity, and can be
used via selectors to include/exclude particular resources.
Annotations are similar, but do not participate in a resources’
identity, and cannot be used to select resources. Annotations are
frequently used to store data about a resource.</p>
<p>Labels are specified in our yaml at <em>spec.template.metadata.labels</em>,
and annotations right next door at
<em>spec.template.metadata.annotations</em>. They will be available to our
Clojure code at /etc/podinfo/labels and /etc/podinfo/annotations
respectively.</p>
<h3 id="sample-use-cases">Sample use cases</h3>
<ul>
<li>Turn on/off a repl in a specific instance.</li>
<li>Turn on/off profiling of a specific instance.</li>
<li>Change the logging level in production, to capture detailed logs during a specific event.</li>
<li>Change caching strategy at runtime.</li>
<li>Change timeouts in production.</li>
<li>Toggle spec verification.</li>
</ul>
<h2 id="wrangling-labels-and-annotations-from-the-shell">Wrangling labels and annotations from the shell.</h2>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Add a label</span>
<span class="nv">$ </span>kubectl label pod my-pod-name a-label<span class="o">=</span>foo
<span class="c"># Show labels</span>
<span class="nv">$ </span>kubectl get pods <span class="nt">--show-labels</span>
<span class="c"># If you only want to show specific labels, use -L=<label1>,<label2></span>
<span class="c"># Update a label</span>
<span class="nv">$ </span>kubectl label pod my-pod-name a-label<span class="o">=</span>bar <span class="nt">--override</span>
<span class="c"># Delete a label</span>
<span class="nv">$ </span>kubectl label pod my-pod-name a-label-
<span class="c"># Add an annotation</span>
<span class="nv">$ </span>kubectl annotatate pod my-pod-name an-annotation<span class="o">=</span>foo
<span class="c"># Show annotations</span>
<span class="nv">$ </span>kubectl describe pod my-pod-name
<span class="c"># Update an annotation</span>
<span class="nv">$ </span>kubectl annotation pod my-pod-name an-annotation<span class="o">=</span>foo <span class="nt">--override</span>
<span class="c"># Delete an annotation</span>
<span class="nv">$ </span>kubectl annotation pod my-pod-name an-annotation-
</code></pre></div></div>
<p>We’ll use the Kubernetes downward-api to expose labels and annotations
directly to our application. We’ll end up with two files (“labels”
and “annotations”) in <em>/etc/podinfo</em>.</p>
<p>First we add the downward api to <em>spec.volumes</em>. Note that
we’re adding both labels and annotations into the same volume. You can
also expose certain container fields as a file consisting of a single
value, <a href="https://kubernetes.io/docs/tasks/inject-data-application/downward-api-volume-expose-pod-information/#store-container-fields">see
here</a>
for more info.</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">volumes</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">podinfo</span>
<span class="na">downwardAPI</span><span class="pi">:</span>
<span class="na">items</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">path</span><span class="pi">:</span> <span class="s2">"</span><span class="s">labels"</span>
<span class="na">fieldRef</span><span class="pi">:</span>
<span class="na">fieldPath</span><span class="pi">:</span> <span class="s">metadata.labels</span>
<span class="pi">-</span> <span class="na">path</span><span class="pi">:</span> <span class="s2">"</span><span class="s">annotations"</span>
<span class="na">fieldRef</span><span class="pi">:</span>
<span class="na">fieldPath</span><span class="pi">:</span> <span class="s">metadata.annotations</span>
</code></pre></div></div>
<p>Now we’re going to specify where we mount it into the container.
We’ll add the following to <em>spec.template.volumeMounts</em>.</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">volumeMounts</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">podinfo</span>
<span class="na">mountPath</span><span class="pi">:</span> <span class="s">/etc/podinfo</span>
<span class="na">readOnly</span><span class="pi">:</span> <span class="no">false</span>
</code></pre></div></div>
<p>Our deps are simple, and are saved as deps.edn locally.</p>
<div class="language-clojure highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="n">deps</span><span class="w"> </span><span class="p">{</span><span class="n">juxt/dirwatch</span><span class="w"> </span><span class="p">{</span><span class="no">:mvn/version</span><span class="w"> </span><span class="s">"0.2.3"</span><span class="p">}}}</span><span class="w">
</span></code></pre></div></div>
<blockquote>
<p><strong>Note:</strong> For juxt/dirwatch, use version 0.2.3, as there is a bug in 0.2.4 that stops it from working.</p>
</blockquote>
<p>Here’s the script we’re going to run. (It’s saved locally as script.clj)</p>
<div class="language-clojure highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">(</span><span class="nf">require</span><span class="w"> </span><span class="o">'</span><span class="p">[</span><span class="n">juxt.dirwatch</span><span class="w"> </span><span class="no">:refer</span><span class="w"> </span><span class="p">(</span><span class="nf">watch-dir</span><span class="p">)])</span><span class="w">
</span><span class="p">(</span><span class="nf">require</span><span class="w"> </span><span class="o">'</span><span class="p">[</span><span class="n">clojure.java.io</span><span class="w"> </span><span class="no">:as</span><span class="w"> </span><span class="n">io</span><span class="p">])</span><span class="w">
</span><span class="p">(</span><span class="nf">require</span><span class="w"> </span><span class="o">'</span><span class="p">[</span><span class="n">clojure.edn</span><span class="w"> </span><span class="no">:as</span><span class="w"> </span><span class="n">edn</span><span class="p">])</span><span class="w">
</span><span class="p">(</span><span class="k">def</span><span class="w"> </span><span class="n">annotations</span><span class="w"> </span><span class="p">(</span><span class="nf">atom</span><span class="w"> </span><span class="p">{}))</span><span class="w">
</span><span class="c1">;; This is just so we can see changes reflected in the log.</span><span class="w">
</span><span class="p">(</span><span class="nf">add-watch</span><span class="w"> </span><span class="n">annotations</span><span class="w"> </span><span class="no">:change</span><span class="w">
</span><span class="p">(</span><span class="k">fn</span><span class="w"> </span><span class="p">[</span><span class="n">ctx</span><span class="w"> </span><span class="n">k</span><span class="w"> </span><span class="n">old-value</span><span class="w"> </span><span class="n">new-value</span><span class="p">]</span><span class="w">
</span><span class="p">(</span><span class="nb">when</span><span class="w"> </span><span class="p">(</span><span class="nb">not=</span><span class="w"> </span><span class="n">old-value</span><span class="w"> </span><span class="n">new-value</span><span class="p">)</span><span class="w">
</span><span class="p">(</span><span class="nb">println</span><span class="w"> </span><span class="n">new-value</span><span class="p">))))</span><span class="w">
</span><span class="p">(</span><span class="k">defn</span><span class="w"> </span><span class="n">load-props</span><span class="w">
</span><span class="s">"The files produced by the downward api are close enough to property
files that we'll use the built in property reader to parse them."</span><span class="w">
</span><span class="p">[</span><span class="n">file-handle</span><span class="p">]</span><span class="w">
</span><span class="p">(</span><span class="nb">with-open</span><span class="w"> </span><span class="p">[</span><span class="o">^</span><span class="n">java.io.Reader</span><span class="w"> </span><span class="n">reader</span><span class="w"> </span><span class="p">(</span><span class="nf">io/reader</span><span class="w"> </span><span class="n">file-handle</span><span class="p">)]</span><span class="w">
</span><span class="p">(</span><span class="k">let</span><span class="w"> </span><span class="p">[</span><span class="n">props</span><span class="w"> </span><span class="p">(</span><span class="nf">java.util.Properties.</span><span class="p">)]</span><span class="w">
</span><span class="p">(</span><span class="nf">.load</span><span class="w"> </span><span class="n">props</span><span class="w"> </span><span class="n">reader</span><span class="p">)</span><span class="w">
</span><span class="p">(</span><span class="nf">->></span><span class="w"> </span><span class="p">(</span><span class="k">for</span><span class="w"> </span><span class="p">[[</span><span class="n">k</span><span class="w"> </span><span class="n">v</span><span class="p">]</span><span class="w"> </span><span class="n">props</span><span class="p">]</span><span class="w">
</span><span class="p">[(</span><span class="nb">keyword</span><span class="w"> </span><span class="n">k</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">edn/read-string</span><span class="w"> </span><span class="n">v</span><span class="p">)])</span><span class="w">
</span><span class="p">(</span><span class="nb">into</span><span class="w"> </span><span class="p">{}</span><span class="w"> </span><span class="p">)))))</span><span class="w">
</span><span class="p">(</span><span class="k">defn</span><span class="w"> </span><span class="n">downward-api-watcher</span><span class="w">
</span><span class="s">"When a file event comes in, reload the atom. Note that we don't use
the given file handle, as it will point to the temporary file."</span><span class="w">
</span><span class="p">[{</span><span class="no">:keys</span><span class="w"> </span><span class="p">[</span><span class="n">file,</span><span class="w"> </span><span class="n">count,</span><span class="w"> </span><span class="n">action</span><span class="p">]}]</span><span class="w">
</span><span class="p">(</span><span class="k">let</span><span class="w"> </span><span class="p">[</span><span class="n">fname</span><span class="w"> </span><span class="p">(</span><span class="nf">.getName</span><span class="w"> </span><span class="n">file</span><span class="p">)]</span><span class="w">
</span><span class="c1">;; Race condition?</span><span class="w">
</span><span class="p">(</span><span class="nb">when</span><span class="w"> </span><span class="p">(</span><span class="nb">=</span><span class="w"> </span><span class="n">fname</span><span class="w"> </span><span class="s">"annotations"</span><span class="p">)</span><span class="w">
</span><span class="p">(</span><span class="nf">reset!</span><span class="w"> </span><span class="n">annotations</span><span class="w"> </span><span class="p">(</span><span class="nf">load-props</span><span class="w"> </span><span class="p">(</span><span class="nf">io/file</span><span class="w"> </span><span class="s">"/etc/podinfo/annotations"</span><span class="p">))))))</span><span class="w">
</span><span class="p">(</span><span class="nf">watch-dir</span><span class="w"> </span><span class="n">downward-api-watcher</span><span class="w">
</span><span class="p">(</span><span class="nf">io/file</span><span class="w"> </span><span class="s">"/etc/podinfo/"</span><span class="p">))</span><span class="w">
</span></code></pre></div></div>
<p>We install it into our cluster via:</p>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>kubectl run <span class="nt">-it</span> <span class="nt">--restart</span><span class="o">=</span>Never <span class="se">\</span>
<span class="nt">--image</span><span class="o">=</span>clojure:openjdk-11-tools-deps <span class="se">\</span>
<span class="nt">--dry-run</span> <span class="se">\</span>
<span class="nt">-oyaml</span> <span class="se">\</span>
<span class="nt">--rm</span><span class="o">=</span><span class="nb">true</span> <span class="se">\</span>
<span class="nt">--command</span> <span class="nt">--</span> <span class="s2">"clojure"</span> <span class="s2">"-Sdeps"</span> <span class="s2">"</span><span class="k">$(</span><span class="nb">cat </span>deps.edn<span class="k">)</span><span class="s2">"</span> <span class="s2">"-e"</span> <span class="s2">"</span><span class="k">$(</span><span class="nb">cat </span>script.clj<span class="k">)</span><span class="s2">"</span>
</code></pre></div></div>
<div class="language-clojure highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">;; Example events</span><span class="w">
</span><span class="p">{</span><span class="no">:file</span><span class="w"> </span><span class="o">#</span><span class="n">object</span><span class="p">[</span><span class="n">java.io.File</span><span class="w"> </span><span class="mi">0</span><span class="n">x6c025238</span><span class="w"> </span><span class="n">/etc/podinfo/..data/labels</span><span class="p">]</span><span class="n">,</span><span class="w"> </span><span class="no">:count</span><span class="w"> </span><span class="mi">1</span><span class="n">,</span><span class="w"> </span><span class="no">:action</span><span class="w"> </span><span class="no">:create</span><span class="p">}</span><span class="w">
</span><span class="p">{</span><span class="no">:file</span><span class="w"> </span><span class="o">#</span><span class="n">object</span><span class="p">[</span><span class="n">java.io.File</span><span class="w"> </span><span class="mi">0</span><span class="n">x26b1ce79</span><span class="w"> </span><span class="n">/etc/podinfo/..data/annotations</span><span class="p">]</span><span class="n">,</span><span class="w"> </span><span class="no">:count</span><span class="w"> </span><span class="mi">1</span><span class="n">,</span><span class="w"> </span><span class="no">:action</span><span class="w"> </span><span class="no">:create</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<p>You can test it as following.</p>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>kubectl apply <span class="nt">-f</span> feature-flag.yaml
deployment.extensions <span class="s2">"feature-flag"</span> created
<span class="nv">$ </span>kubectl get pods
NAME READY STATUS RESTARTS AGE
feature-flag-78db4f4694-cxmrp 1/1 Running 0 6s
<span class="nv">$ </span>kubectl annotate pod feature-flag-78db4f4694-cxmrp <span class="nv">foo</span><span class="o">=</span>bar
pod <span class="s2">"feature-flag-78db4f4694-cxmrp"</span> annotated
</code></pre></div></div>
<p>Here is a yaml listing, with some cut down Clojure code, if you want
to test it simply as a single resource.</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">apiVersion</span><span class="pi">:</span> <span class="s">extensions/v1beta1</span>
<span class="na">kind</span><span class="pi">:</span> <span class="s">Deployment</span>
<span class="na">metadata</span><span class="pi">:</span>
<span class="na">creationTimestamp</span><span class="pi">:</span> <span class="no">null</span>
<span class="na">labels</span><span class="pi">:</span>
<span class="na">run</span><span class="pi">:</span> <span class="s">feature-flag</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">feature-flag</span>
<span class="na">spec</span><span class="pi">:</span>
<span class="na">replicas</span><span class="pi">:</span> <span class="s">1</span>
<span class="na">selector</span><span class="pi">:</span>
<span class="na">matchLabels</span><span class="pi">:</span>
<span class="na">run</span><span class="pi">:</span> <span class="s">feature-flag</span>
<span class="na">strategy</span><span class="pi">:</span> <span class="pi">{}</span>
<span class="na">template</span><span class="pi">:</span>
<span class="na">metadata</span><span class="pi">:</span>
<span class="na">creationTimestamp</span><span class="pi">:</span> <span class="no">null</span>
<span class="na">labels</span><span class="pi">:</span>
<span class="na">run</span><span class="pi">:</span> <span class="s">feature-flag</span>
<span class="na">annotations</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">cloudnativeclojure.org/my_cool_feature="off"</span>
<span class="na">spec</span><span class="pi">:</span>
<span class="na">containers</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">command</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">clojure</span>
<span class="pi">-</span> <span class="s">-Sdeps</span>
<span class="pi">-</span> <span class="s1">'</span><span class="s">{deps</span><span class="nv"> </span><span class="s">{juxt/dirwatch</span><span class="nv"> </span><span class="s">{:mvn/version</span><span class="nv"> </span><span class="s">"0.2.3"}}}'</span>
<span class="pi">-</span> <span class="s">-e</span>
<span class="pi">-</span> <span class="pi">|-</span>
<span class="no">(require '[juxt.dirwatch :refer (watch-dir)])</span>
<span class="no">(watch-dir println (clojure.java.io/file "/etc/podinfo/"))</span>
<span class="no">image: clojure:openjdk-11-tools-deps</span>
<span class="no">name: feature-flag</span>
<span class="no">resources: {}</span>
<span class="no">volumeMounts:</span>
<span class="no">- name: podinfo</span>
<span class="no">mountPath: /etc/podinfo</span>
<span class="no">readOnly: false</span>
<span class="na">volumes</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">podinfo</span>
<span class="na">downwardAPI</span><span class="pi">:</span>
<span class="na">items</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">path</span><span class="pi">:</span> <span class="s2">"</span><span class="s">labels"</span>
<span class="na">fieldRef</span><span class="pi">:</span>
<span class="na">fieldPath</span><span class="pi">:</span> <span class="s">metadata.labels</span>
<span class="pi">-</span> <span class="na">path</span><span class="pi">:</span> <span class="s2">"</span><span class="s">annotations"</span>
<span class="na">fieldRef</span><span class="pi">:</span>
<span class="na">fieldPath</span><span class="pi">:</span> <span class="s">metadata.annotations</span>
<span class="na">status</span><span class="pi">:</span> <span class="pi">{}</span>
</code></pre></div></div>
<p>Thanks to Jonathan Morris, Mia Tyzzer, and Arthur Ulfeldt for reading early drafts of this.</p>UpdateOriginal Clojure on Kubernetes Quickstart2019-01-30T16:56:34+00:002019-01-30T16:56:34+00:00https://cloudnativeclojure.org/getting-started/2019/01/30/original-clj-on-k8s-tutorial<p>My <a href="https://github.com/jwhitlark/clj-on-k8s-quickstart">original tutorial</a>, outdated, but still useful.</p>My original tutorial, outdated, but still useful.