Add registry action to the kubernetes-worker layer

cluster/juju/layers/kubernetes-worker/README.md

@@ -41,6 +41,27 @@ a unit for maintenance.
 
 Resuming the workload will [uncordon](http://kubernetes.io/docs/user-guide/kubectl/kubectl_uncordon/) a paused unit. Workloads will automatically migrate unless otherwise directed via their application declaration.
 
+## Private registry
+
+With the "registry" action that is part for the kubernetes-worker charm, you can very easily create a private docker registry, with authentication, and available over TLS. Please note that the registry deployed with the action is not HA, and uses storage tied to the kubernetes node where the pod is running. So if the registry pod changes is migrated from one node to another for whatever reason, you will need to re-publish the images.
+
+### Example usage
+
+Create the relevant authentication files. Let's say you want user `userA` to authenticate with the password `passwordA`. Then you'll do :
+
+    echo "userA:passwordA" > htpasswd-plain
+    htpasswd -c -b -B htpasswd userA passwordA
+
+(the `htpasswd` program comes with the `apache2-utils` package)
+
+Supposing your registry will be reachable at `myregistry.company.com`, and that you already have your TLS key in the `registry.key` file, and your TLS certificate (with `myregistry.company.com` as Common Name) in the `registry.crt` file, you would then run :
+
+    juju run-action kubernetes-worker/0 registry domain=myregistry.company.com htpasswd="$(base64 -w0 htpasswd)" htpasswd-plain="$(base64 -w0 htpasswd-plain)" tlscert="$(base64 -w0 registry.crt)" tlskey="$(base64 -w0 registry.key)" ingress=true
+
+If you then decide that you want do delete the registry, just run :
+
+    juju run-action kubernetes-worker/0 registry delete=true ingress=true
+
 ## Known Limitations
 
 Kubernetes workers currently only support 'phaux' HA scenarios. Even when configured with an HA cluster string, they will only ever contact the first unit in the cluster map. To enable a proper HA story, kubernetes-worker units are encouraged to proxy through a [kubeapi-load-balancer](https://jujucharms.com/kubeapi-load-balancer)
@@ -48,5 +69,4 @@ application. This enables a HA deployment without the need to
 re-render configuration and disrupt the worker services.
 
 External access to pods must be performed through a [Kubernetes
-Ingress Resource](http://kubernetes.io/docs/user-guide/ingress/). More
-information
+Ingress Resource](http://kubernetes.io/docs/user-guide/ingress/).

cluster/juju/layers/kubernetes-worker/actions.yaml

@@ -14,6 +14,32 @@ microbot:
       delete:
         type: boolean
         default: False
-        description: Removes a microbots deployment, service, and ingress if True.
+        description: Remove a microbots deployment, service, and ingress if True.
 upgrade:
     description: Upgrade the kubernetes snaps
+registry:
+     description: Create a private Docker registry
+     params:
+      htpasswd:
+        type: string
+        description: base64 encoded htpasswd file used for authentication.
+      htpasswd-plain:
+        type: string
+        description: base64 encoded plaintext version of the htpasswd file, needed by docker daemons to authenticate to the registry.
+      tlscert:
+        type: string
+        description: base64 encoded TLS certificate for the registry. Common Name must match the domain name of the registry.
+      tlskey:
+        type: string
+        description: base64 encoded TLS key for the registry.
+      domain:
+        type: string
+        description: The domain name for the registry. Must match the Common Name of the certificate.
+      ingress:
+        type: boolean
+        default: false
+        description: Create an Ingress resource for the registry (or delete resource object if "delete" is True)
+      delete:
+        type: boolean
+        default: false
+        description: Remove a registry replication controller, service, and ingress if True.

cluster/juju/layers/kubernetes-worker/actions/registry

+#!/usr/bin/python3
+#
+# For a usage examples, see README.md
+#
+# TODO
+#
+# - make the action idempotent (i.e. if you run it multiple times, the first
+# run will create/delete the registry, and the reset will be a no-op and won't
+# error out)
+#
+# - take only a plain authentication file, and create the encrypted version in
+# the action
+#
+# - validate the parameters (make sure tlscert is a certificate, that tlskey is a
+# proper key, etc)
+#
+# - when https://bugs.launchpad.net/juju/+bug/1661015 is fixed, handle the
+# base64 encoding the parameters in the action itself
+
+import os
+import sys
+
+from base64 import b64encode
+
+from charmhelpers.core.hookenv import action_get
+from charmhelpers.core.hookenv import action_set
+from charms.templating.jinja2 import render
+from subprocess import call
+
+os.environ['PATH'] += os.pathsep + os.path.join(os.sep, 'snap', 'bin')
+
+deletion = action_get('delete')
+
+context = {}
+
+# These config options must be defined in the case of a creation
+param_error = False
+for param in ('tlscert', 'tlskey', 'domain', 'htpasswd', 'htpasswd-plain'):
+    value = action_get(param)
+    if not value and not deletion:
+        key = "registry-create-parameter-{}".format(param)
+        error = "failure, parameter {} is required".format(param)
+        action_set({key: error})
+        param_error = True
+
+    context[param] = value
+
+# Create the dockercfg template variable
+dockercfg = '{"%s:443": {"auth": "%s", "email": "root@localhost"}}' % \
+            (context['domain'], context['htpasswd-plain'])
+context['dockercfg'] = b64encode(dockercfg.encode()).decode('ASCII')
+
+if param_error:
+    sys.exit(0)
+
+# This one is either true or false, no need to check if it has a "good" value.
+context['ingress'] = action_get('ingress')
+
+# Declare a kubectl template when invoking kubectl
+kubectl = ['kubectl', '--kubeconfig=/root/cdk/kubeconfig']
+
+# Remove deployment if requested
+if deletion:
+    resources = ['svc/kube-registry', 'rc/kube-registry-v0', 'secrets/registry-tls-data',
+                 'secrets/registry-auth-data', 'secrets/registry-access']
+
+    if action_get('ingress'):
+        resources.append('ing/registry-ing')
+
+    delete_command = kubectl + ['delete', '--ignore-not-found=true'] + resources
+    delete_response = call(delete_command)
+    if delete_response == 0:
+        action_set({'registry-delete': 'success'})
+    else:
+        action_set({'registry-delete': 'failure'})
+
+    sys.exit(0)
+
+# Creation request
+render('registry.yaml', '/root/cdk/addons/registry.yaml',
+       context)
+
+create_command = kubectl + ['create', '-f',
+                            '/root/cdk/addons/registry.yaml']
+
+create_response = call(create_command)
+
+if create_response == 0:
+    action_set({'registry-create': 'success'})
+
+    # Create a ConfigMap if it doesn't exist yet, else patch it.
+    # A ConfigMap is needed to change the default value for nginx' client_max_body_size.
+    # The default is 1MB, and this is the maximum size of images that can be
+    # pushed on the registry. 1MB images aren't useful, so we bump this value to 1024MB.
+    cm_name = 'nginx-load-balancer-conf'
+    check_cm_command = kubectl + ['get', 'cm', cm_name]
+    check_cm_response = call(check_cm_command)
+
+    if check_cm_response == 0:
+        # There is an existing ConfigMap, patch it
+        patch = '{"data":{"max-body-size":"1024m"}}'
+        patch_cm_command = kubectl + ['patch', 'cm', cm_name, '-p', patch]
+        patch_cm_response = call(patch_cm_command)
+
+        if patch_cm_response == 0:
+            action_set({'configmap-patch': 'success'})
+        else:
+            action_set({'configmap-patch': 'failure'})
+
+    else:
+        # No existing ConfigMap, create it
+        render('registry-configmap.yaml', '/root/cdk/addons/registry-configmap.yaml',
+               context)
+        create_cm_command = kubectl + ['create', '-f', '/root/cdk/addons/registry-configmap.yaml']
+        create_cm_response = call(create_cm_command)
+
+        if create_cm_response == 0:
+            action_set({'configmap-create': 'success'})
+        else:
+            action_set({'configmap-create': 'failure'})
+
+    # Patch the "default" serviceaccount with an imagePullSecret.
+    # This will allow the docker daemons to authenticate to our private
+    # registry automatically
+    patch = '{"imagePullSecrets":[{"name":"registry-access"}]}'
+    patch_sa_command = kubectl + ['patch', 'sa', 'default', '-p', patch]
+    patch_sa_response = call(patch_sa_command)
+
+    if patch_sa_response == 0:
+        action_set({'serviceaccount-patch': 'success'})
+    else:
+        action_set({'serviceaccount-patch': 'failure'})
+
+
+else:
+    action_set({'registry-create': 'failure'})

cluster/juju/layers/kubernetes-worker/reactive/kubernetes_worker.py

@@ -203,6 +203,11 @@ def install_cni_plugins():
         hookenv.log(install)
         check_call(install)
 
+    # Used by the "registry" action. The action is run on a single worker, but
+    # the registry pod can end up on any worker, so we need this directory on
+    # all the workers.
+    os.makedirs('/srv/registry', exist_ok=True)
+
     set_state('kubernetes-worker.cni-plugins.installed')
 
 

cluster/juju/layers/kubernetes-worker/registry-configmap.yaml

+apiVersion: v1
+data:
+  body-size: 1024m
+kind: ConfigMap
+metadata:
+  name: nginx-load-balancer-conf

cluster/juju/layers/kubernetes-worker/templates/ingress-replication-controller.yaml

 apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: nginx-load-balancer-conf
+---
+apiVersion: v1
 kind: ReplicationController
 metadata:
   name: nginx-ingress-controller
@@ -45,3 +50,4 @@ spec:
         args:
         - /nginx-ingress-controller
         - --default-backend-service=$(POD_NAMESPACE)/default-http-backend
+        - --nginx-configmap=$(POD_NAMESPACE)/nginx-load-balancer-conf

cluster/juju/layers/kubernetes-worker/templates/registry.yaml

+apiVersion: v1
+kind: Secret
+metadata:
+  name: registry-tls-data
+type: Opaque
+data:
+  tls.crt: {{ tlscert }}
+  tls.key: {{ tlskey }}
+---
+apiVersion: v1
+kind: Secret
+metadata:
+  name: registry-auth-data
+type: Opaque
+data:
+  htpasswd: {{ htpasswd }}
+---
+apiVersion: v1
+kind: ReplicationController
+metadata:
+  name: kube-registry-v0
+  labels:
+    k8s-app: kube-registry
+    version: v0
+    kubernetes.io/cluster-service: "true"
+spec:
+  replicas: 1
+  selector:
+    k8s-app: kube-registry
+    version: v0
+  template:
+    metadata:
+      labels:
+        k8s-app: kube-registry
+        version: v0
+        kubernetes.io/cluster-service: "true"
+    spec:
+      containers:
+      - name: registry
+        image: registry:2
+        resources:
+          # keep request = limit to keep this container in guaranteed class
+          limits:
+            cpu: 100m
+            memory: 100Mi
+          requests:
+            cpu: 100m
+            memory: 100Mi
+        env:
+        - name: REGISTRY_HTTP_ADDR
+          value: :5000
+        - name: REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY
+          value: /var/lib/registry
+        - name: REGISTRY_AUTH_HTPASSWD_REALM
+          value: basic_realm
+        - name: REGISTRY_AUTH_HTPASSWD_PATH
+          value: /auth/htpasswd
+        volumeMounts:
+        - name: image-store
+          mountPath: /var/lib/registry
+        - name: auth-dir
+          mountPath: /auth
+        ports:
+        - containerPort: 5000
+          name: registry
+          protocol: TCP
+      volumes:
+      - name: image-store
+        hostPath:
+          path: /srv/registry
+      - name: auth-dir
+        secret:
+          secretName: registry-auth-data
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: kube-registry
+  labels:
+    k8s-app: kube-registry
+    kubernetes.io/cluster-service: "true"
+    kubernetes.io/name: "KubeRegistry"
+spec:
+  selector:
+    k8s-app: kube-registry
+  type: LoadBalancer
+  ports:
+  - name: registry
+    port: 5000
+    protocol: TCP
+---
+apiVersion: v1
+kind: Secret
+metadata:
+  name: registry-access
+data:
+  .dockercfg: {{ dockercfg }}
+type: kubernetes.io/dockercfg
+{%- if ingress %}
+---
+apiVersion: extensions/v1beta1
+kind: Ingress
+metadata:
+  name: registry-ing
+spec:
+  tls:
+  - hosts:
+    - {{ domain }}
+    secretName: registry-tls-data
+  rules:
+  - host: {{ domain }}
+    http:
+      paths:
+      - backend:
+          serviceName: kube-registry
+          servicePort: 5000
+        path: /
+{% endif %}