How to make a Kubernetes app from an OpenStack service? Tale of kuryr-kubernetes' "kubernetization"

---

---

---

---

---

# How to make a Kubernetes app from an OpenStack service?

## Tale of kuryr-kubernetes' "kubernetization"

Michał Dulko - Software Engineer - Red Hat 
Daniel Mellado - Senior Software Engineer - Red Hat 
 
OpenStack Summit Berlin, 15.11.2018

---

# Agenda
--

1. What's Kuryr-Kubernetes?
   * What's CNI?
   * What services it's consisting of?
--

2. Kuryr-Kubernetes requirements
   * Deployment and leader election
   * RBAC
   * Configuration
   * Host access
   * Binary injection

Slides available at: **https://dulek.github.io/kuryr-kubernetization**

---

What's Kuryr-Kubernetes?

---
# Kuryr-Kubernetes

## Logos or mascots

#.left-column[![Insert graphic here](assets/kuryr-logo.png)]

#.right-column[.center[.half[![Insert graphic here](assets/kuryr-logo2.png)]]]

---

# Kuryr-Kubernetes

## Primer

* Kuryr-Kubernetes is part of OpenStack Kuryr project.
--

* Repo: https://git.openstack.org/openstack/kuryr-kubernetes
* Written in Python, *kuryr-kubernetes* on Python Package Index.
--

* Developers from Red Hat, Huawei, Intel, Samsung…
--

* Out of experimental phase in **OpenStack Queens release** with version **0.4.3**.
* Follows **cycle-with-intermediary** release model.

## Goals

* Provide a Container Network Interface plugin that will use OpenStack Neutron
  for Kubernetes Pods networking.
* Provide kube-proxy equivalent that will use OpenStack Octavia for Kubernetes
  Services loadbalancing and Ingress implementation.

---

# Kubernetes and CNI

## Container Network Interface

* Project governed by CNCF (*Cloud Native Computing Foundation*).
* API is based on net.d configuration, executables and environment variables:
  * CNI plugins are configured in net.d directory (default `/etc/cni/net.d`)
    by config files. CNI starts looking from first file lexicographically, e.g.
    `10-kuryr.conf`.
  * CNI plugins are executables that should be run with parameters in env vars.
     - `CNI_COMMAND` - either `ADD`, `DEL` or `VERSION`.
     - `CNI_CONTAINERID` - unique container ID, key when storing state.
     - `CNI_NETNS` - network namespace that interface should be placed in.
     - `CNI_IFNAME` - requested name of the interface in the `CNI_NETNS`.

---

# Kubernetes and CNI

## Container Network Interface

* Project governed by CNCF (*Cloud Native Computing Foundation*).
* API is based on net.d configuration, executables and environment variables:
  * CNI plugins are configured in net.d directory (default `/etc/cni/net.d`)
    by config files. CNI starts looking from first file lexicographically, e.g.
    `10-kuryr.conf`.
  * **CNI plugins are executables that should be run with parameters in env vars.**
     - `CNI_COMMAND` - either `ADD`, `DEL` or `VERSION`.
     - `CNI_CONTAINERID` - unique container ID, key when storing state.
     - `CNI_NETNS` - network namespace that interface should be placed in.
     - `CNI_IFNAME` - requested name of the interface in the `CNI_NETNS`.

---

# Kuryr-Kubernetes

## Motivations

--
* Harness the power of OpenStack Neutron for your Kubernetes cluster!

# Kuryr-Kubernetes

## Motivations

* ~~Harness the power of OpenStack Neutron for your Kubernetes cluster!~~
--

* Use some of Neutron functionalities for Pods networking.
--

* Provide **interconnectivity** between OpenStack VMs and Kubernetes Pods and
  Services.
  * Pod->VM, VM->Pod
  * VM->Services
--

* Avoiding having multiple network overlays in **Kubernetes on OpenStack**
  deployments.

---

# Kuryr-Kubernetes

## Multiple-Overlays

---

# Kuryr-Kubernetes

## Kuryr components

--
* kuryr-controller
  * Responsible for OpenStack API operations.
  * Watches Kubernetes resources (through K8s API) and reacts to events by CRUD
  on OpenStack resources.
  * Passes information about OpenStack resources through annotations on K8s
  resources.

--
* kuryr-daemon
  * Running on every K8s node.
  * Watches for new pods on the node and wires them when requested by
  kuryr-cni.

--
* kuryr-cni
  * Executed by CNI.
  * Passes CNI requests to kuryr-daemon.

---

# Kuryr-Kubernetes

## Kuryr components

---

# Kubernetization
## Translating Kuryr requirements into K8s primitives

---

# Kubernetization

## Why?

Why bother with:

* writing Kubernetes manifests,
* writing Dockerfiles,
* fighting with translating odd requirements into K8s primitives

?
--

Because:

* Container images are great for software distribution.
* Kubernetes is great for application deployment and management.
  * Kuryr is just an app.
* It's super easy for the K8s cluster administrator.
* **Most CNI plugins offer that deployment option.**

---

# Kubernetization

## kuryr-controller

**Requirement:** There needs to be fixed amount of kuryr-controllers. 
**OpenStack solution**: Pacemaker.

**K8s Solution:** K8s Deployments.

```yaml
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: kuryr-controller
spec:
  replicas: 1
  template:
    metadata:
      name: kuryr-controller
    spec:
      containers:
      - image: kuryr/controller
        name: controller
```

---

# Kubernetization

## kuryr-controller

**Requirement:** Leader election for A/P HA. 
**OpenStack solution:** Tooz + etcd/Redis/zookeeper

**K8s Solution:** sidecar container doing leader election

```yaml
spec:
  containers:
  - image: kuryr/controller
    name: kuryr-controller
  - image: gcr.io/google_containers/leader-elector:0.5
    name: leader-elector
    args:
    - "--election=kuryr-controller"
    - "--http=0.0.0.0:8867"
    - "--ttl=5s"
    ports:
    - containerPort: 8867
      protocol: TCP
```

---

# Kubernetization

## kuryr-controller

**Requirement:** Secure access to K8s API

**K8s Solution:** ServiceAccounts

.left-column[
```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kuryr
rules:
- apiGroups: [""]
  resources:
  - pods
  verbs: ["*"]
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: kuryr
```
]

.right-column[
```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kuryr
subjects:
- kind: ServiceAccount
  name: kuryr
---
# In Deployment's Pod spec:
spec:
  serviceAccountName: kuryr-controller
```
]

---

# Kubernetization

## kuryr-controller

**Requirement:** Provide configuration in kuryr.conf file 
**OpenStack solution:** Deployment tool handles that.

**K8s Solution:** ConfigMaps

.left-column[
```yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: kuryr-config
data:
  kuryr.conf: |+
    [DEFAULT]
    debug = true
```
]
.right-column[
```yaml
# In Deployment's Pod spec:
volumes:
- name: config-volume
  configMap:
    name: kuryr-config
    defaultMode: 0666
# In Pod's container spec:
volumeMounts:
- name: config-volume
  mountPath: "/etc/kuryr/kuryr.conf"
  subPath: kuryr.conf
```
]

---

# Kubernetization

## kuryr-controller

**Requirement:** Access to OpenStack API certificates and keys

**K8s Solution:** Secrets + ConfigMaps

.left-column[
```yaml
apiVersion: v1
kind: Secret
metadata:
 name: kuryr-certificates
type: Opaque
data:
 kuryr-ca-bundle.crt: "<base64-cert>"
```
]
.right-column[
```yaml
# In Deployment's Pod spec:
volumes:
- name: certificates-volume
 secret:
 secretName: kuryr-certificates
# In Pod's container spec
volumeMounts:
- name: certificates-volume
 mountPath: "/etc/ssl/certs/os-ca.crt"
 subPath: kuryr-ca-bundle.crt
 readOnly: true
```
]

---

# Kubernetization

## kuryr-controller

**Issue:** Chicken-and-egg problem: how will kuryr-controller pod networking
get wired without kuryr-controller pod running?!

**Solution:** Using host networking for that pod.

```yaml
# In Deployment's Pod spec:
spec:
  hostNetwork: true
  priorityClassName: system-node-critical
```

**Disadvantage:** Port conflicts, security issues.

---

# Kubernetization

## kuryr-daemon

**Requirement:** kuryr-daemon should run on every K8s node.

**Solution:** K8s DaemonSets.

```yaml
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: kuryr-cni
spec:
  template:
    metadata:
      name: kuryr-cni
    spec:
      - name: kuryr-cni
        image: kuryr/cni
```

---

# Kubernetization

## kuryr-daemon

**Requirement:** Provide configuration in kuryr.conf file.

**Solution:** ConfigMaps

<hr />

**Requirement:** Secure access to K8s API

**K8s Solution:** ServiceAccounts

---

# Kubernetization

## kuryr-daemon

**Requirement:** Needs write access to host's kernel networking

**K8s Solution:** host-networking + privileged container + mount
`/var/run/openvswitch` and `/proc`

.left-column[
```yaml
# In DaemonSet's Pod spec:
hostNetwork: true
volumes:
- name: proc
  hostPath:
	path: /proc
- name: openvswitch
  hostPath:
	path: /var/run/openvswitch
```
]

.right-column[
```yaml
# In Pod's container spec:
securityContext:
  privileged: true
volumeMounts:
- name: proc
  mountPath: /host_proc
- name: openvswitch
  mountPath: /var/run/openvswitch
```
]

---

# Kubernetization

## kuryr-daemon

**Requirement:** Pass some info from Pod's spec into the pod.

**Solution:** Pass environment variables.

```yaml
# In Pod's container spec:
env:
- name: KUBERNETES_NODE_NAME
  valueFrom:
	fieldRef:
	  fieldPath: spec.nodeName
- name: KURYR_CNI_POD_NAME
  valueFrom:
	fieldRef:
	  fieldPath: metadata.name
```

---

# Kubernetization

## kuryr-daemon

**Requirement:** Injecting kuryr-cni executable and CNI configuration

**Solution:** Mounting required host directories

```yaml
# In DaemonSet's Pod spec:
volumes:
- name: bin
  hostPath:
	path: "/opt/cni/bin"
- name: net-conf
  hostPath:
	path: "/etc/cni/net.d"
# In Pod's container spec:
volumeMounts:
- name: bin
  mountPath: "/opt/cni/bin"
- name: net-conf
  mountPath: "/etc/cni/net.d"
```

---

# Kubernetization

## Injecting Python executable

###Motivations:

* Easy distribution
* Easy installation
* Easy upgrade
* Other CNI plugins do that

###Challenges:

* It's not a binary, it's a Python script
* Dependencies distribution and conflicts
* Python availability and version mismatches

---

# Kubernetization

## Injecting Python executable - approach #1

Just use [PyInstaller](https://www.pyinstaller.org/) to compile the Python
app and its dependencies into a binary and inject it.

]

---

# Kubernetization

## Injecting Python executable - approach #1

Just use [PyInstaller](https://www.pyinstaller.org/) to compile the Python
app and its dependencies into a binary and inject it.

### Issues

* Complicated build process (need intermediate "builder" container).
* Weird issues with HTTP connection termination.
* PyInstaller changes module paths, so some tricky checks in `os.vif` started to
  fail.

]

---

# Kubernetization

## Injecting Python executable - approach #2

### Solution

**Python virtual environment** - just copy it onto the host.

### Issues

* venvs contain the Python binary - potential CPU architecture mismatches.
* `--relocatable` option simply doesn't work.

---

# Kubernetization

## Injecting Python executable - approach #3 (almost the right one)

### Solution

`docker exec` the kuryr-cni executable inside the container and pass the
stdin and stdout.

<pre>
envs=($(env | grep ^CNI_))
docker exec ${envs[@]/#/--env } -i "${CONTAINERID}" kuryr-cni \
 --config-file /etc/kuryr/kuryr.conf
</pre>

`$CONTAINERID` can be fetched from K8s API.

### Issues

* Docker API >= v1.24 is required to pass env vars.
* Some latency from running command through Docker.
* K8s API unreliable when getting CONTAINERID.

---

# Kubernetization

## Injecting Python executable - approach #4 (or #3.5)

### Solution

Use approach #3, but query Docker API for `$CONTAINERID`.

### Issues

* Docker API >= v1.24 is required to pass env vars.
* Some latency from running command through Docker.
* Some more latency from querying Docker API for CONTAINERID.
* Need to assume that labels added to containers created by K8s pods are a 
  stable API.

---

# Conclusions

## Did we learned something?

* It's working and makes distribution, deployment and management of
  Kuryr-Kubernetes installation easy for the admin.

--
* You shouldn't expect to get this right at first try - there's too many
  variables.

--
  * We've ended up backporting injection solutions through #1 to #4.

--
* Some stuff gets easier when you assume it will only work when application is
  run on Kubernetes cluster.

## New K8s features

* **Operators**

--
  * Simplified: allow you to write a K8s app that will manage your app.
  * Could be useful e.g. with Kuryr in HA mode.
  * No guarantee or ETA when we'll start looking at this.

---

# Kuryr-Kubernetes

## How to use, how to contribute

* Documentation: https://docs.openstack.org/kuryr-kubernetes/latest
* IRC channel: [#openstack-kuryr@Freenode](irc://chat.freenode.net/openstack-kuryr)
* Bugs: https://bugs.launchpad.net/kuryr-kubernetes
* How to contribute: https://wiki.openstack.org/wiki/How_To_Contribute

---

![CC](assets/cc.svg)
.medium-text[
Slides are available on

http://creativecommons.org/licenses/by/4.0/
]

---

# Q&A

## Thank you!

**Slides can be found at:**

- https://dulek.github.io/kuryr-kubernetization
- https://github.com/dulek/kuryr-kubernetization

**Michał Dulko**

![email](assets/email.png)   [mdulko@redhat.com](mailto:mdulko@redhat.com)
![irc](assets/irc.png)   [dulek (freenode)](irc://chat.freenode.net/dulek,isnick)

**Daniel Mellado**

![email](assets/email.png)   [dmellado@redhat.com](mailto:dmellado@redhat.com)
![irc](assets/irc.png)   [dmellado (freenode)](irc://chat.freenode.net/dmellado,isnick)