Skip to content

K
k3s
Commit ed10c233 authored by Michelle Noorali's avatar Michelle Noorali
Browse files
Options

replace contents of docs/proposals with stubs

parent f87edaac
Expand all Hide whitespace changes
Inline Side-by-side
Showing with 35 additions and 4862 deletions
docs/proposals/apiserver-watch.md
View file @ ed10c233
## Abstract
In the current system, most watch requests sent to apiserver are redirected to
etcd. This means that for every watch request the apiserver opens a watch on
etcd.
The purpose of the proposal is to improve the overall performance of the system
by solving the following problems:
- having too many open watches on etcd
- avoiding deserializing/converting the same objects multiple times in different
watch results
In the future, we would also like to add an indexing mechanism to the watch.
Although Indexer is not part of this proposal, it is supposed to be compatible
with it - in the future Indexer should be incorporated into the proposed new
watch solution in apiserver without requiring any redesign.
## High level design
We are going to solve those problems by allowing many clients to watch the same
storage in the apiserver, without being redirected to etcd.
At the high level, apiserver will have a single watch open to etcd, watching all
the objects (of a given type) without any filtering. The changes delivered from
etcd will then be stored in a cache in apiserver. This cache is in fact a
"rolling history window" that will support clients having some amount of latency
between their list and watch calls. Thus it will have a limited capacity and
whenever a new change comes from etcd when a cache is full, the oldest change
will be remove to make place for the new one.
When a client sends a watch request to apiserver, instead of redirecting it to
etcd, it will cause:
- registering a handler to receive all new changes coming from etcd
- iterating though a watch window, starting at the requested resourceVersion
to the head and sending filtered changes directory to the client, blocking
the above until this iteration has caught up
This will be done be creating a go-routine per watcher that will be responsible
for performing the above.
The following section describes the proposal in more details, analyzes some
corner cases and divides the whole design in more fine-grained steps.
## Proposal details
We would like the cache to be __per-resource-type__ and __optional__. Thanks to
it we will be able to:
- have different cache sizes for different resources (e.g. bigger cache
[= longer history] for pods, which can significantly affect performance)
- avoid any overhead for objects that are watched very rarely (e.g. events
are almost not watched at all, but there are a lot of them)
- filter the cache for each watcher more effectively
If we decide to support watches spanning different resources in the future and
we have an efficient indexing mechanisms, it should be relatively simple to unify
the cache to be common for all the resources.
The rest of this section describes the concrete steps that need to be done
to implement the proposal.
1. Since we want the watch in apiserver to be optional for different resource
types, this needs to be self-contained and hidden behind a well defined API.
This should be a layer very close to etcd - in particular all registries:
"pkg/registry/generic/registry" should be built on top of it.
We will solve it by turning tools.EtcdHelper by extracting its interface
and treating this interface as this API - the whole watch mechanisms in
apiserver will be hidden behind that interface.
Thanks to it we will get an initial implementation for free and we will just
need to reimplement few relevant functions (probably just Watch and List).
Moreover, this will not require any changes in other parts of the code.
This step is about extracting the interface of tools.EtcdHelper.
2. Create a FIFO cache with a given capacity. In its "rolling history window"
we will store two things:
- the resourceVersion of the object (being an etcdIndex)
- the object watched from etcd itself (in a deserialized form)
This should be as simple as having an array an treating it as a cyclic buffer.
Obviously resourceVersion of objects watched from etcd will be increasing, but
they are necessary for registering a new watcher that is interested in all the
changes since a given etcdIndex.
Additionally, we should support LIST operation, otherwise clients can never
start watching at now. We may consider passing lists through etcd, however
this will not work once we have Indexer, so we will need that information
in memory anyway.
Thus, we should support LIST operation from the "end of the history" - i.e.
from the moment just after the newest cached watched event. It should be
pretty simple to do, because we can incrementally update this list whenever
the new watch event is watched from etcd.
We may consider reusing existing structures cache.Store or cache.Indexer
("pkg/client/cache") but this is not a hard requirement.
3. Create the new implementation of the API, that will internally have a
single watch open to etcd and will store the data received from etcd in
the FIFO cache - this includes implementing registration of a new watcher
which will start a new go-routine responsible for iterating over the cache
and sending all the objects watcher is interested in (by applying filtering
function) to the watcher.
4. Add a support for processing "error too old" from etcd, which will require:
- disconnect all the watchers
- clear the internal cache and relist all objects from etcd
- start accepting watchers again
5. Enable watch in apiserver for some of the existing resource types - this
should require only changes at the initialization level.
6. The next step will be to incorporate some indexing mechanism, but details
of it are TBD.
### Future optimizations:
1. The implementation of watch in apiserver internally will open a single
watch to etcd, responsible for watching all the changes of objects of a given
resource type. However, this watch can potentially expire at any time and
reconnecting can return "too old resource version". In that case relisting is
necessary. In such case, to avoid LIST requests coming from all watchers at
the same time, we can introduce an additional etcd event type:
[EtcdResync](../../pkg/storage/etcd/etcd_watcher.go#L36)
Whenever relisting will be done to refresh the internal watch to etcd,
EtcdResync event will be send to all the watchers. It will contain the
full list of all the objects the watcher is interested in (appropriately
filtered) as the parameter of this watch event.
Thus, we need to create the EtcdResync event, extend watch.Interface and
its implementations to support it and handle those events appropriately
in places like
[Reflector](../../pkg/client/cache/reflector.go)
However, this might turn out to be unnecessary optimization if apiserver
will always keep up (which is possible in the new design). We will work
out all necessary details at that point.
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/proposals/apiserver-watch.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/design-proposals/apiserver-watch.md](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/apiserver-watch.md)
docs/proposals/cluster-deployment.md
View file @ ed10c233
# Objective
Simplify the cluster provisioning process for a cluster with one master and multiple worker nodes.
It should be secured with SSL and have all the default add-ons. There should not be significant
differences in the provisioning process across deployment targets (cloud provider + OS distribution)
once machines meet the node specification.
# Overview
Cluster provisioning can be broken into a number of phases, each with their own exit criteria.
In some cases, multiple phases will be combined together to more seamlessly automate the cluster setup,
but in all cases the phases can be run sequentially to provision a functional cluster.
It is possible that for some platforms we will provide an optimized flow that combines some of the steps
together, but that is out of scope of this document.
# Deployment flow
**Note**: _Exit critieria_ in the following sections are not intended to list all tests that should pass,
rather list those that must pass.
## Step 1: Provision cluster
**Objective**: Create a set of machines (master + nodes) where we will deploy Kubernetes.
For this phase to be completed successfully, the following requirements must be completed for all nodes:
- Basic connectivity between nodes (i.e. nodes can all ping each other)
- Docker installed (and in production setups should be monitored to be always running)
- One of the supported OS
We will provide a node specification conformance test that will verify if provisioning has been successful.
This step is provider specific and will be implemented for each cloud provider + OS distribution separately
using provider specific technology (cloud formation, deployment manager, PXE boot, etc).
Some OS distributions may meet the provisioning criteria without needing to run any post-boot steps as they
ship with all of the requirements for the node specification by default.
**Substeps** (on the GCE example):
1. Create network
2. Create firewall rules to allow communication inside the cluster
3. Create firewall rule to allow ```ssh``` to all machines
4. Create firewall rule to allow ```https``` to master
5. Create persistent disk for master
6. Create static IP address for master
7. Create master machine
8. Create node machines
9. Install docker on all machines
**Exit critera**:
1. Can ```ssh``` to all machines and run a test docker image
2. Can ```ssh``` to master and nodes and ping other machines
## Step 2: Generate certificates
**Objective**: Generate security certificates used to configure secure communication between client, master and nodes
TODO: Enumerate certificates which have to be generated.
## Step 3: Deploy master
**Objective**: Run kubelet and all the required components (e.g. etcd, apiserver, scheduler, controllers) on the master machine.
**Substeps**:
1. copy certificates
2. copy manifests for static pods:
1. etcd
2. apiserver, controller manager, scheduler
3. run kubelet in docker container (configuration is read from apiserver Config object)
4. run kubelet-checker in docker container
**v1.2 simplifications**:
1. kubelet-runner.sh - we will provide a custom docker image to run kubelet; it will contain
kubelet binary and will run it using ```nsenter``` to workaround problem with mount propagation
1. kubelet config file - we will read kubelet configuration file from disk instead of apiserver; it will
be generated locally and copied to all nodes.
**Exit criteria**:
1. Can run basic API calls (e.g. create, list and delete pods) from the client side (e.g. replication
controller works - user can create RC object and RC manager can create pods based on that)
2. Critical master components works:
1. scheduler
2. controller manager
## Step 4: Deploy nodes
**Objective**: Start kubelet on all nodes and configure kubernetes network.
Each node can be deployed separately and the implementation should make it ~impossible to change this assumption.
### Step 4.1: Run kubelet
**Substeps**:
1. copy certificates
2. run kubelet in docker container (configuration is read from apiserver Config object)
3. run kubelet-checker in docker container
**v1.2 simplifications**:
1. kubelet config file - we will read kubelet configuration file from disk instead of apiserver; it will
be generated locally and copied to all nodes.
**Exit critera**:
1. All nodes are registered, but not ready due to lack of kubernetes networking.
### Step 4.2: Setup kubernetes networking
**Objective**: Configure the Kubernetes networking to allow routing requests to pods and services.
To keep default setup consistent across open source deployments we will use Flannel to configure
kubernetes networking. However, implementation of this step will allow to easily plug in different
network solutions.
**Substeps**:
1. copy manifest for flannel server to master machine
2. create a daemonset with flannel daemon (it will read assigned CIDR and configure network appropriately).
**v1.2 simplifications**:
1. flannel daemon will run as a standalone binary (not in docker container)
2. flannel server will assign CIDRs to nodes outside of kubernetes; this will require restarting kubelet
after reconfiguring network bridge on local machine; this will also require running master nad node differently
(```--configure-cbr0=false``` on node and ```--allocate-node-cidrs=false``` on master), which breaks encapsulation
between nodes
**Exit criteria**:
1. Pods correctly created, scheduled, run and accessible from all nodes.
## Step 5: Add daemons
**Objective:** Start all system daemons (e.g. kube-proxy)
**Substeps:**:
1. Create daemonset for kube-proxy
**Exit criteria**:
1. Services work correctly on all nodes.
## Step 6: Add add-ons
**Objective**: Add default add-ons (e.g. dns, dashboard)
**Substeps:**:
1. Create Deployments (and daemonsets if needed) for all add-ons
## Deployment technology
We will use Ansible as the default technology for deployment orchestration. It has low requirements on the cluster machines
and seems to be popular in kubernetes community which will help us to maintain it.
For simpler UX we will provide simple bash scripts that will wrap all basic commands for deployment (e.g. ```up``` or ```down```)
One disadvantage of using Ansible is that it adds a dependency on a machine which runs deployment scripts. We will workaround
this by distributing deployment scripts via a docker image so that user will run the following command to create a cluster:
```docker run gcr.io/google_containers/deploy_kubernetes:v1.2 up --num-nodes=3 --provider=aws```
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/proposals/cluster-deployment.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/design-proposals/cluster-deployment.md](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/cluster-deployment.md)
docs/proposals/controller-ref.md
View file @ ed10c233
# ControllerRef proposal
Author: gmarek@
Last edit: 2016-05-11
Status: raw
Approvers:
- [ ] briangrant
- [ ] dbsmith
**Table of Contents**
- [Goal of ControllerReference](#goal-of-setreference)
- [Non goals](#non-goals)
- [API and semantic changes](#api-and-semantic-changes)
- [Upgrade/downgrade procedure](#upgradedowngrade-procedure)
- [Orphaning/adoption](#orphaningadoption)
- [Implementation plan (sketch)](#implementation-plan-sketch)
- [Considered alternatives](#considered-alternatives)
# Goal of ControllerReference
Main goal of `ControllerReference` effort is to solve a problem of overlapping controllers that fight over some resources (e.g. `ReplicaSets` fighting with `ReplicationControllers` over `Pods`), which cause serious [problems](https://github.com/kubernetes/kubernetes/issues/24433) such as exploding memory of Controller Manager.
We don’t want to have (just) an in-memory solution, as we don’t want a Controller Manager crash to cause massive changes in object ownership in the system. I.e. we need to persist the information about "owning controller".
Secondary goal of this effort is to improve performance of various controllers and schedulers, by removing the need for expensive lookup for all matching "controllers".
# Non goals
Cascading deletion is not a goal of this effort. Cascading deletion will use `ownerReferences`, which is a [separate effort](garbage-collection.md).
`ControllerRef` will extend `OwnerReference` and reuse machinery written for it (GarbageCollector, adoption/orphaning logic).
# API and semantic changes
There will be a new API field in the `OwnerReference` in which we will store an information if given owner is a managing controller:
```
OwnerReference {
…
Controller bool
…
}
```
From now on by `ControllerRef` we mean an `OwnerReference` with `Controller=true`.
Most controllers (all that manage collections of things defined by label selector) will have slightly changed semantics: currently controller owns an object if its selector matches object’s labels and if it doesn't notice an older controller of the same kind that also matches the object's labels, but after introduction of `ControllerReference` a controller will own an object iff selector matches labels and the `OwnerReference` with `Controller=true`points to it.
If the owner's selector or owned object's labels change, the owning controller will be responsible for orphaning (clearing `Controller` field in the `OwnerReference` and/or deleting `OwnerReference` altogether) objects, after which adoption procedure (setting `Controller` field in one of `OwnerReferencec` and/or adding new `OwnerReferences`) might occur, if another controller has a selector matching.
For debugging purposes we want to add an `adoptionTime` annotation prefixed with `kubernetes.io/` which will keep the time of last controller ownership transfer.
# Upgrade/downgrade procedure
Because `ControllerRef` will be a part of `OwnerReference` effort it will have the same upgrade/downgrade procedures.
# Orphaning/adoption
Because `ControllerRef` will be a part of `OwnerReference` effort it will have the same orphaning/adoption procedures.
Controllers will orphan objects they own in two cases:
* Change of label/selector causing selector to stop matching labels (executed by the controller)
* Deletion of a controller with `Orphaning=true` (executed by the GarbageCollector)
We will need a secondary orphaning mechanism in case of unclean controller deletion:
* GarbageCollector will remove `ControllerRef` from objects that no longer points to existing controllers
Controller will adopt (set `Controller` field in the `OwnerReference` that points to it) an object whose labels match its selector iff:
* there are no `OwnerReferences` with `Controller` set to true in `OwnerReferences` array
* `DeletionTimestamp` is not set
and
* Controller is the first controller that will manage to adopt the Pod from all Controllers that have matching label selector and don't have `DeletionTimestamp` set.
By design there are possible races during adoption if multiple controllers can own a given object.
To prevent re-adoption of an object during deletion the `DeletionTimestamp` will be set when deletion is starting. When a controller has a non-nil `DeletionTimestamp` it won’t take any actions except updating its `Status` (in particular it won’t adopt any objects).
# Implementation plan (sketch):
* Add API field for `Controller`,
* Extend `OwnerReference` adoption procedure to set a `Controller` field in one of the owners,
* Update all affected controllers to respect `ControllerRef`.
Necessary related work:
* `OwnerReferences` are correctly added/deleted,
* GarbageCollector removes dangling references,
* Controllers don't take any meaningful actions when `DeletionTimestamps` is set.
# Considered alternatives
* Generic "ReferenceController": centralized component that managed adoption/orphaning
* Dropped because: hard to write something that will work for all imaginable 3rd party objects, adding hooks to framework makes it possible for users to write their own logic
* Separate API field for `ControllerRef` in the ObjectMeta.
* Dropped because: nontrivial relationship between `ControllerRef` and `OwnerReferences` when it comes to deletion/adoption.
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/proposals/controller-ref.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/design-proposals/controller-ref.md](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/controller-ref.md)
docs/proposals/deploy.md
View file @ ed10c233
<!-- BEGIN MUNGE: GENERATED_TOC -->
- [Deploy through CLI](#deploy-through-cli)
- [Motivation](#motivation)
- [Requirements](#requirements)
- [Related `kubectl` Commands](#related-kubectl-commands)
- [`kubectl run`](#kubectl-run)
- [`kubectl scale` and `kubectl autoscale`](#kubectl-scale-and-kubectl-autoscale)
- [`kubectl rollout`](#kubectl-rollout)
- [`kubectl set`](#kubectl-set)
- [Mutating Operations](#mutating-operations)
- [Example](#example)
- [Support in Deployment](#support-in-deployment)
- [Deployment Status](#deployment-status)
- [Deployment Version](#deployment-version)
- [Pause Deployments](#pause-deployments)
- [Perm-failed Deployments](#perm-failed-deployments)
<!-- END MUNGE: GENERATED_TOC -->
# Deploy through CLI
## Motivation
Users can use [Deployments](../user-guide/deployments.md) or [`kubectl rolling-update`](../user-guide/kubectl/kubectl_rolling-update.md) to deploy in their Kubernetes clusters. A Deployment provides declarative update for Pods and ReplicationControllers, whereas `rolling-update` allows the users to update their earlier deployment without worrying about schemas and configurations. Users need a way that's similar to `rolling-update` to manage their Deployments more easily.
`rolling-update` expects ReplicationController as the only resource type it deals with. It's not trivial to support exactly the same behavior with Deployment, which requires:
- Print out scaling up/down events.
- Stop the deployment if users press Ctrl-c.
- The controller should not make any more changes once the process ends. (Delete the deployment when status.replicas=status.updatedReplicas=spec.replicas)
So, instead, this document proposes another way to support easier deployment management via Kubernetes CLI (`kubectl`).
## Requirements
The followings are operations we need to support for the users to easily managing deployments:
- **Create**: To create deployments.
- **Rollback**: To restore to an earlier version of deployment.
- **Watch the status**: To watch for the status update of deployments.
- **Pause/resume**: To pause a deployment mid-way, and to resume it. (A use case is to support canary deployment.)
- **Version information**: To record and show version information that's meaningful to users. This can be useful for rollback.
## Related `kubectl` Commands
### `kubectl run`
`kubectl run` should support the creation of Deployment (already implemented) and DaemonSet resources.
### `kubectl scale` and `kubectl autoscale`
Users may use `kubectl scale` or `kubectl autoscale` to scale up and down Deployments (both already implemented).
### `kubectl rollout`
`kubectl rollout` supports both Deployment and DaemonSet. It has the following subcommands:
- `kubectl rollout undo` works like rollback; it allows the users to rollback to a previous version of deployment.
- `kubectl rollout pause` allows the users to pause a deployment. See [pause deployments](#pause-deployments).
- `kubectl rollout resume` allows the users to resume a paused deployment.
- `kubectl rollout status` shows the status of a deployment.
- `kubectl rollout history` shows meaningful version information of all previous deployments. See [development version](#deployment-version).
- `kubectl rollout retry` retries a failed deployment. See [perm-failed deployments](#perm-failed-deployments).
### `kubectl set`
`kubectl set` has the following subcommands:
- `kubectl set env` allows the users to set environment variables of Kubernetes resources. It should support any object that contains a single, primary PodTemplate (such as Pod, ReplicationController, ReplicaSet, Deployment, and DaemonSet).
- `kubectl set image` allows the users to update multiple images of Kubernetes resources. Users will use `--container` and `--image` flags to update the image of a container. It should support anything that has a PodTemplate.
`kubectl set` should be used for things that are common and commonly modified. Other possible future commands include:
- `kubectl set volume`
- `kubectl set limits`
- `kubectl set security`
- `kubectl set port`
### Mutating Operations
Other means of mutating Deployments and DaemonSets, including `kubectl apply`, `kubectl edit`, `kubectl replace`, `kubectl patch`, `kubectl label`, and `kubectl annotate`, may trigger rollouts if they modify the pod template.
`kubectl create` and `kubectl delete`, for creating and deleting Deployments and DaemonSets, are also relevant.
### Example
With the commands introduced above, here's an example of deployment management:
```console
# Create a Deployment
$ kubectl run nginx --image=nginx --replicas=2 --generator=deployment/v1beta1
# Watch the Deployment status
$ kubectl rollout status deployment/nginx
# Update the Deployment
$ kubectl set image deployment/nginx --container=nginx --image=nginx:<some-version>
# Pause the Deployment
$ kubectl rollout pause deployment/nginx
# Resume the Deployment
$ kubectl rollout resume deployment/nginx
# Check the change history (deployment versions)
$ kubectl rollout history deployment/nginx
# Rollback to a previous version.
$ kubectl rollout undo deployment/nginx --to-version=<version>
```
## Support in Deployment
### Deployment Status
Deployment status should summarize information about Pods, which includes:
- The number of pods of each version.
- The number of ready/not ready pods.
See issue [#17164](https://github.com/kubernetes/kubernetes/issues/17164).
### Deployment Version
We store previous deployment version information in annotations `rollout.kubectl.kubernetes.io/change-source` and `rollout.kubectl.kubernetes.io/version` of replication controllers of the deployment, to support rolling back changes as well as for the users to view previous changes with `kubectl rollout history`.
- `rollout.kubectl.kubernetes.io/change-source`, which is optional, records the kubectl command of the last mutation made to this rollout. Users may use `--record` in `kubectl` to record current command in this annotation.
- `rollout.kubectl.kubernetes.io/version` records a version number to distinguish the change sequence of a deployment's
replication controllers. A deployment obtains the largest version number from its replication controllers and increments the number by 1 upon update or creation of the deployment, and update the version annotation of its new replication controller.
When the users perform a rollback, i.e. `kubectl rollout undo`, the deployment first looks at its existing replication controllers, regardless of their number of replicas. Then it finds the one with annotation `rollout.kubectl.kubernetes.io/version` that either contains the specified rollback version number or contains the second largest version number among all the replication controllers (current new replication controller should obtain the largest version number) if the user didn't specify any version number (the user wants to rollback to the last change). Lastly, it
starts scaling up that replication controller it's rolling back to, and scaling down the current ones, and then update the version counter and the rollout annotations accordingly.
Note that a deployment's replication controllers use PodTemplate hashes (i.e. the hash of `.spec.template`) to distinguish from each others. When doing rollout or rollback, a deployment reuses existing replication controller if it has the same PodTemplate, and its `rollout.kubectl.kubernetes.io/change-source` and `rollout.kubectl.kubernetes.io/version` annotations will be updated by the new rollout. At this point, the earlier state of this replication controller is lost in history. For example, if we had 3 replication controllers in
deployment history, and then we do a rollout with the same PodTemplate as version 1, then version 1 is lost and becomes version 4 after the rollout.
To make deployment versions more meaningful and readable for the users, we can add more annotations in the future. For example, we can add the following flags to `kubectl` for the users to describe and record their current rollout:
- `--description`: adds `description` annotation to an object when it's created to describe the object.
- `--note`: adds `note` annotation to an object when it's updated to record the change.
- `--commit`: adds `commit` annotation to an object with the commit id.
### Pause Deployments
Users sometimes need to temporarily disable a deployment. See issue [#14516](https://github.com/kubernetes/kubernetes/issues/14516).
### Perm-failed Deployments
The deployment could be marked as "permanently failed" for a given spec hash so that the system won't continue thrashing on a doomed deployment. The users can retry a failed deployment with `kubectl rollout retry`. See issue [#14519](https://github.com/kubernetes/kubernetes/issues/14519).
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/proposals/deploy.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/design-proposals/deploy.md](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/deploy.md)
docs/proposals/federated-ingress.md
View file @ ed10c233
<!-- BEGIN MUNGE: UNVERSIONED_WARNING -->
<!-- BEGIN STRIP_FOR_RELEASE -->
<img src="http://kubernetes.io/kubernetes/img/warning.png" alt="WARNING"
width="25" height="25">
<img src="http://kubernetes.io/kubernetes/img/warning.png" alt="WARNING"
width="25" height="25">
<img src="http://kubernetes.io/kubernetes/img/warning.png" alt="WARNING"
width="25" height="25">
<img src="http://kubernetes.io/kubernetes/img/warning.png" alt="WARNING"
width="25" height="25">
<img src="http://kubernetes.io/kubernetes/img/warning.png" alt="WARNING"
width="25" height="25">
<h2>PLEASE NOTE: This document applies to the HEAD of the source tree</h2>
If you are using a released version of Kubernetes, you should
refer to the docs that go with that version.
Documentation for other releases can be found at
[releases.k8s.io](http://releases.k8s.io).
</strong>
--
<!-- END STRIP_FOR_RELEASE -->
<!-- END MUNGE: UNVERSIONED_WARNING -->
# Kubernetes Federated Ingress
Requirements and High Level Design
Quinton Hoole
July 17, 2016
## Overview/Summary
[Kubernetes Ingress](https://github.com/kubernetes/kubernetes.github.io/blob/master/docs/user-guide/ingress.md)
provides an abstraction for sophisticated L7 load balancing through a
single IP address (and DNS name) across multiple pods in a single
Kubernetes cluster. Multiple alternative underlying implementations
are provided, including one based on GCE L7 load balancing and another
using an in-cluster nginx/HAProxy deployment (for non-GCE
environments). An AWS implementation, based on Elastic Load Balancers
and Route53 is under way by the community.
To extend the above to cover multiple clusters, Kubernetes Federated
Ingress aims to provide a similar/identical API abstraction and,
again, multiple implementations to cover various
cloud-provider-specific as well as multi-cloud scenarios. The general
model is to allow the user to instantiate a single Ingress object via
the Federation API, and have it automatically provision all of the
necessary underlying resources (L7 cloud load balancers, in-cluster
proxies etc) to provide L7 load balancing across a service spanning
multiple clusters.
Four options are outlined:
1. GCP only
1. AWS only
1. Cross-cloud via GCP in-cluster proxies (i.e. clients get to AWS and on-prem via GCP).
1. Cross-cloud via AWS in-cluster proxies (i.e. clients get to GCP and on-prem via AWS).
Option 1 is the:
1. easiest/quickest,
1. most featureful
Recommendations:
+ Suggest tackling option 1 (GCP only) first (target beta in v1.4)
+ Thereafter option 3 (cross-cloud via GCP)
+ We should encourage/facilitate the community to tackle option 2 (AWS-only)
## Options
## Google Cloud Platform only - backed by GCE L7 Load Balancers
This is an option for federations across clusters which all run on Google Cloud Platform (i.e. GCE and/or GKE)
### Features
In summary, all of [GCE L7 Load Balancer](https://cloud.google.com/compute/docs/load-balancing/http/) features:
1. Single global virtual (a.k.a. "anycast") IP address ("VIP" - no dependence on dynamic DNS)
1. Geo-locality for both external and GCP-internal clients
1. Load-based overflow to next-closest geo-locality (i.e. cluster). Based on either queries per second, or CPU load (unfortunately on the first-hop target VM, not the final destination K8s Service).
1. URL-based request direction (different backend services can fulfill each different URL).
1. HTTPS request termination (at the GCE load balancer, with server SSL certs)
### Implementation
1. Federation user creates (federated) Ingress object (the services
backing the ingress object must share the same nodePort, as they
share a single GCP health check).
1. Federated Ingress Controller creates Ingress object in each cluster
in the federation (after [configuring each cluster ingress
controller to share the same ingress UID](https://gist.github.com/bprashanth/52648b2a0b6a5b637f843e7efb2abc97)).
1. Each cluster-level Ingress Controller ("GLBC") creates Google L7
Load Balancer machinery (forwarding rules, target proxy, URL map,
backend service, health check) which ensures that traffic to the
Ingress (backed by a Service), is directed to the nodes in the cluster.
1. KubeProxy redirects to one of the backend Pods (currently round-robin, per KubeProxy instance)
An alternative implementation approach involves lifting the current
Federated Ingress Controller functionality up into the Federation
control plane. This alternative is not considered any any further
detail in this document.
### Outstanding work Items
1. This should in theory all work out of the box. Need to confirm
with a manual setup. ([#29341](https://github.com/kubernetes/kubernetes/issues/29341))
1. Implement Federated Ingress:
1. API machinery (~1 day)
1. Controller (~3 weeks)
1. Add DNS field to Ingress object (currently missing, but needs to be added, independent of federation)
1. API machinery (~1 day)
1. KubeDNS support (~ 1 week?)
### Pros
1. Global VIP is awesome - geo-locality, load-based overflow (but see caveats below)
1. Leverages existing K8s Ingress machinery - not too much to add.
1. Leverages existing Federated Service machinery - controller looks
almost identical, DNS provider also re-used.
### Cons
1. Only works across GCP clusters (but see below for a light at the end of the tunnel, for future versions).
## Amazon Web Services only - backed by Route53
This is an option for AWS-only federations. Parts of this are
apparently work in progress, see e.g.
[AWS Ingress controller](https://github.com/kubernetes/contrib/issues/346)
[[WIP/RFC] Simple ingress -> DNS controller, using AWS
Route53](https://github.com/kubernetes/contrib/pull/841).
### Features
In summary, most of the features of [AWS Elastic Load Balancing](https://aws.amazon.com/elasticloadbalancing/) and [Route53 DNS](https://aws.amazon.com/route53/).
1. Geo-aware DNS direction to closest regional elastic load balancer
1. DNS health checks to route traffic to only healthy elastic load
balancers
1. A variety of possible DNS routing types, including Latency Based Routing, Geo DNS, and Weighted Round Robin
1. Elastic Load Balancing automatically routes traffic across multiple
instances and multiple Availability Zones within the same region.
1. Health checks ensure that only healthy Amazon EC2 instances receive traffic.
### Implementation
1. Federation user creates (federated) Ingress object
1. Federated Ingress Controller creates Ingress object in each cluster in the federation
1. Each cluster-level AWS Ingress Controller creates/updates
1. (regional) AWS Elastic Load Balancer machinery which ensures that traffic to the Ingress (backed by a Service), is directed to one of the nodes in one of the clusters in the region.
1. (global) AWS Route53 DNS machinery which ensures that clients are directed to the closest non-overloaded (regional) elastic load balancer.
1. KubeProxy redirects to one of the backend Pods (currently round-robin, per KubeProxy instance) in the destination K8s cluster.
### Outstanding Work Items
Most of this remains is currently unimplemented ([AWS Ingress controller](https://github.com/kubernetes/contrib/issues/346)
[[WIP/RFC] Simple ingress -> DNS controller, using AWS
Route53](https://github.com/kubernetes/contrib/pull/841).
1. K8s AWS Ingress Controller
1. Re-uses all of the non-GCE specific Federation machinery discussed above under "GCP-only...".
### Pros
1. Geo-locality (via geo-DNS, not VIP)
1. Load-based overflow
1. Real load balancing (same caveats as for GCP above).
1. L7 SSL connection termination.
1. Seems it can be made to work for hybrid with on-premise (using VPC). More research required.
### Cons
1. K8s Ingress Controller still needs to be developed. Lots of work.
1. geo-DNS based locality/failover is not as nice as VIP-based (but very useful, nonetheless)
1. Only works on AWS (initial version, at least).
## Cross-cloud via GCP
### Summary
Use GCP Federated Ingress machinery described above, augmented with additional HA-proxy backends in all GCP clusters to proxy to non-GCP clusters (via either Service External IP's, or VPN directly to KubeProxy or Pods).
### Features
As per GCP-only above, except that geo-locality would be to the closest GCP cluster (and possibly onwards to the closest AWS/on-prem cluster).
### Implementation
TBD - see Summary above in the mean time.
### Outstanding Work
Assuming that GCP-only (see above) is complete:
1. Wire-up the HA-proxy load balancers to redirect to non-GCP clusters
1. Probably some more - additional detailed research and design necessary.
### Pros
1. Works for cross-cloud.
### Cons
1. Traffic to non-GCP clusters proxies through GCP clusters. Additional bandwidth costs (3x?) in those cases.
## Cross-cloud via AWS
In theory the same approach as "Cross-cloud via GCP" above could be used, except that AWS infrastructure would be used to get traffic first to an AWS cluster, and then proxied onwards to non-AWS and/or on-prem clusters.
Detail docs TBD.
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/proposals/federated-ingress.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/design-proposals/federated-ingress.md](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/federated-ingress.md)
docs/proposals/flannel-integration.md
View file @ ed10c233
# Flannel integration with Kubernetes
## Why?
* Networking works out of the box.
* Cloud gateway configuration is regulated by quota.
* Consistent bare metal and cloud experience.
* Lays foundation for integrating with networking backends and vendors.
## How?
Thus:
```
Master | Node1
----------------------------------------------------------------------
{192.168.0.0/16, 256 /24} | docker
| | | restart with podcidr
apiserver <------------------ kubelet (sends podcidr)
| | | here's podcidr, mtu
flannel-server:10253 <------------------ flannel-daemon
Allocates a /24 ------------------> [config iptables, VXLan]
<------------------ [watch subnet leases]
I just allocated ------------------> [config VXLan]
another /24 |
```
## Proposal
Explaining vxlan is out of the scope of this document, however it does take some basic understanding to grok the proposal. Assume some pod wants to communicate across nodes with the above setup. Check the flannel vxlan devices:
```console
node1 $ ip -d link show flannel.1
4: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1410 qdisc noqueue state UNKNOWN mode DEFAULT
link/ether a2:53:86:b5:5f:c1 brd ff:ff:ff:ff:ff:ff
vxlan
node1 $ ip -d link show eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1460 qdisc mq state UP mode DEFAULT qlen 1000
link/ether 42:01:0a:f0:00:04 brd ff:ff:ff:ff:ff:ff
node2 $ ip -d link show flannel.1
4: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1410 qdisc noqueue state UNKNOWN mode DEFAULT
link/ether 56:71:35:66:4a:d8 brd ff:ff:ff:ff:ff:ff
vxlan
node2 $ ip -d link show eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1460 qdisc mq state UP mode DEFAULT qlen 1000
link/ether 42:01:0a:f0:00:03 brd ff:ff:ff:ff:ff:ff
```
Note that we're ignoring cbr0 for the sake of simplicity. Spin-up a container on each node. We're using raw docker for this example only because we want control over where the container lands:
```
node1 $ docker run -it radial/busyboxplus:curl /bin/sh
[ root@5ca3c154cde3:/ ]$ ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue
8: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1410 qdisc noqueue
link/ether 02:42:12:10:20:03 brd ff:ff:ff:ff:ff:ff
inet 192.168.32.3/24 scope global eth0
valid_lft forever preferred_lft forever
node2 $ docker run -it radial/busyboxplus:curl /bin/sh
[ root@d8a879a29f5d:/ ]$ ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue
16: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1410 qdisc noqueue
link/ether 02:42:12:10:0e:07 brd ff:ff:ff:ff:ff:ff
inet 192.168.14.7/24 scope global eth0
valid_lft forever preferred_lft forever
[ root@d8a879a29f5d:/ ]$ ping 192.168.32.3
PING 192.168.32.3 (192.168.32.3): 56 data bytes
64 bytes from 192.168.32.3: seq=0 ttl=62 time=1.190 ms
```
__What happened?__:
From 1000 feet:
* vxlan device driver starts up on node1 and creates a udp tunnel endpoint on 8472
* container 192.168.32.3 pings 192.168.14.7
- what's the MAC of 192.168.14.0?
- L2 miss, flannel looks up MAC of subnet
- Stores `192.168.14.0 <-> 56:71:35:66:4a:d8` in neighbor table
- what's tunnel endpoint of this MAC?
- L3 miss, flannel looks up destination VM ip
- Stores `10.240.0.3 <-> 56:71:35:66:4a:d8` in bridge database
* Sends `[56:71:35:66:4a:d8, 10.240.0.3][vxlan: port, vni][02:42:12:10:20:03, 192.168.14.7][icmp]`
__But will it blend?__
Kubernetes integration is fairly straight-forward once we understand the pieces involved, and can be prioritized as follows:
* Kubelet understands flannel daemon in client mode, flannel server manages independent etcd store on master, node controller backs off CIDR allocation
* Flannel server consults the Kubernetes master for everything network related
* Flannel daemon works through network plugins in a generic way without bothering the kubelet: needs CNI x Kubernetes standardization
The first is accomplished in this PR, while a timeline for 2. and 3. is TDB. To implement the flannel api we can either run a proxy per node and get rid of the flannel server, or service all requests in the flannel server with something like a go-routine per node:
* `/network/config`: read network configuration and return
* `/network/leases`:
- Post: Return a lease as understood by flannel
- Lookip node by IP
- Store node metadata from [flannel request] (https://github.com/coreos/flannel/blob/master/subnet/subnet.go#L34) in annotations
- Return [Lease object] (https://github.com/coreos/flannel/blob/master/subnet/subnet.go#L40) reflecting node cidr
- Get: Handle a watch on leases
* `/network/leases/subnet`:
- Put: This is a request for a lease. If the nodecontroller is allocating CIDRs we can probably just no-op.
* `/network/reservations`: TDB, we can probably use this to accommodate node controller allocating CIDR instead of flannel requesting it
The ick-iest part of this implementation is going to the `GET /network/leases`, i.e. the watch proxy. We can side-step by waiting for a more generic Kubernetes resource. However, we can also implement it as follows:
* Watch all nodes, ignore heartbeats
* On each change, figure out the lease for the node, construct a [lease watch result](https://github.com/coreos/flannel/blob/0bf263826eab1707be5262703a8092c7d15e0be4/subnet/subnet.go#L72), and send it down the watch with the RV from the node
* Implement a lease list that does a similar translation
I say this is gross without an api object because for each node->lease translation one has to store and retrieve the node metadata sent by flannel (eg: VTEP) from node annotations. [Reference implementation](https://github.com/bprashanth/kubernetes/blob/network_vxlan/pkg/kubelet/flannel_server.go) and [watch proxy](https://github.com/bprashanth/kubernetes/blob/network_vxlan/pkg/kubelet/watch_proxy.go).
# Limitations
* Integration is experimental
* Flannel etcd not stored in persistent disk
* CIDR allocation does *not* flow from Kubernetes down to nodes anymore
# Wishlist
This proposal is really just a call for community help in writing a Kubernetes x flannel backend.
* CNI plugin integration
* Flannel daemon in privileged pod
* Flannel server talks to apiserver, described in proposal above
* HTTPs between flannel daemon/server
* Investigate flannel server running on every node (as done in the reference implementation mentioned above)
* Use flannel reservation mode to support node controller podcidr allocation
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/proposals/flannel-integration.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/design-proposals/flannel-integration.md](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/flannel-integration.md)
docs/proposals/high-availability.md
View file @ ed10c233
# High Availability of Scheduling and Controller Components in Kubernetes
This document is deprecated. For more details about running a highly available
cluster master, please see the [admin instructions document](../../docs/admin/high-availability.md).
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/proposals/high-availability.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/design-proposals/high-availability.md](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/high-availability.md)
docs/proposals/initial-resources.md
View file @ ed10c233
## Abstract
Initial Resources is a data-driven feature that based on historical data tries to estimate resource usage of a container without Resources specified
and set them before the container is run. This document describes design of the component.
## Motivation
Since we want to make Kubernetes as simple as possible for its users we don’t want to require setting [Resources](../design/resource-qos.md) for container by its owner.
On the other hand having Resources filled is critical for scheduling decisions.
Current solution to set up Resources to hardcoded value has obvious drawbacks.
We need to implement a component which will set initial Resources to a reasonable value.
## Design
InitialResources component will be implemented as an [admission plugin](../../plugin/pkg/admission/) and invoked right before
[LimitRanger](https://github.com/kubernetes/kubernetes/blob/7c9bbef96ed7f2a192a1318aa312919b861aee00/cluster/gce/config-default.sh#L91).
For every container without Resources specified it will try to predict amount of resources that should be sufficient for it.
So that a pod without specified resources will be treated as
.
InitialResources will set only [request](../design/resource-qos.md#requests-and-limits) (independently for each resource type: cpu, memory) field in the first version to avoid killing containers due to OOM (however the container still may be killed if exceeds requested resources).
To make the component work with LimitRanger the estimated value will be capped by min and max possible values if defined.
It will prevent from situation when the pod is rejected due to too low or too high estimation.
The container won’t be marked as managed by this component in any way, however appropriate event will be exported.
The predicting algorithm should have very low latency to not increase significantly e2e pod startup latency
[#3954](https://github.com/kubernetes/kubernetes/pull/3954).
### Predicting algorithm details
In the first version estimation will be made based on historical data for the Docker image being run in the container (both the name and the tag matters).
CPU/memory usage of each container is exported periodically (by default with 1 minute resolution) to the backend (see more in [Monitoring pipeline](#monitoring-pipeline)).
InitialResources will set Request for both cpu/mem as the 90th percentile of the first (in the following order) set of samples defined in the following way:
* 7 days same image:tag, assuming there is at least 60 samples (1 hour)
* 30 days same image:tag, assuming there is at least 60 samples (1 hour)
* 30 days same image, assuming there is at least 1 sample
If there is still no data the default value will be set by LimitRanger. Same parameters will be configurable with appropriate flags.
#### Example
If we have at least 60 samples from image:tag over the past 7 days, we will use the 90th percentile of all of the samples of image:tag over the past 7 days.
Otherwise, if we have at least 60 samples from image:tag over the past 30 days, we will use the 90th percentile of all of the samples over of image:tag the past 30 days.
Otherwise, if we have at least 1 sample from image over the past 30 days, we will use that the 90th percentile of all of the samples of image over the past 30 days.
Otherwise we will use default value.
### Monitoring pipeline
In the first version there will be available 2 options for backend for predicting algorithm:
* [InfluxDB](../../docs/user-guide/monitoring.md#influxdb-and-grafana) - aggregation will be made in SQL query
* [GCM](../../docs/user-guide/monitoring.md#google-cloud-monitoring) - since GCM is not as powerful as InfluxDB some aggregation will be made on the client side
Both will be hidden under an abstraction layer, so it would be easy to add another option.
The code will be a part of Initial Resources component to not block development, however in the future it should be a part of Heapster.
## Next steps
The first version will be quite simple so there is a lot of possible improvements. Some of them seem to have high priority
and should be introduced shortly after the first version is done:
* observe OOM and then react to it by increasing estimation
* add possibility to specify if estimation should be made, possibly as ```InitialResourcesPolicy``` with options: *always*, *if-not-set*, *never*
* add other features to the model like *namespace*
* remember predefined values for the most popular images like *mysql*, *nginx*, *redis*, etc.
* dry mode, which allows to ask system for resource recommendation for a container without running it
* add estimation as annotations for those containers that already has resources set
* support for other data sources like [Hawkular](http://www.hawkular.org/)
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/proposals/initial-resources.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/design-proposals/initial-resources.md](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/initial-resources.md)
docs/proposals/job.md
View file @ ed10c233
# Job Controller
## Abstract
A proposal for implementing a new controller - Job controller - which will be responsible
for managing pod(s) that require running once to completion even if the machine
the pod is running on fails, in contrast to what ReplicationController currently offers.
Several existing issues and PRs were already created regarding that particular subject:
* Job Controller [#1624](https://github.com/kubernetes/kubernetes/issues/1624)
* New Job resource [#7380](https://github.com/kubernetes/kubernetes/pull/7380)
## Use Cases
1. Be able to start one or several pods tracked as a single entity.
1. Be able to run batch-oriented workloads on Kubernetes.
1. Be able to get the job status.
1. Be able to specify the number of instances performing a job at any one time.
1. Be able to specify the number of successfully finished instances required to finish a job.
## Motivation
Jobs are needed for executing multi-pod computation to completion; a good example
here would be the ability to implement any type of batch oriented tasks.
## Implementation
Job controller is similar to replication controller in that they manage pods.
This implies they will follow the same controller framework that replication
controllers already defined. The biggest difference between a `Job` and a
`ReplicationController` object is the purpose; `ReplicationController`
ensures that a specified number of Pods are running at any one time, whereas
`Job` is responsible for keeping the desired number of Pods to a completion of
a task. This difference will be represented by the `RestartPolicy` which is
required to always take value of `RestartPolicyNever` or `RestartOnFailure`.
The new `Job` object will have the following content:
```go
// Job represents the configuration of a single job.
type Job struct {
TypeMeta
ObjectMeta
// Spec is a structure defining the expected behavior of a job.
Spec JobSpec
// Status is a structure describing current status of a job.
Status JobStatus
}
// JobList is a collection of jobs.
type JobList struct {
TypeMeta
ListMeta
Items []Job
}
```
`JobSpec` structure is defined to contain all the information how the actual job execution
will look like.
```go
// JobSpec describes how the job execution will look like.
type JobSpec struct {
// Parallelism specifies the maximum desired number of pods the job should
// run at any given time. The actual number of pods running in steady state will
// be less than this number when ((.spec.completions - .status.successful) < .spec.parallelism),
// i.e. when the work left to do is less than max parallelism.
Parallelism *int
// Completions specifies the desired number of successfully finished pods the
// job should be run with. Defaults to 1.
Completions *int
// Selector is a label query over pods running a job.
Selector map[string]string
// Template is the object that describes the pod that will be created when
// executing a job.
Template *PodTemplateSpec
}
```
`JobStatus` structure is defined to contain information about pods executing
specified job. The structure holds information about pods currently executing
the job.
```go
// JobStatus represents the current state of a Job.
type JobStatus struct {
Conditions []JobCondition
// CreationTime represents time when the job was created
CreationTime unversioned.Time
// StartTime represents time when the job was started
StartTime unversioned.Time
// CompletionTime represents time when the job was completed
CompletionTime unversioned.Time
// Active is the number of actively running pods.
Active int
// Successful is the number of pods successfully completed their job.
Successful int
// Unsuccessful is the number of pods failures, this applies only to jobs
// created with RestartPolicyNever, otherwise this value will always be 0.
Unsuccessful int
}
type JobConditionType string
// These are valid conditions of a job.
const (
// JobComplete means the job has completed its execution.
JobComplete JobConditionType = "Complete"
)
// JobCondition describes current state of a job.
type JobCondition struct {
Type JobConditionType
Status ConditionStatus
LastHeartbeatTime unversioned.Time
LastTransitionTime unversioned.Time
Reason string
Message string
}
```
## Events
Job controller will be emitting the following events:
* JobStart
* JobFinish
## Future evolution
Below are the possible future extensions to the Job controller:
* Be able to limit the execution time for a job, similarly to ActiveDeadlineSeconds for Pods. *now implemented*
* Be able to create a chain of jobs dependent one on another. *will be implemented in a separate type called Workflow*
* Be able to specify the work each of the workers should execute (see type 1 from
[this comment](https://github.com/kubernetes/kubernetes/issues/1624#issuecomment-97622142))
* Be able to inspect Pods running a Job, especially after a Job has finished, e.g.
by providing pointers to Pods in the JobStatus ([see comment](https://github.com/kubernetes/kubernetes/pull/11746/files#r37142628)).
* help users avoid non-unique label selectors ([see this proposal](../../docs/design/selector-generation.md))
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/proposals/job.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/design-proposals/job.md](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/job.md)
docs/proposals/kubectl-login.md
View file @ ed10c233
# Kubectl Login Subcommand
**Authors**: Eric Chiang (@ericchiang)
## Goals
`kubectl login` is an entrypoint for any user attempting to connect to an
existing server. It should provide a more tailored experience than the existing
`kubectl config` including config validation, auth challenges, and discovery.
Short term the subcommand should recognize and attempt to help:
* New users with an empty configuration trying to connect to a server.
* Users with no credentials, by prompt for any required information.
* Fully configured users who want to validate credentials.
* Users trying to switch servers.
* Users trying to reauthenticate as the same user because credentials have expired.
* Authenticate as a different user to the same server.
Long term `kubectl login` should enable authentication strategies to be
discoverable from a master to avoid the end-user having to know how their
sysadmin configured the Kubernetes cluster.
## Design
The "login" subcommand helps users move towards a fully functional kubeconfig by
evaluating the current state of the kubeconfig and trying to prompt the user for
and validate the necessary information to login to the kubernetes cluster.
This is inspired by a similar tools such as:
* [os login](https://docs.openshift.org/latest/cli_reference/get_started_cli.html#basic-setup-and-login)
* [gcloud auth login](https://cloud.google.com/sdk/gcloud/reference/auth/login)
* [aws configure](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html)
The steps taken are:
1. If no cluster configured, prompt user for cluster information.
2. If no user is configured, discover the authentication strategies supported by the API server.
3. Prompt the user for some information based on the authentication strategy they choose.
4. Attempt to login as a user, including authentication challenges such as OAuth2 flows, and display user info.
Importantly, each step is skipped if the existing configuration is validated or
can be supplied without user interaction (refreshing an OAuth token, redeeming
a Kerberos ticket, etc.). Users with fully configured kubeconfigs will only see
the user they're logged in as, useful for opaque credentials such as X509 certs
or bearer tokens.
The command differs from `kubectl config` by:
* Communicating with the API server to determine if the user is supplying valid auth events.
* Validating input and being opinionated about the input it asks for.
* Triggering authentication challenges for example:
* Basic auth: Actually try to communicate with the API server.
* OpenID Connect: Create an OAuth2 redirect.
However `kubectl login` should still be seen as a supplement to, not a
replacement for, `kubectl config` by helping validate any kubeconfig generated
by the latter command.
## Credential validation
When clusters utilize authorization plugins access decisions are based on the
correct configuration of an auth-N plugin, an auth-Z plugin, and client side
credentials. Being rejected then begs several questions. Is the user's
kubeconfig misconfigured? Is the authorization plugin setup wrong? Is the user
authenticating as a different user than the one they assume?
To help `kubectl login` diagnose misconfigured credentials, responses from the
API server to authenticated requests SHOULD include the `Authentication-Info`
header as defined in [RFC 7615](https://tools.ietf.org/html/rfc7615). The value
will hold name value pairs for `username` and `uid`. Since usernames and IDs
can be arbitrary strings, these values will be escaped using the `quoted-string`
format noted in the RFC.
```
HTTP/1.1 200 OK
Authentication-Info: username="janedoe@example.com", uid="123456"
```
If the user successfully authenticates this header will be set, regardless of
auth-Z decisions. For example a 401 Unauthorized (user didn't provide valid
credentials) would lack this header, while a 403 Forbidden response would
contain it.
## Authentication discovery
A long term goal of `kubectl login` is to facilitate a customized experience
for clusters configured with different auth providers. This will require some
way for the API server to indicate to `kubectl` how a user is expected to
login.
Currently, this document doesn't propose a specific implementation for
discovery. While it'd be preferable to utilize an existing standard (such as the
`WWW-Authenticate` HTTP header), discovery may require a solution custom to the
API server, such as an additional discovery endpoint with a custom type.
## Use in non-interactive session
For the initial implementation, if `kubectl login` requires prompting and is
called from a non-interactive session (determined by if the session is using a
TTY) it errors out, recommending using `kubectl config` instead. In future
updates `kubectl login` may include options for non-interactive sessions so
auth strategies which require custom behavior not built into `kubectl config`,
such as the exchanges in Kerberos or OpenID Connect, can be triggered from
scripts.
## Examples
If kubeconfig isn't configured, `kubectl login` will attempt to fully configure
and validate the client's credentials.
```
$ kubectl login
Cluster URL []: https://172.17.4.99:443
Cluster CA [(defaults to host certs)]: ${PWD}/ssl/ca.pem
Cluster Name ["cluster-1"]:
The kubernetes server supports the following methods:
1. Bearer token
2. Username and password
3. Keystone
4. OpenID Connect
5. TLS client certificate
Enter login method [1]: 4
Logging in using OpenID Connect.
Issuer ["valuefromdiscovery"]: https://accounts.google.com
Issuer CA [(defaults to host certs)]:
Scopes ["profile email"]:
Client ID []: client@localhost:foobar
Client Secret []: *****
Open the following address in a browser.
https://accounts.google.com/o/oauth2/v2/auth?redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&scopes=openid%20email&access_type=offline&...
Enter security code: ****
Logged in as "janedoe@gmail.com"
```
Human readable names are provided by a combination of the auth providers
understood by `kubectl login` and the authenticator discovery. For instance,
Keystone uses basic auth credentials in the same way as a static user file, but
if the discovery indicates that the Keystone plugin is being used it should be
presented to the user differently.
Users with configured credentials will simply auth against the API server and see
who they are. Running this command again simply validates the user's credentials.
```
$ kubectl login
Logged in as "janedoe@gmail.com"
```
Users who are halfway through the flow will start where they left off. For
instance if a user has configured the cluster field but on a user field, they will
be prompted for credentials.
```
$ kubectl login
No auth type configured. The kubernetes server supports the following methods:
1. Bearer token
2. Username and password
3. Keystone
4. OpenID Connect
5. TLS client certificate
Enter login method [1]: 2
Logging in with basic auth. Enter the following fields.
Username: janedoe
Password: ****
Logged in as "janedoe@gmail.com"
```
Users who wish to switch servers can provide the `--switch-cluster` flag which
will prompt the user for new cluster details and switch the current context. It
behaves identically to `kubectl login` when a cluster is not set.
```
$ kubectl login --switch-cluster
# ...
```
Switching users goes through a similar flow attempting to prompt the user for
new credentials to the same server.
```
$ kubectl login --switch-user
# ...
```
## Work to do
Phase 1:
* Provide a simple dialog for configuring authentication.
* Kubectl can trigger authentication actions such as trigging OAuth2 redirects.
* Validation of user credentials thought the `Authentication-Info` endpoint.
Phase 2:
* Update proposal with auth provider discovery mechanism.
* Customize dialog using discovery data.
Further improvements will require adding more authentication providers, and
adapting existing plugins to take advantage of challenge based authentication.
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/proposals/kubectl-login.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/design-proposals/kubectl-login.md](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/kubectl-login.md)
docs/proposals/kubelet-auth.md
View file @ ed10c233
# Kubelet Authentication / Authorization
Author: Jordan Liggitt (jliggitt@redhat.com)
## Overview
The kubelet exposes endpoints which give access to data of varying sensitivity,
and allow performing operations of varying power on the node and within containers.
There is no built-in way to limit or subdivide access to those endpoints,
so deployers must secure the kubelet API using external, ad-hoc methods.
This document proposes a method for authenticating and authorizing access
to the kubelet API, using interfaces and methods that complement the existing
authentication and authorization used by the API server.
## Preliminaries
This proposal assumes the existence of:
* a functioning API server
* the SubjectAccessReview and TokenReview APIs
It also assumes each node is additionally provisioned with the following information:
1. Location of the API server
2. Any CA certificates necessary to trust the API server's TLS certificate
3. Client credentials authorized to make SubjectAccessReview and TokenReview API calls
## API Changes
None
## Kubelet Authentication
Enable starting the kubelet with one or more of the following authentication methods:
* x509 client certificate
* bearer token
* anonymous (current default)
For backwards compatibility, the default is to enable anonymous authentication.
### x509 client certificate
Add a new `--client-ca-file=[file]` option to the kubelet.
When started with this option, the kubelet authenticates incoming requests using x509
client certificates, validated against the root certificates in the provided bundle.
The kubelet will reuse the x509 authenticator already used by the API server.
The master API server can already be started with `--kubelet-client-certificate` and
`--kubelet-client-key` options in order to make authenticated requests to the kubelet.
### Bearer token
Add a new `--authentication-token-webhook=[true|false]` option to the kubelet.
When true, the kubelet authenticates incoming requests with bearer tokens by making
`TokenReview` API calls to the API server.
The kubelet will reuse the webhook authenticator already used by the API server, configured
to call the API server using the connection information already provided to the kubelet.
To improve performance of repeated requests with the same bearer token, the
`--authentication-token-webhook-cache-ttl` option supported by the API server
would be supported.
### Anonymous
Add a new `--anonymous-auth=[true|false]` option to the kubelet.
When true, requests to the secure port that are not rejected by other configured
authentication methods are treated as anonymous requests, and given a username
of `system:anonymous` and a group of `system:unauthenticated`.
## Kubelet Authorization
Add a new `--authorization-mode` option to the kubelet, specifying one of the following modes:
* `Webhook`
* `AlwaysAllow` (current default)
For backwards compatibility, the authorization mode defaults to `AlwaysAllow`.
### Webhook
Webhook mode converts the request to authorization attributes, and makes a `SubjectAccessReview`
API call to check if the authenticated subject is allowed to make a request with those attributes.
This enables authorization policy to be centrally managed by the authorizer configured for the API server.
The kubelet will reuse the webhook authorizer already used by the API server, configured
to call the API server using the connection information already provided to the kubelet.
To improve performance of repeated requests with the same authenticated subject and request attributes,
the same webhook authorizer caching options supported by the API server would be supported:
* `--authorization-webhook-cache-authorized-ttl`
* `--authorization-webhook-cache-unauthorized-ttl`
### AlwaysAllow
This mode allows any authenticated request.
## Future Work
* Add support for CRL revocation for x509 client certificate authentication (http://issue.k8s.io/18982)
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/proposals/kubelet-auth.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/design-proposals/kubelet-auth.md](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/kubelet-auth.md)
docs/proposals/kubelet-hypercontainer-runtime.md
View file @ ed10c233
Kubelet HyperContainer Container Runtime
=======================================
Authors: Pengfei Ni (@feiskyer), Harry Zhang (@resouer)
## Abstract
This proposal aims to support [HyperContainer](http://hypercontainer.io) container
runtime in Kubelet.
## Motivation
HyperContainer is a Hypervisor-agnostic Container Engine that allows you to run Docker images using
hypervisors (KVM, Xen, etc.). By running containers within separate VM instances, it offers a
hardware-enforced isolation, which is required in multi-tenant environments.
## Goals
1. Complete pod/container/image lifecycle management with HyperContainer.
2. Setup network by network plugins.
3. 100% Pass node e2e tests.
4. Easy to deploy for both local dev/test and production clusters.
## Design
The HyperContainer runtime will make use of the kubelet Container Runtime Interface. [Fakti](https://github.com/kubernetes/frakti) implements the CRI interface and exposes
a local endpoint to Kubelet. Fakti communicates with [hyperd](https://github.com/hyperhq/hyperd)
with its gRPC API to manage the lifecycle of sandboxes, containers and images.
![frakti](https://cloud.githubusercontent.com/assets/676637/18940978/6e3e5384-863f-11e6-9132-b638d862fd09.png)
## Limitations
Since pods are running directly inside hypervisor, host network is not supported in HyperContainer
runtime.
## Development
The HyperContainer runtime is maintained by <https://github.com/kubernetes/frakti>.
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/proposals/kubelet-hypercontainer-runtime.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/design-proposals/kubelet-hypercontainer-runtime.md](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/kubelet-hypercontainer-runtime.md)
docs/proposals/kubelet-rkt-runtime.md
View file @ ed10c233
Next generation rkt runtime integration
=======================================
Authors: Euan Kemp (@euank), Yifan Gu (@yifan-gu)
## Abstract
This proposal describes the design and road path for integrating rkt with kubelet with the new container runtime interface.
## Background
Currently, the Kubernetes project supports rkt as a container runtime via an implementation under [pkg/kubelet/rkt package](https://github.com/kubernetes/kubernetes/tree/v1.5.0-alpha.0/pkg/kubelet/rkt).
This implementation, for historical reasons, has required implementing a large amount of logic shared by the original Docker implementation.
In order to make additional container runtime integrations easier, more clearly defined, and more consistent, a new [Container Runtime Interface](https://github.com/kubernetes/kubernetes/blob/v1.5.0-alpha.0/pkg/kubelet/api/v1alpha1/runtime/api.proto) (CRI) is being designed.
The existing runtimes, in order to both prove the correctness of the interface and reduce maintenance burden, are incentivized to move to this interface.
This document proposes how the rkt runtime integration will transition to using the CRI.
## Goals
### Full-featured
The CRI integration must work as well as the existing integration in terms of features.
Until that's the case, the existing integration will continue to be maintained.
### Easy to Deploy
The new integration should not be any more difficult to deploy and configure than the existing integration.
### Easy to Develop
This iteration should be as easy to work and iterate on as the original one.
It will be available in an initial usable form quickly in order to validate the CRI.
## Design
In order to fulfill the above goals, the rkt CRI integration will make the following choices:
### Remain in-process with Kubelet
The current rkt container runtime integration is able to be deployed simply by deploying the kubelet binary.
This is, in no small part, to make it *Easy to Deploy*.
Remaining in-process also helps this integration not regress on performance, one axis of being *Full-Featured*.
### Communicate through gRPC
Although the kubelet and rktlet will be compiled together, the runtime and kubelet will still communicate through gRPC interface for better API abstraction.
For the near short term, they will still talk through a unix socket before we implement a custom gRPC connection that skips the network stack.
### Developed as a Separate Repository
Brian Grant's discussion on splitting the Kubernetes project into [separate repos](https://github.com/kubernetes/kubernetes/issues/24343) is a compelling argument for why it makes sense to split this work into a separate repo.
In order to be *Easy to Develop*, this iteration will be maintained as a separate repository, and re-vendored back in.
This choice will also allow better long-term growth in terms of better issue-management, testing pipelines, and so on.
Unfortunately, in the short term, it's possible that some aspects of this will also cause pain and it's very difficult to weight each side correctly.
### Exec the rkt binary (initially)
While significant work on the rkt [api-service](https://coreos.com/rkt/docs/latest/subcommands/api-service.html) has been made,
it has also been a source of problems and additional complexity,
and was never transitioned to entirely.
In addition, the rkt cli has historically been the primary interface to the rkt runtime.
The initial integration will execute the rkt binary directly for app creation/start/stop/removal, as well as image pulling/removal.
The creation of pod sanbox is also done via rkt command line, but it will run under `systemd-run` so it's monitored by the init process.
In the future, some of these decisions are expected to be changed such that rkt is vendored as a library dependency for all operations, and other init systems will be supported as well.
## Roadmap and Milestones
1. rktlet integrate with kubelet to support basic pod/container lifecycle (pod creation, container creation/start/stop, pod stop/removal) [[Done]](https://github.com/kubernetes-incubator/rktlet/issues/9)
2. rktlet integrate with kubelet to support more advanced features:
- Support kubelet networking, host network
- Support mount / volumes [[#33526]](https://github.com/kubernetes/kubernetes/issues/33526)
- Support exposing ports
- Support privileged containers
- Support selinux options [[#33139]](https://github.com/kubernetes/kubernetes/issues/33139)
- Support attach [[#29579]](https://github.com/kubernetes/kubernetes/issues/29579)
- Support exec [[#29579]](https://github.com/kubernetes/kubernetes/issues/29579)
- Support logging [[#33111]](https://github.com/kubernetes/kubernetes/pull/33111)
3. rktlet integrate with kubelet, pass 100% e2e and node e2e tests, with nspawn stage1.
4. rktlet integrate with kubelet, pass 100% e2e and node e2e tests, with kvm stage1.
5. Revendor rktlet into `pkg/kubelet/rktshim`, and start deprecating the `pkg/kubelet/rkt` package.
6. Eventually replace the current `pkg/kubelet/rkt` package.
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/proposals/kubelet-rkt-runtime.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/design-proposals/kubelet-rkt-runtime.md](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/kubelet-rkt-runtime.md)
docs/proposals/kubelet-tls-bootstrap.md
View file @ ed10c233
# Kubelet TLS bootstrap
Author: George Tankersley (george.tankersley@coreos.com)
## Preface
This document describes a method for a kubelet to bootstrap itself
into a TLS-secured cluster. Crucially, it automates the provision and
distribution of signed certificates.
## Overview
When a kubelet runs for the first time, it must be given TLS assets
or generate them itself. In the first case, this is a burden on the cluster
admin and a significant logistical barrier to secure Kubernetes rollouts. In
the second, the kubelet must self-sign its certificate and forfeits many of the
advantages of a PKI system. Instead, we propose that the kubelet generate a
private key and a CSR for submission to a cluster-level certificate signing
process.
## Preliminaries
We assume the existence of a functioning control plane. The
apiserver should be configured for TLS initially or possess the ability to
generate valid TLS credentials for itself. If secret information is passed in
the request (e.g. auth tokens supplied with the request or included in
ExtraInfo) then all communications from the node to the apiserver must take
place over a verified TLS connection.
Each node is additionally provisioned with the following information:
1. Location of the apiserver
2. Any CA certificates necessary to trust the apiserver's TLS certificate
3. Access tokens (if needed) to communicate with the CSR endpoint
These should not change often and are thus simple to include in a static
provisioning script.
## API Changes
### CertificateSigningRequest Object
We introduce a new API object to represent PKCS#10 certificate signing
requests. It will be accessible under:
`/apis/certificates/v1beta1/certificatesigningrequests/mycsr`
It will have the following structure:
```go
// Describes a certificate signing request
type CertificateSigningRequest struct {
unversioned.TypeMeta `json:",inline"`
api.ObjectMeta `json:"metadata,omitempty"`
// The certificate request itself and any additional information.
Spec CertificateSigningRequestSpec `json:"spec,omitempty"`
// Derived information about the request.
Status CertificateSigningRequestStatus `json:"status,omitempty"`
}
// This information is immutable after the request is created.
type CertificateSigningRequestSpec struct {
// Base64-encoded PKCS#10 CSR data
Request string `json:"request"`
// Any extra information the node wishes to send with the request.
ExtraInfo []string `json:"extrainfo,omitempty"`
}
// This information is derived from the request by Kubernetes and cannot be
// modified by users. All information is optional since it might not be
// available in the underlying request. This is intended to aid approval
// decisions.
type CertificateSigningRequestStatus struct {
// Information about the requesting user (if relevant)
// See user.Info interface for details
Username string `json:"username,omitempty"`
UID string `json:"uid,omitempty"`
Groups []string `json:"groups,omitempty"`
// Fingerprint of the public key in request
Fingerprint string `json:"fingerprint,omitempty"`
// Subject fields from the request
Subject internal.Subject `json:"subject,omitempty"`
// DNS SANs from the request
Hostnames []string `json:"hostnames,omitempty"`
// IP SANs from the request
IPAddresses []string `json:"ipaddresses,omitempty"`
Conditions []CertificateSigningRequestCondition `json:"conditions,omitempty"`
}
type RequestConditionType string
// These are the possible states for a certificate request.
const (
Approved RequestConditionType = "Approved"
Denied RequestConditionType = "Denied"
)
type CertificateSigningRequestCondition struct {
// request approval state, currently Approved or Denied.
Type RequestConditionType `json:"type"`
// brief reason for the request state
Reason string `json:"reason,omitempty"`
// human readable message with details about the request state
Message string `json:"message,omitempty"`
// If request was approved, the controller will place the issued certificate here.
Certificate []byte `json:"certificate,omitempty"`
}
type CertificateSigningRequestList struct {
unversioned.TypeMeta `json:",inline"`
unversioned.ListMeta `json:"metadata,omitempty"`
Items []CertificateSigningRequest `json:"items,omitempty"`
}
```
We also introduce CertificateSigningRequestList to allow listing all the CSRs in the cluster:
```go
type CertificateSigningRequestList struct {
api.TypeMeta
api.ListMeta
Items []CertificateSigningRequest
}
```
## Certificate Request Process
### Node intialization
When the kubelet executes it checks a location on disk for TLS assets
(currently `/var/run/kubernetes/kubelet.{key,crt}` by default). If it finds
them, it proceeds. If there are no TLS assets, the kubelet generates a keypair
and self-signed certificate. We propose the following optional behavior:
1. Generate a keypair
2. Generate a CSR for that keypair with CN set to the hostname (or
`--hostname-override` value) and DNS/IP SANs supplied with whatever values
the host knows for itself.
3. Post the CSR to the CSR API endpoint.
4. Set a watch on the CSR object to be notified of approval or rejection.
### Controller response
The apiserver persists the CertificateSigningRequests and exposes the List of
all CSRs for an administrator to approve or reject.
A new certificate controller watches for certificate requests. It must first
validate the signature on each CSR and add `Condition=Denied` on
any requests with invalid signatures (with Reason and Message incidicating
such). For valid requests, the controller will derive the information in
`CertificateSigningRequestStatus` and update that object. The controller should
watch for updates to the approval condition of any CertificateSigningRequest.
When a request is approved (signified by Conditions containing only Approved)
the controller should generate and sign a certificate based on that CSR, then
update the condition with the certificate data using the `/approval`
subresource.
### Manual CSR approval
An administrator using `kubectl` or another API client can query the
CertificateSigningRequestList and update the approval condition of
CertificateSigningRequests. The default state is empty, indicating that there
has been no decision so far. A state of "Approved" indicates that the admin has
approved the request and the certificate controller should issue the
certificate. A state of "Denied" indicates that admin has denied the
request. An admin may also supply Reason and Message fields to explain the
rejection.
## kube-apiserver support
The apiserver will present the new endpoints mentioned above and support the
relevant object types.
## kube-controller-manager support
To handle certificate issuance, the controller-manager will need access to CA
signing assets. This could be as simple as a private key and a config file or
as complex as a PKCS#11 client and supplementary policy system. For now, we
will add flags for a signing key, a certificate, and a basic policy file.
## kubectl support
To support manual CSR inspection and approval, we will add support for listing,
inspecting, and approving or denying CertificateSigningRequests to kubectl. The
interaction will be similar to
[salt-key](https://docs.saltstack.com/en/latest/ref/cli/salt-key.html).
Specifically, the admin will have the ability to retrieve the full list of
pending CSRs, inspect their contents, and set their approval conditions to one
of:
1. **Approved** if the controller should issue the cert
2. **Denied** if the controller should not issue the cert
The suggested command for listing is `kubectl get csrs`. The approve/deny
interactions can be accomplished with normal updates, but would be more
conveniently accessed by direct subresource updates. We leave this for future
updates to kubectl.
## Security Considerations
### Endpoint Access Control
The ability to post CSRs to the signing endpoint should be controlled. As a
simple solution we propose that each node be provisioned with an auth token
(possibly static across the cluster) that is scoped via ABAC to only allow
access to the CSR endpoint.
### Expiration & Revocation
The node is responsible for monitoring its own certificate expiration date.
When the certificate is close to expiration, the kubelet should begin repeating
this flow until it successfully obtains a new certificate. If the expiring
certificate has not been revoked and the previous certificate request is still
approved, then it may do so using the same keypair unless the cluster policy
(see "Future Work") requires fresh keys.
Revocation is for the most part an unhandled problem in Go, requiring each
application to produce its own logic around a variety of parsing functions. For
now, our suggested best practice is to issue only short-lived certificates. In
the future it may make sense to add CRL support to the apiserver's client cert
auth.
## Future Work
- revocation UI in kubectl and CRL support at the apiserver
- supplemental policy (e.g. cluster CA only issues 30-day certs for hostnames *.k8s.example.com, each new cert must have fresh keys, ...)
- fully automated provisioning (using a handshake protocol or external list of authorized machines)
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/proposals/kubelet-tls-bootstrap.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/design-proposals/kubelet-tls-bootstrap.md](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/kubelet-tls-bootstrap.md)
docs/proposals/local-cluster-ux.md
View file @ ed10c233
# Kubernetes Local Cluster Experience
This proposal attempts to improve the existing local cluster experience for kubernetes.
The current local cluster experience is sub-par and often not functional.
There are several options to setup a local cluster (docker, vagrant, linux processes, etc) and we do not test any of them continuously.
Here are some highlighted issues:
- Docker based solution breaks with docker upgrades, does not support DNS, and many kubelet features are not functional yet inside a container.
- Vagrant based solution are too heavy and have mostly failed on OS X.
- Local linux cluster is poorly documented and is undiscoverable.
From an end user perspective, they want to run a kubernetes cluster. They care less about *how* a cluster is setup locally and more about what they can do with a functional cluster.
## Primary Goals
From a high level the goal is to make it easy for a new user to run a Kubernetes cluster and play with curated examples that require least amount of knowledge about Kubernetes.
These examples will only use kubectl and only a subset of Kubernetes features that are available will be exposed.
- Works across multiple OSes - OS X, Linux and Windows primarily.
- Single command setup and teardown UX.
- Unified UX across OSes
- Minimal dependencies on third party software.
- Minimal resource overhead.
- Eliminate any other alternatives to local cluster deployment.
## Secondary Goals
- Enable developers to use the local cluster for kubernetes development.
## Non Goals
- Simplifying kubernetes production deployment experience. [Kube-deploy](https://github.com/kubernetes/kube-deploy) is attempting to tackle this problem.
- Supporting all possible deployment configurations of Kubernetes like various types of storage, networking, etc.
## Local cluster requirements
- Includes all the master components & DNS (Apiserver, scheduler, controller manager, etcd and kube dns)
- Basic auth
- Service accounts should be setup
- Kubectl should be auto-configured to use the local cluster
- Tested & maintained as part of Kubernetes core
## Existing solutions
Following are some of the existing solutions that attempt to simplify local cluster deployments.
### [Spread](https://github.com/redspread/spread)
Spread's UX is great!
It is adapted from monokube and includes DNS as well.
It satisfies almost all the requirements, excepting that of requiring docker to be pre-installed.
It has a loose dependency on docker.
New releases of docker might break this setup.
### [Kmachine](https://github.com/skippbox/kmachine)
Kmachine is adapted from docker-machine.
It exposes the entire docker-machine CLI.
It is possible to repurpose Kmachine to meet all our requirements.
### [Monokube](https://github.com/polvi/monokube)
Single binary that runs all kube master components.
Does not include DNS.
This is only a part of the overall local cluster solution.
### Vagrant
The kube-up.sh script included in Kubernetes release supports a few Vagrant based local cluster deployments.
kube-up.sh is not user friendly.
It typically takes a long time for the cluster to be set up using vagrant and often times is unsuccessful on OS X.
The [Core OS single machine guide](https://coreos.com/kubernetes/docs/latest/kubernetes-on-vagrant-single.html) uses Vagrant as well and it just works.
Since we are targeting a single command install/teardown experience, vagrant needs to be an implementation detail and not be exposed to our users.
## Proposed Solution
To avoid exposing users to third party software and external dependencies, we will build a toolbox that will be shipped with all the dependencies including all kubernetes components, hypervisor, base image, kubectl, etc.
*Note: Docker provides a [similar toolbox](https://www.docker.com/products/docker-toolbox).*
This "Localkube" tool will be referred to as "Minikube" in this proposal to avoid ambiguity against Spread's existing ["localkube"](https://github.com/redspread/localkube).
The final name of this tool is TBD. Suggestions are welcome!
Minikube will provide a unified CLI to interact with the local cluster.
The CLI will support only a few operations:
- **Start** - creates & starts a local cluster along with setting up kubectl & networking (if necessary)
- **Stop** - suspends the local cluster & preserves cluster state
- **Delete** - deletes the local cluster completely
- **Upgrade** - upgrades internal components to the latest available version (upgrades are not guaranteed to preserve cluster state)
For running and managing the kubernetes components themselves, we can re-use [Spread's localkube](https://github.com/redspread/localkube).
Localkube is a self-contained go binary that includes all the master components including DNS and runs them using multiple go threads.
Each Kubernetes release will include a localkube binary that has been tested exhaustively.
To support Windows and OS X, minikube will use [libmachine](https://github.com/docker/machine/tree/master/libmachine) internally to create and destroy virtual machines.
Minikube will be shipped with an hypervisor (virtualbox) in the case of OS X.
Minikube will include a base image that will be well tested.
In the case of Linux, since the cluster can be run locally, we ideally want to avoid setting up a VM.
Since docker is the only fully supported runtime as of Kubernetes v1.2, we can initially use docker to run and manage localkube.
There is risk of being incompatible with the existing version of docker.
By using a VM, we can avoid such incompatibility issues though.
Feedback from the community will be helpful here.
If the goal is to run outside of a VM, we can have minikube prompt the user if docker is unavailable or version is incompatible.
Alternatives to docker for running the localkube core includes using [rkt](https://coreos.com/rkt/docs/latest/), setting up systemd services, or a System V Init script depending on the distro.
To summarize the pipeline is as follows:
##### OS X / Windows
minikube -> libmachine -> virtualbox/hyper V -> linux VM -> localkube
##### Linux
minikube -> docker -> localkube
### Alternatives considered
#### Bring your own docker
##### Pros
- Kubernetes users will probably already have it
- No extra work for us
- Only one VM/daemon, we can just reuse the existing one
##### Cons
- Not designed to be wrapped, may be unstable
- Might make configuring networking difficult on OS X and Windows
- Versioning and updates will be challenging. We can mitigate some of this with testing at HEAD, but we'll - inevitably hit situations where it's infeasible to work with multiple versions of docker.
- There are lots of different ways to install docker, networking might be challenging if we try to support many paths.
#### Vagrant
##### Pros
- We control the entire experience
- Networking might be easier to build
- Docker can't break us since we'll include a pinned version of Docker
- Easier to support rkt or hyper in the future
- Would let us run some things outside of containers (kubelet, maybe ingress/load balancers)
##### Cons
- More work
- Extra resources (if the user is also running docker-machine)
- Confusing if there are two docker daemons (images built in one can't be run in another)
- Always needs a VM, even on Linux
- Requires installing and possibly understanding Vagrant.
## Releases & Distribution
- Minikube will be released independent of Kubernetes core in order to facilitate fixing of issues that are outside of Kubernetes core.
- The latest version of Minikube is guaranteed to support the latest release of Kubernetes, including documentation.
- The Google Cloud SDK will package minikube and provide utilities for configuring kubectl to use it, but will not in any other way wrap minikube.
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/proposals/local-cluster-ux.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/design-proposals/local-cluster-ux.md](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/local-cluster-ux.md)
docs/proposals/multiple-schedulers.md
View file @ ed10c233
# Multi-Scheduler in Kubernetes
**Status**: Design & Implementation in progress.
> Contact @HaiyangDING for questions & suggestions.
## Motivation
In current Kubernetes design, there is only one default scheduler in a Kubernetes cluster.
However it is common that multiple types of workload, such as traditional batch, DAG batch, streaming and user-facing production services,
are running in the same cluster and they need to be scheduled in different ways. For example, in
[Omega](http://research.google.com/pubs/pub41684.html) batch workload and service workload are scheduled by two types of schedulers:
the batch workload is scheduled by a scheduler which looks at the current usage of the cluster to improve the resource usage rate
and the service workload is scheduled by another one which considers the reserved resources in the
cluster and many other constraints since their performance must meet some higher SLOs.
[Mesos](http://mesos.apache.org/) has done a great work to support multiple schedulers by building a
two-level scheduling structure. This proposal describes how Kubernetes is going to support multi-scheduler
so that users could be able to run their user-provided scheduler(s) to enable some customized scheduling
behavior as they need. As previously discussed in [#11793](https://github.com/kubernetes/kubernetes/issues/11793),
[#9920](https://github.com/kubernetes/kubernetes/issues/9920) and [#11470](https://github.com/kubernetes/kubernetes/issues/11470),
the design of the multiple scheduler should be generic and includes adding a scheduler name annotation to separate the pods.
It is worth mentioning that the proposal does not address the question of how the scheduler name annotation gets
set although it is reasonable to anticipate that it would be set by a component like admission controller/initializer,
as the doc currently does.
Before going to the details of this proposal, below lists a number of the methods to extend the scheduler:
- Write your own scheduler and run it along with Kubernetes native scheduler. This is going to be detailed in this proposal
- Use the callout approach such as the one implemented in [#13580](https://github.com/kubernetes/kubernetes/issues/13580)
- Recompile the scheduler with a new policy
- Restart the scheduler with a new [scheduler policy config file](../../examples/scheduler-policy-config.json)
- Or maybe in future dynamically link a new policy into the running scheduler
## Challenges in multiple schedulers
- Separating the pods
Each pod should be scheduled by only one scheduler. As for implementation, a pod should
have an additional field to tell by which scheduler it wants to be scheduled. Besides,
each scheduler, including the default one, should have a unique logic of how to add unscheduled
pods to its to-be-scheduled pod queue. Details will be explained in later sections.
- Dealing with conflicts
Different schedulers are essentially separated processes. When all schedulers try to schedule
their pods onto the nodes, there might be conflicts.
One example of the conflicts is resource racing: Suppose there be a `pod1` scheduled by
`my-scheduler` requiring 1 CPU's *request*, and a `pod2` scheduled by `kube-scheduler` (k8s native
scheduler, acting as default scheduler) requiring 2 CPU's *request*, while `node-a` only has 2.5
free CPU's, if both schedulers all try to put their pods on `node-a`, then one of them would eventually
fail when Kubelet on `node-a` performs the create action due to insufficient CPU resources.
This conflict is complex to deal with in api-server and etcd. Our current solution is to let Kubelet
to do the conflict check and if the conflict happens, effected pods would be put back to scheduler
and waiting to be scheduled again. Implementation details are in later sections.
## Where to start: initial design
We definitely want the multi-scheduler design to be a generic mechanism. The following lists the changes
we want to make in the first step.
- Add an annotation in pod template: `scheduler.alpha.kubernetes.io/name: scheduler-name`, this is used to
separate pods between schedulers. `scheduler-name` should match one of the schedulers' `scheduler-name`
- Add a `scheduler-name` to each scheduler. It is done by hardcode or as command-line argument. The
Kubernetes native scheduler (now `kube-scheduler` process) would have the name as `kube-scheduler`
- The `scheduler-name` plays an important part in separating the pods between different schedulers.
Pods are statically dispatched to different schedulers based on `scheduler.alpha.kubernetes.io/name: scheduler-name`
annotation and there should not be any conflicts between different schedulers handling their pods, i.e. one pod must
NOT be claimed by more than one scheduler. To be specific, a scheduler can add a pod to its queue if and only if:
1. The pod has no nodeName, **AND**
2. The `scheduler-name` specified in the pod's annotation `scheduler.alpha.kubernetes.io/name: scheduler-name`
matches the `scheduler-name` of the scheduler.
The only one exception is the default scheduler. Any pod that has no `scheduler.alpha.kubernetes.io/name: scheduler-name`
annotation is assumed to be handled by the "default scheduler". In the first version of the multi-scheduler feature,
the default scheduler would be the Kubernetes built-in scheduler with `scheduler-name` as `kube-scheduler`.
The Kubernetes build-in scheduler will claim any pod which has no `scheduler.alpha.kubernetes.io/name: scheduler-name`
annotation or which has `scheduler.alpha.kubernetes.io/name: kube-scheduler`. In the future, it may be possible to
change which scheduler is the default for a given cluster.
- Dealing with conflicts. All schedulers must use predicate functions that are at least as strict as
the ones that Kubelet applies when deciding whether to accept a pod, otherwise Kubelet and scheduler
may get into an infinite loop where Kubelet keeps rejecting a pod and scheduler keeps re-scheduling
it back the same node. To make it easier for people who write new schedulers to obey this rule, we will
create a library containing the predicates Kubelet uses. (See issue [#12744](https://github.com/kubernetes/kubernetes/issues/12744).)
In summary, in the initial version of this multi-scheduler design, we will achieve the following:
- If a pod has the annotation `scheduler.alpha.kubernetes.io/name: kube-scheduler` or the user does not explicitly
sets this annotation in the template, it will be picked up by default scheduler
- If the annotation is set and refers to a valid `scheduler-name`, it will be picked up by the scheduler of
specified `scheduler-name`
- If the annotation is set but refers to an invalid `scheduler-name`, the pod will not be picked by any scheduler.
The pod will keep PENDING.
### An example
```yaml
kind: Pod
apiVersion: v1
metadata:
name: pod-abc
labels:
foo: bar
annotations:
scheduler.alpha.kubernetes.io/name: my-scheduler
```
This pod will be scheduled by "my-scheduler" and ignored by "kube-scheduler". If there is no running scheduler
of name "my-scheduler", the pod will never be scheduled.
## Next steps
1. Use admission controller to add and verify the annotation, and do some modification if necessary. For example, the
admission controller might add the scheduler annotation based on the namespace of the pod, and/or identify if
there are conflicting rules, and/or set a default value for the scheduler annotation, and/or reject pods on
which the client has set a scheduler annotation that does not correspond to a running scheduler.
2. Dynamic launching scheduler(s) and registering to admission controller (as an external call). This also
requires some work on authorization and authentication to control what schedulers can write the /binding
subresource of which pods.
3. Optimize the behaviors of priority functions in multi-scheduler scenario. In the case where multiple schedulers have
the same predicate and priority functions (for example, when using multiple schedulers for parallelism rather than to
customize the scheduling policies), all schedulers would tend to pick the same node as "best" when scheduling identical
pods and therefore would be likely to conflict on the Kubelet. To solve this problem, we can pass
an optional flag such as `--randomize-node-selection=N` to scheduler, setting this flag would cause the scheduler to pick
randomly among the top N nodes instead of the one with the highest score.
## Other issues/discussions related to scheduler design
- [#13580](https://github.com/kubernetes/kubernetes/pull/13580): scheduler extension
- [#17097](https://github.com/kubernetes/kubernetes/issues/17097): policy config file in pod template
- [#16845](https://github.com/kubernetes/kubernetes/issues/16845): scheduling groups of pods
- [#17208](https://github.com/kubernetes/kubernetes/issues/17208): guide to writing a new scheduler
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/proposals/multiple-schedulers.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/design-proposals/multiple-schedulers.md](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/multiple-schedulers.md)
docs/proposals/node-allocatable.md
View file @ ed10c233
# Node Allocatable Resources
**Issue:** https://github.com/kubernetes/kubernetes/issues/13984
## Overview
Currently Node.Status has Capacity, but no concept of node Allocatable. We need additional
parameters to serve several purposes:
1. Kubernetes metrics provides "/docker-daemon", "/kubelet",
"/kube-proxy", "/system" etc. raw containers for monitoring system component resource usage
patterns and detecting regressions. Eventually we want to cap system component usage to a certain
limit / request. However this is not currently feasible due to a variety of reasons including:
1. Docker still uses tons of computing resources (See
[#16943](https://github.com/kubernetes/kubernetes/issues/16943))
2. We have not yet defined the minimal system requirements, so we cannot control Kubernetes
nodes or know about arbitrary daemons, which can make the system resources
unmanageable. Even with a resource cap we cannot do a full resource management on the
node, but with the proposed parameters we can mitigate really bad resource over commits
3. Usage scales with the number of pods running on the node
2. For external schedulers (such as mesos, hadoop, etc.) integration, they might want to partition
compute resources on a given node, limiting how much Kubelet can use. We should provide a
mechanism by which they can query kubelet, and reserve some resources for their own purpose.
### Scope of proposal
This proposal deals with resource reporting through the [`Allocatable` field](#allocatable) for more
reliable scheduling, and minimizing resource over commitment. This proposal *does not* cover
resource usage enforcement (e.g. limiting kubernetes component usage), pod eviction (e.g. when
reservation grows), or running multiple Kubelets on a single node.
## Design
### Definitions
![image](node-allocatable.png)
1. **Node Capacity** - Already provided as
[`NodeStatus.Capacity`](https://htmlpreview.github.io/?https://github.com/kubernetes/kubernetes/blob/HEAD/docs/api-reference/v1/definitions.html#_v1_nodestatus),
this is total capacity read from the node instance, and assumed to be constant.
2. **System-Reserved** (proposed) - Compute resources reserved for processes which are not managed by
Kubernetes. Currently this covers all the processes lumped together in the `/system` raw
container.
3. **Kubelet Allocatable** - Compute resources available for scheduling (including scheduled &
unscheduled resources). This value is the focus of this proposal. See [below](#api-changes) for
more details.
4. **Kube-Reserved** (proposed) - Compute resources reserved for Kubernetes components such as the
docker daemon, kubelet, kube proxy, etc.
### API changes
#### Allocatable
Add `Allocatable` (4) to
[`NodeStatus`](https://htmlpreview.github.io/?https://github.com/kubernetes/kubernetes/blob/HEAD/docs/api-reference/v1/definitions.html#_v1_nodestatus):
```
type NodeStatus struct {
...
// Allocatable represents schedulable resources of a node.
Allocatable ResourceList `json:"allocatable,omitempty"`
...
}
```
Allocatable will be computed by the Kubelet and reported to the API server. It is defined to be:
```
[Allocatable] = [Node Capacity] - [Kube-Reserved] - [System-Reserved]
```
The scheduler will use `Allocatable` in place of `Capacity` when scheduling pods, and the Kubelet
will use it when performing admission checks.
*Note: Since kernel usage can fluctuate and is out of kubernetes control, it will be reported as a
separate value (probably via the metrics API). Reporting kernel usage is out-of-scope for this
proposal.*
#### Kube-Reserved
`KubeReserved` is the parameter specifying resources reserved for kubernetes components (4). It is
provided as a command-line flag to the Kubelet at startup, and therefore cannot be changed during
normal Kubelet operation (this may change in the [future](#future-work)).
The flag will be specified as a serialized `ResourceList`, with resources defined by the API
`ResourceName` and values specified in `resource.Quantity` format, e.g.:
```
--kube-reserved=cpu=500m,memory=5Mi
```
Initially we will only support CPU and memory, but will eventually support more resources. See
[#16889](https://github.com/kubernetes/kubernetes/pull/16889) for disk accounting.
If KubeReserved is not set it defaults to a sane value (TBD) calculated from machine capacity. If it
is explicitly set to 0 (along with `SystemReserved`), then `Allocatable == Capacity`, and the system
behavior is equivalent to the 1.1 behavior with scheduling based on Capacity.
#### System-Reserved
In the initial implementation, `SystemReserved` will be functionally equivalent to
[`KubeReserved`](#system-reserved), but with a different semantic meaning. While KubeReserved
designates resources set aside for kubernetes components, SystemReserved designates resources set
aside for non-kubernetes components (currently this is reported as all the processes lumped
together in the `/system` raw container).
## Issues
### Kubernetes reservation is smaller than kubernetes component usage
**Solution**: Initially, do nothing (best effort). Let the kubernetes daemons overflow the reserved
resources and hope for the best. If the node usage is less than Allocatable, there will be some room
for overflow and the node should continue to function. If the node has been scheduled to capacity
(worst-case scenario) it may enter an unstable state, which is the current behavior in this
situation.
In the [future](#future-work) we may set a parent cgroup for kubernetes components, with limits set
according to `KubeReserved`.
### Version discrepancy
**API server / scheduler is not allocatable-resources aware:** If the Kubelet rejects a Pod but the
scheduler expects the Kubelet to accept it, the system could get stuck in an infinite loop
scheduling a Pod onto the node only to have Kubelet repeatedly reject it. To avoid this situation,
we will do a 2-stage rollout of `Allocatable`. In stage 1 (targeted for 1.2), `Allocatable` will
be reported by the Kubelet and the scheduler will be updated to use it, but Kubelet will continue
to do admission checks based on `Capacity` (same as today). In stage 2 of the rollout (targeted
for 1.3 or later), the Kubelet will start doing admission checks based on `Allocatable`.
**API server expects `Allocatable` but does not receive it:** If the kubelet is older and does not
provide `Allocatable` in the `NodeStatus`, then `Allocatable` will be
[defaulted](../../pkg/api/v1/defaults.go) to
`Capacity` (which will yield today's behavior of scheduling based on capacity).
### 3rd party schedulers
The community should be notified that an update to schedulers is recommended, but if a scheduler is
not updated it falls under the above case of "scheduler is not allocatable-resources aware".
## Future work
1. Convert kubelet flags to Config API - Prerequisite to (2). See
[#12245](https://github.com/kubernetes/kubernetes/issues/12245).
2. Set cgroup limits according KubeReserved - as described in the [overview](#overview)
3. Report kernel usage to be considered with scheduling decisions.
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/proposals/node-allocatable.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/design-proposals/node-allocatable.md](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/node-allocatable.md)
docs/proposals/performance-related-monitoring.md
View file @ ed10c233
# Performance Monitoring
## Reason for this document
This document serves as a place to gather information about past performance regressions, their reason and impact and discuss ideas to avoid similar regressions in the future.
Main reason behind doing this is to understand what kind of monitoring needs to be in place to keep Kubernetes fast.
## Known past and present performance issues
### Higher logging level causing scheduler stair stepping
Issue https://github.com/kubernetes/kubernetes/issues/14216 was opened because @spiffxp observed a regression in scheduler performance in 1.1 branch in comparison to `old` 1.0
cut. In the end it turned out the be caused by `--v=4` (instead of default `--v=2`) flag in the scheduler together with the flag `--logtostderr` which disables batching of
log lines and a number of logging without explicit V level. This caused weird behavior of the whole component.
Because we now know that logging may have big performance impact we should consider instrumenting logging mechanism and compute statistics such as number of logged messages,
total and average size of them. Each binary should be responsible for exposing its metrics. An unaccounted but way too big number of days, if not weeks, of engineering time was
lost because of this issue.
### Adding per-pod probe-time, which increased the number of PodStatus updates, causing major slowdown
In September 2015 we tried to add per-pod probe times to the PodStatus. It caused (https://github.com/kubernetes/kubernetes/issues/14273) a massive increase in both number and
total volume of object (PodStatus) changes. It drastically increased the load on API server which wasn’t able to handle new number of requests quickly enough, violating our
response time SLO. We had to revert this change.
### Late Ready->Running PodPhase transition caused test failures as it seemed like slowdown
In late September we encountered a strange problem (https://github.com/kubernetes/kubernetes/issues/14554): we observed an increased observed latencies in small clusters (few
Nodes). It turned out that it’s caused by an added latency between PodRunning and PodReady phases. This was not a real regression, but our tests thought it were, which shows
how careful we need to be.
### Huge number of handshakes slows down API server
It was a long standing issue for performance and is/was an important bottleneck for scalability (https://github.com/kubernetes/kubernetes/issues/13671). The bug directly
causing this problem was incorrect (from the golangs standpoint) handling of TCP connections. Secondary issue was that elliptic curve encryption (only one available in go 1.4)
is unbelievably slow.
## Proposed metrics/statistics to gather/compute to avoid problems
### Cluster-level metrics
Basic ideas:
- number of Pods/ReplicationControllers/Services in the cluster
- number of running replicas of master components (if they are replicated)
- current elected master of ectd cluster (if running distributed version)
- nuber of master component restarts
- number of lost Nodes
### Logging monitoring
Log spam is a serious problem and we need to keep it under control. Simplest way to check for regressions, suggested by @brendandburns, is to compute the rate in which log files
grow in e2e tests.
Basic ideas:
- log generation rate (B/s)
### REST call monitoring
We do measure REST call duration in the Density test, but we need an API server monitoring as well, to avoid false failures caused e.g. by the network traffic. We already have
some metrics in place (https://github.com/kubernetes/kubernetes/blob/master/pkg/apiserver/metrics/metrics.go), but we need to revisit the list and add some more.
Basic ideas:
- number of calls per verb, client, resource type
- latency distribution per verb, client, resource type
- number of calls that was rejected per client, resource type and reason (invalid version number, already at maximum number of requests in flight)
- number of relists in various watchers
### Rate limit monitoring
Reverse of REST call monitoring done in the API server. We need to know when a given component increases a pressure it puts on the API server. As a proxy for number of
requests sent we can track how saturated are rate limiters. This has additional advantage of giving us data needed to fine-tune rate limiter constants.
Because we have rate limiting on both ends (client and API server) we should monitor number of inflight requests in API server and how it relates to `max-requests-inflight`.
Basic ideas:
- percentage of used non-burst limit,
- amount of time in last hour with depleted burst tokens,
- number of inflight requests in API server.
### Network connection monitoring
During development we observed incorrect use/reuse of HTTP connections multiple times already. We should at least monitor number of created connections.
### ETCD monitoring
@xiang-90 and @hongchaodeng - you probably have way more experience on what'd be good to look at from the ETCD perspective.
Basic ideas:
- ETCD memory footprint
- number of objects per kind
- read/write latencies per kind
- number of requests from the API server
- read/write counts per key (it may be too heavy though)
### Resource consumption
On top of all things mentioned above we need to monitor changes in resource usage in both: cluster components (API server, Kubelet, Scheduler, etc.) and system add-ons
(Heapster, L7 load balancer, etc.). Monitoring memory usage is tricky, because if no limits are set, system won't apply memory pressure to processes, which makes their memory
footprint constantly grow. We argue that monitoring usage in tests still makes sense, as tests should be repeatable, and if memory usage will grow drastically between two runs
it most likely can be attributed to some kind of regression (assuming that nothing else has changed in the environment).
Basic ideas:
- CPU usage
- memory usage
### Other saturation metrics
We should monitor other aspects of the system, which may indicate saturation of some component.
Basic ideas:
- queue length for queues in the system,
- wait time for WaitGroups.
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/proposals/performance-related-monitoring.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/design-proposals/performance-related-monitoring.md](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/performance-related-monitoring.md)
docs/proposals/pod-lifecycle-event-generator.md
View file @ ed10c233
# Kubelet: Pod Lifecycle Event Generator (PLEG)
In Kubernetes, Kubelet is a per-node daemon that manages the pods on the node,
driving the pod states to match their pod specifications (specs). To achieve
this, Kubelet needs to react to changes in both (1) pod specs and (2) the
container states. For the former, Kubelet watches the pod specs changes from
multiple sources; for the latter, Kubelet polls the container runtime
periodically (e.g., 10s) for the latest states for all containers.
Polling incurs non-negligible overhead as the number of pods/containers increases,
and is exacerbated by Kubelet's parallelism -- one worker (goroutine) per pod, which
queries the container runtime individually. Periodic, concurrent, large number
of requests causes high CPU usage spikes (even when there is no spec/state
change), poor performance, and reliability problems due to overwhelmed container
runtime. Ultimately, it limits Kubelet's scalability.
(Related issues reported by users: [#10451](https://issues.k8s.io/10451),
[#12099](https://issues.k8s.io/12099), [#12082](https://issues.k8s.io/12082))
## Goals and Requirements
The goal of this proposal is to improve Kubelet's scalability and performance
by lowering the pod management overhead.
- Reduce unnecessary work during inactivity (no spec/state changes)
- Lower the concurrent requests to the container runtime.
The design should be generic so that it can support different container runtimes
(e.g., Docker and rkt).
## Overview
This proposal aims to replace the periodic polling with a pod lifecycle event
watcher.
![pleg](pleg.png)
## Pod Lifecycle Event
A pod lifecycle event interprets the underlying container state change at the
pod-level abstraction, making it container-runtime-agnostic. The abstraction
shields Kubelet from the runtime specifics.
```go
type PodLifeCycleEventType string
const (
ContainerStarted PodLifeCycleEventType = "ContainerStarted"
ContainerStopped PodLifeCycleEventType = "ContainerStopped"
NetworkSetupCompleted PodLifeCycleEventType = "NetworkSetupCompleted"
NetworkFailed PodLifeCycleEventType = "NetworkFailed"
)
// PodLifecycleEvent is an event reflects the change of the pod state.
type PodLifecycleEvent struct {
// The pod ID.
ID types.UID
// The type of the event.
Type PodLifeCycleEventType
// The accompanied data which varies based on the event type.
Data interface{}
}
```
Using Docker as an example, starting of a POD infra container would be
translated to a NetworkSetupCompleted`pod lifecycle event.
## Detect Changes in Container States Via Relisting
In order to generate pod lifecycle events, PLEG needs to detect changes in
container states. We can achieve this by periodically relisting all containers
(e.g., docker ps). Although this is similar to Kubelet's polling today, it will
only be performed by a single thread (PLEG). This means that we still
benefit from not having all pod workers hitting the container runtime
concurrently. Moreover, only the relevant pod worker would be woken up
to perform a sync.
The upside of relying on relisting is that it is container runtime-agnostic,
and requires no external dependency.
### Relist period
The shorter the relist period is, the sooner that Kubelet can detect the
change. Shorter relist period also implies higher cpu usage. Moreover, the
relist latency depends on the underlying container runtime, and usually
increases as the number of containers/pods grows. We should set a default
relist period based on measurements. Regardless of what period we set, it will
likely be significantly shorter than the current pod sync period (10s), i.e.,
Kubelet will detect container changes sooner.
## Impact on the Pod Worker Control Flow
Kubelet is responsible for dispatching an event to the appropriate pod
worker based on the pod ID. Only one pod worker would be woken up for
each event.
Today, the pod syncing routine in Kubelet is idempotent as it always
examines the pod state and the spec, and tries to drive to state to
match the spec by performing a series of operations. It should be
noted that this proposal does not intend to change this property --
the sync pod routine would still perform all necessary checks,
regardless of the event type. This trades some efficiency for
reliability and eliminate the need to build a state machine that is
compatible with different runtimes.
## Leverage Upstream Container Events
Instead of relying on relisting, PLEG can leverage other components which
provide container events, and translate these events into pod lifecycle
events. This will further improve Kubelet's responsiveness and reduce the
resource usage caused by frequent relisting.
The upstream container events can come from:
(1). *Event stream provided by each container runtime*
Docker's API exposes an [event
stream](https://docs.docker.com/reference/api/docker_remote_api_v1.17/#monitor-docker-s-events).
Nonetheless, rkt does not support this yet, but they will eventually support it
(see [coreos/rkt#1193](https://github.com/coreos/rkt/issues/1193)).
(2). *cgroups event stream by cAdvisor*
cAdvisor is integrated in Kubelet to provide container stats. It watches cgroups
containers using inotify and exposes an event stream. Even though it does not
support rkt yet, it should be straightforward to add such a support.
Option (1) may provide richer sets of events, but option (2) has the advantage
to be more universal across runtimes, as long as the container runtime uses
cgroups. Regardless of what one chooses to implement now, the container event
stream should be easily swappable with a clearly defined interface.
Note that we cannot solely rely on the upstream container events due to the
possibility of missing events. PLEG should relist infrequently to ensure no
events are missed.
## Generate Expected Events
*This is optional for PLEGs which performs only relisting, but required for
PLEGs that watch upstream events.*
A pod worker's actions could lead to pod lifecycle events (e.g.,
create/kill a container), which the worker would not observe until
later. The pod worker should ignore such events to avoid unnecessary
work.
For example, assume a pod has two containers, A and B. The worker
- Creates container A
- Receives an event `(ContainerStopped, B)`
- Receives an event `(ContainerStarted, A)`
The worker should ignore the `(ContainerStarted, A)` event since it is
expected. Arguably, the worker could process `(ContainerStopped, B)`
as soon as it receives the event, before observing the creation of
A. However, it is desirable to wait until the expected event
`(ContainerStarted, A)` is observed to keep a consistent per-pod view
at the worker. Therefore, the control flow of a single pod worker
should adhere to the following rules:
1. Pod worker should process the events sequentially.
2. Pod worker should not start syncing until it observes the outcome of its own
actions in the last sync to maintain a consistent view.
In other words, a pod worker should record the expected events, and
only wake up to perform the next sync until all expectations are met.
- Creates container A, records an expected event `(ContainerStarted, A)`
- Receives `(ContainerStopped, B)`; stores the event and goes back to sleep.
- Receives `(ContainerStarted, A)`; clears the expectation. Proceeds to handle
`(ContainerStopped, B)`.
We should set an expiration time for each expected events to prevent the worker
from being stalled indefinitely by missing events.
## TODOs for v1.2
For v1.2, we will add a generic PLEG which relists periodically, and leave
adopting container events for future work. We will also *not* implement the
optimization that generate and filters out expected events to minimize
redundant syncs.
- Add a generic PLEG using relisting. Modify the container runtime interface
to provide all necessary information to detect container state changes
in `GetPods()` (#13571).
- Benchmark docker to adjust relising frequency.
- Fix/adapt features that rely on frequent, periodic pod syncing.
* Liveness/Readiness probing: Create a separate probing manager using
explicitly container probing period [#10878](https://issues.k8s.io/10878).
* Instruct pod workers to set up a wake-up call if syncing failed, so that
it can retry.
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/proposals/pod-lifecycle-event-generator.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/design-proposals/pod-lifecycle-event-generator.md](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/pod-lifecycle-event-generator.md)
docs/proposals/release-notes.md
View file @ ed10c233
# Kubernetes Release Notes
[djmm@google.com](mailto:djmm@google.com)<BR>
Last Updated: 2016-04-06
<!-- BEGIN MUNGE: GENERATED_TOC -->
- [Kubernetes Release Notes](#kubernetes-release-notes)
- [Objective](#objective)
- [Background](#background)
- [The Problem](#the-problem)
- [The (general) Solution](#the-general-solution)
- [Then why not just list *every* change that was submitted, CHANGELOG-style?](#then-why-not-just-list-every-change-that-was-submitted-changelog-style)
- [Options](#options)
- [Collection Design](#collection-design)
- [Publishing Design](#publishing-design)
- [Location](#location)
- [Layout](#layout)
- [Alpha/Beta/Patch Releases](#alphabetapatch-releases)
- [Major/Minor Releases](#majorminor-releases)
- [Work estimates](#work-estimates)
- [Caveats / Considerations](#caveats--considerations)
<!-- END MUNGE: GENERATED_TOC -->
## Objective
Define a process and design tooling for collecting, arranging and publishing
release notes for Kubernetes releases, automating as much of the process as
possible.
The goal is to introduce minor changes to the development workflow
in a way that is mostly frictionless and allows for the capture of release notes
as PRs are submitted to the repository.
This direct association of release notes to PRs captures the intention of
release visibility of the PR at the point an idea is submitted upstream.
The release notes can then be more easily collected and published when the
release is ready.
## Background
### The Problem
Release notes are often an afterthought and clarifying and finalizing them
is often left until the very last minute at the time the release is made.
This is usually long after the feature or bug fix was added and is no longer on
the mind of the author. Worse, the collecting and summarizing of the
release is often left to those who may know little or nothing about these
individual changes!
Writing and editing release notes at the end of the cycle can be a rushed,
interrupt-driven and often stressful process resulting in incomplete,
inconsistent release notes often with errors and omissions.
### The (general) Solution
Like most things in the development/release pipeline, the earlier you do it,
the easier it is for everyone and the better the outcome. Gather your release
notes earlier in the development cycle, at the time the features and fixes are
added.
#### Then why not just list *every* change that was submitted, CHANGELOG-style?
On larger projects like Kubernetes, showing every single change (PR) would mean
hundreds of entries. The goal is to highlight the major changes for a release.
## Options
1. Use of pre-commit and other local git hooks
* Experiments here using `prepare-commit-msg` and `commit-msg` git hook files
were promising but less than optimal due to the fact that they would
require input/confirmation with each commit and there may be multiple
commits in a push and eventual PR.
1. Use of [github templates](https://github.com/blog/2111-issue-and-pull-request-templates)
* Templates provide a great way to pre-fill PR comments, but there are no
server-side hooks available to parse and/or easily check the contents of
those templates to ensure that checkboxes were checked or forms were filled
in.
1. Use of labels enforced by mungers/bots
* We already make great use of mungers/bots to manage labels on PRs and it
fits very nicely in the existing workflow
## Collection Design
The munger/bot option fits most cleanly into the existing workflow.
All `release-note-*` labeling is managed on the master branch PR only.
No `release-note-*` labels are needed on cherry-pick PRs and no information
will be collected from that cherry-pick PR.
The only exception to this rule is when a PR is not a cherry-pick and is
targeted directly to the non-master branch. In this case, a `release-note-*`
label is required for that non-master PR.
1. New labels added to github: `release-note-none`, maybe others for new release note categories - see Layout section below
1. A [new munger](https://github.com/kubernetes/kubernetes/issues/23409) that will:
* Add a `release-note-label-needed` label to all new master branch PRs
* Block merge by the submit queue on all PRs labeled as `release-note-label-needed`
* Auto-remove `release-note-label-needed` when one of the `release-note-*` labels is added
## Publishing Design
### Location
With v1.2.0, the release notes were moved from their previous [github releases](https://github.com/kubernetes/kubernetes/releases)
location to [CHANGELOG.md](../../CHANGELOG.md). Going forward this seems like a good plan.
Other projects do similarly.
The kubernetes.tar.gz download link is also displayed along with the release notes
in [CHANGELOG.md](../../CHANGELOG.md).
Is there any reason to continue publishing anything to github releases if
the complete release story is published in [CHANGELOG.md](../../CHANGELOG.md)?
### Layout
Different types of releases will generally have different requirements in
terms of layout. As expected, major releases like v1.2.0 are going
to require much more detail than the automated release notes will provide.
The idea is that these mechanisms will provide 100% of the release note
content for alpha, beta and most minor releases and bootstrap the content
with a release note 'template' for the authors of major releases like v1.2.0.
The authors can then collaborate and edit the higher level sections of the
release notes in a PR, updating [CHANGELOG.md](../../CHANGELOG.md) as needed.
v1.2.0 demonstrated the need, at least for major releases like v1.2.0, for
several sections in the published release notes.
In order to provide a basic layout for release notes in the future,
new releases can bootstrap [CHANGELOG.md](../../CHANGELOG.md) with the following template types:
#### Alpha/Beta/Patch Releases
These are automatically generated from `release-note*` labels, but can be modified as needed.
```
Action Required
* PR titles from the release-note-action-required label
Other notable changes
* PR titles from the release-note label
```
#### Major/Minor Releases
```
Major Themes
* Add to or delete this section
Other notable improvements
* Add to or delete this section
Experimental Features
* Add to or delete this section
Action Required
* PR titles from the release-note-action-required label
Known Issues
* Add to or delete this section
Provider-specific Notes
* Add to or delete this section
Other notable changes
* PR titles from the release-note label
```
## Work estimates
* The [new munger](https://github.com/kubernetes/kubernetes/issues/23409)
* Owner: @eparis
* Time estimate: Mostly done
* Updates to the tool that collects, organizes, publishes and sends release
notifications.
* Owner: @david-mcmahon
* Time estimate: A few days
## Caveats / Considerations
* As part of the planning and development workflow how can we capture
release notes for bigger features?
[#23070](https://github.com/kubernetes/kubernetes/issues/23070)
* For now contributors should simply use the first PR that enables a new
feature by default. We'll revisit if this does not work well.
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/proposals/release-notes.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/design-proposals/release-notes.md](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/release-notes.md)
docs/proposals/rescheduler.md
View file @ ed10c233
# Rescheduler design space
@davidopp, @erictune, @briangrant
July 2015
## Introduction and definition
A rescheduler is an agent that proactively causes currently-running
Pods to be moved, so as to optimize some objective function for
goodness of the layout of Pods in the cluster. (The objective function
doesn't have to be expressed mathematically; it may just be a
collection of ad-hoc rules, but in principle there is an objective
function. Implicitly an objective function is described by the
scheduler's predicate and priority functions.) It might be triggered
to run every N minutes, or whenever some event happens that is known
to make the objective function worse (for example, whenever any Pod goes
PENDING for a long time.)
## Motivation and use cases
A rescheduler is useful because without a rescheduler, scheduling
decisions are only made at the time Pods are created. But later on,
the state of the cell may have changed in some way such that it would
be better to move the Pod to another node.
There are two categories of movements a rescheduler might trigger: coalescing
and spreading.
### Coalesce Pods
This is the most common use case. Cluster layout changes over time. For
example, run-to-completion Pods terminate, producing free space in their wake, but that space
is fragmented. This fragmentation might prevent a PENDING Pod from scheduling
(there are enough free resource for the Pod in aggregate across the cluster,
but not on any single node). A rescheduler can coalesce free space like a
disk defragmenter, thereby producing enough free space on a node for a PENDING
Pod to schedule. In some cases it can do this just by moving Pods into existing
holes, but often it will need to evict (and reschedule) running Pods in order to
create a large enough hole.
A second use case for a rescheduler to coalesce pods is when it becomes possible
to support the running Pods on a fewer number of nodes. The rescheduler can
gradually move Pods off of some set of nodes to make those nodes empty so
that they can then be shut down/removed. More specifically,
the system could do a simulation to see whether after removing a node from the
cluster, will the Pods that were on that node be able to reschedule,
either directly or with the help of the rescheduler; if the answer is
yes, then you can safely auto-scale down (assuming services will still
meeting their application-level SLOs).
### Spread Pods
The main use cases for spreading Pods revolve around relieving congestion on (a) highly
utilized node(s). For example, some process might suddenly start receiving a significantly
above-normal amount of external requests, leading to starvation of best-effort
Pods on the node. We can use the rescheduler to move the best-effort Pods off of the
node. (They are likely to have generous eviction SLOs, so are more likely to be movable
than the Pod that is experiencing the higher load, but in principle we might move either.)
Or even before any node becomes overloaded, we might proactively re-spread Pods from nodes
with high-utilization, to give them some buffer against future utilization spikes. In either
case, the nodes we move the Pods onto might have been in the system for a long time or might
have been added by the cluster auto-scaler specifically to allow the rescheduler to
rebalance utilization.
A second spreading use case is to separate antagonists.
Sometimes the processes running in two different Pods on the same node
may have unexpected antagonistic
behavior towards one another. A system component might monitor for such
antagonism and ask the rescheduler to move one of the antagonists to a new node.
### Ranking the use cases
The vast majority of users probably only care about rescheduling for three scenarios:
1. Move Pods around to get a PENDING Pod to schedule
1. Redistribute Pods onto new nodes added by a cluster auto-scaler when there are no PENDING Pods
1. Move Pods around when CPU starvation is detected on a node
## Design considerations and design space
Because rescheduling is disruptive--it causes one or more
already-running Pods to die when they otherwise wouldn't--a key
constraint on rescheduling is that it must be done subject to
disruption SLOs. There are a number of ways to specify these SLOs--a
global rate limit across all Pods, a rate limit across a set of Pods
defined by some particular label selector, a maximum number of Pods
that can be down at any one time among a set defined by some
particular label selector, etc. These policies are presumably part of
the Rescheduler's configuration.
There are a lot of design possibilities for a rescheduler. To explain
them, it's easiest to start with the description of a baseline
rescheduler, and then describe possible modifications. The Baseline
rescheduler
* only kicks in when there are one or more PENDING Pods for some period of time; its objective function is binary: completely happy if there are no PENDING Pods, and completely unhappy if there are PENDING Pods; it does not try to optimize for any other aspect of cluster layout
* is not a scheduler -- it simply identifies a node where a PENDING Pod could fit if one or more Pods on that node were moved out of the way, and then kills those Pods to make room for the PENDING Pod, which will then be scheduled there by the regular scheduler(s). [obviously this killing operation must be able to specify "don't allow the killed Pod to reschedule back to whence it was killed" otherwise the killing is pointless] Of course it should only do this if it is sure the killed Pods will be able to reschedule into already-free space in the cluster. Note that although it is not a scheduler, the Rescheduler needs to be linked with the predicate functions of the scheduling algorithm(s) so that it can know (1) that the PENDING Pod would actually schedule into the hole it has identified once the hole is created, and (2) that the evicted Pod(s) will be able to schedule somewhere else in the cluster.
Possible variations on this Baseline rescheduler are
1. it can kill the Pod(s) whose space it wants **and also schedule the Pod that will take that space and reschedule the Pod(s) that were killed**, rather than just killing the Pod(s) whose space it wants and relying on the regular scheduler(s) to schedule the Pod that will take that space (and to reschedule the Pod(s) that were evicted)
1. it can run continuously in the background to optimize general cluster layout instead of just trying to get a PENDING Pod to schedule
1. it can try to move groups of Pods instead of using a one-at-a-time / greedy approach
1. it can formulate multi-hop plans instead of single-hop
A key design question for a Rescheduler is how much knowledge it needs about the scheduling policies used by the cluster's scheduler(s).
* For the Baseline rescheduler, it needs to know the predicate functions used by the cluster's scheduler(s) else it can't know how to create a hole that the PENDING Pod will fit into, nor be sure that the evicted Pod(s) will be able to reschedule elsewhere.
* If it is going to run continuously in the background to optimize cluster layout but is still only going to kill Pods, then it still needs to know the predicate functions for the reason mentioned above. In principle it doesn't need to know the priority functions; it could just randomly kill Pods and rely on the regular scheduler to put them back in better places. However, this is a rather inexact approach. Thus it is useful for the rescheduler to know the priority functions, or at least some subset of them, so it can be sure that an action it takes will actually improve the cluster layout.
* If it is going to run continuously in the background to optimize cluster layout and is going to act as a scheduler rather than just killing Pods, then it needs to know the predicate functions and some compatible (but not necessarily identical) priority functions One example of a case where "compatible but not identical" might be useful is if the main scheduler(s) has a very simple scheduling policy optimized for low scheduling latency, and the Rescheduler having a more sophisticated/optimal scheduling policy that requires more computation time. The main thing to avoid is for the scheduler(s) and rescheduler to have incompatible priority functions, as this will cause them to "fight" (though it still can't lead to an infinite loop, since the scheduler(s) only ever touches a Pod once).
## Appendix: Integrating rescheduler with cluster auto-scaler (scale up)
For scaling up the cluster, a reasonable workflow might be:
1. pod horizontal auto-scaler decides to add one or more Pods to a service, based on the metrics it is observing
1. the Pod goes PENDING due to lack of a suitable node with sufficient resources
1. rescheduler notices the PENDING Pod and determines that the Pod cannot schedule just by rearranging existing Pods (while respecting SLOs)
1. rescheduler triggers cluster auto-scaler to add a node of the appropriate type for the PENDING Pod
1. the PENDING Pod schedules onto the new node (and possibly the rescheduler also moves other Pods onto that node)
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/proposals/rescheduler.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/design-proposals/rescheduler.md](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/rescheduler.md)
docs/proposals/rescheduling-for-critical-pods.md
View file @ ed10c233
# Rescheduler: guaranteed scheduling of critical addons
## Motivation
In addition to Kubernetes core components like api-server, scheduler, controller-manager running on a master machine
there is a bunch of addons which due to various reasons have to run on a regular cluster node, not the master.
Some of them are critical to have fully functional cluster: Heapster, DNS, UI. Users can break their cluster
by evicting a critical addon (either manually or as a side effect of an other operation like upgrade)
which possibly can become pending (for example when the cluster is highly utilized).
To avoid such situation we want to have a mechanism which guarantees that
critical addons are scheduled assuming the cluster is big enough.
This possibly may affect other pods (including production user’s applications).
## Design
Rescheduler will ensure that critical addons are always scheduled.
In the first version it will implement only this policy, but later we may want to introduce other policies.
It will be a standalone component running on master machine similarly to scheduler.
Those components will share common logic (initially rescheduler will in fact import some of scheduler packages).
### Guaranteed scheduling of critical addons
Rescheduler will observe critical addons
(with annotation `scheduler.alpha.kubernetes.io/critical-pod`).
If one of them is marked by scheduler as unschedulable (pod condition `PodScheduled` set to `false`, the reason set to `Unschedulable`)
the component will try to find a space for the addon by evicting some pods and then the scheduler will schedule the addon.
#### Scoring nodes
Initially we want to choose a random node with enough capacity
(chosen as described in [Evicting pods](rescheduling-for-critical-pods.md#evicting-pods)) to schedule given addons.
Later we may want to introduce some heuristic:
* minimize number of evicted pods with violation of disruption budget or shortened termination grace period
* minimize number of affected pods by choosing a node on which we have to evict less pods
* increase probability of scheduling of evicted pods by preferring a set of pods with the smallest total sum of requests
* avoid nodes which are ‘non-drainable’ (according to drain logic), for example on which there is a pod which doesn’t belong to any RC/RS/Deployment
#### Evicting pods
There are 2 mechanism which possibly can delay a pod eviction: Disruption Budget and Termination Grace Period.
While removing a pod we will try to avoid violating Disruption Budget, though we can’t guarantee it
since there is a chance that it would block this operation for longer period of time.
We will also try to respect Termination Grace Period, though without any guarantee.
In case we have to remove a pod with termination grace period longer than 10s it will be shortened to 10s.
The proposed order while choosing a node to schedule a critical addon and pods to remove:
1. a node where the critical addon pod can fit after evicting only pods satisfying both
(1) their disruption budget will not be violated by such eviction and (2) they have grace period <= 10 seconds
1. a node where the critical addon pod can fit after evicting only pods whose disruption budget will not be violated by such eviction
1. any node where the critical addon pod can fit after evicting some pods
### Interaction with Scheduler
To avoid situation when Scheduler will schedule another pod into the space prepared for the critical addon,
the chosen node has to be temporarily excluded from a list of nodes considered by Scheduler while making decisions.
For this purpose the node will get a temporary
[Taint](../../docs/design/taint-toleration-dedicated.md) “CriticalAddonsOnly”
and each critical addon has to have defined toleration for this taint.
After Rescheduler has no more work to do: all critical addons are scheduled or cluster is too small for them,
all taints will be removed.
### Interaction with Cluster Autoscaler
Rescheduler possibly can duplicate the responsibility of Cluster Autoscaler:
both components are taking action when there is unschedulable pod.
It may cause the situation when CA will add extra node for a pending critical addon
and Rescheduler will evict some running pods to make a space for the addon.
This situation would be rare and usually an extra node would be anyway needed for evicted pods.
In the worst case CA will add and then remove the node.
To not complicate architecture by introducing interaction between those 2 components we accept this overlap.
We want to ensure that CA won’t remove nodes with critical addons by adding appropriate logic there.
### Rescheduler control loop
The rescheduler control loop will be as follow:
* while there is an unschedulable critical addon do the following:
* choose a node on which the addon should be scheduled (as described in Evicting pods)
* add taint to the node to prevent scheduler from using it
* delete pods which blocks the addon from being scheduled
* wait until scheduler will schedule the critical addon
* if there is no more critical addons for which we can help, ensure there is no node with the taint
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/proposals/rescheduling-for-critical-pods.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/design-proposals/rescheduling-for-critical-pods.md](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/rescheduling-for-critical-pods.md)
docs/proposals/resource-metrics-api.md
View file @ ed10c233
# Resource Metrics API
This document describes API part of MVP version of Resource Metrics API effort in Kubernetes.
Once the agreement will be made the document will be extended to also cover implementation details.
The shape of the effort may be also a subject of changes once we will have more well-defined use cases.
## Goal
The goal for the effort is to provide resource usage metrics for pods and nodes through the API server.
This will be a stable, versioned API which core Kubernetes components can rely on.
In the first version only the well-defined use cases will be handled,
although the API should be easily extensible for potential future use cases.
## Main use cases
This section describes well-defined use cases which should be handled in the first version.
Use cases which are not listed below are out of the scope of MVP version of Resource Metrics API.
#### Horizontal Pod Autoscaler
HPA uses the latest value of cpu usage as an average aggregated across 1 minute
(the window may change in the future). The data for a given set of pods
(defined either by pod list or label selector) should be accesible in one request
due to performance issues.
#### Scheduler
Scheduler in order to schedule best-effort pods requires node level resource usage metrics
as an average aggregated across 1 minute (the window may change in the future).
The metrics should be available for all resources supported in the scheduler.
Currently the scheduler does not need this information, because it schedules best-effort pods
without considering node usage. But having the metrics available in the API server is a blocker
for adding the ability to take node usage into account when scheduling best-effort pods.
## Other considered use cases
This section describes the other considered use cases and explains why they are out
of the scope of the MVP version.
#### Custom metrics in HPA
HPA requires the latest value of application level metrics.
The design of the pipeline for collecting application level metrics should
be revisited and it's not clear whether application level metrics should be
available in API server so the use case initially won't be supported.
#### Cluster Federation
The Cluster Federation control system might want to consider cluster-level usage (in addition to cluster-level request)
of running pods when choosing where to schedule new pods. Although
Cluster Federation is still in design,
we expect the metrics API described here to be sufficient. Cluster-level usage can be
obtained by summing over usage of all nodes in the cluster.
#### kubectl top
This feature is not yet specified/implemented although it seems reasonable to provide users information
about resource usage on pod/node level.
Since this feature has not been fully specified yet it will be not supported initially in the API although
it will be probably possible to provide a reasonable implementation of the feature anyway.
#### Kubernetes dashboard
[Kubernetes dashboard](https://github.com/kubernetes/dashboard) in order to draw graphs requires resource usage
in timeseries format from relatively long period of time. The aggregations should be also possible on various levels
including replication controllers, deployments, services, etc.
Since the use case is complicated it will not be supported initially in the API and they will query Heapster
directly using some custom API there.
## Proposed API
Initially the metrics API will be in a separate [API group](api-group.md) called ```metrics```.
Later if we decided to have Node and Pod in different API groups also
NodeMetrics and PodMetrics should be in different API groups.
#### Schema
The proposed schema is as follow. Each top-level object has `TypeMeta` and `ObjectMeta` fields
to be compatible with Kubernetes API standards.
```go
type NodeMetrics struct {
unversioned.TypeMeta
ObjectMeta
// The following fields define time interval from which metrics were
// collected in the following format [Timestamp-Window, Timestamp].
Timestamp unversioned.Time
Window unversioned.Duration
// The memory usage is the memory working set.
Usage v1.ResourceList
}
type PodMetrics struct {
unversioned.TypeMeta
ObjectMeta
// The following fields define time interval from which metrics were
// collected in the following format [Timestamp-Window, Timestamp].
Timestamp unversioned.Time
Window unversioned.Duration
// Metrics for all containers are collected within the same time window.
Containers []ContainerMetrics
}
type ContainerMetrics struct {
// Container name corresponding to the one from v1.Pod.Spec.Containers.
Name string
// The memory usage is the memory working set.
Usage v1.ResourceList
}
```
By default `Usage` is the mean from samples collected within the returned time window.
The default time window is 1 minute.
#### Endpoints
All endpoints are GET endpoints, rooted at `/apis/metrics/v1alpha1/`.
There won't be support for the other REST methods.
The list of supported endpoints:
- `/nodes` - all node metrics; type `[]NodeMetrics`
- `/nodes/{node}` - metrics for a specified node; type `NodeMetrics`
- `/namespaces/{namespace}/pods` - all pod metrics within namespace with support for `all-namespaces`; type `[]PodMetrics`
- `/namespaces/{namespace}/pods/{pod}` - metrics for a specified pod; type `PodMetrics`
The following query parameters are supported:
- `labelSelector` - restrict the list of returned objects by labels (list endpoints only)
In the future we may want to introduce the following params:
`aggregator` (`max`, `min`, `95th`, etc.) and `window` (`1h`, `1d`, `1w`, etc.)
which will allow to get the other aggregates over the custom time window.
## Further improvements
Depending on the further requirements the following features may be added:
- support for more metrics
- support for application level metrics
- watch for metrics
- possibility to query for window sizes and aggregation functions (though single window size/aggregation function per request)
- cluster level metrics
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/proposals/resource-metrics-api.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/design-proposals/resource-metrics-api.md](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/resource-metrics-api.md)
docs/proposals/runtime-pod-cache.md
View file @ ed10c233
# Kubelet: Runtime Pod Cache
This proposal builds on top of the Pod Lifecycle Event Generator (PLEG) proposed
in [#12802](https://issues.k8s.io/12802). It assumes that Kubelet subscribes to
the pod lifecycle event stream to eliminate periodic polling of pod
states. Please see [#12802](https://issues.k8s.io/12802). for the motivation and
design concept for PLEG.
Runtime pod cache is an in-memory cache which stores the *status* of
all pods, and is maintained by PLEG. It serves as a single source of
truth for internal pod status, freeing Kubelet from querying the
container runtime.
## Motivation
With PLEG, Kubelet no longer needs to perform comprehensive state
checking for all pods periodically. It only instructs a pod worker to
start syncing when there is a change of its pod status. Nevertheless,
during each sync, a pod worker still needs to construct the pod status
by examining all containers (whether dead or alive) in the pod, due to
the lack of the caching of previous states. With the integration of
pod cache, we can further improve Kubelet's CPU usage by
1. Lowering the number of concurrent requests to the container
runtime since pod workers no longer have to query the runtime
individually.
2. Lowering the total number of inspect requests because there is no
need to inspect containers with no state changes.
***Don't we already have a [container runtime cache]
(https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/container/runtime_cache.go)?***
The runtime cache is an optimization that reduces the number of `GetPods()`
calls from the workers. However,
* The cache does not store all information necessary for a worker to
complete a sync (e.g., `docker inspect`); workers still need to inspect
containers individually to generate `api.PodStatus`.
* Workers sometimes need to bypass the cache in order to retrieve the
latest pod state.
This proposal generalizes the cache and instructs PLEG to populate the cache, so
that the content is always up-to-date.
**Why can't each worker cache its own pod status?**
The short answer is yes, they can. The longer answer is that localized
caching limits the use of the cache content -- other components cannot
access it. This often leads to caching at multiple places and/or passing
objects around, complicating the control flow.
## Runtime Pod Cache
![pod cache](pod-cache.png)
Pod cache stores the `PodStatus` for all pods on the node. `PodStatus` encompasses
all the information required from the container runtime to generate
`api.PodStatus` for a pod.
```go
// PodStatus represents the status of the pod and its containers.
// api.PodStatus can be derived from examining PodStatus and api.Pod.
type PodStatus struct {
ID types.UID
Name string
Namespace string
IP string
ContainerStatuses []*ContainerStatus
}
// ContainerStatus represents the status of a container.
type ContainerStatus struct {
ID ContainerID
Name string
State ContainerState
CreatedAt time.Time
StartedAt time.Time
FinishedAt time.Time
ExitCode int
Image string
ImageID string
Hash uint64
RestartCount int
Reason string
Message string
}
```
`PodStatus` is defined in the container runtime interface, hence is
runtime-agnostic.
PLEG is responsible for updating the entries pod cache, hence always keeping
the cache up-to-date.
1. Detect change of container state
2. Inspect the pod for details
3. Update the pod cache with the new PodStatus
- If there is no real change of the pod entry, do nothing
- Otherwise, generate and send out the corresponding pod lifecycle event
Note that in (3), PLEG can check if there is any disparity between the old
and the new pod entry to filter out duplicated events if needed.
### Evict cache entries
Note that the cache represents all the pods/containers known by the container
runtime. A cache entry should only be evicted if the pod is no longer visible
by the container runtime. PLEG is responsible for deleting entries in the
cache.
### Generate `api.PodStatus`
Because pod cache stores the up-to-date `PodStatus` of the pods, Kubelet can
generate the `api.PodStatus` by interpreting the cache entry at any
time. To avoid sending intermediate status (e.g., while a pod worker
is restarting a container), we will instruct the pod worker to generate a new
status at the beginning of each sync.
### Cache contention
Cache contention should not be a problem when the number of pods is
small. When Kubelet scales, we can always shard the pods by ID to
reduce contention.
### Disk management
The pod cache is not capable to fulfill the needs of container/image garbage
collectors as they may demand more than pod-level information. These components
will still need to query the container runtime directly at times. We may
consider extending the cache for these use cases, but they are beyond the scope
of this proposal.
## Impact on Pod Worker Control Flow
A pod worker may perform various operations (e.g., start/kill a container)
during a sync. They will expect to see the results of such operations reflected
in the cache in the next sync. Alternately, they can bypass the cache and
query the container runtime directly to get the latest status. However, this
is not desirable since the cache is introduced exactly to eliminate unnecessary,
concurrent queries. Therefore, a pod worker should be blocked until all expected
results have been updated to the cache by PLEG.
Depending on the type of PLEG (see [#12802](https://issues.k8s.io/12802)) in
use, the methods to check whether a requirement is met can differ. For a
PLEG that solely relies on relisting, a pod worker can simply wait until the
relist timestamp is newer than the end of the worker's last sync. On the other
hand, if pod worker knows what events to expect, they can also block until the
events are observed.
It should be noted that `api.PodStatus` will only be generated by the pod
worker *after* the cache has been updated. This means that the perceived
responsiveness of Kubelet (from querying the API server) will be affected by
how soon the cache can be populated. For the pure-relisting PLEG, the relist
period can become the bottleneck. On the other hand, A PLEG which watches the
upstream event stream (and knows how what events to expect) is not restricted
by such periods and should improve Kubelet's perceived responsiveness.
## TODOs for v1.2
- Redefine container runtime types ([#12619](https://issues.k8s.io/12619)):
and introduce `PodStatus`. Refactor dockertools and rkt to use the new type.
- Add cache and instruct PLEG to populate it.
- Refactor Kubelet to use the cache.
- Deprecate the old runtime cache.
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/proposals/runtime-pod-cache.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/design-proposals/runtime-pod-cache.md](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/runtime-pod-cache.md)
docs/proposals/runtimeconfig.md
View file @ ed10c233
# Overview
Proposes adding a `--feature-config` to core kube system components:
apiserver , scheduler, controller-manager, kube-proxy, and selected addons.
This flag will be used to enable/disable alpha features on a per-component basis.
## Motivation
Motivation is enabling/disabling features that are not tied to
an API group. API groups can be selectively enabled/disabled in the
apiserver via existing `--runtime-config` flag on apiserver, but there is
currently no mechanism to toggle alpha features that are controlled by
e.g. annotations. This means the burden of controlling whether such
features are enabled in a particular cluster is on feature implementors;
they must either define some ad hoc mechanism for toggling (e.g. flag
on component binary) or else toggle the feature on/off at compile time.
By adding a`--feature-config` to all kube-system components, alpha features
can be toggled on a per-component basis by passing `enableAlphaFeature=true|false`
to `--feature-config` for each component that the feature touches.
## Design
The following components will all get a `--feature-config` flag,
which loads a `config.ConfigurationMap`:
- kube-apiserver
- kube-scheduler
- kube-controller-manager
- kube-proxy
- kube-dns
(Note kubelet is omitted, it's dynamic config story is being addressed
by #29459). Alpha features that are not accessed via an alpha API
group should define an `enableFeatureName` flag and use it to toggle
activation of the feature in each system component that the feature
uses.
## Suggested conventions
This proposal only covers adding a mechanism to toggle features in
system components. Implementation details will still depend on the alpha
feature's owner(s). The following are suggested conventions:
- Naming for feature config entries should follow the pattern
"enable<FeatureName>=true".
- Features that touch multiple components should reserve the same key
in each component to toggle on/off.
- Alpha features should be disabled by default. Beta features may
be enabled by default. Refer to docs/devel/api_changes.md#alpha-beta-and-stable-versions
for more detailed guidance on alpha vs. beta.
## Upgrade support
As the primary motivation for cluster config is toggling alpha
features, upgrade support is not in scope. Enabling or disabling
a feature is necessarily a breaking change, so config should
not be altered in a running cluster.
## Future work
1. The eventual plan is for component config to be managed by versioned
APIs and not flags (#12245). When that is added, toggling of features
could be handled by versioned component config and the component flags
deprecated.
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/proposals/runtimeconfig.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/design-proposals/runtimeconfig.md](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/runtimeconfig.md)
docs/proposals/scalability-testing.md
View file @ ed10c233
## Background
We have a goal to be able to scale to 1000-node clusters by end of 2015.
As a result, we need to be able to run some kind of regression tests and deliver
a mechanism so that developers can test their changes with respect to performance.
Ideally, we would like to run performance tests also on PRs - although it might
be impossible to run them on every single PR, we may introduce a possibility for
a reviewer to trigger them if the change has non obvious impact on the performance
(something like "k8s-bot run scalability tests please" should be feasible).
However, running performance tests on 1000-node clusters (or even bigger in the
future is) is a non-starter. Thus, we need some more sophisticated infrastructure
to simulate big clusters on relatively small number of machines and/or cores.
This document describes two approaches to tackling this problem.
Once we have a better understanding of their consequences, we may want to
decide to drop one of them, but we are not yet in that position.
## Proposal 1 - Kubmark
In this proposal we are focusing on scalability testing of master components.
We do NOT focus on node-scalability - this issue should be handled separately.
Since we do not focus on the node performance, we don't need real Kubelet nor
KubeProxy - in fact we don't even need to start real containers.
All we actually need is to have some Kubelet-like and KubeProxy-like components
that will be simulating the load on apiserver that their real equivalents are
generating (e.g. sending NodeStatus updated, watching for pods, watching for
endpoints (KubeProxy), etc.).
What needs to be done:
1. Determine what requests both KubeProxy and Kubelet are sending to apiserver.
2. Create a KubeletSim that is generating the same load on apiserver that the
real Kubelet, but is not starting any containers. In the initial version we
can assume that pods never die, so it is enough to just react on the state
changes read from apiserver.
TBD: Maybe we can reuse a real Kubelet for it by just injecting some "fake"
interfaces to it?
3. Similarly create a KubeProxySim that is generating the same load on apiserver
as a real KubeProxy. Again, since we are not planning to talk to those
containers, it basically doesn't need to do anything apart from that.
TBD: Maybe we can reuse a real KubeProxy for it by just injecting some "fake"
interfaces to it?
4. Refactor kube-up/kube-down scripts (or create new ones) to allow starting
a cluster with KubeletSim and KubeProxySim instead of real ones and put
a bunch of them on a single machine.
5. Create a load generator for it (probably initially it would be enough to
reuse tests that we use in gce-scalability suite).
## Proposal 2 - Oversubscribing
The other method we are proposing is to oversubscribe the resource,
or in essence enable a single node to look like many separate nodes even though
they reside on a single host. This is a well established pattern in many different
cluster managers (for more details see
http://www.uscms.org/SoftwareComputing/Grid/WMS/glideinWMS/doc.prd/index.html ).
There are a couple of different ways to accomplish this, but the most viable method
is to run privileged kubelet pods under a hosts kubelet process. These pods then
register back with the master via the introspective service using modified names
as not to collide.
Complications may currently exist around container tracking and ownership in docker.
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/proposals/scalability-testing.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/design-proposals/scalability-testing.md](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/scalability-testing.md)
docs/proposals/secret-configmap-downwarapi-file-mode.md
View file @ ed10c233
# Secrets, configmaps and downwardAPI file mode bits
Author: Rodrigo Campos (@rata), Tim Hockin (@thockin)
Date: July 2016
Status: Design in progress
# Goal
Allow users to specify permission mode bits for a secret/configmap/downwardAPI
file mounted as a volume. For example, if a secret has several keys, a user
should be able to specify the permission mode bits for any file, and they may
all have different modes.
Let me say that with "permission" I only refer to the file mode here and I may
use them interchangeably. This is not about the file owners, although let me
know if you prefer to discuss that here too.
# Motivation
There is currently no way to set permissions on secret files mounted as volumes.
This can be a problem for applications that enforce files to have permissions
only for the owner (like fetchmail, ssh, pgpass file in postgres[1], etc.) and
it's just not possible to run them without changing the file mode. Also,
in-house applications may have this restriction too.
It doesn't seem totally wrong if someone wants to make a secret, that is
sensitive information, not world-readable (or group, too) as it is by default.
Although it's already in a container that is (hopefully) running only one
process and it might not be so bad. But people running more than one process in
a container asked for this too[2].
For example, my use case is that we are migrating to kubernetes, the migration
is in progress (and will take a while) and we have migrated our deployment web
interface to kubernetes. But this interface connects to the servers via ssh, so
it needs the ssh keys, and ssh will only work if the ssh key file mode is the
one it expects.
This was asked on the mailing list here[2] and here[3], too.
[1]: https://www.postgresql.org/docs/9.1/static/libpq-pgpass.html
[2]: https://groups.google.com/forum/#!topic/kubernetes-dev/eTnfMJSqmaM
[3]: https://groups.google.com/forum/#!topic/google-containers/EcaOPq4M758
# Alternatives considered
Several alternatives have been considered:
* Add a mode to the API definition when using secrets: this is backward
compatible as described in (docs/devel/api_changes.md) IIUC and seems like the
way to go. Also @thockin said in the ML that he would consider such an
approach. But it might be worth to consider if we want to do the same for
configmaps or owners, but there is no need to do it now either.
* Change the default file mode for secrets: I think this is unacceptable as it
is stated in the api_changes doc. And besides it doesn't feel correct IMHO, it
is technically one option. The argument for this might be that world and group
readable for a secret is not a nice default, we already take care of not
writing it to disk, etc. but the file is created world-readable anyways. Such a
default change has been done recently: the default was 0444 in kubernetes <= 1.2
and is now 0644 in kubernetes >= 1.3 (and the file is not a regular file,
it's a symlink now). This change was done here to minimize differences between
configmaps and secrets: https://github.com/kubernetes/kubernetes/pull/25285. But
doing it again, and changing to something more restrictive (now is 0644 and it
should be 0400 to work with ssh and most apps) seems too risky, it's even more
restrictive than in k8s 1.2. Specially if there is no way to revert to the old
permissions and some use case is broken by this. And if we are adding a way to
change it, like in the option above, there is no need to rush changing the
default. So I would discard this.
* We don't want to people be able to change this, at least for now, and the
ones who do, suggest that do it as a "postStart" command. This is acceptable
if we don't want to change kubernetes core for some reason, although there
seem to be valid use cases. But if the user want's to use the "postStart" for
something else, then it is more disturbing to do both things (have a script
in the docker image that deals with this, but is not probably concern of the
project so it's not nice, or specify several commands by using "sh").
# Proposed implementation
The proposed implementation goes with the first alternative: adding a `mode`
to the API.
There will be a `defaultMode`, type `int`, in: `type SecretVolumeSource`, `type
ConfigMapVolumeSource` and `type DownwardAPIVolumeSource`. And a `mode`, type
`int` too, in `type KeyToPath` and `DownwardAPIVolumeFile`.
The mask provided in any of these fields will be ANDed with 0777 to disallow
setting sticky and setuid bits. It's not clear that use case is needed nor
really understood. And directories within the volume will be created as before
and are not affected by this setting.
In other words, the fields will look like this:
```
type SecretVolumeSource struct {
// Name of the secret in the pod's namespace to use.
SecretName string `json:"secretName,omitempty"`
// If unspecified, each key-value pair in the Data field of the referenced
// Secret will be projected into the volume as a file whose name is the
// key and content is the value. If specified, the listed keys will be
// projected into the specified paths, and unlisted keys will not be
// present. If a key is specified which is not present in the Secret,
// the volume setup will error. Paths must be relative and may not contain
// the '..' path or start with '..'.
Items []KeyToPath `json:"items,omitempty"`
// Mode bits to use on created files by default. The used mode bits will
// be the provided AND 0777.
// Directories within the path are not affected by this setting
DefaultMode int32 `json:"defaultMode,omitempty"`
}
type ConfigMapVolumeSource struct {
LocalObjectReference `json:",inline"`
// If unspecified, each key-value pair in the Data field of the referenced
// ConfigMap will be projected into the volume as a file whose name is the
// key and content is the value. If specified, the listed keys will be
// projected into the specified paths, and unlisted keys will not be
// present. If a key is specified which is not present in the ConfigMap,
// the volume setup will error. Paths must be relative and may not contain
// the '..' path or start with '..'.
Items []KeyToPath `json:"items,omitempty"`
// Mode bits to use on created files by default. The used mode bits will
// be the provided AND 0777.
// Directories within the path are not affected by this setting
DefaultMode int32 `json:"defaultMode,omitempty"`
}
type KeyToPath struct {
// The key to project.
Key string `json:"key"`
// The relative path of the file to map the key to.
// May not be an absolute path.
// May not contain the path element '..'.
// May not start with the string '..'.
Path string `json:"path"`
// Mode bits to use on this file. The used mode bits will be the
// provided AND 0777.
Mode int32 `json:"mode,omitempty"`
}
type DownwardAPIVolumeSource struct {
// Items is a list of DownwardAPIVolume file
Items []DownwardAPIVolumeFile `json:"items,omitempty"`
// Mode bits to use on created files by default. The used mode bits will
// be the provided AND 0777.
// Directories within the path are not affected by this setting
DefaultMode int32 `json:"defaultMode,omitempty"`
}
type DownwardAPIVolumeFile struct {
// Required: Path is the relative path name of the file to be created. Must not be absolute or contain the '..' path. Must be utf-8 encoded. The first item of the relative path must not start with '..'
Path string `json:"path"`
// Required: Selects a field of the pod: only annotations, labels, name and namespace are supported.
FieldRef *ObjectFieldSelector `json:"fieldRef,omitempty"`
// Selects a resource of the container: only resources limits and requests
// (limits.cpu, limits.memory, requests.cpu and requests.memory) are currently supported.
ResourceFieldRef *ResourceFieldSelector `json:"resourceFieldRef,omitempty"`
// Mode bits to use on this file. The used mode bits will be the
// provided AND 0777.
Mode int32 `json:"mode,omitempty"`
}
```
Adding it there allows the user to change the mode bits of every file in the
object, so it achieves the goal, while having the option to have a default and
not specify all files in the object.
The are two downside:
* The files are symlinks pointint to the real file, and the realfile
permissions are only set. The symlink has the classic symlink permissions.
This is something already present in 1.3, and it seems applications like ssh
work just fine with that. Something worth mentioning, but doesn't seem to be
an issue.
* If the secret/configMap/downwardAPI is mounted in more than one container,
the file permissions will be the same on all. This is already the case for
Key mappings and doesn't seem like a big issue either.
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/proposals/secret-configmap-downwarapi-file-mode.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/design-proposals/secret-configmap-downwarapi-file-mode.md](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/secret-configmap-downwarapi-file-mode.md)
docs/proposals/self-hosted-kubelet.md
View file @ ed10c233
# Proposal: Self-hosted kubelet
## Abstract
In a self-hosted Kubernetes deployment (see [this
comment](https://github.com/kubernetes/kubernetes/issues/246#issuecomment-64533959)
for background on self hosted kubernetes), we have the initial bootstrap problem.
When running self-hosted components, there needs to be a mechanism for pivoting
from the initial bootstrap state to the kubernetes-managed (self-hosted) state.
In the case of a self-hosted kubelet, this means pivoting from the initial
kubelet defined and run on the host, to the kubelet pod which has been scheduled
to the node.
This proposal presents a solution to the kubelet bootstrap, and assumes a
functioning control plane (e.g. an apiserver, controller-manager, scheduler, and
etcd cluster), and a kubelet that can securely contact the API server. This
functioning control plane can be temporary, and not necessarily the "production"
control plane that will be used after the initial pivot / bootstrap.
## Background and Motivation
In order to understand the goals of this proposal, one must understand what
"self-hosted" means. This proposal defines "self-hosted" as a kubernetes cluster
that is installed and managed by the kubernetes installation itself. This means
that each kubernetes component is described by a kubernetes manifest (Daemonset,
Deployment, etc) and can be updated via kubernetes.
The overall goal of this proposal is to make kubernetes easier to install and
upgrade. We can then treat kubernetes itself just like any other application
hosted in a kubernetes cluster, and have access to easy upgrades, monitoring,
and durability for core kubernetes components themselves.
We intend to achieve this by using kubernetes to manage itself. However, in
order to do that we must first "bootstrap" the cluster, by using kubernetes to
install kubernetes components. This is where this proposal fits in, by
describing the necessary modifications, and required procedures, needed to run a
self-hosted kubelet.
The approach being proposed for a self-hosted kubelet is a "pivot" style
installation. This procedure assumes a short-lived “bootstrap” kubelet will run
and start a long-running “self-hosted” kubelet. Once the self-hosted kubelet is
running the bootstrap kubelet will exit. As part of this, we propose introducing
a new `--bootstrap` flag to the kubelet. The behaviour of that flag will be
explained in detail below.
## Proposal
We propose adding a new flag to the kubelet, the `--bootstrap` flag, which is
assumed to be used in conjunction with the `--lock-file` flag. The `--lock-file`
flag is used to ensure only a single kubelet is running at any given time during
this pivot process. When the `--bootstrap` flag is provided, after the kubelet
acquires the file lock, it will begin asynchronously waiting on
[inotify](http://man7.org/linux/man-pages/man7/inotify.7.html) events. Once an
"open" event is received, the kubelet will assume another kubelet is attempting
to take control and will exit by calling `exit(0)`.
Thus, the initial bootstrap becomes:
1. "bootstrap" kubelet is started by $init system.
1. "bootstrap" kubelet pulls down "self-hosted" kubelet as a pod from a
daemonset
1. "self-hosted" kubelet attempts to acquire the file lock, causing "bootstrap"
kubelet to exit
1. "self-hosted" kubelet acquires lock and takes over
1. "bootstrap" kubelet is restarted by $init system and blocks on acquiring the
file lock
During an upgrade of the kubelet, for simplicity we will consider 3 kubelets,
namely "bootstrap", "v1", and "v2". We imagine the following scenario for
upgrades:
1. Cluster administrator introduces "v2" kubelet daemonset
1. "v1" kubelet pulls down and starts "v2"
1. Cluster administrator removes "v1" kubelet daemonset
1. "v1" kubelet is killed
1. Both "bootstrap" and "v2" kubelets race for file lock
1. If "v2" kubelet acquires lock, process has completed
1. If "bootstrap" kubelet acquires lock, it is assumed that "v2" kubelet will
fail a health check and be killed. Once restarted, it will try to acquire the
lock, triggering the "bootstrap" kubelet to exit.
Alternatively, it would also be possible via this mechanism to delete the "v1"
daemonset first, allow the "bootstrap" kubelet to take over, and then introduce
the "v2" kubelet daemonset, effectively eliminating the race between "bootstrap"
and "v2" for lock acquisition, and the reliance on the failing health check
procedure.
Eventually this could be handled by a DaemonSet upgrade policy.
This will allow a "self-hosted" kubelet with minimal new concepts introduced
into the core Kubernetes code base, and remains flexible enough to work well
with future [bootstrapping
services](https://github.com/kubernetes/kubernetes/issues/5754).
## Production readiness considerations / Out of scope issues
* Deterministically pulling and running kubelet pod: we would prefer not to have
to loop until we finally get a kubelet pod.
* It is possible that the bootstrap kubelet version is incompatible with the
newer versions that were run in the node. For example, the cgroup
configurations might be incompatible. In the beginning, we will require
cluster admins to keep the configuration in sync. Since we want the bootstrap
kubelet to come up and run even if the API server is not available, we should
persist the configuration for bootstrap kubelet on the node. Once we have
checkpointing in kubelet, we will checkpoint the updated config and have the
bootstrap kubelet use the updated config, if it were to take over.
* Currently best practice when upgrading the kubelet on a node is to drain all
pods first. Automatically draining of the node during kubelet upgrade is out
of scope for this proposal. It is assumed that either the cluster
administrator or the daemonset upgrade policy will handle this.
## Other discussion
Various similar approaches have been discussed
[here](https://github.com/kubernetes/kubernetes/issues/246#issuecomment-64533959)
and
[here](https://github.com/kubernetes/kubernetes/issues/23073#issuecomment-198478997).
Other discussion around the kubelet being able to be run inside a container is
[here](https://github.com/kubernetes/kubernetes/issues/4869). Note this isn't a
strict requirement as the kubelet could be run in a chroot jail via rkt fly or
other such similar approach.
Additionally, [Taints and
Tolerations](../../docs/design/taint-toleration-dedicated.md), whose design has
already been accepted, would make the overall kubelet bootstrap more
deterministic. With this, we would also need the ability for a kubelet to
register itself with a given taint when it first contacts the API server. Given
that, a kubelet could register itself with a given taint such as
“component=kubelet”, and a kubelet pod could exist that has a toleration to that
taint, ensuring it is the only pod the “bootstrap” kubelet runs.
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/proposals/self-hosted-kubelet.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/design-proposals/self-hosted-kubelet.md](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/self-hosted-kubelet.md)
docs/proposals/selinux-enhancements.md
View file @ ed10c233
## Abstract
Presents a proposal for enhancing the security of Kubernetes clusters using
SELinux and simplifying the implementation of SELinux support within the
Kubelet by removing the need to label the Kubelet directory with an SELinux
context usable from a container.
## Motivation
The current Kubernetes codebase relies upon the Kubelet directory being
labeled with an SELinux context usable from a container. This means that a
container escaping namespace isolation will be able to use any file within the
Kubelet directory without defeating kernel
[MAC (mandatory access control)](https://en.wikipedia.org/wiki/Mandatory_access_control).
In order to limit the attack surface, we should enhance the Kubelet to relabel
any bind-mounts into containers into a usable SELinux context without depending
on the Kubelet directory's SELinux context.
## Constraints and Assumptions
1. No API changes allowed
2. Behavior must be fully backward compatible
3. No new admission controllers - make incremental improvements without huge
refactorings
## Use Cases
1. As a cluster operator, I want to avoid having to label the Kubelet
directory with a label usable from a container, so that I can limit the
attack surface available to a container escaping its namespace isolation
2. As a user, I want to run a pod without an SELinux context explicitly
specified and be isolated using MCS (multi-category security) on systems
where SELinux is enabled, so that the pods on each host are isolated from
one another
3. As a user, I want to run a pod that uses the host IPC or PID namespace and
want the system to do the right thing with regard to SELinux, so that no
unnecessary relabel actions are performed
### Labeling the Kubelet directory
As previously stated, the current codebase relies on the Kubelet directory
being labeled with an SELinux context usable from a container. The Kubelet
uses the SELinux context of this directory to determine what SELinux context
`tmpfs` mounts (provided by the EmptyDir memory-medium option) should receive.
The problem with this is that it opens an attack surface to a container that
escapes its namespace isolation; such a container would be able to use any
file in the Kubelet directory without defeating kernel MAC.
### SELinux when no context is specified
When no SELinux context is specified, Kubernetes should just do the right
thing, where doing the right thing is defined as isolating pods with a node-
unique set of categories. Node-uniqueness means unique among the pods
scheduled onto the node. Long-term, we want to have a cluster-wide allocator
for MCS labels. Node-unique MCS labels are a good middle ground that is
possible without a new, large, feature.
### SELinux and host IPC and PID namespaces
Containers in pods that use the host IPC or PID namespaces need access to
other processes and IPC mechanisms on the host. Therefore, these containers
should be run with the `spc_t` SELinux type by the container runtime. The
`spc_t` type is an unconfined type that other SELinux domains are allowed to
connect to. In the case where a pod uses one of these host namespaces, it
should be unnecessary to relabel the pod's volumes.
## Analysis
### Libcontainer SELinux library
Docker and rkt both use the libcontainer SELinux library. This library
provides a method, `GetLxcContexts`, that returns the a unique SELinux
contexts for container processes and files used by them. `GetLxcContexts`
reads the base SELinux context information from a file at `/etc/selinux/<policy-
name>/contexts/lxc_contexts` and then adds a process-unique MCS label.
Docker and rkt both leverage this call to determine the 'starting' SELinux
contexts for containers.
### Docker
Docker's behavior when no SELinux context is defined for a container is to
give the container a node-unique MCS label.
#### Sharing IPC namespaces
On the Docker runtime, the containers in a Kubernetes pod share the IPC and
PID namespaces of the pod's infra container.
Docker's behavior for containers sharing these namespaces is as follows: if a
container B shares the IPC namespace of another container A, container B is
given the SELinux context of container A. Therefore, for Kubernetes pods
running on docker, in a vacuum the containers in a pod should have the same
SELinux context.
[**Known issue**](https://bugzilla.redhat.com/show_bug.cgi?id=1377869): When
the seccomp profile is set on a docker container that shares the IPC namespace
of another container, that container will not receive the other container's
SELinux context.
#### Host IPC and PID namespaces
In the case of a pod that shares the host IPC or PID namespace, this flag is
simply ignored and the container receives the `spc_t` SELinux type. The
`spc_t` type is unconfined, and so no relabeling needs to be done for volumes
for these pods. Currently, however, there is code which relabels volumes into
explicitly specified SELinux contexts for these pods. This code is unnecessary
and should be removed.
#### Relabeling bind-mounts
Docker is capable of relabeling bind-mounts into containers using the `:Z`
bind-mount flag. However, in the current implementation of the docker runtime
in Kubernetes, the `:Z` option is only applied when the pod's SecurityContext
contains an SELinux context. We could easily implement the correct behaviors
by always setting `:Z` on systems where SELinux is enabled.
### rkt
rkt's behavior when no SELinux context is defined for a pod is similar to
Docker's -- an SELinux context with a node-unique MCS label is given to the
containers of a pod.
#### Sharing IPC namespaces
Containers (apps, in rkt terminology) in rkt pods share an IPC and PID
namespace by default.
#### Relabeling bind-mounts
Bind-mounts into rkt pods are automatically relabeled into the pod's SELinux
context.
#### Host IPC and PID namespaces
Using the host IPC and PID namespaces is not currently supported by rkt.
## Proposed Changes
### Refactor `pkg/util/selinux`
1. The `selinux` package should provide a method `SELinuxEnabled` that returns
whether SELinux is enabled, and is built for all platforms (the
libcontainer SELinux is only built on linux)
2. The `SelinuxContextRunner` interface should be renamed to `SELinuxRunner`
and be changed to have the same method names and signatures as the
libcontainer methods its implementations wrap
3. The `SELinuxRunner` interface only needs `Getfilecon`, which is used by
the rkt code
```go
package selinux
// Note: the libcontainer SELinux package is only built for Linux, so it is
// necessary to have a NOP wrapper which is built for non-Linux platforms to
// allow code that links to this package not to differentiate its own methods
// for Linux and non-Linux platforms.
//
// SELinuxRunner wraps certain libcontainer SELinux calls. For more
// information, see:
//
// https://github.com/opencontainers/runc/blob/master/libcontainer/selinux/selinux.go
type SELinuxRunner interface {
// Getfilecon returns the SELinux context for the given path or returns an
// error.
Getfilecon(path string) (string, error)
}
```
### Kubelet Changes
1. The `relabelVolumes` method in `kubelet_volumes.go` is not needed and can
be removed
2. The `GenerateRunContainerOptions` method in `kubelet_pods.go` should no
longer call `relabelVolumes`
3. The `makeHostsMount` method in `kubelet_pods.go` should set the
`SELinuxRelabel` attribute of the mount for the pod's hosts file to `true`
### Changes to `pkg/kubelet/dockertools/`
1. The `makeMountBindings` should be changed to:
1. No longer accept the `podHasSELinuxLabel` parameter
2. Always use the `:Z` bind-mount flag when SELinux is enabled and the mount
has the `SELinuxRelabel` attribute set to `true`
2. The `runContainer` method should be changed to always use the `:Z`
bind-mount flag on the termination message mount when SELinux is enabled
### Changes to `pkg/kubelet/rkt`
The should not be any required changes for the rkt runtime; we should test to
ensure things work as expected under rkt.
### Changes to volume plugins and infrastructure
1. The `VolumeHost` interface contains a method called `GetRootContext`; this
is an artifact of the old assumptions about the Kubelet directory's SELinux
context and can be removed
2. The `empty_dir.go` file should be changed to be completely agnostic of
SELinux; no behavior in this plugin needs to be differentiated when SELinux
is enabled
### Changes to `pkg/controller/...`
The `VolumeHost` abstraction is used in a couple of PV controllers as NOP
implementations. These should be altered to no longer include `GetRootContext`.
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/proposals/selinux-enhancements.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/design-proposals/selinux-enhancements.md](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/selinux-enhancements.md)
docs/proposals/service-discovery.md
View file @ ed10c233
# Service Discovery Proposal
## Goal of this document
To consume a service, a developer needs to know the full URL and a description of the API. Kubernetes contains the host and port information of a service, but it lacks the scheme and the path information needed if the service is not bound at the root. In this document we propose some standard kubernetes service annotations to fix these gaps. It is important that these annotations are a standard to allow for standard service discovery across Kubernetes implementations. Note that the example largely speaks to consuming WebServices but that the same concepts apply to other types of services.
## Endpoint URL, Service Type
A URL can accurately describe the location of a Service. A generic URL is of the following form
scheme:[//[user:password@]host[:port]][/]path[?query][#fragment]
however for the purpose of service discovery we can simplify this to the following form
scheme:[//host[:port]][/]path
If a user and/or password is required then this information can be passed using Kubernetes Secrets. Kubernetes contains the host and port of each service but it lacks the scheme and path.
`Service Path` - Every Service has one (or more) endpoint. As a rule the endpoint should be located at the root "/" of the location URL, i.e. `http://172.100.1.52/`. There are cases where this is not possible and the actual service endpoint could be located at `http://172.100.1.52/cxfcdi`. The Kubernetes metadata for a service does not capture the path part, making it hard to consume this service.
`Service Scheme` - Services can be deployed using different schemes. Some popular schemes include `http`,`https`,`file`,`ftp` and `jdbc`.
`Service Protocol` - Services use different protocols that clients need to speak in order to communicate with the service, some examples of service level protocols are SOAP, REST (Yes, technically REST isn’t a protocol but an architectural style). For service consumers it can be hard to tell what protocol is expected.
## Service Description
The API of a service is the point of interaction with a service consumer. The description of the API is an essential piece of information at creation time of the service consumer. It has become common to publish a service definition document on a know location on the service itself. This 'well known' place it not very standard, so it is proposed the service developer provides the service description path and the type of Definition Language (DL) used.
`Service Description Path` - To facilitate the consumption of the service by client, the location this document would be greatly helpful to the service consumer. In some cases the client side code can be generated from such a document. It is assumed that the service description document is published somewhere on the service endpoint itself.
`Service Description Language` - A number of Definition Languages (DL) have been developed to describe the service. Some of examples are `WSDL`, `WADL` and `Swagger`. In order to consume a description document it is good to know the type of DL used.
## Standard Service Annotations
Kubernetes allows the creation of Service Annotations. Here we propose the use of the following standard annotations
* `api.service.kubernetes.io/path` - the path part of the service endpoint url. An example value could be `cxfcdi`,
* `api.service.kubernetes.io/scheme` - the scheme part of the service endpoint url. Some values could be `http` or `https`.
* `api.service.kubernetes.io/protocol` - the protocol of the service. Known values are `SOAP`, `XML-RPC` and `REST`,
* `api.service.kubernetes.io/description-path` - the path part of the service description document’s endpoint. It is a pretty safe assumption that the service self-documents. An example value for a swagger 2.0 document can be `cxfcdi/swagger.json`,
* `api.kubernetes.io/description-language` - the type of Description Language used. Known values are `WSDL`, `WADL`, `SwaggerJSON`, `SwaggerYAML`.
The fragment below is taken from the service section of the kubernetes.json were these annotations are used
...
"objects" : [ {
"apiVersion" : "v1",
"kind" : "Service",
"metadata" : {
"annotations" : {
"api.service.kubernetes.io/protocol" : "REST",
"api.service.kubernetes.io/scheme" "http",
"api.service.kubernetes.io/path" : "cxfcdi",
"api.service.kubernetes.io/description-path" : "cxfcdi/swagger.json",
"api.service.kubernetes.io/description-language" : "SwaggerJSON"
},
...
## Conclusion
Five service annotations are proposed as a standard way to describe a service endpoint. These five annotation are promoted as a Kubernetes standard, so that services can be discovered and a service catalog can be build to facilitate service consumers.
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/proposals/service-discovery.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/design-proposals/service-discovery.md](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/service-discovery.md)
docs/proposals/service-external-name.md
View file @ ed10c233
# Service externalName
Author: Tim Hockin (@thockin), Rodrigo Campos (@rata), Rudi C (@therc)
Date: August 2016
Status: Implementation in progress
# Goal
Allow a service to have a CNAME record in the cluster internal DNS service. For
example, the lookup for a `db` service could return a CNAME that points to the
RDS resource `something.rds.aws.amazon.com`. No proxying is involved.
# Motivation
There were many related issues, but we'll try to summarize them here. More info
is on GitHub issues/PRs: #13748, #11838, #13358, #23921
One motivation is to present as native cluster services, services that are
hosted externally. Some cloud providers, like AWS, hand out hostnames (IPs are
not static) and the user wants to refer to these services using regular
Kubernetes tools. This was requested in bugs, at least for AWS, for RedShift,
RDS, Elasticsearch Service, ELB, etc.
Other users just want to use an external service, for example `oracle`, with dns
name `oracle-1.testdev.mycompany.com`, without having to keep DNS in sync, and
are fine with a CNAME.
Another use case is to "integrate" some services for local development. For
example, consider a search service running in Kubernetes in staging, let's say
`search-1.stating.mycompany.com`. It's running on AWS, so it resides behind an
ELB (which has no static IP, just a hostname). A developer is building an app
that consumes `search-1`, but doesn't want to run it on their machine (before
Kubernetes, they didn't, either). They can just create a service that has a
CNAME to the `search-1` endpoint in staging and be happy as before.
Also, Openshift needs this for "service refs". Service ref is really just the
three use cases mentioned above, but in the future a way to automatically inject
"service ref"s into namespaces via "service catalog"[1] might be considered. And
service ref is the natural way to integrate an external service, since it takes
advantage of native DNS capabilities already in wide use.
[1]: https://github.com/kubernetes/kubernetes/pull/17543
# Alternatives considered
In the issues linked above, some alternatives were also considered. A partial
summary of them follows.
One option is to add the hostname to endpoints, as proposed in
https://github.com/kubernetes/kubernetes/pull/11838. This is problematic, as
endpoints are used in many places and users assume the required fields (such as
IP address) are always present and valid (and check that, too). If the field is
not required anymore or if there is just a hostname instead of the IP,
applications could break. Even assuming those cases could be solved, the
hostname will have to be resolved, which presents further questions and issues:
the timeout to use, whether the lookup is synchronous or asynchronous, dealing
with DNS TTL and more. One imperfect approach was to only resolve the hostname
upon creation, but this was considered not a great idea. A better approach
would be at a higher level, maybe a service type.
There are more ideas described in #13748, but all raised further issues,
ranging from using another upstream DNS server to creating a Name object
associated with DNSs.
# Proposed solution
The proposed solution works at the service layer, by adding a new `externalName`
type for services. This will create a CNAME record in the internal cluster DNS
service. No virtual IP or proxying is involved.
Using a CNAME gets rid of unnecessary DNS lookups. There's no need for the
Kubernetes control plane to issue them, to pick a timeout for them and having to
refresh them when the TTL for a record expires. It's way simpler to implement,
while solving the right problem. And addressing it at the service layer avoids
all the complications mentioned above about doing it at the endpoints layer.
The solution was outlined by Tim Hockin in
https://github.com/kubernetes/kubernetes/issues/13748#issuecomment-230397975
Currently a ServiceSpec looks like this, with comments edited for clarity:
```
type ServiceSpec struct {
Ports []ServicePort
// If not specified, the associated Endpoints object is not automatically managed
Selector map[string]string
// "", a real IP, or "None". If not specified, this is default allocated. If "None", this Service is not load-balanced
ClusterIP string
// ClusterIP, NodePort, LoadBalancer. Only applies if clusterIP != "None"
Type ServiceType
// Only applies if clusterIP != "None"
ExternalIPs []string
SessionAffinity ServiceAffinity
// Only applies to type=LoadBalancer
LoadBalancerIP string
LoadBalancerSourceRanges []string
```
The proposal is to change it to:
```
type ServiceSpec struct {
Ports []ServicePort
// If not specified, the associated Endpoints object is not automatically managed
+ // Only applies if type is ClusterIP, NodePort, or LoadBalancer. If type is ExternalName, this is ignored.
Selector map[string]string
// "", a real IP, or "None". If not specified, this is default allocated. If "None", this Service is not load-balanced.
+ // Only applies if type is ClusterIP, NodePort, or LoadBalancer. If type is ExternalName, this is ignored.
ClusterIP string
- // ClusterIP, NodePort, LoadBalancer. Only applies if clusterIP != "None"
+ // ExternalName, ClusterIP, NodePort, LoadBalancer. Only applies if clusterIP != "None"
Type ServiceType
+ // Only applies if type is ExternalName
+ ExternalName string
// Only applies if clusterIP != "None"
ExternalIPs []string
SessionAffinity ServiceAffinity
// Only applies to type=LoadBalancer
LoadBalancerIP string
LoadBalancerSourceRanges []string
```
For example, it can be used like this:
```
apiVersion: v1
kind: Service
metadata:
name: my-rds
spec:
ports:
- port: 12345
type: ExternalName
externalName: myapp.rds.whatever.aws.says
```
There is one issue to take into account, that no other alternative considered
fixes, either: TLS. If the service is a CNAME for an endpoint that uses TLS,
connecting with the Kubernetes name `my-service.my-ns.svc.cluster.local` may
result in a failure during server certificate validation. This is acknowledged
and left for future consideration. For the time being, users and administrators
might need to ensure that the server certificates also mentions the Kubernetes
name as an alternate host name.
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/proposals/service-external-name.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/design-proposals/service-external-name.md](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/service-external-name.md)
docs/proposals/volume-hostpath-qualifiers.md
View file @ ed10c233
# Support HostPath volume existence qualifiers
## Introduction
A Host volume source is probably the simplest volume type to define, needing
only a single path. However, that simplicity comes with many assumptions and
caveats.
This proposal describes one of the issues associated with Host volumes &mdash;
their silent and implicit creation of directories on the host &mdash; and
proposes a solution.
## Problem
Right now, under Docker, when a bindmount references a hostPath, that path will
be created as an empty directory, owned by root, if it does not already exist.
This is rarely what the user actually wants because hostPath volumes are
typically used to express a dependency on an existing external file or
directory.
This concern was raised during the [initial
implementation](https://github.com/docker/docker/issues/1279#issuecomment-22965058)
of this behavior in Docker and it was suggested that orchestration systems
could better manage volume creation than Docker, but Docker does so as well
anyways.
To fix this problem, I propose allowing a pod to specify whether a given
hostPath should exist prior to the pod running, whether it should be created,
and what it should exist as.
I also propose the inclusion of a default value which matches the current
behavior to ensure backwards compatibility.
To understand exactly when this behavior will or won't be correct, it's
important to look at the use-cases of Host Volumes.
The table below broadly classifies the use-case of Host Volumes and asserts
whether this change would be of benefit to that use-case.
### HostPath volume Use-cases
| Use-case | Description | Examples | Benefits from this change? | Why? |
|:---------|:------------|:---------|:--------------------------:|:-----|
| Accessing an external system, data, or configuration | Data or a unix socket is created by a process on the host, and a pod within kubernetes consumes it | [fluentd-es-addon](https://github.com/kubernetes/kubernetes/blob/74b01041cc3feb2bb731cc243ab0e4515bef9a84/cluster/saltbase/salt/fluentd-es/fluentd-es.yaml#L30), [addon-manager](https://github.com/kubernetes/kubernetes/blob/808f3ecbe673b4127627a457dc77266ede49905d/cluster/gce/coreos/kube-manifests/kube-addon-manager.yaml#L23), [kube-proxy](https://github.com/kubernetes/kubernetes/blob/010c976ce8dd92904a7609483c8e794fd8e94d4e/cluster/saltbase/salt/kube-proxy/kube-proxy.manifest#L65), etc | :white_check_mark: | Fails faster and with more useful messages, and won't run when basic assumptions are false (e.g. that docker is the runtime and the docker.sock exists) |
| Providing data to external systems | Some pods wish to publish data to the host for other systems to consume, sometimes to a generic directory and sometimes to more component-specific ones | Kubelet core components which bindmount their logs out to `/var/log/*.log` so logrotate and other tools work with them | :white_check_mark: | Sometimes, but not always. It's directory-specific whether it not existing will be a problem. |
| Communicating between instances and versions of yourself | A pod can use a hostPath directory as a sort of cache and, as opposed to an emptyDir, persist the directory between versions of itself | [etcd](https://github.com/kubernetes/kubernetes/blob/fac54c9b22eff5c5052a8e3369cf8416a7827d36/cluster/saltbase/salt/etcd/etcd.manifest#L84), caches | :x: | It's pretty much always okay to create them |
### Other motivating factors
One additional motivating factor for this change is that under the rkt runtime
paths are not created when they do not exist. This change moves the management
of these volumes into the Kubelet to the benefit of the rkt container runtime.
## Proposed API Change
### Host Volume
I propose that the
[`v1.HostPathVolumeSource`](https://github.com/kubernetes/kubernetes/blob/d26b4ca2859aa667ad520fb9518e0db67b74216a/pkg/api/types.go#L447-L451)
object be changed to include the following additional field:
`Type` - An optional string of `exists|file|device|socket|directory` - If not
set, it will default to a backwards-compatible default behavior described
below.
| Value | Behavior |
|:------|:---------|
| *unset* | If nothing exists at the given path, an empty directory will be created there. Otherwise, behaves like `exists` |
| `exists` | If nothing exists at the given path, the pod will fail to run and provide an informative error message |
| `file` | If a file does not exist at the given path, the pod will fail to run and provide an informative error message |
| `device` | If a block or character device does not exist at the given path, the pod will fail to run and provide an informative error message |
| `socket` | If a socket does not exist at the given path, the pod will fail to run and provide an informative error message |
| `directory` | If a directory does not exist at the given path, the pod will fail to run and provide an informative error message |
Additional possible values, which are proposed to be excluded:
|Value | Behavior | Reason for exclusion |
|:-----|:---------|:---------------------|
| `new-directory` | Like `auto`, but the given path must be a directory if it exists | `auto` mostly fills this use-case |
| `character-device` | | Granularity beyond `device` shouldn't matter often |
| `block-device` | | Granularity beyond `device` shouldn't matter often |
| `new-file` | Like file, but if nothing exist an empty file is created instead | In general, bindmounting the parent directory of the file you intend to create addresses this usecase |
| `optional` | If a path does not exist, then do not create any container-mount at all | This would better be handled by a new field entirely if this behavior is desirable |
### Why not as part of any other volume types?
This feature does not make sense for any of the other volume types simply
because all of the other types are already fully qualified. For example, NFS
volumes are known to always be in existence else they will not mount.
Similarly, EmptyDir volumes will always exist as a directory.
Only the HostVolume and SubPath means of referencing a path have the potential
to reference arbitrary incorrect or nonexistent things without erroring out.
### Alternatives
One alternative is to augment Host Volumes with a `MustExist` bool and provide
no further granularity. This would allow toggling between the `auto` and
`exists` behaviors described above. This would likely cover the "90%" use-case
and would be a simpler API. It would be sufficient for all of the examples
linked above in my opionion.
## Kubelet implementation
It's proposed that prior to starting a pod, the Kubelet validates that the
given path meets the qualifications of its type. Namely, if the type is `auto`
the Kubelet will create an empty directory if none exists there, and for each
of the others the Kubelet will perform the given validation prior to running
the pod. This validation might be done by a volume plugin, but further
technical consideration (out of scope of this proposal) is needed.
## Possible concerns
### Permissions
This proposal does not attempt to change the state of volume permissions. Currently, a HostPath volume is created with `root` ownership and `755` permissions. This behavior will be retained. An argument for this behavior is given [here](volumes.md#shared-storage-hostpath).
### SELinux
This proposal should not impact SELinux relabeling. Verifying the presence and
type of a given path will be logically separate from SELinux labeling.
Similarly, creating the directory when it doesn't exist will happen before any
SELinux operations and should not impact it.
### Containerized Kubelet
A containerized kubelet would have difficulty creating directories. The
implementation will likely respect the `containerized` flag, or similar,
allowing it to either break out or be "/rootfs/" aware and thus operate as
desired.
### Racy Validation
Ideally the validation would be done at the time the bindmounts are created,
else it's possible for a given path or directory to change in the duration from
when it's validated and the container runtime attempts to create said mount.
The only way to solve this problem is to integrate these sorts of qualification
into container runtimes themselves.
I don't think this problem is severe enough that we need to push to solve it;
rather I think we can simply accept this minor race, and if runtimes eventually
allow this we can begin to leverage them.
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/proposals/volume-hostpath-qualifiers.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/design-proposals/volume-hostpath-qualifiers.md](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/volume-hostpath-qualifiers.md)
docs/proposals/volume-ownership-management.md
View file @ ed10c233
## Volume plugins and idempotency
Currently, volume plugins have a `SetUp` method which is called in the context of a higher-level
workflow within the kubelet which has externalized the problem of managing the ownership of volumes.
This design has a number of drawbacks that can be mitigated by completely internalizing all concerns
of volume setup behind the volume plugin `SetUp` method.
### Known issues with current externalized design
1. The ownership management is currently repeatedly applied, which breaks packages that require
special permissions in order to work correctly
2. There is a gap between files being mounted/created by volume plugins and when their ownership
is set correctly; race conditions exist around this
3. Solving the correct application of ownership management in an externalized model is difficult
and makes it clear that the a transaction boundary is being broken by the externalized design
### Additional issues with externalization
Fully externalizing any one concern of volumes is difficult for a number of reasons:
1. Many types of idempotence checks exist, and are used in a variety of combinations and orders
2. Workflow in the kubelet becomes much more complex to handle:
1. composition of plugins
2. correct timing of application of ownership management
3. callback to volume plugins when we know the whole `SetUp` flow is complete and correct
4. callback to touch sentinel files
5. etc etc
3. We want to support fully external volume plugins -- would require complex orchestration / chatty
remote API
## Proposed implementation
Since all of the ownership information is known in advance of the call to the volume plugin `SetUp`
method, we can easily internalize these concerns into the volume plugins and pass the ownership
information to `SetUp`.
The volume `Builder` interface's `SetUp` method changes to accept the group that should own the
volume. Plugins become responsible for ensuring that the correct group is applied. The volume
`Attributes` struct can be modified to remove the `SupportsOwnershipManagement` field.
```go
package volume
type Builder interface {
// other methods omitted
// SetUp prepares and mounts/unpacks the volume to a self-determined
// directory path and returns an error. The group ID that should own the volume
// is passed as a parameter. Plugins may choose to ignore the group ID directive
// in the event that they do not support it (example: NFS). A group ID of -1
// indicates that the group ownership of the volume should not be modified by the plugin.
//
// SetUp will be called multiple times and should be idempotent.
SetUp(gid int64) error
}
```
Each volume plugin will have to change to support the new `SetUp` signature. The existing
ownership management code will be refactored into a library that volume plugins can use:
```
package volume
func ManageOwnership(path string, fsGroup int64) error {
// 1. recursive chown of path
// 2. make path +setgid
}
```
The workflow from the Kubelet's perspective for handling volume setup and refresh becomes:
```go
// go-ish pseudocode
func mountExternalVolumes(pod) error {
podVolumes := make(kubecontainer.VolumeMap)
for i := range pod.Spec.Volumes {
volSpec := &pod.Spec.Volumes[i]
var fsGroup int64 = 0
if pod.Spec.SecurityContext != nil &&
pod.Spec.SecurityContext.FSGroup != nil {
fsGroup = *pod.Spec.SecurityContext.FSGroup
} else {
fsGroup = -1
}
// Try to use a plugin for this volume.
plugin := volume.NewSpecFromVolume(volSpec)
builder, err := kl.newVolumeBuilderFromPlugins(plugin, pod)
if err != nil {
return err
}
if builder == nil {
return errUnsupportedVolumeType
}
err := builder.SetUp(fsGroup)
if err != nil {
return nil
}
}
return nil
}
```
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/proposals/volume-ownership-management.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/design-proposals/volume-ownership-management.md](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/volume-ownership-management.md)
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment