Skip to content

K
k3s
Commit 97f1f599 authored by k8s-merge-robot's avatar k8s-merge-robot
Browse files
Options

Merge pull request #24231 from mikebrow/design-docs-80col-updates

Automatic merge from submit-queue Cleans up line wrap at 80 cols and some minor editing issues Address line wrap issue #1488. Also cleans up other minor editing issues in the docs/design/* tree such as spelling errors. Signed-off-by: 's avatarmikebrow <brownwm@us.ibm.com>
parents 2347d0f0 1053be8e
Expand all Hide whitespace changes
Inline Side-by-side
Showing with 919 additions and 536 deletions
docs/design/README.md
View file @ 97f1f599
......@@ -34,19 +34,59 @@ Documentation for other releases can be found at
# Kubernetes Design Overview
Kubernetes is a system for managing containerized applications across multiple hosts, providing basic mechanisms for deployment, maintenance, and scaling of applications.
Kubernetes establishes robust declarative primitives for maintaining the desired state requested by the user. We see these primitives as the main value added by Kubernetes. Self-healing mechanisms, such as auto-restarting, re-scheduling, and replicating containers require active controllers, not just imperative orchestration.
Kubernetes is primarily targeted at applications composed of multiple containers, such as elastic, distributed micro-services. It is also designed to facilitate migration of non-containerized application stacks to Kubernetes. It therefore includes abstractions for grouping containers in both loosely coupled and tightly coupled formations, and provides ways for containers to find and communicate with each other in relatively familiar ways.
Kubernetes enables users to ask a cluster to run a set of containers. The system automatically chooses hosts to run those containers on. While Kubernetes's scheduler is currently very simple, we expect it to grow in sophistication over time. Scheduling is a policy-rich, topology-aware, workload-specific function that significantly impacts availability, performance, and capacity. The scheduler needs to take into account individual and collective resource requirements, quality of service requirements, hardware/software/policy constraints, affinity and anti-affinity specifications, data locality, inter-workload interference, deadlines, and so on. Workload-specific requirements will be exposed through the API as necessary.
Kubernetes is intended to run on a number of cloud providers, as well as on physical hosts.
A single Kubernetes cluster is not intended to span multiple availability zones. Instead, we recommend building a higher-level layer to replicate complete deployments of highly available applications across multiple zones (see [the multi-cluster doc](../admin/multi-cluster.md) and [cluster federation proposal](../proposals/federation.md) for more details).
Finally, Kubernetes aspires to be an extensible, pluggable, building-block OSS platform and toolkit. Therefore, architecturally, we want Kubernetes to be built as a collection of pluggable components and layers, with the ability to use alternative schedulers, controllers, storage systems, and distribution mechanisms, and we're evolving its current code in that direction. Furthermore, we want others to be able to extend Kubernetes functionality, such as with higher-level PaaS functionality or multi-cluster layers, without modification of core Kubernetes source. Therefore, its API isn't just (or even necessarily mainly) targeted at end users, but at tool and extension developers. Its APIs are intended to serve as the foundation for an open ecosystem of tools, automation systems, and higher-level API layers. Consequently, there are no "internal" inter-component APIs. All APIs are visible and available, including the APIs used by the scheduler, the node controller, the replication-controller manager, Kubelet's API, etc. There's no glass to break -- in order to handle more complex use cases, one can just access the lower-level APIs in a fully transparent, composable manner.
Kubernetes is a system for managing containerized applications across multiple
hosts, providing basic mechanisms for deployment, maintenance, and scaling of
applications.
Kubernetes establishes robust declarative primitives for maintaining the desired
state requested by the user. We see these primitives as the main value added by
Kubernetes. Self-healing mechanisms, such as auto-restarting, re-scheduling, and
replicating containers require active controllers, not just imperative
orchestration.
Kubernetes is primarily targeted at applications composed of multiple
containers, such as elastic, distributed micro-services. It is also designed to
facilitate migration of non-containerized application stacks to Kubernetes. It
therefore includes abstractions for grouping containers in both loosely coupled
and tightly coupled formations, and provides ways for containers to find and
communicate with each other in relatively familiar ways.
Kubernetes enables users to ask a cluster to run a set of containers. The system
automatically chooses hosts to run those containers on. While Kubernetes's
scheduler is currently very simple, we expect it to grow in sophistication over
time. Scheduling is a policy-rich, topology-aware, workload-specific function
that significantly impacts availability, performance, and capacity. The
scheduler needs to take into account individual and collective resource
requirements, quality of service requirements, hardware/software/policy
constraints, affinity and anti-affinity specifications, data locality,
inter-workload interference, deadlines, and so on. Workload-specific
requirements will be exposed through the API as necessary.
Kubernetes is intended to run on a number of cloud providers, as well as on
physical hosts.
A single Kubernetes cluster is not intended to span multiple availability zones.
Instead, we recommend building a higher-level layer to replicate complete
deployments of highly available applications across multiple zones (see
[the multi-cluster doc](../admin/multi-cluster.md) and [cluster federation proposal](../proposals/federation.md)
for more details).
Finally, Kubernetes aspires to be an extensible, pluggable, building-block OSS
platform and toolkit. Therefore, architecturally, we want Kubernetes to be built
as a collection of pluggable components and layers, with the ability to use
alternative schedulers, controllers, storage systems, and distribution
mechanisms, and we're evolving its current code in that direction. Furthermore,
we want others to be able to extend Kubernetes functionality, such as with
higher-level PaaS functionality or multi-cluster layers, without modification of
core Kubernetes source. Therefore, its API isn't just (or even necessarily
mainly) targeted at end users, but at tool and extension developers. Its APIs
are intended to serve as the foundation for an open ecosystem of tools,
automation systems, and higher-level API layers. Consequently, there are no
"internal" inter-component APIs. All APIs are visible and available, including
the APIs used by the scheduler, the node controller, the replication-controller
manager, Kubelet's API, etc. There's no glass to break -- in order to handle
more complex use cases, one can just access the lower-level APIs in a fully
transparent, composable manner.
For more about the Kubernetes architecture, see [architecture](architecture.md).
......
docs/design/admission_control.md
View file @ 97f1f599
......@@ -43,24 +43,30 @@ Documentation for other releases can be found at
## Background
High level goals:
* Enable an easy-to-use mechanism to provide admission control to cluster.
* Enable a provider to support multiple admission control strategies or author
their own.
* Ensure any rejected request can propagate errors back to the caller with why
the request failed.
* Enable an easy-to-use mechanism to provide admission control to cluster
* Enable a provider to support multiple admission control strategies or author their own
* Ensure any rejected request can propagate errors back to the caller with why the request failed
Authorization via policy is focused on answering if a user is authorized to perform an action.
Authorization via policy is focused on answering if a user is authorized to
perform an action.
Admission Control is focused on if the system will accept an authorized action.
Kubernetes may choose to dismiss an authorized action based on any number of admission control strategies.
Kubernetes may choose to dismiss an authorized action based on any number of
admission control strategies.
This proposal documents the basic design, and describes how any number of admission control plug-ins could be injected.
This proposal documents the basic design, and describes how any number of
admission control plug-ins could be injected.
Implementation of specific admission control strategies are handled in separate documents.
Implementation of specific admission control strategies are handled in separate
documents.
## kube-apiserver
The kube-apiserver takes the following OPTIONAL arguments to enable admission control
The kube-apiserver takes the following OPTIONAL arguments to enable admission
control:
| Option | Behavior |
| ------ | -------- |
......@@ -72,7 +78,8 @@ An **AdmissionControl** plug-in is an implementation of the following interface:
```go
package admission
// Attributes is an interface used by a plug-in to make an admission decision on a individual request.
// Attributes is an interface used by a plug-in to make an admission decision
// on a individual request.
type Attributes interface {
GetNamespace() string
GetKind() string
......@@ -88,8 +95,8 @@ type Interface interface {
}
```
A **plug-in** must be compiled with the binary, and is registered as an available option by providing a name, and implementation
of admission.Interface.
A **plug-in** must be compiled with the binary, and is registered as an
available option by providing a name, and implementation of admission.Interface.
```go
func init() {
......@@ -97,9 +104,12 @@ func init() {
}
```
Invocation of admission control is handled by the **APIServer** and not individual **RESTStorage** implementations.
Invocation of admission control is handled by the **APIServer** and not
individual **RESTStorage** implementations.
This design assumes that **Issue 297** is adopted, and as a consequence, the general framework of the APIServer request/response flow will ensure the following:
This design assumes that **Issue 297** is adopted, and as a consequence, the
general framework of the APIServer request/response flow will ensure the
following:
1. Incoming request
2. Authenticate user
......
docs/design/admission_control_limit_range.md
View file @ 97f1f599
......@@ -36,7 +36,8 @@ Documentation for other releases can be found at
## Background
This document proposes a system for enforcing resource requirements constraints as part of admission control.
This document proposes a system for enforcing resource requirements constraints
as part of admission control.
## Use cases
......@@ -64,7 +65,8 @@ const (
LimitTypeContainer LimitType = "Container"
)
// LimitRangeItem defines a min/max usage limit for any resource that matches on kind.
// LimitRangeItem defines a min/max usage limit for any resource that matches
// on kind.
type LimitRangeItem struct {
// Type of resource that this limit applies to.
Type LimitType `json:"type,omitempty"`
......@@ -72,29 +74,38 @@ type LimitRangeItem struct {
Max ResourceList `json:"max,omitempty"`
// Min usage constraints on this kind by resource name.
Min ResourceList `json:"min,omitempty"`
// Default resource requirement limit value by resource name if resource limit is omitted.
// Default resource requirement limit value by resource name if resource limit
// is omitted.
Default ResourceList `json:"default,omitempty"`
// DefaultRequest is the default resource requirement request value by resource name if resource request is omitted.
// DefaultRequest is the default resource requirement request value by
// resource name if resource request is omitted.
DefaultRequest ResourceList `json:"defaultRequest,omitempty"`
// MaxLimitRequestRatio if specified, the named resource must have a request and limit that are both non-zero where limit divided by request is less than or equal to the enumerated value; this represents the max burst for the named resource.
// MaxLimitRequestRatio if specified, the named resource must have a request
// and limit that are both non-zero where limit divided by request is less
// than or equal to the enumerated value; this represents the max burst for
// the named resource.
MaxLimitRequestRatio ResourceList `json:"maxLimitRequestRatio,omitempty"`
}
// LimitRangeSpec defines a min/max usage limit for resources that match on kind.
// LimitRangeSpec defines a min/max usage limit for resources that match
// on kind.
type LimitRangeSpec struct {
// Limits is the list of LimitRangeItem objects that are enforced.
Limits []LimitRangeItem `json:"limits"`
}
// LimitRange sets resource usage limits for each kind of resource in a Namespace.
// LimitRange sets resource usage limits for each kind of resource in a
// Namespace.
type LimitRange struct {
TypeMeta `json:",inline"`
// Standard object's metadata.
// More info: http://releases.k8s.io/HEAD/docs/devel/api-conventions.md#metadata
// More info:
// http://releases.k8s.io/HEAD/docs/devel/api-conventions.md#metadata
ObjectMeta `json:"metadata,omitempty"`
// Spec defines the limits enforced.
// More info: http://releases.k8s.io/HEAD/docs/devel/api-conventions.md#spec-and-status
// More info:
// http://releases.k8s.io/HEAD/docs/devel/api-conventions.md#spec-and-status
Spec LimitRangeSpec `json:"spec,omitempty"`
}
......@@ -102,24 +113,29 @@ type LimitRange struct {
type LimitRangeList struct {
TypeMeta `json:",inline"`
// Standard list metadata.
// More info: http://releases.k8s.io/HEAD/docs/devel/api-conventions.md#types-kinds
// More info:
// http://releases.k8s.io/HEAD/docs/devel/api-conventions.md#types-kinds
ListMeta `json:"metadata,omitempty"`
// Items is a list of LimitRange objects.
// More info: http://releases.k8s.io/HEAD/docs/design/admission_control_limit_range.md
// More info:
// http://releases.k8s.io/HEAD/docs/design/admission_control_limit_range.md
Items []LimitRange `json:"items"`
}
```
### Validation
Validation of a **LimitRange** enforces that for a given named resource the following rules apply:
Validation of a **LimitRange** enforces that for a given named resource the
following rules apply:
Min (if specified) <= DefaultRequest (if specified) <= Default (if specified) <= Max (if specified)
Min (if specified) <= DefaultRequest (if specified) <= Default (if specified)
<= Max (if specified)
### Default Value Behavior
The following default value behaviors are applied to a LimitRange for a given named resource.
The following default value behaviors are applied to a LimitRange for a given
named resource.
```
if LimitRangeItem.Default[resourceName] is undefined
......@@ -137,11 +153,14 @@ if LimitRangeItem.DefaultRequest[resourceName] is undefined
## AdmissionControl plugin: LimitRanger
The **LimitRanger** plug-in introspects all incoming pod requests and evaluates the constraints defined on a LimitRange.
The **LimitRanger** plug-in introspects all incoming pod requests and evaluates
the constraints defined on a LimitRange.
If a constraint is not specified for an enumerated resource, it is not enforced or tracked.
If a constraint is not specified for an enumerated resource, it is not enforced
or tracked.
To enable the plug-in and support for LimitRange, the kube-apiserver must be configured as follows:
To enable the plug-in and support for LimitRange, the kube-apiserver must be
configured as follows:
```console
$ kube-apiserver --admission-control=LimitRanger
......@@ -158,7 +177,7 @@ Supported Resources:
Supported Constraints:
Per container, the following must hold true
Per container, the following must hold true:
| Constraint | Behavior |
| ---------- | -------- |
......@@ -168,8 +187,10 @@ Per container, the following must hold true
Supported Defaults:
1. Default - if the named resource has no enumerated value, the Limit is equal to the Default
2. DefaultRequest - if the named resource has no enumerated value, the Request is equal to the DefaultRequest
1. Default - if the named resource has no enumerated value, the Limit is equal
to the Default
2. DefaultRequest - if the named resource has no enumerated value, the Request
is equal to the DefaultRequest
**Type: Pod**
......@@ -190,7 +211,8 @@ Across all containers in pod, the following must hold true
## Run-time configuration
The default ```LimitRange``` that is applied via Salt configuration will be updated as follows:
The default ```LimitRange``` that is applied via Salt configuration will be
updated as follows:
```
apiVersion: "v1"
......@@ -219,7 +241,8 @@ the following would happen.
1. The incoming container cpu would request 250m with a limit of 500m.
2. The incoming container memory would request 250Mi with a limit of 500Mi
3. If the container is later resized, it's cpu would be constrained to between .1 and 1 and the ratio of limit to request could not exceed 4.
3. If the container is later resized, it's cpu would be constrained to between
.1 and 1 and the ratio of limit to request could not exceed 4.
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/design/admission_control_limit_range.md?pixel)]()
......
docs/design/admission_control_resource_quota.md
View file @ 97f1f599
......@@ -36,7 +36,8 @@ Documentation for other releases can be found at
## Background
This document describes a system for enforcing hard resource usage limits per namespace as part of admission control.
This document describes a system for enforcing hard resource usage limits per
namespace as part of admission control.
## Use cases
......@@ -103,7 +104,7 @@ type ResourceQuotaList struct {
## Quota Tracked Resources
The following resources are supported by the quota system.
The following resources are supported by the quota system:
| Resource | Description |
| ------------ | ----------- |
......@@ -116,16 +117,19 @@ The following resources are supported by the quota system.
| secrets | Total number of secrets |
| persistentvolumeclaims | Total number of persistent volume claims |
If a third-party wants to track additional resources, it must follow the resource naming conventions prescribed
by Kubernetes. This means the resource must have a fully-qualified name (i.e. mycompany.org/shinynewresource)
If a third-party wants to track additional resources, it must follow the
resource naming conventions prescribed by Kubernetes. This means the resource
must have a fully-qualified name (i.e. mycompany.org/shinynewresource)
## Resource Requirements: Requests vs Limits
If a resource supports the ability to distinguish between a request and a limit for a resource,
the quota tracking system will only cost the request value against the quota usage. If a resource
is tracked by quota, and no request value is provided, the associated entity is rejected as part of admission.
If a resource supports the ability to distinguish between a request and a limit
for a resource, the quota tracking system will only cost the request value
against the quota usage. If a resource is tracked by quota, and no request value
is provided, the associated entity is rejected as part of admission.
For an example, consider the following scenarios relative to tracking quota on CPU:
For an example, consider the following scenarios relative to tracking quota on
CPU:
| Pod | Container | Request CPU | Limit CPU | Result |
| --- | --------- | ----------- | --------- | ------ |
......@@ -134,13 +138,14 @@ For an example, consider the following scenarios relative to tracking quota on C
| Y | C2 | none | 500m | The quota usage is incremented 500m since request will default to limit |
| Z | C3 | none | none | The pod is rejected since it does not enumerate a request. |
The rationale for accounting for the requested amount of a resource versus the limit is the belief
that a user should only be charged for what they are scheduled against in the cluster. In addition,
attempting to track usage against actual usage, where request < actual < limit, is considered highly
volatile.
The rationale for accounting for the requested amount of a resource versus the
limit is the belief that a user should only be charged for what they are
scheduled against in the cluster. In addition, attempting to track usage against
actual usage, where request < actual < limit, is considered highly volatile.
As a consequence of this decision, the user is able to spread its usage of a resource across multiple tiers
of service. Let's demonstrate this via an example with a 4 cpu quota.
As a consequence of this decision, the user is able to spread its usage of a
resource across multiple tiers of service. Let's demonstrate this via an
example with a 4 cpu quota.
The quota may be allocated as follows:
......@@ -150,48 +155,62 @@ The quota may be allocated as follows:
| Y | C2 | 2 | 2 | Guaranteed | 2 |
| Z | C3 | 1 | 3 | Burstable | 1 |
It is possible that the pods may consume 9 cpu over a given time period depending on the nodes available cpu
that held pod X and Z, but since we scheduled X and Z relative to the request, we only track the requesting
value against their allocated quota. If one wants to restrict the ratio between the request and limit,
it is encouraged that the user define a **LimitRange** with **LimitRequestRatio** to control burst out behavior.
This would in effect, let an administrator keep the difference between request and limit more in line with
It is possible that the pods may consume 9 cpu over a given time period
depending on the nodes available cpu that held pod X and Z, but since we
scheduled X and Z relative to the request, we only track the requesting value
against their allocated quota. If one wants to restrict the ratio between the
request and limit, it is encouraged that the user define a **LimitRange** with
**LimitRequestRatio** to control burst out behavior. This would in effect, let
an administrator keep the difference between request and limit more in line with
tracked usage if desired.
## Status API
A REST API endpoint to update the status section of the **ResourceQuota** is exposed. It requires an atomic compare-and-swap
in order to keep resource usage tracking consistent.
A REST API endpoint to update the status section of the **ResourceQuota** is
exposed. It requires an atomic compare-and-swap in order to keep resource usage
tracking consistent.
## Resource Quota Controller
A resource quota controller monitors observed usage for tracked resources in the **Namespace**.
A resource quota controller monitors observed usage for tracked resources in the
**Namespace**.
If there is observed difference between the current usage stats versus the current **ResourceQuota.Status**, the controller
posts an update of the currently observed usage metrics to the **ResourceQuota** via the /status endpoint.
If there is observed difference between the current usage stats versus the
current **ResourceQuota.Status**, the controller posts an update of the
currently observed usage metrics to the **ResourceQuota** via the /status
endpoint.
The resource quota controller is the only component capable of monitoring and recording usage updates after a DELETE operation
since admission control is incapable of guaranteeing a DELETE request actually succeeded.
The resource quota controller is the only component capable of monitoring and
recording usage updates after a DELETE operation since admission control is
incapable of guaranteeing a DELETE request actually succeeded.
## AdmissionControl plugin: ResourceQuota
The **ResourceQuota** plug-in introspects all incoming admission requests.
To enable the plug-in and support for ResourceQuota, the kube-apiserver must be configured as follows:
To enable the plug-in and support for ResourceQuota, the kube-apiserver must be
configured as follows:
```
$ kube-apiserver --admission-control=ResourceQuota
```
It makes decisions by evaluating the incoming object against all defined **ResourceQuota.Status.Hard** resource limits in the request
namespace. If acceptance of the resource would cause the total usage of a named resource to exceed its hard limit, the request is denied.
It makes decisions by evaluating the incoming object against all defined
**ResourceQuota.Status.Hard** resource limits in the request namespace. If
acceptance of the resource would cause the total usage of a named resource to
exceed its hard limit, the request is denied.
If the incoming request does not cause the total usage to exceed any of the enumerated hard resource limits, the plug-in will post a
**ResourceQuota.Status** document to the server to atomically update the observed usage based on the previously read
**ResourceQuota.ResourceVersion**. This keeps incremental usage atomically consistent, but does introduce a bottleneck (intentionally)
into the system.
If the incoming request does not cause the total usage to exceed any of the
enumerated hard resource limits, the plug-in will post a
**ResourceQuota.Status** document to the server to atomically update the
observed usage based on the previously read **ResourceQuota.ResourceVersion**.
This keeps incremental usage atomically consistent, but does introduce a
bottleneck (intentionally) into the system.
To optimize system performance, it is encouraged that all resource quotas are tracked on the same **ResourceQuota** document in a **Namespace**. As a result, its encouraged to impose a cap on the total number of individual quotas that are tracked in the **Namespace**
to 1 in the **ResourceQuota** document.
To optimize system performance, it is encouraged that all resource quotas are
tracked on the same **ResourceQuota** document in a **Namespace**. As a result,
it is encouraged to impose a cap on the total number of individual quotas that
are tracked in the **Namespace** to 1 in the **ResourceQuota** document.
## kubectl
......@@ -199,7 +218,7 @@ kubectl is modified to support the **ResourceQuota** resource.
`kubectl describe` provides a human-readable output of quota.
For example,
For example:
```console
$ kubectl create -f docs/admin/resourcequota/namespace.yaml
......
docs/design/architecture.md
View file @ 97f1f599
......@@ -34,49 +34,84 @@ Documentation for other releases can be found at
# Kubernetes architecture
A running Kubernetes cluster contains node agents (`kubelet`) and master components (APIs, scheduler, etc), on top of a distributed storage solution. This diagram shows our desired eventual state, though we're still working on a few things, like making `kubelet` itself (all our components, really) run within containers, and making the scheduler 100% pluggable.
A running Kubernetes cluster contains node agents (`kubelet`) and master
components (APIs, scheduler, etc), on top of a distributed storage solution.
This diagram shows our desired eventual state, though we're still working on a
few things, like making `kubelet` itself (all our components, really) run within
containers, and making the scheduler 100% pluggable.
![Architecture Diagram](architecture.png?raw=true "Architecture overview")
## The Kubernetes Node
When looking at the architecture of the system, we'll break it down to services that run on the worker node and services that compose the cluster-level control plane.
When looking at the architecture of the system, we'll break it down to services
that run on the worker node and services that compose the cluster-level control
plane.
The Kubernetes node has the services necessary to run application containers and be managed from the master systems.
The Kubernetes node has the services necessary to run application containers and
be managed from the master systems.
Each node runs Docker, of course. Docker takes care of the details of downloading images and running containers.
Each node runs Docker, of course. Docker takes care of the details of
downloading images and running containers.
### `kubelet`
The `kubelet` manages [pods](../user-guide/pods.md) and their containers, their images, their volumes, etc.
The `kubelet` manages [pods](../user-guide/pods.md) and their containers, their
images, their volumes, etc.
### `kube-proxy`
Each node also runs a simple network proxy and load balancer (see the [services FAQ](https://github.com/kubernetes/kubernetes/wiki/Services-FAQ) for more details). This reflects `services` (see [the services doc](../user-guide/services.md) for more details) as defined in the Kubernetes API on each node and can do simple TCP and UDP stream forwarding (round robin) across a set of backends.
Each node also runs a simple network proxy and load balancer (see the
[services FAQ](https://github.com/kubernetes/kubernetes/wiki/Services-FAQ) for
more details). This reflects `services` (see
[the services doc](../user-guide/services.md) for more details) as defined in
the Kubernetes API on each node and can do simple TCP and UDP stream forwarding
(round robin) across a set of backends.
Service endpoints are currently found via [DNS](../admin/dns.md) or through environment variables (both [Docker-links-compatible](https://docs.docker.com/userguide/dockerlinks/) and Kubernetes `{FOO}_SERVICE_HOST` and `{FOO}_SERVICE_PORT` variables are supported). These variables resolve to ports managed by the service proxy.
Service endpoints are currently found via [DNS](../admin/dns.md) or through
environment variables (both
[Docker-links-compatible](https://docs.docker.com/userguide/dockerlinks/) and
Kubernetes `{FOO}_SERVICE_HOST` and `{FOO}_SERVICE_PORT` variables are
supported). These variables resolve to ports managed by the service proxy.
## The Kubernetes Control Plane
The Kubernetes control plane is split into a set of components. Currently they all run on a single _master_ node, but that is expected to change soon in order to support high-availability clusters. These components work together to provide a unified view of the cluster.
The Kubernetes control plane is split into a set of components. Currently they
all run on a single _master_ node, but that is expected to change soon in order
to support high-availability clusters. These components work together to provide
a unified view of the cluster.
### `etcd`
All persistent master state is stored in an instance of `etcd`. This provides a great way to store configuration data reliably. With `watch` support, coordinating components can be notified very quickly of changes.
All persistent master state is stored in an instance of `etcd`. This provides a
great way to store configuration data reliably. With `watch` support,
coordinating components can be notified very quickly of changes.
### Kubernetes API Server
The apiserver serves up the [Kubernetes API](../api.md). It is intended to be a CRUD-y server, with most/all business logic implemented in separate components or in plug-ins. It mainly processes REST operations, validates them, and updates the corresponding objects in `etcd` (and eventually other stores).
The apiserver serves up the [Kubernetes API](../api.md). It is intended to be a
CRUD-y server, with most/all business logic implemented in separate components
or in plug-ins. It mainly processes REST operations, validates them, and updates
the corresponding objects in `etcd` (and eventually other stores).
### Scheduler
The scheduler binds unscheduled pods to nodes via the `/binding` API. The scheduler is pluggable, and we expect to support multiple cluster schedulers and even user-provided schedulers in the future.
The scheduler binds unscheduled pods to nodes via the `/binding` API. The
scheduler is pluggable, and we expect to support multiple cluster schedulers and
even user-provided schedulers in the future.
### Kubernetes Controller Manager Server
All other cluster-level functions are currently performed by the Controller Manager. For instance, `Endpoints` objects are created and updated by the endpoints controller, and nodes are discovered, managed, and monitored by the node controller. These could eventually be split into separate components to make them independently pluggable.
All other cluster-level functions are currently performed by the Controller
Manager. For instance, `Endpoints` objects are created and updated by the
endpoints controller, and nodes are discovered, managed, and monitored by the
node controller. These could eventually be split into separate components to
make them independently pluggable.
The [`replicationcontroller`](../user-guide/replication-controller.md) is a mechanism that is layered on top of the simple [`pod`](../user-guide/pods.md) API. We eventually plan to port it to a generic plug-in mechanism, once one is implemented.
The [`replicationcontroller`](../user-guide/replication-controller.md) is a
mechanism that is layered on top of the simple [`pod`](../user-guide/pods.md)
API. We eventually plan to port it to a generic plug-in mechanism, once one is
implemented.
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
......
docs/design/clustering/README.md
View file @ 97f1f599
......@@ -33,7 +33,8 @@ Documentation for other releases can be found at
<!-- END MUNGE: UNVERSIONED_WARNING -->
This directory contains diagrams for the clustering design doc.
This depends on the `seqdiag` [utility](http://blockdiag.com/en/seqdiag/index.html). Assuming you have a non-borked python install, this should be installable with
This depends on the `seqdiag` [utility](http://blockdiag.com/en/seqdiag/index.html).
Assuming you have a non-borked python install, this should be installable with:
```sh
pip install seqdiag
......@@ -43,7 +44,8 @@ Just call `make` to regenerate the diagrams.
## Building with Docker
If you are on a Mac or your pip install is messed up, you can easily build with docker.
If you are on a Mac or your pip install is messed up, you can easily build with
docker:
```sh
make docker
......@@ -51,13 +53,18 @@ make docker
The first run will be slow but things should be fast after that.
To clean up the docker containers that are created (and other cruft that is left around) you can run `make docker-clean`.
To clean up the docker containers that are created (and other cruft that is left
around) you can run `make docker-clean`.
If you are using boot2docker and get warnings about clock skew (or if things aren't building for some reason) then you can fix that up with `make fix-clock-skew`.
If you are using boot2docker and get warnings about clock skew (or if things
aren't building for some reason) then you can fix that up with
`make fix-clock-skew`.
## Automatically rebuild on file changes
If you have the fswatch utility installed, you can have it monitor the file system and automatically rebuild when files have changed. Just do a `make watch`.
If you have the fswatch utility installed, you can have it monitor the file
system and automatically rebuild when files have changed. Just do a
`make watch`.
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
......
docs/design/command_execution_port_forwarding.md
View file @ 97f1f599
......@@ -36,14 +36,13 @@ Documentation for other releases can be found at
## Abstract
This describes an approach for providing support for:
- executing commands in containers, with stdin/stdout/stderr streams attached
- port forwarding to containers
This document describes how to use Kubernetes to execute commands in containers,
with stdin/stdout/stderr streams attached and how to implement port forwarding
to the containers.
## Background
There are several related issues/PRs:
See the following related issues/PRs:
- [Support attach](http://issue.k8s.io/1521)
- [Real container ssh](http://issue.k8s.io/1513)
......@@ -77,34 +76,39 @@ won't be able to work with this mechanism, unless adapters can be written.
## Constraints and Assumptions
- SSH support is not currently in scope
- CGroup confinement is ultimately desired, but implementing that support is not currently in scope
- SELinux confinement is ultimately desired, but implementing that support is not currently in scope
- SSH support is not currently in scope.
- CGroup confinement is ultimately desired, but implementing that support is not
currently in scope.
- SELinux confinement is ultimately desired, but implementing that support is
not currently in scope.
## Use Cases
- As a user of a Kubernetes cluster, I want to run arbitrary commands in a container, attaching my local stdin/stdout/stderr to the container
- As a user of a Kubernetes cluster, I want to be able to connect to local ports on my computer and have them forwarded to ports in the container
- A user of a Kubernetes cluster wants to run arbitrary commands in a
container with local stdin/stdout/stderr attached to the container.
- A user of a Kubernetes cluster wants to connect to local ports on his computer
and have them forwarded to ports in a container.
## Process Flow
### Remote Command Execution Flow
1. The client connects to the Kubernetes Master to initiate a remote command execution
request
2. The Master proxies the request to the Kubelet where the container lives
3. The Kubelet executes nsenter + the requested command and streams stdin/stdout/stderr back and forth between the client and the container
1. The client connects to the Kubernetes Master to initiate a remote command
execution request.
2. The Master proxies the request to the Kubelet where the container lives.
3. The Kubelet executes nsenter + the requested command and streams
stdin/stdout/stderr back and forth between the client and the container.
### Port Forwarding Flow
1. The client connects to the Kubernetes Master to initiate a remote command execution
request
2. The Master proxies the request to the Kubelet where the container lives
3. The client listens on each specified local port, awaiting local connections
4. The client connects to one of the local listening ports
4. The client notifies the Kubelet of the new connection
5. The Kubelet executes nsenter + socat and streams data back and forth between the client and the port in the container
1. The client connects to the Kubernetes Master to initiate a remote command
execution request.
2. The Master proxies the request to the Kubelet where the container lives.
3. The client listens on each specified local port, awaiting local connections.
4. The client connects to one of the local listening ports.
4. The client notifies the Kubelet of the new connection.
5. The Kubelet executes nsenter + socat and streams data back and forth between
the client and the port in the container.
## Design Considerations
......@@ -177,7 +181,10 @@ functionality. We need to make sure that users are not allowed to execute
remote commands or do port forwarding to containers they aren't allowed to
access.
Additional work is required to ensure that multiple command execution or port forwarding connections from different clients are not able to see each other's data. This can most likely be achieved via SELinux labeling and unique process contexts.
Additional work is required to ensure that multiple command execution or port
forwarding connections from different clients are not able to see each other's
data. This can most likely be achieved via SELinux labeling and unique process
contexts.
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
......
docs/design/configmap.md
View file @ 97f1f599
......@@ -36,8 +36,8 @@ Documentation for other releases can be found at
## Abstract
The `ConfigMap` API resource stores data used for the configuration of applications deployed on
Kubernetes.
The `ConfigMap` API resource stores data used for the configuration of
applications deployed on Kubernetes.
The main focus of this resource is to:
......@@ -47,71 +47,74 @@ The main focus of this resource is to:
## Motivation
A `Secret`-like API resource is needed to store configuration data that pods can consume.
A `Secret`-like API resource is needed to store configuration data that pods can
consume.
Goals of this design:
1. Describe a `ConfigMap` API resource
2. Describe the semantics of consuming `ConfigMap` as environment variables
3. Describe the semantics of consuming `ConfigMap` as files in a volume
1. Describe a `ConfigMap` API resource.
2. Describe the semantics of consuming `ConfigMap` as environment variables.
3. Describe the semantics of consuming `ConfigMap` as files in a volume.
## Use Cases
1. As a user, I want to be able to consume configuration data as environment variables
2. As a user, I want to be able to consume configuration data as files in a volume
3. As a user, I want my view of configuration data in files to be eventually consistent with changes
to the data
1. As a user, I want to be able to consume configuration data as environment
variables.
2. As a user, I want to be able to consume configuration data as files in a
volume.
3. As a user, I want my view of configuration data in files to be eventually
consistent with changes to the data.
### Consuming `ConfigMap` as Environment Variables
Many programs read their configuration from environment variables. `ConfigMap` should be possible
to consume in environment variables. The rough series of events for consuming `ConfigMap` this way
is:
A series of events for consuming `ConfigMap` as environment variables:
1. A `ConfigMap` object is created
2. A pod that consumes the configuration data via environment variables is created
3. The pod is scheduled onto a node
4. The kubelet retrieves the `ConfigMap` resource(s) referenced by the pod and starts the container
processes with the appropriate data in environment variables
1. Create a `ConfigMap` object.
2. Create a pod to consume the configuration data via environment variables.
3. The pod is scheduled onto a node.
4. The Kubelet retrieves the `ConfigMap` resource(s) referenced by the pod and
starts the container processes with the appropriate configuration data from
environment variables.
### Consuming `ConfigMap` in Volumes
Many programs read their configuration from configuration files. `ConfigMap` should be possible
to consume in a volume. The rough series of events for consuming `ConfigMap` this way
is:
A series of events for consuming `ConfigMap` as configuration files in a volume:
1. A `ConfigMap` object is created
2. A new pod using the `ConfigMap` via the volume plugin is created
3. The pod is scheduled onto a node
4. The Kubelet creates an instance of the volume plugin and calls its `Setup()` method
5. The volume plugin retrieves the `ConfigMap` resource(s) referenced by the pod and projects
the appropriate data into the volume
1. Create a `ConfigMap` object.
2. Create a new pod using the `ConfigMap` via a volume plugin.
3. The pod is scheduled onto a node.
4. The Kubelet creates an instance of the volume plugin and calls its `Setup()`
method.
5. The volume plugin retrieves the `ConfigMap` resource(s) referenced by the pod
and projects the appropriate configuration data into the volume.
### Consuming `ConfigMap` Updates
Any long-running system has configuration that is mutated over time. Changes made to configuration
data must be made visible to pods consuming data in volumes so that they can respond to those
changes.
Any long-running system has configuration that is mutated over time. Changes
made to configuration data must be made visible to pods consuming data in
volumes so that they can respond to those changes.
The `resourceVersion` of the `ConfigMap` object will be updated by the API server every time the
object is modified. After an update, modifications will be made visible to the consumer container:
The `resourceVersion` of the `ConfigMap` object will be updated by the API
server every time the object is modified. After an update, modifications will be
made visible to the consumer container:
1. A `ConfigMap` object is created
2. A new pod using the `ConfigMap` via the volume plugin is created
3. The pod is scheduled onto a node
4. During the sync loop, the Kubelet creates an instance of the volume plugin and calls its
`Setup()` method
5. The volume plugin retrieves the `ConfigMap` resource(s) referenced by the pod and projects
the appropriate data into the volume
6. The `ConfigMap` referenced by the pod is updated
7. During the next iteration of the `syncLoop`, the Kubelet creates an instance of the volume plugin
and calls its `Setup()` method
8. The volume plugin projects the updated data into the volume atomically
1. Create a `ConfigMap` object.
2. Create a new pod using the `ConfigMap` via the volume plugin.
3. The pod is scheduled onto a node.
4. During the sync loop, the Kubelet creates an instance of the volume plugin
and calls its `Setup()` method.
5. The volume plugin retrieves the `ConfigMap` resource(s) referenced by the pod
and projects the appropriate data into the volume.
6. The `ConfigMap` referenced by the pod is updated.
7. During the next iteration of the `syncLoop`, the Kubelet creates an instance
of the volume plugin and calls its `Setup()` method.
8. The volume plugin projects the updated data into the volume atomically.
It is the consuming pod's responsibility to make use of the updated data once it is made visible.
It is the consuming pod's responsibility to make use of the updated data once it
is made visible.
Because environment variables cannot be updated without restarting a container, configuration data
consumed in environment variables will not be updated.
Because environment variables cannot be updated without restarting a container,
configuration data consumed in environment variables will not be updated.
### Advantages
......@@ -133,8 +136,8 @@ type ConfigMap struct {
TypeMeta `json:",inline"`
ObjectMeta `json:"metadata,omitempty"`
// Data contains the configuration data. Each key must be a valid DNS_SUBDOMAIN or leading
// dot followed by valid DNS_SUBDOMAIN.
// Data contains the configuration data. Each key must be a valid
// DNS_SUBDOMAIN or leading dot followed by valid DNS_SUBDOMAIN.
Data map[string]string `json:"data,omitempty"`
}
......@@ -146,7 +149,8 @@ type ConfigMapList struct {
}
```
A `Registry` implementation for `ConfigMap` will be added to `pkg/registry/configmap`.
A `Registry` implementation for `ConfigMap` will be added to
`pkg/registry/configmap`.
### Environment Variables
......@@ -174,8 +178,8 @@ type ConfigMapKeySelector struct {
### Volume Source
A new `ConfigMapVolumeSource` type of volume source containing the `ConfigMap` object will be
added to the `VolumeSource` struct in the API:
A new `ConfigMapVolumeSource` type of volume source containing the `ConfigMap`
object will be added to the `VolumeSource` struct in the API:
```go
package api
......@@ -209,13 +213,14 @@ type KeyToPath struct {
}
```
**Note:** The update logic used in the downward API volume plug-in will be extracted and re-used in
the volume plug-in for `ConfigMap`.
**Note:** The update logic used in the downward API volume plug-in will be
extracted and re-used in the volume plug-in for `ConfigMap`.
### Changes to Secret
We will update the Secret volume plugin to have a similar API to the new ConfigMap volume plugin.
The secret volume plugin will also begin updating secret content in the volume when secrets change.
We will update the Secret volume plugin to have a similar API to the new
`ConfigMap` volume plugin. The secret volume plugin will also begin updating
secret content in the volume when secrets change.
## Examples
......@@ -281,7 +286,8 @@ spec:
#### Consuming `ConfigMap` as Volumes
`redis-volume-config` is intended to be used as a volume containing a config file:
`redis-volume-config` is intended to be used as a volume containing a config
file:
```yaml
apiVersion: v1
......@@ -320,8 +326,8 @@ spec:
## Future Improvements
In the future, we may add the ability to specify an init-container that can watch the volume
contents for updates and respond to changes when they occur.
In the future, we may add the ability to specify an init-container that can
watch the volume contents for updates and respond to changes when they occur.
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/design/configmap.md?pixel)]()
......
docs/design/control-plane-resilience.md
View file @ 97f1f599
......@@ -54,7 +54,7 @@ ideas.
* **High availability:** continuing to be available and work correctly
even if some components are down or uncontactable. This typically
involves multiple replicas of critical services, and a reliable way
to find available replicas. Note that it's possible (but not
to find available replicas. Note that it's possible (but not
desirable) to have high
availability properties (e.g. multiple replicas) in the absence of
self-healing properties (e.g. if a replica fails, nothing replaces
......@@ -109,11 +109,11 @@ ideas.
## Relative Priorities
1. **(Possibly manual) recovery from catastrophic failures:** having a Kubernetes cluster, and all
applications running inside it, disappear forever perhaps is the worst
possible failure mode. So it is critical that we be able to
recover the applications running inside a cluster from such
failures in some well-bounded time period.
1. **(Possibly manual) recovery from catastrophic failures:** having a
Kubernetes cluster, and all applications running inside it, disappear forever
perhaps is the worst possible failure mode. So it is critical that we be able to
recover the applications running inside a cluster from such failures in some
well-bounded time period.
1. In theory a cluster can be recovered by replaying all API calls
that have ever been executed against it, in order, but most
often that state has been lost, and/or is scattered across
......@@ -121,12 +121,12 @@ ideas.
probably infeasible.
1. In theory a cluster can also be recovered to some relatively
recent non-corrupt backup/snapshot of the disk(s) backing the
etcd cluster state. But we have no default consistent
etcd cluster state. But we have no default consistent
backup/snapshot, verification or restoration process. And we
don't routinely test restoration, so even if we did routinely
perform and verify backups, we have no hard evidence that we
can in practise effectively recover from catastrophic cluster
failure or data corruption by restoring from these backups. So
failure or data corruption by restoring from these backups. So
there's more work to be done here.
1. **Self-healing:** Most major cloud providers provide the ability to
easily and automatically replace failed virtual machines within a
......@@ -144,7 +144,6 @@ ideas.
addition](https://github.com/coreos/etcd/blob/master/Documentation/runtime-configuration.md#add-a-new-member)
or [backup and
recovery](https://github.com/coreos/etcd/blob/master/Documentation/admin_guide.md#disaster-recovery)).
1. and boot disks are either:
1. truely persistent (i.e. remote persistent disks), or
1. reconstructible (e.g. using boot-from-snapshot,
......@@ -157,7 +156,7 @@ ideas.
quorum members). In environments where cloud-assisted automatic
self-healing might be infeasible (e.g. on-premise bare-metal
deployments), it also gives cluster administrators more time to
respond (e.g. replace/repair failed machines) without incurring
respond (e.g. replace/repair failed machines) without incurring
system downtime.
## Design and Status (as of December 2015)
......@@ -174,7 +173,7 @@ ideas.
Multiple stateless, self-hosted, self-healing API servers behind a HA
load balancer, built out by the default "kube-up" automation on GCE,
AWS and basic bare metal (BBM). Note that the single-host approach of
AWS and basic bare metal (BBM). Note that the single-host approach of
hving etcd listen only on localhost to ensure that onyl API server can
connect to it will no longer work, so alternative security will be
needed in the regard (either using firewall rules, SSL certs, or
......@@ -189,13 +188,13 @@ design doc.
<td>
No scripted self-healing or HA on GCE, AWS or basic bare metal
currently exists in the OSS distro. To be clear, "no self healing"
currently exists in the OSS distro. To be clear, "no self healing"
means that even if multiple e.g. API servers are provisioned for HA
purposes, if they fail, nothing replaces them, so eventually the
system will fail. Self-healing and HA can be set up
system will fail. Self-healing and HA can be set up
manually by following documented instructions, but this is not
currently an automated process, and it is not tested as part of
continuous integration. So it's probably safest to assume that it
continuous integration. So it's probably safest to assume that it
doesn't actually work in practise.
</td>
......@@ -205,8 +204,8 @@ doesn't actually work in practise.
<td>
Multiple self-hosted, self healing warm standby stateless controller
managers and schedulers with leader election and automatic failover of API server
clients, automatically installed by default "kube-up" automation.
managers and schedulers with leader election and automatic failover of API
server clients, automatically installed by default "kube-up" automation.
</td>
<td>As above.</td>
......@@ -218,47 +217,49 @@ clients, automatically installed by default "kube-up" automation.
Multiple (3-5) etcd quorum members behind a load balancer with session
affinity (to prevent clients from being bounced from one to another).
Regarding self-healing, if a node running etcd goes down, it is always necessary to do three
things:
Regarding self-healing, if a node running etcd goes down, it is always necessary
to do three things:
<ol>
<li>allocate a new node (not necessary if running etcd as a pod, in
which case specific measures are required to prevent user pods from
interfering with system pods, for example using node selectors as
described in <A HREF=")
<li>start an etcd replica on that new node,
described in <A HREF="),
<li>start an etcd replica on that new node, and
<li>have the new replica recover the etcd state.
</ol>
In the case of local disk (which fails in concert with the machine), the etcd
state must be recovered from the other replicas. This is called <A HREF="https://github.com/coreos/etcd/blob/master/Documentation/runtime-configuration.md#add-a-new-member">dynamic member
addition</A>.
In the case of remote persistent disk, the etcd state can be recovered
by attaching the remote persistent disk to the replacement node, thus
the state is recoverable even if all other replicas are down.
state must be recovered from the other replicas. This is called
<A HREF="https://github.com/coreos/etcd/blob/master/Documentation/runtime-configuration.md#add-a-new-member">
dynamic member addition</A>.
In the case of remote persistent disk, the etcd state can be recovered by
attaching the remote persistent disk to the replacement node, thus the state is
recoverable even if all other replicas are down.
There are also significant performance differences between local disks and remote
persistent disks. For example, the <A HREF="https://cloud.google.com/compute/docs/disks/#comparison_of_disk_types">sustained throughput
local disks in GCE is approximatley 20x that of remote disks</A>.
Hence we suggest that self-healing be provided by remotely mounted persistent disks in
non-performance critical, single-zone cloud deployments. For
performance critical installations, faster local SSD's should be used,
in which case remounting on node failure is not an option, so
<A HREF="https://github.com/coreos/etcd/blob/master/Documentation/runtime-configuration.md ">etcd runtime configuration</A>
should be used to replace the failed machine. Similarly, for
cross-zone self-healing, cloud persistent disks are zonal, so
automatic
<A HREF="https://github.com/coreos/etcd/blob/master/Documentation/runtime-configuration.md">runtime configuration</A>
is required. Similarly, basic bare metal deployments cannot generally
rely on
remote persistent disks, so the same approach applies there.
persistent disks. For example, the
<A HREF="https://cloud.google.com/compute/docs/disks/#comparison_of_disk_types">
sustained throughput local disks in GCE is approximatley 20x that of remote
disks</A>.
Hence we suggest that self-healing be provided by remotely mounted persistent
disks in non-performance critical, single-zone cloud deployments. For
performance critical installations, faster local SSD's should be used, in which
case remounting on node failure is not an option, so
<A HREF="https://github.com/coreos/etcd/blob/master/Documentation/runtime-configuration.md ">
etcd runtime configuration</A> should be used to replace the failed machine.
Similarly, for cross-zone self-healing, cloud persistent disks are zonal, so
automatic <A HREF="https://github.com/coreos/etcd/blob/master/Documentation/runtime-configuration.md">
runtime configuration</A> is required. Similarly, basic bare metal deployments
cannot generally rely on remote persistent disks, so the same approach applies
there.
</td>
<td>
<A HREF="http://kubernetes.io/v1.1/docs/admin/high-availability.html">
Somewhat vague instructions exist</A>
on how to set some of this up manually in a self-hosted
configuration. But automatic bootstrapping and self-healing is not
described (and is not implemented for the non-PD cases). This all
still needs to be automated and continuously tested.
Somewhat vague instructions exist</A> on how to set some of this up manually in
a self-hosted configuration. But automatic bootstrapping and self-healing is not
described (and is not implemented for the non-PD cases). This all still needs to
be automated and continuously tested.
</td>
</tr>
</table>
......
docs/design/extending-api.md
View file @ 97f1f599
......@@ -34,59 +34,62 @@ Documentation for other releases can be found at
# Adding custom resources to the Kubernetes API server
This document describes the design for implementing the storage of custom API types in the Kubernetes API Server.
This document describes the design for implementing the storage of custom API
types in the Kubernetes API Server.
## Resource Model
### The ThirdPartyResource
The `ThirdPartyResource` resource describes the multiple versions of a custom resource that the user wants to add
to the Kubernetes API. `ThirdPartyResource` is a non-namespaced resource; attempting to place it in a namespace
will return an error.
The `ThirdPartyResource` resource describes the multiple versions of a custom
resource that the user wants to add to the Kubernetes API. `ThirdPartyResource`
is a non-namespaced resource; attempting to place it in a namespace will return
an error.
Each `ThirdPartyResource` resource has the following:
* Standard Kubernetes object metadata.
* ResourceKind - The kind of the resources described by this third party resource.
* ResourceKind - The kind of the resources described by this third party
resource.
* Description - A free text description of the resource.
* APIGroup - An API group that this resource should be placed into.
* Versions - One or more `Version` objects.
### The `Version` Object
The `Version` object describes a single concrete version of a custom resource. The `Version` object currently
only specifies:
The `Version` object describes a single concrete version of a custom resource.
The `Version` object currently only specifies:
* The `Name` of the version.
* The `APIGroup` this version should belong to.
## Expectations about third party objects
Every object that is added to a third-party Kubernetes object store is expected to contain Kubernetes
compatible [object metadata](../devel/api-conventions.md#metadata). This requirement enables the
Kubernetes API server to provide the following features:
* Filtering lists of objects via label queries
* `resourceVersion`-based optimistic concurrency via compare-and-swap
* Versioned storage
* Event recording
* Integration with basic `kubectl` command line tooling
* Watch for resource changes
The `Kind` for an instance of a third-party object (e.g. CronTab) below is expected to be
programmatically convertible to the name of the resource using
the following conversion. Kinds are expected to be of the form `<CamelCaseKind>`, and the
`APIVersion` for the object is expected to be `<api-group>/<api-version>`. To
prevent collisions, it's expected that you'll use a fully qualified domain
name for the API group, e.g. `example.com`.
Every object that is added to a third-party Kubernetes object store is expected
to contain Kubernetes compatible [object metadata](../devel/api-conventions.md#metadata).
This requirement enables the Kubernetes API server to provide the following
features:
* Filtering lists of objects via label queries.
* `resourceVersion`-based optimistic concurrency via compare-and-swap.
* Versioned storage.
* Event recording.
* Integration with basic `kubectl` command line tooling.
* Watch for resource changes.
The `Kind` for an instance of a third-party object (e.g. CronTab) below is
expected to be programmatically convertible to the name of the resource using
the following conversion. Kinds are expected to be of the form
`<CamelCaseKind>`, and the `APIVersion` for the object is expected to be
`<api-group>/<api-version>`. To prevent collisions, it's expected that you'll
use a fully qualified domain name for the API group, e.g. `example.com`.
For example `stable.example.com/v1`
'CamelCaseKind' is the specific type name.
To convert this into the `metadata.name` for the `ThirdPartyResource` resource instance,
the `<domain-name>` is copied verbatim, the `CamelCaseKind` is
then converted
using '-' instead of capitalization ('camel-case'), with the first character being assumed to be
capitalized. In pseudo code:
To convert this into the `metadata.name` for the `ThirdPartyResource` resource
instance, the `<domain-name>` is copied verbatim, the `CamelCaseKind` is then
converted using '-' instead of capitalization ('camel-case'), with the first
character being assumed to be capitalized. In pseudo code:
```go
var result string
......@@ -98,17 +101,20 @@ for ix := range kindName {
}
```
As a concrete example, the resource named `camel-case-kind.example.com` defines resources of Kind `CamelCaseKind`, in
the APIGroup with the prefix `example.com/...`.
As a concrete example, the resource named `camel-case-kind.example.com` defines
resources of Kind `CamelCaseKind`, in the APIGroup with the prefix
`example.com/...`.
The reason for this is to enable rapid lookup of a `ThirdPartyResource` object given the kind information.
This is also the reason why `ThirdPartyResource` is not namespaced.
The reason for this is to enable rapid lookup of a `ThirdPartyResource` object
given the kind information. This is also the reason why `ThirdPartyResource` is
not namespaced.
## Usage
When a user creates a new `ThirdPartyResource`, the Kubernetes API Server reacts by creating a new, namespaced
RESTful resource path. For now, non-namespaced objects are not supported. As with existing built-in objects,
deleting a namespace deletes all third party resources in that namespace.
When a user creates a new `ThirdPartyResource`, the Kubernetes API Server reacts
by creating a new, namespaced RESTful resource path. For now, non-namespaced
objects are not supported. As with existing built-in objects, deleting a
namespace deletes all third party resources in that namespace.
For example, if a user creates:
......@@ -143,14 +149,15 @@ Now that this schema has been created, a user can `POST`:
to: `/apis/stable.example.com/v1/namespaces/default/crontabs`
and the corresponding data will be stored into etcd by the APIServer, so that when the user issues:
and the corresponding data will be stored into etcd by the APIServer, so that
when the user issues:
```
GET /apis/stable.example.com/v1/namespaces/default/crontabs/my-new-cron-object`
```
And when they do that, they will get back the same data, but with additional Kubernetes metadata
(e.g. `resourceVersion`, `createdTimestamp`) filled in.
And when they do that, they will get back the same data, but with additional
Kubernetes metadata (e.g. `resourceVersion`, `createdTimestamp`) filled in.
Likewise, to list all resources, a user can issue:
......@@ -178,29 +185,35 @@ and get back:
}
```
Because all objects are expected to contain standard Kubernetes metadata fields, these
list operations can also use label queries to filter requests down to specific subsets.
Likewise, clients can use watch endpoints to watch for changes to stored objects.
Because all objects are expected to contain standard Kubernetes metadata fields,
these list operations can also use label queries to filter requests down to
specific subsets.
Likewise, clients can use watch endpoints to watch for changes to stored
objects.
## Storage
In order to store custom user data in a versioned fashion inside of etcd, we need to also introduce a
`Codec`-compatible object for persistent storage in etcd. This object is `ThirdPartyResourceData` and it contains:
* Standard API Metadata
In order to store custom user data in a versioned fashion inside of etcd, we
need to also introduce a `Codec`-compatible object for persistent storage in
etcd. This object is `ThirdPartyResourceData` and it contains:
* Standard API Metadata.
* `Data`: The raw JSON data for this custom object.
### Storage key specification
Each custom object stored by the API server needs a custom key in storage, this is described below:
Each custom object stored by the API server needs a custom key in storage, this
is described below:
#### Definitions
* `resource-namespace`: the namespace of the particular resource that is being stored
* `resource-namespace`: the namespace of the particular resource that is
being stored
* `resource-name`: the name of the particular resource being stored
* `third-party-resource-namespace`: the namespace of the `ThirdPartyResource` resource that represents the type for the specific instance being stored
* `third-party-resource-name`: the name of the `ThirdPartyResource` resource that represents the type for the specific instance being stored
* `third-party-resource-namespace`: the namespace of the `ThirdPartyResource`
resource that represents the type for the specific instance being stored
* `third-party-resource-name`: the name of the `ThirdPartyResource` resource
that represents the type for the specific instance being stored
#### Key
......
docs/design/federation-phase-1.md
View file @ 97f1f599
......@@ -71,7 +71,8 @@ unified view.
Here are the functionality requirements derived from above use cases:
+ Clients of the federation control plane API server can register and deregister clusters.
+ Clients of the federation control plane API server can register and deregister
clusters.
+ Workloads should be spread to different clusters according to the
workload distribution policy.
+ Pods are able to discover and connect to services hosted in other
......@@ -90,7 +91,7 @@ Here are the functionality requirements derived from above use cases:
It’s difficult to have a perfect design with one click that implements
all the above requirements. Therefore we will go with an iterative
approach to design and build the system. This document describes the
phase one of the whole work. In phase one we will cover only the
phase one of the whole work. In phase one we will cover only the
following objectives:
+ Define the basic building blocks and API objects of control plane
......@@ -130,9 +131,9 @@ description of each module contained in above diagram.
The API Server in the Ubernetes control plane works just like the API
Server in K8S. It talks to a distributed key-value store to persist,
retrieve and watch API objects. This store is completely distinct
retrieve and watch API objects. This store is completely distinct
from the kubernetes key-value stores (etcd) in the underlying
kubernetes clusters. We still use `etcd` as the distributed
kubernetes clusters. We still use `etcd` as the distributed
storage so customers don’t need to learn and manage a different
storage system, although it is envisaged that other storage systems
(consol, zookeeper) will probably be developedand supported over
......@@ -141,16 +142,16 @@ time.
## Ubernetes Scheduler
The Ubernetes Scheduler schedules resources onto the underlying
Kubernetes clusters. For example it watches for unscheduled Ubernetes
Kubernetes clusters. For example it watches for unscheduled Ubernetes
replication controllers (those that have not yet been scheduled onto
underlying Kubernetes clusters) and performs the global scheduling
work. For each unscheduled replication controller, it calls policy
work. For each unscheduled replication controller, it calls policy
engine to decide how to spit workloads among clusters. It creates a
Kubernetes Replication Controller on one ore more underlying cluster,
and post them back to `etcd` storage.
One sublety worth noting here is that the scheduling decision is
arrived at by combining the application-specific request from the user (which might
One sublety worth noting here is that the scheduling decision is arrived at by
combining the application-specific request from the user (which might
include, for example, placement constraints), and the global policy specified
by the federation administrator (for example, "prefer on-premise
clusters over AWS clusters" or "spread load equally across clusters").
......@@ -165,9 +166,9 @@ performs the following two kinds of work:
corresponding API objects on the underlying K8S clusters.
1. It periodically retrieves the available resources metrics from the
underlying K8S cluster, and updates them as object status of the
`cluster` API object. An alternative design might be to run a pod
`cluster` API object. An alternative design might be to run a pod
in each underlying cluster that reports metrics for that cluster to
the Ubernetes control plane. Which approach is better remains an
the Ubernetes control plane. Which approach is better remains an
open topic of discussion.
## Ubernetes Service Controller
......@@ -187,7 +188,7 @@ Cluster is a new first-class API object introduced in this design. For
each registered K8S cluster there will be such an API resource in
control plane. The way clients register or deregister a cluster is to
send corresponding REST requests to following URL:
`/api/{$version}/clusters`. Because control plane is behaving like a
`/api/{$version}/clusters`. Because control plane is behaving like a
regular K8S client to the underlying clusters, the spec of a cluster
object contains necessary properties like K8S cluster address and
credentials. The status of a cluster API object will contain
......@@ -294,7 +295,7 @@ $version.clusterStatus
**For simplicity we didn’t introduce a separate “cluster metrics” API
object here**. The cluster resource metrics are stored in cluster
status section, just like what we did to nodes in K8S. In phase one it
only contains available CPU resources and memory resources. The
only contains available CPU resources and memory resources. The
cluster controller will periodically poll the underlying cluster API
Server to get cluster capability. In phase one it gets the metrics by
simply aggregating metrics from all nodes. In future we will improve
......@@ -315,7 +316,7 @@ Below is the state transition diagram.
## Replication Controller
A global workload submitted to control plane is represented as an
Ubernetes replication controller. When a replication controller
Ubernetes replication controller. When a replication controller
is submitted to control plane, clients need a way to express its
requirements or preferences on clusters. Depending on different use
cases it may be complex. For example:
......@@ -327,7 +328,7 @@ cases it may be complex. For example:
(use case: workload )
+ Seventy percent of this workload should be scheduled to cluster Foo,
and thirty percent should be scheduled to cluster Bar (use case:
vendor lock-in avoidance). In phase one, we only introduce a
vendor lock-in avoidance). In phase one, we only introduce a
_clusterSelector_ field to filter acceptable clusters. In default
case there is no such selector and it means any cluster is
acceptable.
......@@ -376,7 +377,7 @@ clusters. How to handle this will be addressed after phase one.
The Service API object exposed by Ubernetes is similar to service
objects on Kubernetes. It defines the access to a group of pods. The
Ubernetes service controller will create corresponding Kubernetes
service objects on underlying clusters. These are detailed in a
service objects on underlying clusters. These are detailed in a
separate design document: [Federated Services](federated-services.md).
## Pod
......@@ -389,7 +390,8 @@ order to keep the Ubernetes API compatible with the Kubernetes API.
## Scheduling
The below diagram shows how workloads are scheduled on the Ubernetes control plane:
The below diagram shows how workloads are scheduled on the Ubernetes control\
plane:
1. A replication controller is created by the client.
1. APIServer persists it into the storage.
......@@ -425,8 +427,8 @@ proposed solutions like resource reservation mechanisms.
This part has been included in the section “Federated Service” of
document
“[Ubernetes Cross-cluster Load Balancing and Service Discovery Requirements and System Design](federated-services.md))”. Please
refer to that document for details.
“[Ubernetes Cross-cluster Load Balancing and Service Discovery Requirements and System Design](federated-services.md))”.
Please refer to that document for details.
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
......
docs/design/identifiers.md
View file @ 97f1f599
......@@ -34,95 +34,111 @@ Documentation for other releases can be found at
# Identifiers and Names in Kubernetes
A summarization of the goals and recommendations for identifiers in Kubernetes. Described in [GitHub issue #199](http://issue.k8s.io/199).
A summarization of the goals and recommendations for identifiers in Kubernetes.
Described in GitHub issue [#199](http://issue.k8s.io/199).
## Definitions
UID
: A non-empty, opaque, system-generated value guaranteed to be unique in time and space; intended to distinguish between historical occurrences of similar entities.
`UID`: A non-empty, opaque, system-generated value guaranteed to be unique in time
and space; intended to distinguish between historical occurrences of similar
entities.
Name
: A non-empty string guaranteed to be unique within a given scope at a particular time; used in resource URLs; provided by clients at creation time and encouraged to be human friendly; intended to facilitate creation idempotence and space-uniqueness of singleton objects, distinguish distinct entities, and reference particular entities across operations.
`Name`: A non-empty string guaranteed to be unique within a given scope at a
particular time; used in resource URLs; provided by clients at creation time and
encouraged to be human friendly; intended to facilitate creation idempotence and
space-uniqueness of singleton objects, distinguish distinct entities, and
reference particular entities across operations.
[rfc1035](http://www.ietf.org/rfc/rfc1035.txt)/[rfc1123](http://www.ietf.org/rfc/rfc1123.txt) label (DNS_LABEL)
: An alphanumeric (a-z, and 0-9) string, with a maximum length of 63 characters, with the '-' character allowed anywhere except the first or last character, suitable for use as a hostname or segment in a domain name
[rfc1035](http://www.ietf.org/rfc/rfc1035.txt)/[rfc1123](http://www.ietf.org/rfc/rfc1123.txt) `label` (DNS_LABEL):
An alphanumeric (a-z, and 0-9) string, with a maximum length of 63 characters,
with the '-' character allowed anywhere except the first or last character,
suitable for use as a hostname or segment in a domain name.
[rfc1035](http://www.ietf.org/rfc/rfc1035.txt)/[rfc1123](http://www.ietf.org/rfc/rfc1123.txt) subdomain (DNS_SUBDOMAIN)
: One or more lowercase rfc1035/rfc1123 labels separated by '.' with a maximum length of 253 characters
[rfc1035](http://www.ietf.org/rfc/rfc1035.txt)/[rfc1123](http://www.ietf.org/rfc/rfc1123.txt) `subdomain` (DNS_SUBDOMAIN):
One or more lowercase rfc1035/rfc1123 labels separated by '.' with a maximum
length of 253 characters.
[rfc4122](http://www.ietf.org/rfc/rfc4122.txt) universally unique identifier (UUID)
: A 128 bit generated value that is extremely unlikely to collide across time and space and requires no central coordination
[rfc4122](http://www.ietf.org/rfc/rfc4122.txt) `universally unique identifier` (UUID):
A 128 bit generated value that is extremely unlikely to collide across time and
space and requires no central coordination.
[rfc6335](https://tools.ietf.org/rfc/rfc6335.txt) port name (IANA_SVC_NAME)
: An alphanumeric (a-z, and 0-9) string, with a maximum length of 15 characters, with the '-' character allowed anywhere except the first or the last character or adjacent to another '-' character, it must contain at least a (a-z) character
[rfc6335](https://tools.ietf.org/rfc/rfc6335.txt) `port name` (IANA_SVC_NAME):
An alphanumeric (a-z, and 0-9) string, with a maximum length of 15 characters,
with the '-' character allowed anywhere except the first or the last character
or adjacent to another '-' character, it must contain at least a (a-z)
character.
## Objectives for names and UIDs
1. Uniquely identify (via a UID) an object across space and time
2. Uniquely name (via a name) an object across space
3. Provide human-friendly names in API operations and/or configuration files
4. Allow idempotent creation of API resources (#148) and enforcement of space-uniqueness of singleton objects
5. Allow DNS names to be automatically generated for some objects
1. Uniquely identify (via a UID) an object across space and time.
2. Uniquely name (via a name) an object across space.
3. Provide human-friendly names in API operations and/or configuration files.
4. Allow idempotent creation of API resources (#148) and enforcement of
space-uniqueness of singleton objects.
5. Allow DNS names to be automatically generated for some objects.
## General design
1. When an object is created via an API, a Name string (a DNS_SUBDOMAIN) must be specified. Name must be non-empty and unique within the apiserver. This enables idempotent and space-unique creation operations. Parts of the system (e.g. replication controller) may join strings (e.g. a base name and a random suffix) to create a unique Name. For situations where generating a name is impractical, some or all objects may support a param to auto-generate a name. Generating random names will defeat idempotency.
1. When an object is created via an API, a Name string (a DNS_SUBDOMAIN) must
be specified. Name must be non-empty and unique within the apiserver. This
enables idempotent and space-unique creation operations. Parts of the system
(e.g. replication controller) may join strings (e.g. a base name and a random
suffix) to create a unique Name. For situations where generating a name is
impractical, some or all objects may support a param to auto-generate a name.
Generating random names will defeat idempotency.
* Examples: "guestbook.user", "backend-x4eb1"
2. When an object is created via an API, a Namespace string (a DNS_SUBDOMAIN? format TBD via #1114) may be specified. Depending on the API receiver, namespaces might be validated (e.g. apiserver might ensure that the namespace actually exists). If a namespace is not specified, one will be assigned by the API receiver. This assignment policy might vary across API receivers (e.g. apiserver might have a default, kubelet might generate something semi-random).
2. When an object is created via an API, a Namespace string (a DNS_SUBDOMAIN?
format TBD via #1114) may be specified. Depending on the API receiver,
namespaces might be validated (e.g. apiserver might ensure that the namespace
actually exists). If a namespace is not specified, one will be assigned by the
API receiver. This assignment policy might vary across API receivers (e.g.
apiserver might have a default, kubelet might generate something semi-random).
* Example: "api.k8s.example.com"
3. Upon acceptance of an object via an API, the object is assigned a UID (a UUID). UID must be non-empty and unique across space and time.
3. Upon acceptance of an object via an API, the object is assigned a UID
(a UUID). UID must be non-empty and unique across space and time.
* Example: "01234567-89ab-cdef-0123-456789abcdef"
## Case study: Scheduling a pod
Pods can be placed onto a particular node in a number of ways. This case
study demonstrates how the above design can be applied to satisfy the
objectives.
Pods can be placed onto a particular node in a number of ways. This case study
demonstrates how the above design can be applied to satisfy the objectives.
### A pod scheduled by a user through the apiserver
1. A user submits a pod with Namespace="" and Name="guestbook" to the apiserver.
2. The apiserver validates the input.
1. A default Namespace is assigned.
2. The pod name must be space-unique within the Namespace.
3. Each container within the pod has a name which must be space-unique within the pod.
3. Each container within the pod has a name which must be space-unique within
the pod.
3. The pod is accepted.
1. A new UID is assigned.
4. The pod is bound to a node.
1. The kubelet on the node is passed the pod's UID, Namespace, and Name.
5. Kubelet validates the input.
6. Kubelet runs the pod.
1. Each container is started up with enough metadata to distinguish the pod from whence it came.
2. Each attempt to run a container is assigned a UID (a string) that is unique across time.
* This may correspond to Docker's container ID.
1. Each container is started up with enough metadata to distinguish the pod
from whence it came.
2. Each attempt to run a container is assigned a UID (a string) that is
unique across time. * This may correspond to Docker's container ID.
### A pod placed by a config file on the node
1. A config file is stored on the node, containing a pod with UID="", Namespace="", and Name="cadvisor".
1. A config file is stored on the node, containing a pod with UID="",
Namespace="", and Name="cadvisor".
2. Kubelet validates the input.
1. Since UID is not provided, kubelet generates one.
2. Since Namespace is not provided, kubelet generates one.
1. The generated namespace should be deterministic and cluster-unique for the source, such as a hash of the hostname and file path.
1. The generated namespace should be deterministic and cluster-unique for
the source, such as a hash of the hostname and file path.
* E.g. Namespace="file-f4231812554558a718a01ca942782d81"
3. Kubelet runs the pod.
1. Each container is started up with enough metadata to distinguish the pod from whence it came.
2. Each attempt to run a container is assigned a UID (a string) that is unique across time.
1. Each container is started up with enough metadata to distinguish the pod
from whence it came.
2. Each attempt to run a container is assigned a UID (a string) that is
unique across time.
1. This may correspond to Docker's container ID.
......
docs/design/metadata-policy.md
View file @ 97f1f599
......@@ -38,21 +38,24 @@ Documentation for other releases can be found at
This document describes a new API resource, `MetadataPolicy`, that configures an
admission controller to take one or more actions based on an object's metadata.
Initially the metadata fields that the predicates can examine are labels and annotations,
and the actions are to add one or more labels and/or annotations, or to reject creation/update
of the object. In the future other actions might be supported, such as applying an initializer.
The first use of `MetadataPolicy` will be to decide which scheduler should schedule a pod
in a [multi-scheduler](../proposals/multiple-schedulers.md) Kubernetes system. In particular, the
policy will add the scheduler name annotation to a pod based on an annotation that
is already on the pod that indicates the QoS of the pod.
(That annotation was presumably set by a simpler admission controller that
uses code, rather than configuration, to map the resource requests and limits of a pod
to QoS, and attaches the corresponding annotation.)
We anticipate a number of other uses for `MetadataPolicy`, such as defaulting for
labels and annotations, prohibiting/requiring particular labels or annotations, or
choosing a scheduling policy within a scheduler. We do not discuss them in this doc.
Initially the metadata fields that the predicates can examine are labels and
annotations, and the actions are to add one or more labels and/or annotations,
or to reject creation/update of the object. In the future other actions might be
supported, such as applying an initializer.
The first use of `MetadataPolicy` will be to decide which scheduler should
schedule a pod in a [multi-scheduler](../proposals/multiple-schedulers.md)
Kubernetes system. In particular, the policy will add the scheduler name
annotation to a pod based on an annotation that is already on the pod that
indicates the QoS of the pod. (That annotation was presumably set by a simpler
admission controller that uses code, rather than configuration, to map the
resource requests and limits of a pod to QoS, and attaches the corresponding
annotation.)
We anticipate a number of other uses for `MetadataPolicy`, such as defaulting
for labels and annotations, prohibiting/requiring particular labels or
annotations, or choosing a scheduling policy within a scheduler. We do not
discuss them in this doc.
## API
......@@ -126,7 +129,8 @@ type MetadataPolicyList struct {
## Implementation plan
1. Create `MetadataPolicy` API resource
1. Create admission controller that implements policies defined in `MetadataPolicy`
1. Create admission controller that implements policies defined in
`MetadataPolicy`
1. Create admission controller that sets annotation
`scheduler.alpha.kubernetes.io/qos: <QoS>`
(where `QOS` is one of `Guaranteed, Burstable, BestEffort`)
......@@ -134,30 +138,32 @@ based on pod's resource request and limit.
## Future work
Longer-term we will have QoS be set on create and update by the registry, similar to `Pending` phase today,
instead of having an admission controller (that runs before the one that takes `MetadataPolicy` as input)
do it.
Longer-term we will have QoS be set on create and update by the registry,
similar to `Pending` phase today, instead of having an admission controller
(that runs before the one that takes `MetadataPolicy` as input) do it.
We plan to eventually move from having an admission controller
set the scheduler name as a pod annotation, to using the initializer concept. In particular, the
scheduler will be an initializer, and the admission controller that decides which scheduler to use
will add the scheduler's name to the list of initializers for the pod (presumably the scheduler
will be the last initializer to run on each pod).
The admission controller would still be configured using the `MetadataPolicy` described here, only the
mechanism the admission controller uses to record its decision of which scheduler to use would change.
We plan to eventually move from having an admission controller set the scheduler
name as a pod annotation, to using the initializer concept. In particular, the
scheduler will be an initializer, and the admission controller that decides
which scheduler to use will add the scheduler's name to the list of initializers
for the pod (presumably the scheduler will be the last initializer to run on
each pod). The admission controller would still be configured using the
`MetadataPolicy` described here, only the mechanism the admission controller
uses to record its decision of which scheduler to use would change.
## Related issues
The main issue for multiple schedulers is #11793. There was also a lot of discussion
in PRs #17197 and #17865.
The main issue for multiple schedulers is #11793. There was also a lot of
discussion in PRs #17197 and #17865.
We could use the approach described here to choose a scheduling
policy within a single scheduler, as opposed to choosing a scheduler, a desire mentioned in #9920.
Issue #17097 describes a scenario unrelated to scheduler-choosing where `MetadataPolicy` could be used.
Issue #17324 proposes to create a generalized API for matching
"claims" to "service classes"; matching a pod to a scheduler would be one use for such an API.
We could use the approach described here to choose a scheduling policy within a
single scheduler, as opposed to choosing a scheduler, a desire mentioned in
# 9920. Issue #17097 describes a scenario unrelated to scheduler-choosing where
`MetadataPolicy` could be used. Issue #17324 proposes to create a generalized
API for matching "claims" to "service classes"; matching a pod to a scheduler
would be one use for such an API.
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
......
docs/design/networking.md
View file @ 97f1f599
......@@ -44,9 +44,9 @@ There are 4 distinct networking problems to solve:
## Model and motivation
Kubernetes deviates from the default Docker networking model (though as of
Docker 1.8 their network plugins are getting closer). The goal is for each pod
Docker 1.8 their network plugins are getting closer). The goal is for each pod
to have an IP in a flat shared networking namespace that has full communication
with other physical computers and containers across the network. IP-per-pod
with other physical computers and containers across the network. IP-per-pod
creates a clean, backward-compatible model where pods can be treated much like
VMs or physical hosts from the perspectives of port allocation, networking,
naming, service discovery, load balancing, application configuration, and
......@@ -71,15 +71,15 @@ among other problems.
All containers within a pod behave as if they are on the same host with regard
to networking. They can all reach each other’s ports on localhost. This offers
simplicity (static ports know a priori), security (ports bound to localhost
are visible within the pod but never outside it), and performance. This also
are visible within the pod but never outside it), and performance. This also
reduces friction for applications moving from the world of uncontainerized apps
on physical or virtual hosts. People running application stacks together on
on physical or virtual hosts. People running application stacks together on
the same host have already figured out how to make ports not conflict and have
arranged for clients to find them.
The approach does reduce isolation between containers within a pod &mdash;
ports could conflict, and there can be no container-private ports, but these
seem to be relatively minor issues with plausible future workarounds. Besides,
seem to be relatively minor issues with plausible future workarounds. Besides,
the premise of pods is that containers within a pod share some resources
(volumes, cpu, ram, etc.) and therefore expect and tolerate reduced isolation.
Additionally, the user can control what containers belong to the same pod
......@@ -88,7 +88,7 @@ whereas, in general, they don't control what pods land together on a host.
## Pod to pod
Because every pod gets a "real" (not machine-private) IP address, pods can
communicate without proxies or translations. The pod can use well-known port
communicate without proxies or translations. The pod can use well-known port
numbers and can avoid the use of higher-level service discovery systems like
DNS-SD, Consul, or Etcd.
......@@ -98,7 +98,7 @@ each pod has its own IP address that other pods can know. By making IP addresses
and ports the same both inside and outside the pods, we create a NAT-less, flat
address space. Running "ip addr show" should work as expected. This would enable
all existing naming/discovery mechanisms to work out of the box, including
self-registration mechanisms and applications that distribute IP addresses. We
self-registration mechanisms and applications that distribute IP addresses. We
should be optimizing for inter-pod network communication. Within a pod,
containers are more likely to use communication through volumes (e.g., tmpfs) or
IPC.
......@@ -141,7 +141,7 @@ gcloud compute routes add "${NODE_NAMES[$i]}" \
--next-hop-instance-zone "${ZONE}" &
```
GCE itself does not know anything about these IPs, though. This means that when
GCE itself does not know anything about these IPs, though. This means that when
a pod tries to egress beyond GCE's project the packets must be SNAT'ed
(masqueraded) to the VM's IP, which GCE recognizes and allows.
......@@ -161,26 +161,26 @@ to serve the purpose outside of GCE.
## Pod to service
The [service](../user-guide/services.md) abstraction provides a way to group pods under a
common access policy (e.g. load-balanced). The implementation of this creates a
common access policy (e.g. load-balanced). The implementation of this creates a
virtual IP which clients can access and which is transparently proxied to the
pods in a Service. Each node runs a kube-proxy process which programs
pods in a Service. Each node runs a kube-proxy process which programs
`iptables` rules to trap access to service IPs and redirect them to the correct
backends. This provides a highly-available load-balancing solution with low
backends. This provides a highly-available load-balancing solution with low
performance overhead by balancing client traffic from a node on that same node.
## External to internal
So far the discussion has been about how to access a pod or service from within
the cluster. Accessing a pod from outside the cluster is a bit more tricky. We
the cluster. Accessing a pod from outside the cluster is a bit more tricky. We
want to offer highly-available, high-performance load balancing to target
Kubernetes Services. Most public cloud providers are simply not flexible enough
Kubernetes Services. Most public cloud providers are simply not flexible enough
yet.
The way this is generally implemented is to set up external load balancers (e.g.
GCE's ForwardingRules or AWS's ELB) which target all nodes in a cluster. When
GCE's ForwardingRules or AWS's ELB) which target all nodes in a cluster. When
traffic arrives at a node it is recognized as being part of a particular Service
and routed to an appropriate backend Pod. This does mean that some traffic will
get double-bounced on the network. Once cloud providers have better offerings
and routed to an appropriate backend Pod. This does mean that some traffic will
get double-bounced on the network. Once cloud providers have better offerings
we can take advantage of those.
## Challenges and future work
......@@ -207,7 +207,13 @@ External IP assignment would also simplify DNS support (see below).
### IPv6
IPv6 would be a nice option, also, but we can't depend on it yet. Docker support is in progress: [Docker issue #2974](https://github.com/dotcloud/docker/issues/2974), [Docker issue #6923](https://github.com/dotcloud/docker/issues/6923), [Docker issue #6975](https://github.com/dotcloud/docker/issues/6975). Additionally, direct ipv6 assignment to instances doesn't appear to be supported by major cloud providers (e.g., AWS EC2, GCE) yet. We'd happily take pull requests from people running Kubernetes on bare metal, though. :-)
IPv6 would be a nice option, also, but we can't depend on it yet. Docker support
is in progress: [Docker issue #2974](https://github.com/dotcloud/docker/issues/2974),
[Docker issue #6923](https://github.com/dotcloud/docker/issues/6923),
[Docker issue #6975](https://github.com/dotcloud/docker/issues/6975).
Additionally, direct ipv6 assignment to instances doesn't appear to be supported
by major cloud providers (e.g., AWS EC2, GCE) yet. We'd happily take pull
requests from people running Kubernetes on bare metal, though. :-)
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
......
docs/design/principles.md
View file @ 97f1f599
......@@ -43,26 +43,57 @@ See also the [API conventions](../devel/api-conventions.md).
* All APIs should be declarative.
* API objects should be complementary and composable, not opaque wrappers.
* The control plane should be transparent -- there are no hidden internal APIs.
* The cost of API operations should be proportional to the number of objects intentionally operated upon. Therefore, common filtered lookups must be indexed. Beware of patterns of multiple API calls that would incur quadratic behavior.
* Object status must be 100% reconstructable by observation. Any history kept must be just an optimization and not required for correct operation.
* Cluster-wide invariants are difficult to enforce correctly. Try not to add them. If you must have them, don't enforce them atomically in master components, that is contention-prone and doesn't provide a recovery path in the case of a bug allowing the invariant to be violated. Instead, provide a series of checks to reduce the probability of a violation, and make every component involved able to recover from an invariant violation.
* Low-level APIs should be designed for control by higher-level systems. Higher-level APIs should be intent-oriented (think SLOs) rather than implementation-oriented (think control knobs).
* The cost of API operations should be proportional to the number of objects
intentionally operated upon. Therefore, common filtered lookups must be indexed.
Beware of patterns of multiple API calls that would incur quadratic behavior.
* Object status must be 100% reconstructable by observation. Any history kept
must be just an optimization and not required for correct operation.
* Cluster-wide invariants are difficult to enforce correctly. Try not to add
them. If you must have them, don't enforce them atomically in master components,
that is contention-prone and doesn't provide a recovery path in the case of a
bug allowing the invariant to be violated. Instead, provide a series of checks
to reduce the probability of a violation, and make every component involved able
to recover from an invariant violation.
* Low-level APIs should be designed for control by higher-level systems.
Higher-level APIs should be intent-oriented (think SLOs) rather than
implementation-oriented (think control knobs).
## Control logic
* Functionality must be *level-based*, meaning the system must operate correctly given the desired state and the current/observed state, regardless of how many intermediate state updates may have been missed. Edge-triggered behavior must be just an optimization.
* Assume an open world: continually verify assumptions and gracefully adapt to external events and/or actors. Example: we allow users to kill pods under control of a replication controller; it just replaces them.
* Do not define comprehensive state machines for objects with behaviors associated with state transitions and/or "assumed" states that cannot be ascertained by observation.
* Don't assume a component's decisions will not be overridden or rejected, nor for the component to always understand why. For example, etcd may reject writes. Kubelet may reject pods. The scheduler may not be able to schedule pods. Retry, but back off and/or make alternative decisions.
* Components should be self-healing. For example, if you must keep some state (e.g., cache) the content needs to be periodically refreshed, so that if an item does get erroneously stored or a deletion event is missed etc, it will be soon fixed, ideally on timescales that are shorter than what will attract attention from humans.
* Component behavior should degrade gracefully. Prioritize actions so that the most important activities can continue to function even when overloaded and/or in states of partial failure.
* Functionality must be *level-based*, meaning the system must operate correctly
given the desired state and the current/observed state, regardless of how many
intermediate state updates may have been missed. Edge-triggered behavior must be
just an optimization.
* Assume an open world: continually verify assumptions and gracefully adapt to
external events and/or actors. Example: we allow users to kill pods under
control of a replication controller; it just replaces them.
* Do not define comprehensive state machines for objects with behaviors
associated with state transitions and/or "assumed" states that cannot be
ascertained by observation.
* Don't assume a component's decisions will not be overridden or rejected, nor
for the component to always understand why. For example, etcd may reject writes.
Kubelet may reject pods. The scheduler may not be able to schedule pods. Retry,
but back off and/or make alternative decisions.
* Components should be self-healing. For example, if you must keep some state
(e.g., cache) the content needs to be periodically refreshed, so that if an item
does get erroneously stored or a deletion event is missed etc, it will be soon
fixed, ideally on timescales that are shorter than what will attract attention
from humans.
* Component behavior should degrade gracefully. Prioritize actions so that the
most important activities can continue to function even when overloaded and/or
in states of partial failure.
## Architecture
* Only the apiserver should communicate with etcd/store, and not other components (scheduler, kubelet, etc.).
* Only the apiserver should communicate with etcd/store, and not other
components (scheduler, kubelet, etc.).
* Compromising a single node shouldn't compromise the cluster.
* Components should continue to do what they were last told in the absence of new instructions (e.g., due to network partition or component outage).
* All components should keep all relevant state in memory all the time. The apiserver should write through to etcd/store, other components should write through to the apiserver, and they should watch for updates made by other clients.
* Components should continue to do what they were last told in the absence of
new instructions (e.g., due to network partition or component outage).
* All components should keep all relevant state in memory all the time. The
apiserver should write through to etcd/store, other components should write
through to the apiserver, and they should watch for updates made by other
clients.
* Watch is preferred over polling.
## Extensibility
......@@ -72,13 +103,23 @@ TODO: pluggability
## Bootstrapping
* [Self-hosting](http://issue.k8s.io/246) of all components is a goal.
* Minimize the number of dependencies, particularly those required for steady-state operation.
* Minimize the number of dependencies, particularly those required for
steady-state operation.
* Stratify the dependencies that remain via principled layering.
* Break any circular dependencies by converting hard dependencies to soft dependencies.
* Also accept that data from other components from another source, such as local files, which can then be manually populated at bootstrap time and then continuously updated once those other components are available.
* Break any circular dependencies by converting hard dependencies to soft
dependencies.
* Also accept that data from other components from another source, such as
local files, which can then be manually populated at bootstrap time and then
continuously updated once those other components are available.
* State should be rediscoverable and/or reconstructable.
* Make it easy to run temporary, bootstrap instances of all components in order to create the runtime state needed to run the components in the steady state; use a lock (master election for distributed components, file lock for local components like Kubelet) to coordinate handoff. We call this technique "pivoting".
* Have a solution to restart dead components. For distributed components, replication works well. For local components such as Kubelet, a process manager or even a simple shell loop works.
* Make it easy to run temporary, bootstrap instances of all components in
order to create the runtime state needed to run the components in the steady
state; use a lock (master election for distributed components, file lock for
local components like Kubelet) to coordinate handoff. We call this technique
"pivoting".
* Have a solution to restart dead components. For distributed components,
replication works well. For local components such as Kubelet, a process manager
or even a simple shell loop works.
## Availability
......
docs/design/scheduler_extender.md
View file @ 97f1f599
......@@ -34,11 +34,26 @@ Documentation for other releases can be found at
# Scheduler extender
There are three ways to add new scheduling rules (predicates and priority functions) to Kubernetes: (1) by adding these rules to the scheduler and recompiling (described here: https://github.com/kubernetes/kubernetes/blob/master/docs/devel/scheduler.md), (2) implementing your own scheduler process that runs instead of, or alongside of, the standard Kubernetes scheduler, (3) implementing a "scheduler extender" process that the standard Kubernetes scheduler calls out to as a final pass when making scheduling decisions.
This document describes the third approach. This approach is needed for use cases where scheduling decisions need to be made on resources not directly managed by the standard Kubernetes scheduler. The extender helps make scheduling decisions based on such resources. (Note that the three approaches are not mutually exclusive.)
When scheduling a pod, the extender allows an external process to filter and prioritize nodes. Two separate http/https calls are issued to the extender, one for "filter" and one for "prioritize" actions. To use the extender, you must create a scheduler policy configuration file. The configuration specifies how to reach the extender, whether to use http or https and the timeout.
There are three ways to add new scheduling rules (predicates and priority
functions) to Kubernetes: (1) by adding these rules to the scheduler and
recompiling (described here:
https://github.com/kubernetes/kubernetes/blob/master/docs/devel/scheduler.md),
(2) implementing your own scheduler process that runs instead of, or alongside
of, the standard Kubernetes scheduler, (3) implementing a "scheduler extender"
process that the standard Kubernetes scheduler calls out to as a final pass when
making scheduling decisions.
This document describes the third approach. This approach is needed for use
cases where scheduling decisions need to be made on resources not directly
managed by the standard Kubernetes scheduler. The extender helps make scheduling
decisions based on such resources. (Note that the three approaches are not
mutually exclusive.)
When scheduling a pod, the extender allows an external process to filter and
prioritize nodes. Two separate http/https calls are issued to the extender, one
for "filter" and one for "prioritize" actions. To use the extender, you must
create a scheduler policy configuration file. The configuration specifies how to
reach the extender, whether to use http or https and the timeout.
```go
// Holds the parameters used to communicate with the extender. If a verb is unspecified/empty,
......@@ -94,7 +109,10 @@ A sample scheduler policy file with extender configuration:
}
```
Arguments passed to the FilterVerb endpoint on the extender are the set of nodes filtered through the k8s predicates and the pod. Arguments passed to the PrioritizeVerb endpoint on the extender are the set of nodes filtered through the k8s predicates and extender predicates and the pod.
Arguments passed to the FilterVerb endpoint on the extender are the set of nodes
filtered through the k8s predicates and the pod. Arguments passed to the
PrioritizeVerb endpoint on the extender are the set of nodes filtered through
the k8s predicates and extender predicates and the pod.
```go
// ExtenderArgs represents the arguments needed by the extender to filter/prioritize
......@@ -107,9 +125,12 @@ type ExtenderArgs struct {
}
```
The "filter" call returns a list of nodes (api.NodeList). The "prioritize" call returns priorities for each node (schedulerapi.HostPriorityList).
The "filter" call returns a list of nodes (api.NodeList). The "prioritize" call
returns priorities for each node (schedulerapi.HostPriorityList).
The "filter" call may prune the set of nodes based on its predicates. Scores returned by the "prioritize" call are added to the k8s scores (computed through its priority functions) and used for final host selection.
The "filter" call may prune the set of nodes based on its predicates. Scores
returned by the "prioritize" call are added to the k8s scores (computed through
its priority functions) and used for final host selection.
Multiple extenders can be configured in the scheduler policy.
......
docs/design/simple-rolling-update.md
View file @ 97f1f599
......@@ -34,32 +34,47 @@ Documentation for other releases can be found at
## Simple rolling update
This is a lightweight design document for simple [rolling update](../user-guide/kubectl/kubectl_rolling-update.md) in `kubectl`.
This is a lightweight design document for simple
[rolling update](../user-guide/kubectl/kubectl_rolling-update.md) in `kubectl`.
Complete execution flow can be found [here](#execution-details). See the [example of rolling update](../user-guide/update-demo/) for more information.
Complete execution flow can be found [here](#execution-details). See the
[example of rolling update](../user-guide/update-demo/) for more information.
### Lightweight rollout
Assume that we have a current replication controller named `foo` and it is running image `image:v1`
Assume that we have a current replication controller named `foo` and it is
running image `image:v1`
`kubectl rolling-update foo [foo-v2] --image=myimage:v2`
If the user doesn't specify a name for the 'next' replication controller, then the 'next' replication controller is renamed to
If the user doesn't specify a name for the 'next' replication controller, then
the 'next' replication controller is renamed to
the name of the original replication controller.
Obviously there is a race here, where if you kill the client between delete foo, and creating the new version of 'foo' you might be surprised about what is there, but I think that's ok.
See [Recovery](#recovery) below
Obviously there is a race here, where if you kill the client between delete foo,
and creating the new version of 'foo' you might be surprised about what is
there, but I think that's ok. See [Recovery](#recovery) below
If the user does specify a name for the 'next' replication controller, then the 'next' replication controller is retained with its existing name,
and the old 'foo' replication controller is deleted. For the purposes of the rollout, we add a unique-ifying label `kubernetes.io/deployment` to both the `foo` and `foo-next` replication controllers.
The value of that label is the hash of the complete JSON representation of the`foo-next` or`foo` replication controller. The name of this label can be overridden by the user with the `--deployment-label-key` flag.
If the user does specify a name for the 'next' replication controller, then the
'next' replication controller is retained with its existing name, and the old
'foo' replication controller is deleted. For the purposes of the rollout, we add
a unique-ifying label `kubernetes.io/deployment` to both the `foo` and
`foo-next` replication controllers. The value of that label is the hash of the
complete JSON representation of the`foo-next` or`foo` replication controller.
The name of this label can be overridden by the user with the
`--deployment-label-key` flag.
#### Recovery
If a rollout fails or is terminated in the middle, it is important that the user be able to resume the roll out.
To facilitate recovery in the case of a crash of the updating process itself, we add the following annotations to each replication controller in the `kubernetes.io/` annotation namespace:
* `desired-replicas` The desired number of replicas for this replication controller (either N or zero)
* `update-partner` A pointer to the replication controller resource that is the other half of this update (syntax `<name>` the namespace is assumed to be identical to the namespace of this replication controller.)
If a rollout fails or is terminated in the middle, it is important that the user
be able to resume the roll out. To facilitate recovery in the case of a crash of
the updating process itself, we add the following annotations to each
replication controller in the `kubernetes.io/` annotation namespace:
* `desired-replicas` The desired number of replicas for this replication
controller (either N or zero)
* `update-partner` A pointer to the replication controller resource that is
the other half of this update (syntax `<name>` the namespace is assumed to be
identical to the namespace of this replication controller.)
Recovery is achieved by issuing the same command again:
......@@ -67,9 +82,12 @@ Recovery is achieved by issuing the same command again:
kubectl rolling-update foo [foo-v2] --image=myimage:v2
```
Whenever the rolling update command executes, the kubectl client looks for replication controllers called `foo` and `foo-next`, if they exist, an attempt is
made to roll `foo` to `foo-next`. If `foo-next` does not exist, then it is created, and the rollout is a new rollout. If `foo` doesn't exist, then
it is assumed that the rollout is nearly completed, and `foo-next` is renamed to `foo`. Details of the execution flow are given below.
Whenever the rolling update command executes, the kubectl client looks for
replication controllers called `foo` and `foo-next`, if they exist, an attempt
is made to roll `foo` to `foo-next`. If `foo-next` does not exist, then it is
created, and the rollout is a new rollout. If `foo` doesn't exist, then it is
assumed that the rollout is nearly completed, and `foo-next` is renamed to
`foo`. Details of the execution flow are given below.
### Aborting a rollout
......@@ -82,22 +100,28 @@ This is really just semantic sugar for:
`kubectl rolling-update foo-v2 foo`
With the added detail that it moves the `desired-replicas` annotation from `foo-v2` to `foo`
With the added detail that it moves the `desired-replicas` annotation from
`foo-v2` to `foo`
### Execution Details
For the purposes of this example, assume that we are rolling from `foo` to `foo-next` where the only change is an image update from `v1` to `v2`
For the purposes of this example, assume that we are rolling from `foo` to
`foo-next` where the only change is an image update from `v1` to `v2`
If the user doesn't specify a `foo-next` name, then it is either discovered from the `update-partner` annotation on `foo`. If that annotation doesn't exist,
then `foo-next` is synthesized using the pattern `<controller-name>-<hash-of-next-controller-JSON>`
If the user doesn't specify a `foo-next` name, then it is either discovered from
the `update-partner` annotation on `foo`. If that annotation doesn't exist,
then `foo-next` is synthesized using the pattern
`<controller-name>-<hash-of-next-controller-JSON>`
#### Initialization
* If `foo` and `foo-next` do not exist:
* Exit, and indicate an error to the user, that the specified controller doesn't exist.
* Exit, and indicate an error to the user, that the specified controller
doesn't exist.
* If `foo` exists, but `foo-next` does not:
* Create `foo-next` populate it with the `v2` image, set `desired-replicas` to `foo.Spec.Replicas`
* Create `foo-next` populate it with the `v2` image, set
`desired-replicas` to `foo.Spec.Replicas`
* Goto Rollout
* If `foo-next` exists, but `foo` does not:
* Assume that we are in the rename phase.
......@@ -105,7 +129,8 @@ then `foo-next` is synthesized using the pattern `<controller-name>-<hash-of-nex
* If both `foo` and `foo-next` exist:
* Assume that we are in a partial rollout
* If `foo-next` is missing the `desired-replicas` annotation
* Populate the `desired-replicas` annotation to `foo-next` using the current size of `foo`
* Populate the `desired-replicas` annotation to `foo-next` using the
current size of `foo`
* Goto Rollout
#### Rollout
......@@ -125,11 +150,13 @@ then `foo-next` is synthesized using the pattern `<controller-name>-<hash-of-nex
#### Abort
* If `foo-next` doesn't exist
* Exit and indicate to the user that they may want to simply do a new rollout with the old version
* Exit and indicate to the user that they may want to simply do a new
rollout with the old version
* If `foo` doesn't exist
* Exit and indicate not found to the user
* Otherwise, `foo-next` and `foo` both exist
* Set `desired-replicas` annotation on `foo` to match the annotation on `foo-next`
* Set `desired-replicas` annotation on `foo` to match the annotation on
`foo-next`
* Goto Rollout with `foo` and `foo-next` trading places.
......
docs/devel/adding-an-APIGroup.md
View file @ 97f1f599
......@@ -35,30 +35,55 @@ Documentation for other releases can be found at
Adding an API Group
===============
This document includes the steps to add an API group. You may also want to take a look at PR [#16621](https://github.com/kubernetes/kubernetes/pull/16621) and PR [#13146](https://github.com/kubernetes/kubernetes/pull/13146), which add API groups.
This document includes the steps to add an API group. You may also want to take
a look at PR [#16621](https://github.com/kubernetes/kubernetes/pull/16621) and
PR [#13146](https://github.com/kubernetes/kubernetes/pull/13146), which add API
groups.
Please also read about [API conventions](api-conventions.md) and [API changes](api_changes.md) before adding an API group.
Please also read about [API conventions](api-conventions.md) and
[API changes](api_changes.md) before adding an API group.
### Your core group package:
We plan on improving the way the types are factored in the future; see [#16062](https://github.com/kubernetes/kubernetes/pull/16062) for the directions in which this might evolve.
We plan on improving the way the types are factored in the future; see
[#16062](https://github.com/kubernetes/kubernetes/pull/16062) for the directions
in which this might evolve.
1. Create a folder in pkg/apis to hold you group. Create types.go in pkg/apis/`<group>`/ and pkg/apis/`<group>`/`<version>`/ to define API objects in your group;
1. Create a folder in pkg/apis to hold you group. Create types.go in
pkg/apis/`<group>`/ and pkg/apis/`<group>`/`<version>`/ to define API objects
in your group;
2. Create pkg/apis/`<group>`/{register.go, `<version>`/register.go} to register this group's API objects to the encoding/decoding scheme (e.g., [pkg/apis/extensions/register.go](../../pkg/apis/extensions/register.go) and [pkg/apis/extensions/v1beta1/register.go](../../pkg/apis/extensions/v1beta1/register.go);
2. Create pkg/apis/`<group>`/{register.go, `<version>`/register.go} to register
this group's API objects to the encoding/decoding scheme (e.g.,
[pkg/apis/extensions/register.go](../../pkg/apis/extensions/register.go) and
[pkg/apis/extensions/v1beta1/register.go](../../pkg/apis/extensions/v1beta1/register.go);
3. Add a pkg/apis/`<group>`/install/install.go, which is responsible for adding the group to the `latest` package, so that other packages can access the group's meta through `latest.Group`. You probably only need to change the name of group and version in the [example](../../pkg/apis/extensions/install/install.go)). You need to import this `install` package in {pkg/master, pkg/client/unversioned}/import_known_versions.go, if you want to make your group accessible to other packages in the kube-apiserver binary, binaries that uses the client package.
3. Add a pkg/apis/`<group>`/install/install.go, which is responsible for adding
the group to the `latest` package, so that other packages can access the group's
meta through `latest.Group`. You probably only need to change the name of group
and version in the [example](../../pkg/apis/extensions/install/install.go)). You
need to import this `install` package in {pkg/master,
pkg/client/unversioned}/import_known_versions.go, if you want to make your group
accessible to other packages in the kube-apiserver binary, binaries that uses
the client package.
Step 2 and 3 are mechanical, we plan on autogenerate these using the cmd/libs/go2idl/ tool.
Step 2 and 3 are mechanical, we plan on autogenerate these using the
cmd/libs/go2idl/ tool.
### Scripts changes and auto-generated code:
1. Generate conversions and deep-copies:
1. Add your "group/" or "group/version" into hack/after-build/{update-generated-conversions.sh, update-generated-deep-copies.sh, verify-generated-conversions.sh, verify-generated-deep-copies.sh};
2. Make sure your pkg/apis/`<group>`/`<version>` directory has a doc.go file with the comment `// +genconversion=true`, to catch the attention of our gen-conversion script.
1. Add your "group/" or "group/version" into
hack/after-build/{update-generated-conversions.sh,
update-generated-deep-copies.sh, verify-generated-conversions.sh,
verify-generated-deep-copies.sh};
2. Make sure your pkg/apis/`<group>`/`<version>` directory has a doc.go file
with the comment `// +genconversion=true`, to catch the attention of our
gen-conversion script.
3. Run hack/update-all.sh.
2. Generate files for Ugorji codec:
1. Touch types.generated.go in pkg/apis/`<group>`{/, `<version>`};
......@@ -71,19 +96,29 @@ Step 2 and 3 are mechanical, we plan on autogenerate these using the cmd/libs/go
### Client (optional):
We are overhauling pkg/client, so this section might be outdated; see [#15730](https://github.com/kubernetes/kubernetes/pull/15730) for how the client package might evolve. Currently, to add your group to the client package, you need to
We are overhauling pkg/client, so this section might be outdated; see
[#15730](https://github.com/kubernetes/kubernetes/pull/15730) for how the client
package might evolve. Currently, to add your group to the client package, you
need to:
1. Create pkg/client/unversioned/`<group>`.go, define a group client interface and implement the client. You can take pkg/client/unversioned/extensions.go as a reference.
1. Create pkg/client/unversioned/`<group>`.go, define a group client interface
and implement the client. You can take pkg/client/unversioned/extensions.go as a
reference.
2. Add the group client interface to the `Interface` in pkg/client/unversioned/client.go and add method to fetch the interface. Again, you can take how we add the Extensions group there as an example.
2. Add the group client interface to the `Interface` in
pkg/client/unversioned/client.go and add method to fetch the interface. Again,
you can take how we add the Extensions group there as an example.
3. If you need to support the group in kubectl, you'll also need to modify pkg/kubectl/cmd/util/factory.go.
3. If you need to support the group in kubectl, you'll also need to modify
pkg/kubectl/cmd/util/factory.go.
### Make the group/version selectable in unit tests (optional):
1. Add your group in pkg/api/testapi/testapi.go, then you can access the group in tests through testapi.`<group>`;
1. Add your group in pkg/api/testapi/testapi.go, then you can access the group
in tests through testapi.`<group>`;
2. Add your "group/version" to `KUBE_API_VERSIONS` and `KUBE_TEST_API_VERSIONS` in hack/test-go.sh.
2. Add your "group/version" to `KUBE_API_VERSIONS` and `KUBE_TEST_API_VERSIONS`
in hack/test-go.sh.
TODO: Add a troubleshooting section.
......
docs/devel/api-conventions.md
View file @ 97f1f599
This source diff could not be displayed because it is too large. You can view the blob instead.
docs/devel/automation.md
View file @ 97f1f599
......@@ -36,8 +36,9 @@ Documentation for other releases can be found at
## Overview
Kubernetes uses a variety of automated tools in an attempt to relieve developers of repetitive, low
brain power work. This document attempts to describe these processes.
Kubernetes uses a variety of automated tools in an attempt to relieve developers
of repetitive, low brain power work. This document attempts to describe these
processes.
## Submit Queue
......@@ -47,8 +48,11 @@ In an effort to
* maintain e2e stability
* load test githubs label feature
We have added an automated [submit-queue](https://github.com/kubernetes/contrib/blob/master/mungegithub/mungers/submit-queue.go) to the
[github "munger"](https://github.com/kubernetes/contrib/tree/master/mungegithub) for kubernetes.
We have added an automated [submit-queue]
(https://github.com/kubernetes/contrib/blob/master/mungegithub/mungers/submit-queue.go)
to the
[github "munger"](https://github.com/kubernetes/contrib/tree/master/mungegithub)
for kubernetes.
The submit-queue does the following:
......@@ -76,59 +80,76 @@ A PR is considered "ready for merging" if it matches the following:
* it has passed the Jenkins e2e test
* it has the `e2e-not-required` label
Note that the combined whitelist/committer list is available at [submit-queue.k8s.io](http://submit-queue.k8s.io)
Note that the combined whitelist/committer list is available at
[submit-queue.k8s.io](http://submit-queue.k8s.io)
### Merge process
Merges _only_ occur when the `critical builds` (Jenkins e2e for gce, gke, scalability, upgrade) are passing.
We're open to including more builds here, let us know...
Merges _only_ occur when the `critical builds` (Jenkins e2e for gce, gke,
scalability, upgrade) are passing. We're open to including more builds here, let
us know...
Merges are serialized, so only a single PR is merged at a time, to ensure against races.
Merges are serialized, so only a single PR is merged at a time, to ensure
against races.
If the PR has the `e2e-not-required` label, it is simply merged.
If the PR does not have this label, e2e tests are re-run, if these new tests pass, the PR is merged.
If the PR has the `e2e-not-required` label, it is simply merged. If the PR does
not have this label, e2e tests are re-run, if these new tests pass, the PR is
merged.
If e2e flakes or is currently buggy, the PR will not be merged, but it will be re-run on the following
pass.
If e2e flakes or is currently buggy, the PR will not be merged, but it will be
re-run on the following pass.
## Github Munger
We also run a [github "munger"](https://github.com/kubernetes/contrib/tree/master/mungegithub)
We also run a [github "munger."]
(https://github.com/kubernetes/contrib/tree/master/mungegithub)
This runs repeatedly over github pulls and issues and runs modular "mungers" similar to "mungedocs"
This runs repeatedly over github pulls and issues and runs modular "mungers"
similar to "mungedocs."
Currently this runs:
* blunderbuss - Tries to automatically find an owner for a PR without an owner, uses mapping file here:
* blunderbuss - Tries to automatically find an owner for a PR without an
owner, uses mapping file here:
https://github.com/kubernetes/contrib/blob/master/mungegithub/blunderbuss.yml
* needs-rebase - Adds `needs-rebase` to PRs that aren't currently mergeable, and removes it from those that are.
* needs-rebase - Adds `needs-rebase` to PRs that aren't currently mergeable,
and removes it from those that are.
* size - Adds `size/xs` - `size/xxl` labels to PRs
* ok-to-test - Adds the `ok-to-test` message to PRs that have an `lgtm` but the e2e-builder would otherwise not test due to whitelist
* ping-ci - Attempts to ping the ci systems (Travis) if they are missing from a PR.
* lgtm-after-commit - Removes the `lgtm` label from PRs where there are commits that are newer than the `lgtm` label
* ok-to-test - Adds the `ok-to-test` message to PRs that have an `lgtm` but
the e2e-builder would otherwise not test due to whitelist
* ping-ci - Attempts to ping the ci systems (Travis) if they are missing from
a PR.
* lgtm-after-commit - Removes the `lgtm` label from PRs where there are
commits that are newer than the `lgtm` label
In the works:
* issue-detector - machine learning for determining if an issue that has been filed is a `support` issue, `bug` or `feature`
* issue-detector - machine learning for determining if an issue that has been
filed is a `support` issue, `bug` or `feature`
Please feel free to unleash your creativity on this tool, send us new mungers that you think will help support the Kubernetes development process.
Please feel free to unleash your creativity on this tool, send us new mungers
that you think will help support the Kubernetes development process.
## PR builder
We also run a robotic PR builder that attempts to run e2e tests for each PR.
Before a PR from an unknown user is run, the PR builder bot (`k8s-bot`) asks to a message from a
contributor that a PR is "ok to test", the contributor replies with that message. Contributors can also
add users to the whitelist by replying with the message "add to whitelist" ("please" is optional, but
remember to treat your robots with kindness...)
Before a PR from an unknown user is run, the PR builder bot (`k8s-bot`) asks to
a message from a contributor that a PR is "ok to test", the contributor replies
with that message. Contributors can also add users to the whitelist by replying
with the message "add to whitelist" ("please" is optional, but remember to treat
your robots with kindness...)
If a PR is approved for testing, and tests either haven't run, or need to be re-run, you can ask the
PR builder to re-run the tests. To do this, reply to the PR with a message that begins with `@k8s-bot test this`, this should trigger a re-build/re-test.
If a PR is approved for testing, and tests either haven't run, or need to be
re-run, you can ask the PR builder to re-run the tests. To do this, reply to the
PR with a message that begins with `@k8s-bot test this`, this should trigger a
re-build/re-test.
## FAQ:
#### How can I ask my PR to be tested again for Jenkins failures?
Right now you have to ask a contributor (this may be you!) to re-run the test with "@k8s-bot test this"
Right now you have to ask a contributor (this may be you!) to re-run the test
with "@k8s-bot test this"
### How can I kick Travis to re-test on a failure?
......
docs/devel/cherry-picks.md
View file @ 97f1f599
......@@ -40,46 +40,54 @@ depending on the point in the release cycle.
## Propose a Cherry Pick
1. Cherrypicks are [managed with labels and milestones](pull-requests.md#release-notes)
1. All label/milestone accounting happens on PRs on master. There's nothing to do on PRs targeted to the release branches.
1. When you want a PR to be merged to the release branch, make the following label changes to the **master** branch PR:
1. Cherrypicks are [managed with labels and milestones]
(pull-requests.md#release-notes)
1. All label/milestone accounting happens on PRs on master. There's nothing to
do on PRs targeted to the release branches.
1. When you want a PR to be merged to the release branch, make the following
label changes to the **master** branch PR:
* Remove release-note-label-needed
* Add an appropriate release-note-(!label-needed) label
* Add an appropriate milestone
* Add the `cherrypick-candidate` label
* The PR title is the **release note** you want published at release time and
note that PR titles are mutable and should reflect a release note
friendly message for any `release-note-*` labeled PRs.
note that PR titles are mutable and should reflect a release note
friendly message for any `release-note-*` labeled PRs.
### How do cherrypick-candidates make it to the release branch?
1. **BATCHING:** After a branch is first created and before the X.Y.0 release
* Branch owners review the list of `cherrypick-candidate` labeled PRs.
* PRs batched up and merged to the release branch get a `cherrypick-approved` label and lose the `cherrypick-candidate` label.
* PRs that won't be merged to the release branch, lose the `cherrypick-candidate` label.
* PRs batched up and merged to the release branch get a `cherrypick-approved`
label and lose the `cherrypick-candidate` label.
* PRs that won't be merged to the release branch, lose the
`cherrypick-candidate` label.
1. **INDIVIDUAL CHERRYPICKS:** After the first X.Y.0 on a branch
* Run the cherry pick script. This example applies a master branch PR #98765 to the remote branch `upstream/release-3.14`:
`hack/cherry_pick_pull.sh upstream/release-3.14 98765`
* Run the cherry pick script. This example applies a master branch PR #98765
to the remote branch `upstream/release-3.14`:
`hack/cherry_pick_pull.sh upstream/release-3.14 98765`
* Your cherrypick PR (targeted to the branch) will immediately get the
`do-not-merge` label. The branch owner will triage PRs targeted to
the branch and label the ones to be merged by applying the `lgtm`
label.
`do-not-merge` label. The branch owner will triage PRs targeted to
the branch and label the ones to be merged by applying the `lgtm`
label.
There is an [issue](https://github.com/kubernetes/kubernetes/issues/23347) open tracking the tool to automate the batching procedure.
There is an [issue](https://github.com/kubernetes/kubernetes/issues/23347) open
tracking the tool to automate the batching procedure.
#### Cherrypicking a doc change
If you are cherrypicking a change which adds a doc, then you also need to run
`build/versionize-docs.sh` in the release branch to versionize that doc.
Ideally, just running `hack/cherry_pick_pull.sh` should be enough, but we are not there
yet: [#18861](https://github.com/kubernetes/kubernetes/issues/18861)
Ideally, just running `hack/cherry_pick_pull.sh` should be enough, but we are
not there yet: [#18861](https://github.com/kubernetes/kubernetes/issues/18861)
To cherrypick PR 123456 to release-3.14, run the following commands after running `hack/cherry_pick_pull.sh` and before merging the PR:
To cherrypick PR 123456 to release-3.14, run the following commands after
running `hack/cherry_pick_pull.sh` and before merging the PR:
```
$ git checkout -b automated-cherry-pick-of-#123456-upstream-release-3.14
origin/automated-cherry-pick-of-#123456-upstream-release-3.14
origin/automated-cherry-pick-of-#123456-upstream-release-3.14
$ ./build/versionize-docs.sh release-3.14
$ git commit -a -m "Running versionize docs"
$ git push origin automated-cherry-pick-of-#123456-upstream-release-3.14
......@@ -97,9 +105,9 @@ requested - this should not be the norm, but it may happen.
See the [cherrypick queue dashboard](http://cherrypick.k8s.io/#/queue) for
status of PRs labeled as `cherrypick-candidate`.
[Contributor License Agreements](http://releases.k8s.io/HEAD/CONTRIBUTING.md) is considered implicit
for all code within cherry-pick pull requests, ***unless there is a large
conflict***.
[Contributor License Agreements](http://releases.k8s.io/HEAD/CONTRIBUTING.md) is
considered implicit for all code within cherry-pick pull requests, ***unless
there is a large conflict***.
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
......
docs/devel/client-libraries.md
View file @ 97f1f599
......@@ -40,7 +40,8 @@ Documentation for other releases can be found at
### User Contributed
*Note: Libraries provided by outside parties are supported by their authors, not the core Kubernetes team*
*Note: Libraries provided by outside parties are supported by their authors, not
the core Kubernetes team*
* [Clojure](https://github.com/yanatan16/clj-kubernetes-api)
* [Java (OSGi)](https://bitbucket.org/amdatulabs/amdatu-kubernetes)
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment