Skip to content

K
k3s
Commit bc8f7e2c authored by Michelle Noorali's avatar Michelle Noorali
Browse files
Options

replace contents of docs/devel with stubs

parent 4495af38
Expand all Hide whitespace changes
Inline Side-by-side
Showing with 42 additions and 4790 deletions
docs/devel/README.md
View file @ bc8f7e2c
# Kubernetes Developer Guide This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/devel/README.md](https://github.com/kubernetes/community/blob/master/contributors/devel/README.md)
The developer guide is for anyone wanting to either write code which directly accesses the
Kubernetes API, or to contribute directly to the Kubernetes project.
It assumes some familiarity with concepts in the [User Guide](../user-guide/README.md) and the [Cluster Admin
Guide](../admin/README.md).
## The process of developing and contributing code to the Kubernetes project
* **On Collaborative Development** ([collab.md](collab.md)): Info on pull requests and code reviews.
* **GitHub Issues** ([issues.md](issues.md)): How incoming issues are reviewed and prioritized.
* **Pull Request Process** ([pull-requests.md](pull-requests.md)): When and why pull requests are closed.
* **Kubernetes On-Call Rotations** ([on-call-rotations.md](on-call-rotations.md)): Descriptions of on-call rotations for build and end-user support.
* **Faster PR reviews** ([faster_reviews.md](faster_reviews.md)): How to get faster PR reviews.
* **Getting Recent Builds** ([getting-builds.md](getting-builds.md)): How to get recent builds including the latest builds that pass CI.
* **Automated Tools** ([automation.md](automation.md)): Descriptions of the automation that is running on our github repository.
## Setting up your dev environment, coding, and debugging
* **Development Guide** ([development.md](development.md)): Setting up your development environment.
* **Hunting flaky tests** ([flaky-tests.md](flaky-tests.md)): We have a goal of 99.9% flake free tests.
Here's how to run your tests many times.
* **Logging Conventions** ([logging.md](logging.md)): Glog levels.
* **Profiling Kubernetes** ([profiling.md](profiling.md)): How to plug in go pprof profiler to Kubernetes.
* **Instrumenting Kubernetes with a new metric**
([instrumentation.md](instrumentation.md)): How to add a new metrics to the
Kubernetes code base.
* **Coding Conventions** ([coding-conventions.md](coding-conventions.md)):
Coding style advice for contributors.
* **Document Conventions** ([how-to-doc.md](how-to-doc.md))
Document style advice for contributors.
* **Running a cluster locally** ([running-locally.md](running-locally.md)):
A fast and lightweight local cluster deployment for development.
## Developing against the Kubernetes API
* The [REST API documentation](../api-reference/README.md) explains the REST
API exposed by apiserver.
* **Annotations** ([docs/user-guide/annotations.md](../user-guide/annotations.md)): are for attaching arbitrary non-identifying metadata to objects.
Programs that automate Kubernetes objects may use annotations to store small amounts of their state.
* **API Conventions** ([api-conventions.md](api-conventions.md)):
Defining the verbs and resources used in the Kubernetes API.
* **API Client Libraries** ([client-libraries.md](client-libraries.md)):
A list of existing client libraries, both supported and user-contributed.
## Writing plugins
* **Authentication Plugins** ([docs/admin/authentication.md](../admin/authentication.md)):
The current and planned states of authentication tokens.
* **Authorization Plugins** ([docs/admin/authorization.md](../admin/authorization.md)):
Authorization applies to all HTTP requests on the main apiserver port.
This doc explains the available authorization implementations.
* **Admission Control Plugins** ([admission_control](../design/admission_control.md))
## Building releases
See the [kubernetes/release](https://github.com/kubernetes/release) repository for details on creating releases and related tools and helper scripts.
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/devel/README.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
docs/devel/adding-an-APIGroup.md
View file @ bc8f7e2c
Adding an API Group This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/devel/adding-an-APIGroup.md](https://github.com/kubernetes/community/blob/master/contributors/devel/adding-an-APIGroup.md)
===============
This document includes the steps to add an API group. You may also want to take
a look at PR [#16621](https://github.com/kubernetes/kubernetes/pull/16621) and
PR [#13146](https://github.com/kubernetes/kubernetes/pull/13146), which add API
groups.
Please also read about [API conventions](api-conventions.md) and
[API changes](api_changes.md) before adding an API group.
### Your core group package:
We plan on improving the way the types are factored in the future; see
[#16062](https://github.com/kubernetes/kubernetes/pull/16062) for the directions
in which this might evolve.
1. Create a folder in pkg/apis to hold your group. Create types.go in
pkg/apis/`<group>`/ and pkg/apis/`<group>`/`<version>`/ to define API objects
in your group;
2. Create pkg/apis/`<group>`/{register.go, `<version>`/register.go} to register
this group's API objects to the encoding/decoding scheme (e.g.,
[pkg/apis/authentication/register.go](../../pkg/apis/authentication/register.go) and
[pkg/apis/authentication/v1beta1/register.go](../../pkg/apis/authentication/v1beta1/register.go);
3. Add a pkg/apis/`<group>`/install/install.go, which is responsible for adding
the group to the `latest` package, so that other packages can access the group's
meta through `latest.Group`. You probably only need to change the name of group
and version in the [example](../../pkg/apis/authentication/install/install.go)). You
need to import this `install` package in {pkg/master,
pkg/client/unversioned}/import_known_versions.go, if you want to make your group
accessible to other packages in the kube-apiserver binary, binaries that uses
the client package.
Step 2 and 3 are mechanical, we plan on autogenerate these using the
cmd/libs/go2idl/ tool.
### Scripts changes and auto-generated code:
1. Generate conversions and deep-copies:
1. Add your "group/" or "group/version" into
cmd/libs/go2idl/conversion-gen/main.go;
2. Make sure your pkg/apis/`<group>`/`<version>` directory has a doc.go file
with the comment `// +k8s:deepcopy-gen=package,register`, to catch the
attention of our generation tools.
3. Make sure your `pkg/apis/<group>/<version>` directory has a doc.go file
with the comment `// +k8s:conversion-gen=<internal-pkg>`, to catch the
attention of our generation tools. For most APIs the only target you
need is `k8s.io/kubernetes/pkg/apis/<group>` (your internal API).
3. Make sure your `pkg/apis/<group>` and `pkg/apis/<group>/<version>` directories
have a doc.go file with the comment `+groupName=<group>.k8s.io`, to correctly
generate the DNS-suffixed group name.
5. Run hack/update-all.sh.
2. Generate files for Ugorji codec:
1. Touch types.generated.go in pkg/apis/`<group>`{/, `<version>`};
2. Run hack/update-codecgen.sh.
3. Generate protobuf objects:
1. Add your group to `cmd/libs/go2idl/go-to-protobuf/protobuf/cmd.go` to
`New()` in the `Packages` field
2. Run hack/update-generated-protobuf.sh
### Client (optional):
We are overhauling pkg/client, so this section might be outdated; see
[#15730](https://github.com/kubernetes/kubernetes/pull/15730) for how the client
package might evolve. Currently, to add your group to the client package, you
need to:
1. Create pkg/client/unversioned/`<group>`.go, define a group client interface
and implement the client. You can take pkg/client/unversioned/extensions.go as a
reference.
2. Add the group client interface to the `Interface` in
pkg/client/unversioned/client.go and add method to fetch the interface. Again,
you can take how we add the Extensions group there as an example.
3. If you need to support the group in kubectl, you'll also need to modify
pkg/kubectl/cmd/util/factory.go.
### Make the group/version selectable in unit tests (optional):
1. Add your group in pkg/api/testapi/testapi.go, then you can access the group
in tests through testapi.`<group>`;
2. Add your "group/version" to `KUBE_TEST_API_VERSIONS` in
hack/make-rules/test.sh and hack/make-rules/test-integration.sh
TODO: Add a troubleshooting section.
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/devel/adding-an-APIGroup.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
docs/devel/automation.md
View file @ bc8f7e2c
# Kubernetes Development Automation This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/devel/automation.md](https://github.com/kubernetes/community/blob/master/contributors/devel/automation.md)
## Overview
Kubernetes uses a variety of automated tools in an attempt to relieve developers
of repetitive, low brain power work. This document attempts to describe these
processes.
## Submit Queue
In an effort to
* reduce load on core developers
* maintain e2e stability
* load test github's label feature
We have added an automated [submit-queue]
(https://github.com/kubernetes/contrib/blob/master/mungegithub/mungers/submit-queue.go)
to the
[github "munger"](https://github.com/kubernetes/contrib/tree/master/mungegithub)
for kubernetes.
The submit-queue does the following:
```go
for _, pr := range readyToMergePRs() {
if testsAreStable() {
if retestPR(pr) == success {
mergePR(pr)
}
}
}
```
The status of the submit-queue is [online.](http://submit-queue.k8s.io/)
### Ready to merge status
The submit-queue lists what it believes are required on the [merge requirements tab](http://submit-queue.k8s.io/#/info) of the info page. That may be more up to date.
A PR is considered "ready for merging" if it matches the following:
* The PR must have the label "cla: yes" or "cla: human-approved"
* The PR must be mergeable. aka cannot need a rebase
* All of the following github statuses must be green
* Jenkins GCE Node e2e
* Jenkins GCE e2e
* Jenkins unit/integration
* The PR cannot have any prohibited future milestones (such as a v1.5 milestone during v1.4 code freeze)
* The PR must have the "lgtm" label. The "lgtm" label is automatically applied
following a review comment consisting of only "LGTM" (case-insensitive)
* The PR must not have been updated since the "lgtm" label was applied
* The PR must not have the "do-not-merge" label
### Merge process
Merges _only_ occur when the [critical builds](http://submit-queue.k8s.io/#/e2e)
are passing. We're open to including more builds here, let us know...
Merges are serialized, so only a single PR is merged at a time, to ensure
against races.
If the PR has the `retest-not-required` label, it is simply merged. If the PR does
not have this label the e2e, unit/integration, and node tests are re-run. If these
tests pass a second time, the PR will be merged as long as the `critical builds` are
green when this PR finishes retesting.
## Github Munger
We run [github "mungers"](https://github.com/kubernetes/contrib/tree/master/mungegithub).
This runs repeatedly over github pulls and issues and runs modular "mungers"
similar to "mungedocs." The mungers include the 'submit-queue' referenced above along
with numerous other functions. See the README in the link above.
Please feel free to unleash your creativity on this tool, send us new mungers
that you think will help support the Kubernetes development process.
### Closing stale pull-requests
Github Munger will close pull-requests that don't have human activity in the
last 90 days. It will warn about this process 60 days before closing the
pull-request, and warn again 30 days later. One way to prevent this from
happening is to add the "keep-open" label on the pull-request.
Feel free to re-open and maybe add the "keep-open" label if this happens to a
valid pull-request. It may also be a good opportunity to get more attention by
verifying that it is properly assigned and/or mention people that might be
interested. Commenting on the pull-request will also keep it open for another 90
days.
## PR builder
We also run a robotic PR builder that attempts to run tests for each PR.
Before a PR from an unknown user is run, the PR builder bot (`k8s-bot`) asks to
a message from a contributor that a PR is "ok to test", the contributor replies
with that message. ("please" is optional, but remember to treat your robots with
kindness...)
## FAQ:
#### How can I ask my PR to be tested again for Jenkins failures?
PRs should only need to be manually re-tested if you believe there was a flake
during the original test. All flakes should be filed as an
[issue](https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Akind%2Fflake).
Once you find or file a flake a contributer (this may be you!) should request
a retest with "@k8s-bot test this issue: #NNNNN", where NNNNN is replaced with
the issue number you found or filed.
Any pushes of new code to the PR will automatically trigger a new test. No human
interraction is required.
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/devel/automation.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
docs/devel/bazel.md
View file @ bc8f7e2c
# Build with Bazel This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/devel/bazel.md](https://github.com/kubernetes/community/blob/master/contributors/devel/bazel.md)
Building with bazel is currently experimental. Automanaged BUILD rules have the
tag "automanaged" and are maintained by
[gazel](https://github.com/mikedanese/gazel). Instructions for installing bazel
can be found [here](https://www.bazel.io/versions/master/docs/install.html).
To build docker images for the components, run:
```
$ bazel build //build-tools/...
```
To run many of the unit tests, run:
```
$ bazel test //cmd/... //build-tools/... //pkg/... //federation/... //plugin/...
```
To update automanaged build files, run:
```
$ ./hack/update-bazel.sh
```
**NOTES**: `update-bazel.sh` only works if check out directory of Kubernetes is "$GOPATH/src/k8s.io/kubernetes".
To update a single build file, run:
```
$ # get gazel
$ go get -u github.com/mikedanese/gazel
$ # .e.g. ./pkg/kubectl/BUILD
$ gazel -root="${YOUR_KUBE_ROOT_PATH}" ./pkg/kubectl
```
Updating BUILD file for a package will be required when:
* Files are added to or removed from a package
* Import dependencies change for a package
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/devel/bazel.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
docs/devel/cherry-picks.md
View file @ bc8f7e2c
# Overview This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md](https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md)
This document explains cherry picks are managed on release branches within the
Kubernetes projects. Patches are either applied in batches or individually
depending on the point in the release cycle.
## Propose a Cherry Pick
1. Cherrypicks are [managed with labels and milestones]
(pull-requests.md#release-notes)
1. To get a PR merged to the release branch, first ensure the following labels
are on the original **master** branch PR:
* An appropriate milestone (e.g. v1.3)
* The `cherrypick-candidate` label
1. If `release-note-none` is set on the master PR, the cherrypick PR will need
to set the same label to confirm that no release note is needed.
1. `release-note` labeled PRs generate a release note using the PR title by
default OR the release-note block in the PR template if filled in.
* See the [PR template](../../.github/PULL_REQUEST_TEMPLATE.md) for more
details.
* PR titles and body comments are mutable and can be modified at any time
prior to the release to reflect a release note friendly message.
### How do cherrypick-candidates make it to the release branch?
1. **BATCHING:** After a branch is first created and before the X.Y.0 release
* Branch owners review the list of `cherrypick-candidate` labeled PRs.
* PRs batched up and merged to the release branch get a `cherrypick-approved`
label and lose the `cherrypick-candidate` label.
* PRs that won't be merged to the release branch, lose the
`cherrypick-candidate` label.
1. **INDIVIDUAL CHERRYPICKS:** After the first X.Y.0 on a branch
* Run the cherry pick script. This example applies a master branch PR #98765
to the remote branch `upstream/release-3.14`:
`hack/cherry_pick_pull.sh upstream/release-3.14 98765`
* Your cherrypick PR (targeted to the branch) will immediately get the
`do-not-merge` label. The branch owner will triage PRs targeted to
the branch and label the ones to be merged by applying the `lgtm`
label.
There is an [issue](https://github.com/kubernetes/kubernetes/issues/23347) open
tracking the tool to automate the batching procedure.
## Cherry Pick Review
Cherry pick pull requests are reviewed differently than normal pull requests. In
particular, they may be self-merged by the release branch owner without fanfare,
in the case the release branch owner knows the cherry pick was already
requested - this should not be the norm, but it may happen.
## Searching for Cherry Picks
See the [cherrypick queue dashboard](http://cherrypick.k8s.io/#/queue) for
status of PRs labeled as `cherrypick-candidate`.
[Contributor License Agreements](http://releases.k8s.io/HEAD/CONTRIBUTING.md) is
considered implicit for all code within cherry-pick pull requests, ***unless
there is a large conflict***.
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/devel/cherry-picks.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
docs/devel/cli-roadmap.md
View file @ bc8f7e2c
# Kubernetes CLI/Configuration Roadmap This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/devel/cli-roadmap.md](https://github.com/kubernetes/community/blob/master/contributors/devel/cli-roadmap.md)
See github issues with the following labels:
* [area/app-config-deployment](https://github.com/kubernetes/kubernetes/labels/area/app-config-deployment)
* [component/kubectl](https://github.com/kubernetes/kubernetes/labels/component/kubectl)
* [component/clientlib](https://github.com/kubernetes/kubernetes/labels/component/clientlib)
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/devel/cli-roadmap.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
docs/devel/client-libraries.md
View file @ bc8f7e2c
## Kubernetes API client libraries This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/devel/client-libraries.md](https://github.com/kubernetes/community/blob/master/contributors/devel/client-libraries.md)
### Supported
* [Go](https://github.com/kubernetes/client-go)
### User Contributed
*Note: Libraries provided by outside parties are supported by their authors, not
the core Kubernetes team*
* [Clojure](https://github.com/yanatan16/clj-kubernetes-api)
* [Java (OSGi)](https://bitbucket.org/amdatulabs/amdatu-kubernetes)
* [Java (Fabric8, OSGi)](https://github.com/fabric8io/kubernetes-client)
* [Node.js](https://github.com/tenxcloud/node-kubernetes-client)
* [Node.js](https://github.com/godaddy/kubernetes-client)
* [Perl](https://metacpan.org/pod/Net::Kubernetes)
* [PHP](https://github.com/devstub/kubernetes-api-php-client)
* [PHP](https://github.com/maclof/kubernetes-client)
* [Python](https://github.com/eldarion-gondor/pykube)
* [Ruby](https://github.com/Ch00k/kuber)
* [Ruby](https://github.com/abonas/kubeclient)
* [Scala](https://github.com/doriordan/skuber)
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/devel/client-libraries.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
docs/devel/coding-conventions.md
View file @ bc8f7e2c
# Coding Conventions This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/devel/coding-conventions.md](https://github.com/kubernetes/community/blob/master/contributors/devel/coding-conventions.md)
Updated: 5/3/2016
**Table of Contents**
<!-- BEGIN MUNGE: GENERATED_TOC -->
- [Coding Conventions](#coding-conventions)
- [Code conventions](#code-conventions)
- [Testing conventions](#testing-conventions)
- [Directory and file conventions](#directory-and-file-conventions)
- [Coding advice](#coding-advice)
<!-- END MUNGE: GENERATED_TOC -->
## Code conventions
- Bash
- https://google.github.io/styleguide/shell.xml
- Ensure that build, release, test, and cluster-management scripts run on
OS X
- Go
- Ensure your code passes the [presubmit checks](development.md#hooks)
- [Go Code Review
Comments](https://github.com/golang/go/wiki/CodeReviewComments)
- [Effective Go](https://golang.org/doc/effective_go.html)
- Comment your code.
- [Go's commenting
conventions](http://blog.golang.org/godoc-documenting-go-code)
- If reviewers ask questions about why the code is the way it is, that's a
sign that comments might be helpful.
- Command-line flags should use dashes, not underscores
- Naming
- Please consider package name when selecting an interface name, and avoid
redundancy.
- e.g.: `storage.Interface` is better than `storage.StorageInterface`.
- Do not use uppercase characters, underscores, or dashes in package
names.
- Please consider parent directory name when choosing a package name.
- so pkg/controllers/autoscaler/foo.go should say `package autoscaler`
not `package autoscalercontroller`.
- Unless there's a good reason, the `package foo` line should match
the name of the directory in which the .go file exists.
- Importers can use a different name if they need to disambiguate.
- Locks should be called `lock` and should never be embedded (always `lock
sync.Mutex`). When multiple locks are present, give each lock a distinct name
following Go conventions - `stateLock`, `mapLock` etc.
- [API changes](api_changes.md)
- [API conventions](api-conventions.md)
- [Kubectl conventions](kubectl-conventions.md)
- [Logging conventions](logging.md)
## Testing conventions
- All new packages and most new significant functionality must come with unit
tests
- Table-driven tests are preferred for testing multiple scenarios/inputs; for
example, see [TestNamespaceAuthorization](../../test/integration/auth/auth_test.go)
- Significant features should come with integration (test/integration) and/or
[end-to-end (test/e2e) tests](e2e-tests.md)
- Including new kubectl commands and major features of existing commands
- Unit tests must pass on OS X and Windows platforms - if you use Linux
specific features, your test case must either be skipped on windows or compiled
out (skipped is better when running Linux specific commands, compiled out is
required when your code does not compile on Windows).
- Avoid relying on Docker hub (e.g. pull from Docker hub). Use gcr.io instead.
- Avoid waiting for a short amount of time (or without waiting) and expect an
asynchronous thing to happen (e.g. wait for 1 seconds and expect a Pod to be
running). Wait and retry instead.
- See the [testing guide](testing.md) for additional testing advice.
## Directory and file conventions
- Avoid package sprawl. Find an appropriate subdirectory for new packages.
(See [#4851](http://issues.k8s.io/4851) for discussion.)
- Libraries with no more appropriate home belong in new package
subdirectories of pkg/util
- Avoid general utility packages. Packages called "util" are suspect. Instead,
derive a name that describes your desired function. For example, the utility
functions dealing with waiting for operations are in the "wait" package and
include functionality like Poll. So the full name is wait.Poll
- All filenames should be lowercase
- Go source files and directories use underscores, not dashes
- Package directories should generally avoid using separators as much as
possible (when packages are multiple words, they usually should be in nested
subdirectories).
- Document directories and filenames should use dashes rather than underscores
- Contrived examples that illustrate system features belong in
/docs/user-guide or /docs/admin, depending on whether it is a feature primarily
intended for users that deploy applications or cluster administrators,
respectively. Actual application examples belong in /examples.
- Examples should also illustrate [best practices for configuration and
using the system](../user-guide/config-best-practices.md)
- Third-party code
- Go code for normal third-party dependencies is managed using
[Godeps](https://github.com/tools/godep)
- Other third-party code belongs in `/third_party`
- forked third party Go code goes in `/third_party/forked`
- forked _golang stdlib_ code goes in `/third_party/golang`
- Third-party code must include licenses
- This includes modified third-party code and excerpts, as well
## Coding advice
- Go
- [Go landmines](https://gist.github.com/lavalamp/4bd23295a9f32706a48f)
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/devel/coding-conventions.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
docs/devel/collab.md
View file @ bc8f7e2c
# On Collaborative Development This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/devel/collab.md](https://github.com/kubernetes/community/blob/master/contributors/devel/collab.md)
Kubernetes is open source, but many of the people working on it do so as their
day job. In order to avoid forcing people to be "at work" effectively 24/7, we
want to establish some semi-formal protocols around development. Hopefully these
rules make things go more smoothly. If you find that this is not the case,
please complain loudly.
## Patches welcome
First and foremost: as a potential contributor, your changes and ideas are
welcome at any hour of the day or night, weekdays, weekends, and holidays.
Please do not ever hesitate to ask a question or send a PR.
## Code reviews
All changes must be code reviewed. For non-maintainers this is obvious, since
you can't commit anyway. But even for maintainers, we want all changes to get at
least one review, preferably (for non-trivial changes obligatorily) from someone
who knows the areas the change touches. For non-trivial changes we may want two
reviewers. The primary reviewer will make this decision and nominate a second
reviewer, if needed. Except for trivial changes, PRs should not be committed
until relevant parties (e.g. owners of the subsystem affected by the PR) have
had a reasonable chance to look at PR in their local business hours.
Most PRs will find reviewers organically. If a maintainer intends to be the
primary reviewer of a PR they should set themselves as the assignee on GitHub
and say so in a reply to the PR. Only the primary reviewer of a change should
actually do the merge, except in rare cases (e.g. they are unavailable in a
reasonable timeframe).
If a PR has gone 2 work days without an owner emerging, please poke the PR
thread and ask for a reviewer to be assigned.
Except for rare cases, such as trivial changes (e.g. typos, comments) or
emergencies (e.g. broken builds), maintainers should not merge their own
changes.
Expect reviewers to request that you avoid [common go style
mistakes](https://github.com/golang/go/wiki/CodeReviewComments) in your PRs.
## Assigned reviews
Maintainers can assign reviews to other maintainers, when appropriate. The
assignee becomes the shepherd for that PR and is responsible for merging the PR
once they are satisfied with it or else closing it. The assignee might request
reviews from non-maintainers.
## Merge hours
Maintainers will do merges of appropriately reviewed-and-approved changes during
their local "business hours" (typically 7:00 am Monday to 5:00 pm (17:00h)
Friday). PRs that arrive over the weekend or on holidays will only be merged if
there is a very good reason for it and if the code review requirements have been
met. Concretely this means that nobody should merge changes immediately before
going to bed for the night.
There may be discussion an even approvals granted outside of the above hours,
but merges will generally be deferred.
If a PR is considered complex or controversial, the merge of that PR should be
delayed to give all interested parties in all timezones the opportunity to
provide feedback. Concretely, this means that such PRs should be held for 24
hours before merging. Of course "complex" and "controversial" are left to the
judgment of the people involved, but we trust that part of being a committer is
the judgment required to evaluate such things honestly, and not be motivated by
your desire (or your cube-mate's desire) to get their code merged. Also see
"Holds" below, any reviewer can issue a "hold" to indicate that the PR is in
fact complicated or complex and deserves further review.
PRs that are incorrectly judged to be merge-able, may be reverted and subject to
re-review, if subsequent reviewers believe that they in fact are controversial
or complex.
## Holds
Any maintainer or core contributor who wants to review a PR but does not have
time immediately may put a hold on a PR simply by saying so on the PR discussion
and offering an ETA measured in single-digit days at most. Any PR that has a
hold shall not be merged until the person who requested the hold acks the
review, withdraws their hold, or is overruled by a preponderance of maintainers.
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/devel/collab.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
docs/devel/community-expectations.md
View file @ bc8f7e2c
## Community Expectations This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/devel/community-expectations.md](https://github.com/kubernetes/community/blob/master/contributors/devel/community-expectations.md)
Kubernetes is a community project. Consequently, it is wholly dependent on
its community to provide a productive, friendly and collaborative environment.
The first and foremost goal of the Kubernetes community to develop orchestration
technology that radically simplifies the process of creating reliable
distributed systems. However a second, equally important goal is the creation
of a community that fosters easy, agile development of such orchestration
systems.
We therefore describe the expectations for
members of the Kubernetes community. This document is intended to be a living one
that evolves as the community evolves via the same PR and code review process
that shapes the rest of the project. It currently covers the expectations
of conduct that govern all members of the community as well as the expectations
around code review that govern all active contributors to Kubernetes.
### Code of Conduct
The most important expectation of the Kubernetes community is that all members
abide by the Kubernetes [community code of conduct](../../code-of-conduct.md).
Only by respecting each other can we develop a productive, collaborative
community.
### Code review
As a community we believe in the [value of code review for all contributions](collab.md).
Code review increases both the quality and readability of our codebase, which
in turn produces high quality software.
However, the code review process can also introduce latency for contributors
and additional work for reviewers that can frustrate both parties.
Consequently, as a community we expect that all active participants in the
community will also be active reviewers.
We ask that active contributors to the project participate in the code review process
in areas where that contributor has expertise. Active
contributors are considered to be anyone who meets any of the following criteria:
* Sent more than two pull requests (PRs) in the previous one month, or more
than 20 PRs in the previous year.
* Filed more than three issues in the previous month, or more than 30 issues in
the previous 12 months.
* Commented on more than pull requests in the previous month, or
more than 50 pull requests in the previous 12 months.
* Marked any PR as LGTM in the previous month.
* Have *collaborator* permissions in the Kubernetes github project.
In addition to these community expectations, any community member who wants to
be an active reviewer can also add their name to an *active reviewer* file
(location tbd) which will make them an active reviewer for as long as they
are included in the file.
#### Expectations of reviewers: Review comments
Because reviewers are often the first points of contact between new members of
the community and can significantly impact the first impression of the
Kubernetes community, reviewers are especially important in shaping the
Kubernetes community. Reviewers are highly encouraged to review the
[code of conduct](../../code-of-conduct.md) and are strongly encouraged to go above
and beyond the code of conduct to promote a collaborative, respectful
Kubernetes community.
#### Expectations of reviewers: Review latency
Reviewers are expected to respond in a timely fashion to PRs that are assigned
to them. Reviewers are expected to respond to an *active* PRs with reasonable
latency, and if reviewers fail to respond, those PRs may be assigned to other
reviewers.
*Active* PRs are considered those which have a proper CLA (`cla:yes`) label
and do not need rebase to be merged. PRs that do not have a proper CLA, or
require a rebase are not considered active PRs.
## Thanks
Many thanks in advance to everyone who contributes their time and effort to
making Kubernetes both a successful system as well as a successful community.
The strength of our software shines in the strengths of each individual
community member. Thanks!
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/devel/community-expectations.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
docs/devel/container-runtime-interface.md
View file @ bc8f7e2c
# CRI: the Container Runtime Interface This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/devel/container-runtime-interface.md](https://github.com/kubernetes/community/blob/master/contributors/devel/container-runtime-interface.md)
## What is CRI?
CRI (_Container Runtime Interface_) consists of a
[protobuf API](../../pkg/kubelet/api/v1alpha1/runtime/api.proto),
specifications/requirements (to-be-added),
and [libraries] (https://github.com/kubernetes/kubernetes/tree/master/pkg/kubelet/server/streaming)
for container runtimes to integrate with kubelet on a node. CRI is currently in Alpha.
In the future, we plan to add more developer tools such as the CRI validation
tests.
## Why develop CRI?
Prior to the existence of CRI, container runtimes (e.g., `docker`, `rkt`) were
integrated with kubelet through implementing an internal, high-level interface
in kubelet. The entrance barrier for runtimes was high because the integration
required understanding the internals of kubelet and contributing to the main
Kubernetes repository. More importantly, this would not scale because every new
addition incurs a significant maintenance overhead in the main kubernetes
repository.
Kubernetes aims to be extensible. CRI is one small, yet important step to enable
pluggable container runtimes and build a healthier ecosystem.
## How to use CRI?
1. Start the image and runtime services on your node. You can have a single
service acting as both image and runtime services.
2. Set the kubelet flags
- Pass the unix socket(s) to which your services listen to kubelet:
`--container-runtime-endpoint` and `--image-service-endpoint`.
- Enable CRI in kubelet by`--experimental-cri=true`.
- Use the "remote" runtime by `--container-runtime=remote`.
Please see the [Status Update](#status-update) section for known issues for
each release.
Note that CRI is still in its early stages. We are actively incorporating
feedback from early developers to improve the API. Developers should expect
occasional API breaking changes.
## Does Kubelet use CRI today?
No, but we are working on it.
The first step is to switch kubelet to integrate with Docker via CRI by
default. The current [Docker CRI implementation](https://github.com/kubernetes/kubernetes/blob/release-1.5/pkg/kubelet/dockershim)
already passes most end-to-end tests, and has mandatory PR builders to prevent
regressions. While we are expanding the test coverage gradually, it is
difficult to test on all combinations of OS distributions, platforms, and
plugins. There are also many experimental or even undocumented features relied
upon by some users. We would like to **encourage the community to help test
this Docker-CRI integration and report bugs and/or missing features** to
smooth the transition in the near future. Please file a Github issue and
include @kubernetes/sig-node for any CRI problem.
### How to test the new Docker CRI integration?
Start kubelet with the following flags:
- Use the Docker container runtime by `--container-runtime=docker`(the default).
- Enable CRI in kubelet by`--experimental-cri=true`.
Please also see the [known issues](#docker-cri-1.5-known-issues) before trying
out.
## Design docs and proposals
We plan to add CRI specifications/requirements in the near future. For now,
these proposals and design docs are the best sources to understand CRI
besides discussions on Github issues.
- [Original proposal](https://github.com/kubernetes/kubernetes/blob/release-1.5/docs/proposals/container-runtime-interface-v1.md)
- [Exec/attach/port-forward streaming requests](https://docs.google.com/document/d/1OE_QoInPlVCK9rMAx9aybRmgFiVjHpJCHI9LrfdNM_s/edit?usp=sharing)
- [Container stdout/stderr logs](https://github.com/kubernetes/kubernetes/blob/release-1.5/docs/proposals/kubelet-cri-logging.md)
- Networking: The CRI runtime handles network plugins and the
setup/teardown of the pod sandbox.
## Work-In-Progress CRI runtimes
- [cri-o](https://github.com/kubernetes-incubator/cri-o)
- [rktlet](https://github.com/kubernetes-incubator/rktlet)
- [frakti](https://github.com/kubernetes/frakti)
## [Status update](#status-update)
### Kubernetes v1.5 release (CRI v1alpha1)
- [v1alpha1 version](https://github.com/kubernetes/kubernetes/blob/release-1.5/pkg/kubelet/api/v1alpha1/runtime/api.proto) of CRI is released.
#### [CRI known issues](#cri-1.5-known-issues):
- [#27097](https://github.com/kubernetes/kubernetes/issues/27097): Container
metrics are not yet defined in CRI.
- [#36401](https://github.com/kubernetes/kubernetes/issues/36401): The new
container log path/format is not yet supported by the logging pipeline
(e.g., fluentd, GCL).
- CRI may not be compatible with other experimental features (e.g., Seccomp).
- Streaming server needs to be hardened.
- [#36666](https://github.com/kubernetes/kubernetes/issues/36666):
Authentication.
- [#36187](https://github.com/kubernetes/kubernetes/issues/36187): Avoid
including user data in the redirect URL.
#### [Docker CRI integration known issues](#docker-cri-1.5-known-issues)
- Docker compatibility: Support only Docker v1.11 and v1.12.
- Network:
- [#35457](https://github.com/kubernetes/kubernetes/issues/35457): Does
not support host ports.
- [#37315](https://github.com/kubernetes/kubernetes/issues/37315): Does
not support bandwidth shaping.
- Exec/attach/port-forward (streaming requests):
- [#35747](https://github.com/kubernetes/kubernetes/issues/35747): Does
not support `nsenter` as the exec handler (`--exec-handler=nsenter`).
- Also see (#cri-1.5-known-issues) for limitations on CRI streaming.
## Contacts
- Email: sig-node (kubernetes-sig-node@googlegroups.com)
- Slack: https://kubernetes.slack.com/messages/sig-node
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/devel/container-runtime-interface.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
docs/devel/controllers.md
View file @ bc8f7e2c
# Writing Controllers This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/devel/controllers.md](https://github.com/kubernetes/community/blob/master/contributors/devel/controllers.md)
A Kubernetes controller is an active reconciliation process. That is, it watches some object for the world's desired
state, and it watches the world's actual state, too. Then, it sends instructions to try and make the world's current
state be more like the desired state.
The simplest implementation of this is a loop:
```go
for {
desired := getDesiredState()
current := getCurrentState()
makeChanges(desired, current)
}
```
Watches, etc, are all merely optimizations of this logic.
## Guidelines
When you’re writing controllers, there are few guidelines that will help make sure you get the results and performance
you’re looking for.
1. Operate on one item at a time. If you use a `workqueue.Interface`, you’ll be able to queue changes for a
particular resource and later pop them in multiple “worker” gofuncs with a guarantee that no two gofuncs will
work on the same item at the same time.
Many controllers must trigger off multiple resources (I need to "check X if Y changes"), but nearly all controllers
can collapse those into a queue of “check this X” based on relationships. For instance, a ReplicaSetController needs
to react to a pod being deleted, but it does that by finding the related ReplicaSets and queuing those.
1. Random ordering between resources. When controllers queue off multiple types of resources, there is no guarantee
of ordering amongst those resources.
Distinct watches are updated independently. Even with an objective ordering of “created resourceA/X” and “created
resourceB/Y”, your controller could observe “created resourceB/Y” and “created resourceA/X”.
1. Level driven, not edge driven. Just like having a shell script that isn’t running all the time, your controller
may be off for an indeterminate amount of time before running again.
If an API object appears with a marker value of `true`, you can’t count on having seen it turn from `false` to `true`,
only that you now observe it being `true`. Even an API watch suffers from this problem, so be sure that you’re not
counting on seeing a change unless your controller is also marking the information it last made the decision on in
the object's status.
1. Use `SharedInformers`. `SharedInformers` provide hooks to receive notifications of adds, updates, and deletes for
a particular resource. They also provide convenience functions for accessing shared caches and determining when a
cache is primed.
Use the factory methods down in https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/framework/informers/factory.go
to ensure that you are sharing the same instance of the cache as everyone else.
This saves us connections against the API server, duplicate serialization costs server-side, duplicate deserialization
costs controller-side, and duplicate caching costs controller-side.
You may see other mechanisms like reflectors and deltafifos driving controllers. Those were older mechanisms that we
later used to build the `SharedInformers`. You should avoid using them in new controllers
1. Never mutate original objects! Caches are shared across controllers, this means that if you mutate your "copy"
(actually a reference or shallow copy) of an object, you’ll mess up other controllers (not just your own).
The most common point of failure is making a shallow copy, then mutating a map, like `Annotations`. Use
`api.Scheme.Copy` to make a deep copy.
1. Wait for your secondary caches. Many controllers have primary and secondary resources. Primary resources are the
resources that you’ll be updating `Status` for. Secondary resources are resources that you’ll be managing
(creating/deleting) or using for lookups.
Use the `framework.WaitForCacheSync` function to wait for your secondary caches before starting your primary sync
functions. This will make sure that things like a Pod count for a ReplicaSet isn’t working off of known out of date
information that results in thrashing.
1. There are other actors in the system. Just because you haven't changed an object doesn't mean that somebody else
hasn't.
Don't forget that the current state may change at any moment--it's not sufficient to just watch the desired state.
If you use the absence of objects in the desired state to indicate that things in the current state should be deleted,
make sure you don't have a bug in your observation code (e.g., act before your cache has filled).
1. Percolate errors to the top level for consistent re-queuing. We have a `workqueue.RateLimitingInterface` to allow
simple requeuing with reasonable backoffs.
Your main controller func should return an error when requeuing is necessary. When it isn’t, it should use
`utilruntime.HandleError` and return nil instead. This makes it very easy for reviewers to inspect error handling
cases and to be confident that your controller doesn’t accidentally lose things it should retry for.
1. Watches and Informers will “sync”. Periodically, they will deliver every matching object in the cluster to your
`Update` method. This is good for cases where you may need to take additional action on the object, but sometimes you
know there won’t be more work to do.
In cases where you are *certain* that you don't need to requeue items when there are no new changes, you can compare the
resource version of the old and new objects. If they are the same, you skip requeuing the work. Be careful when you
do this. If you ever skip requeuing your item on failures, you could fail, not requeue, and then never retry that
item again.
## Rough Structure
Overall, your controller should look something like this:
```go
type Controller struct{
// podLister is secondary cache of pods which is used for object lookups
podLister cache.StoreToPodLister
// queue is where incoming work is placed to de-dup and to allow "easy" rate limited requeues on errors
queue workqueue.RateLimitingInterface
}
func (c *Controller) Run(threadiness int, stopCh chan struct{}){
// don't let panics crash the process
defer utilruntime.HandleCrash()
// make sure the work queue is shutdown which will trigger workers to end
defer dsc.queue.ShutDown()
glog.Infof("Starting <NAME> controller")
// wait for your secondary caches to fill before starting your work
if !framework.WaitForCacheSync(stopCh, c.podStoreSynced) {
return
}
// start up your worker threads based on threadiness. Some controllers have multiple kinds of workers
for i := 0; i < threadiness; i++ {
// runWorker will loop until "something bad" happens. The .Until will then rekick the worker
// after one second
go wait.Until(c.runWorker, time.Second, stopCh)
}
// wait until we're told to stop
<-stopCh
glog.Infof("Shutting down <NAME> controller")
}
func (c *Controller) runWorker() {
// hot loop until we're told to stop. processNextWorkItem will automatically wait until there's work
// available, so we don't don't worry about secondary waits
for c.processNextWorkItem() {
}
}
// processNextWorkItem deals with one key off the queue. It returns false when it's time to quit.
func (c *Controller) processNextWorkItem() bool {
// pull the next work item from queue. It should be a key we use to lookup something in a cache
key, quit := c.queue.Get()
if quit {
return false
}
// you always have to indicate to the queue that you've completed a piece of work
defer c.queue.Done(key)
// do your work on the key. This method will contains your "do stuff" logic"
err := c.syncHandler(key.(string))
if err == nil {
// if you had no error, tell the queue to stop tracking history for your key. This will
// reset things like failure counts for per-item rate limiting
c.queue.Forget(key)
return true
}
// there was a failure so be sure to report it. This method allows for pluggable error handling
// which can be used for things like cluster-monitoring
utilruntime.HandleError(fmt.Errorf("%v failed with : %v", key, err))
// since we failed, we should requeue the item to work on later. This method will add a backoff
// to avoid hotlooping on particular items (they're probably still not going to work right away)
// and overall controller protection (everything I've done is broken, this controller needs to
// calm down or it can starve other useful work) cases.
c.queue.AddRateLimited(key)
return true
}
```
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/devel/controllers.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
docs/devel/development.md
View file @ bc8f7e2c
# Development Guide This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/devel/development.md](https://github.com/kubernetes/community/blob/master/contributors/devel/development.md)
This document is intended to be the canonical source of truth for things like
supported toolchain versions for building Kubernetes. If you find a
requirement that this doc does not capture, please
[submit an issue](https://github.com/kubernetes/kubernetes/issues) on github. If
you find other docs with references to requirements that are not simply links to
this doc, please [submit an issue](https://github.com/kubernetes/kubernetes/issues).
This document is intended to be relative to the branch in which it is found.
It is guaranteed that requirements will change over time for the development
branch, but release branches of Kubernetes should not change.
## Building Kubernetes with Docker
Official releases are built using Docker containers. To build Kubernetes using
Docker please follow [these instructions]
(http://releases.k8s.io/HEAD/build-tools/README.md).
## Building Kubernetes on a local OS/shell environment
Many of the Kubernetes development helper scripts rely on a fairly up-to-date
GNU tools environment, so most recent Linux distros should work just fine
out-of-the-box. Note that Mac OS X ships with somewhat outdated BSD-based tools,
some of which may be incompatible in subtle ways, so we recommend
[replacing those with modern GNU tools]
(https://www.topbug.net/blog/2013/04/14/install-and-use-gnu-command-line-tools-in-mac-os-x/).
### Go development environment
Kubernetes is written in the [Go](http://golang.org) programming language.
To build Kubernetes without using Docker containers, you'll need a Go
development environment. Builds for Kubernetes 1.0 - 1.2 require Go version
1.4.2. Builds for Kubernetes 1.3 and higher require Go version 1.6.0. If you
haven't set up a Go development environment, please follow [these
instructions](http://golang.org/doc/code.html) to install the go tools.
Set up your GOPATH and add a path entry for go binaries to your PATH. Typically
added to your ~/.profile:
```sh
export GOPATH=$HOME/go
export PATH=$PATH:$GOPATH/bin
```
### Godep dependency management
Kubernetes build and test scripts use [godep](https://github.com/tools/godep) to
manage dependencies.
#### Install godep
Ensure that [mercurial](http://mercurial.selenic.com/wiki/Download) is
installed on your system. (some of godep's dependencies use the mercurial
source control system). Use `apt-get install mercurial` or `yum install
mercurial` on Linux, or [brew.sh](http://brew.sh) on OS X, or download directly
from mercurial.
Install godep and go-bindata (may require sudo):
```sh
go get -u github.com/tools/godep
go get -u github.com/jteeuwen/go-bindata/go-bindata
```
Note:
At this time, godep version >= v63 is known to work in the Kubernetes project.
To check your version of godep:
```sh
$ godep version
godep v74 (linux/amd64/go1.6.2)
```
Developers planning to managing dependencies in the `vendor/` tree may want to
explore alternative environment setups. See
[using godep to manage dependencies](godep.md).
### Local build using make
To build Kubernetes using your local Go development environment (generate linux
binaries):
```sh
make
```
You may pass build options and packages to the script as necessary. For example,
to build with optimizations disabled for enabling use of source debug tools:
```sh
make GOGCFLAGS="-N -l"
```
To build binaries for all platforms:
```sh
make cross
```
### How to update the Go version used to test & build k8s
The kubernetes project tries to stay on the latest version of Go so it can
benefit from the improvements to the language over time and can easily
bump to a minor release version for security updates.
Since kubernetes is mostly built and tested in containers, there are a few
unique places you need to update the go version.
- The image for cross compiling in [build-tools/build-image/cross/](../../build-tools/build-image/cross/). The `VERSION` file and `Dockerfile`.
- Update [dockerized-e2e-runner.sh](https://github.com/kubernetes/test-infra/blob/master/jenkins/dockerized-e2e-runner.sh) to run a kubekins-e2e with the desired go version, which requires pushing [e2e-image](https://github.com/kubernetes/test-infra/tree/master/jenkins/e2e-image) and [test-image](https://github.com/kubernetes/test-infra/tree/master/jenkins/test-image) images that are `FROM` the desired go version.
- The docker image being run in [gotest-dockerized.sh](https://github.com/kubernetes/test-infra/tree/master/jenkins/gotest-dockerized.sh).
- The cross tag `KUBE_BUILD_IMAGE_CROSS_TAG` in [build-tools/common.sh](../../build-tools/common.sh)
## Workflow
Below, we outline one of the more common git workflows that core developers use.
Other git workflows are also valid.
### Visual overview
![Git workflow](git_workflow.png)
### Fork the main repository
1. Go to https://github.com/kubernetes/kubernetes
2. Click the "Fork" button (at the top right)
### Clone your fork
The commands below require that you have $GOPATH set ([$GOPATH
docs](https://golang.org/doc/code.html#GOPATH)). We highly recommend you put
Kubernetes' code into your GOPATH. Note: the commands below will not work if
there is more than one directory in your `$GOPATH`.
```sh
mkdir -p $GOPATH/src/k8s.io
cd $GOPATH/src/k8s.io
# Replace "$YOUR_GITHUB_USERNAME" below with your github username
git clone https://github.com/$YOUR_GITHUB_USERNAME/kubernetes.git
cd kubernetes
git remote add upstream 'https://github.com/kubernetes/kubernetes.git'
```
### Create a branch and make changes
```sh
git checkout -b my-feature
# Make your code changes
```
### Keeping your development fork in sync
```sh
git fetch upstream
git rebase upstream/master
```
Note: If you have write access to the main repository at
github.com/kubernetes/kubernetes, you should modify your git configuration so
that you can't accidentally push to upstream:
```sh
git remote set-url --push upstream no_push
```
### Committing changes to your fork
Before committing any changes, please link/copy the pre-commit hook into your
.git directory. This will keep you from accidentally committing non-gofmt'd Go
code. This hook will also do a build and test whether documentation generation
scripts need to be executed.
The hook requires both Godep and etcd on your `PATH`.
```sh
cd kubernetes/.git/hooks/
ln -s ../../hooks/pre-commit .
```
Then you can commit your changes and push them to your fork:
```sh
git commit
git push -f origin my-feature
```
### Creating a pull request
1. Visit https://github.com/$YOUR_GITHUB_USERNAME/kubernetes
2. Click the "Compare & pull request" button next to your "my-feature" branch.
3. Check out the pull request [process](pull-requests.md) for more details
**Note:** If you have write access, please refrain from using the GitHub UI for creating PRs, because GitHub will create the PR branch inside the main repository rather than inside your fork.
### Getting a code review
Once your pull request has been opened it will be assigned to one or more
reviewers. Those reviewers will do a thorough code review, looking for
correctness, bugs, opportunities for improvement, documentation and comments,
and style.
Very small PRs are easy to review. Very large PRs are very difficult to
review. Github has a built-in code review tool, which is what most people use.
At the assigned reviewer's discretion, a PR may be switched to use
[Reviewable](https://reviewable.k8s.io) instead. Once a PR is switched to
Reviewable, please ONLY send or reply to comments through reviewable. Mixing
code review tools can be very confusing.
See [Faster Reviews](faster_reviews.md) for some thoughts on how to streamline
the review process.
### When to retain commits and when to squash
Upon merge, all git commits should represent meaningful milestones or units of
work. Use commits to add clarity to the development and review process.
Before merging a PR, squash any "fix review feedback", "typo", and "rebased"
sorts of commits. It is not imperative that every commit in a PR compile and
pass tests independently, but it is worth striving for. For mass automated
fixups (e.g. automated doc formatting), use one or more commits for the
changes to tooling and a final commit to apply the fixup en masse. This makes
reviews much easier.
## Testing
Three basic commands let you run unit, integration and/or e2e tests:
```sh
cd kubernetes
make test # Run every unit test
make test WHAT=pkg/util/cache GOFLAGS=-v # Run tests of a package verbosely
make test-integration # Run integration tests, requires etcd
make test-e2e # Run e2e tests
```
See the [testing guide](testing.md) and [end-to-end tests](e2e-tests.md) for additional information and scenarios.
## Regenerating the CLI documentation
```sh
hack/update-generated-docs.sh
```
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/devel/development.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
docs/devel/e2e-node-tests.md
View file @ bc8f7e2c
# Node End-To-End tests This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/devel/e2e-node-tests.md](https://github.com/kubernetes/community/blob/master/contributors/devel/e2e-node-tests.md)
Node e2e tests are component tests meant for testing the Kubelet code on a custom host environment.
Tests can be run either locally or against a host running on GCE.
Node e2e tests are run as both pre- and post- submit tests by the Kubernetes project.
*Note: Linux only. Mac and Windows unsupported.*
*Note: There is no scheduler running. The e2e tests have to do manual scheduling, e.g. by using `framework.PodClient`.*
# Running tests
## Locally
Why run tests *Locally*? Much faster than running tests Remotely.
Prerequisites:
- [Install etcd](https://github.com/coreos/etcd/releases) on your PATH
- Verify etcd is installed correctly by running `which etcd`
- Or make etcd binary available and executable at `/tmp/etcd`
- [Install ginkgo](https://github.com/onsi/ginkgo) on your PATH
- Verify ginkgo is installed correctly by running `which ginkgo`
From the Kubernetes base directory, run:
```sh
make test-e2e-node
```
This will: run the *ginkgo* binary against the subdirectory *test/e2e_node*, which will in turn:
- Ask for sudo access (needed for running some of the processes)
- Build the Kubernetes source code
- Pre-pull docker images used by the tests
- Start a local instance of *etcd*
- Start a local instance of *kube-apiserver*
- Start a local instance of *kubelet*
- Run the test using the locally started processes
- Output the test results to STDOUT
- Stop *kubelet*, *kube-apiserver*, and *etcd*
## Remotely
Why Run tests *Remotely*? Tests will be run in a customized pristine environment. Closely mimics what will be done
as pre- and post- submit testing performed by the project.
Prerequisites:
- [join the googlegroup](https://groups.google.com/forum/#!forum/kubernetes-dev)
`kubernetes-dev@googlegroups.com`
- *This provides read access to the node test images.*
- Setup a [Google Cloud Platform](https://cloud.google.com/) account and project with Google Compute Engine enabled
- Install and setup the [gcloud sdk](https://cloud.google.com/sdk/downloads)
- Verify the sdk is setup correctly by running `gcloud compute instances list` and `gcloud compute images list --project kubernetes-node-e2e-images`
Run:
```sh
make test-e2e-node REMOTE=true
```
This will:
- Build the Kubernetes source code
- Create a new GCE instance using the default test image
- Instance will be called **test-e2e-node-containervm-v20160321-image**
- Lookup the instance public ip address
- Copy a compressed archive file to the host containing the following binaries:
- ginkgo
- kubelet
- kube-apiserver
- e2e_node.test (this binary contains the actual tests to be run)
- Unzip the archive to a directory under **/tmp/gcloud**
- Run the tests using the `ginkgo` command
- Starts etcd, kube-apiserver, kubelet
- The ginkgo command is used because this supports more features than running the test binary directly
- Output the remote test results to STDOUT
- `scp` the log files back to the local host under /tmp/_artifacts/e2e-node-containervm-v20160321-image
- Stop the processes on the remote host
- **Leave the GCE instance running**
**Note: Subsequent tests run using the same image will *reuse the existing host* instead of deleting it and
provisioning a new one. To delete the GCE instance after each test see
*[DELETE_INSTANCE](#delete-instance-after-tests-run)*.**
# Additional Remote Options
## Run tests using different images
This is useful if you want to run tests against a host using a different OS distro or container runtime than
provided by the default image.
List the available test images using gcloud.
```sh
make test-e2e-node LIST_IMAGES=true
```
This will output a list of the available images for the default image project.
Then run:
```sh
make test-e2e-node REMOTE=true IMAGES="<comma-separated-list-images>"
```
## Run tests against a running GCE instance (not an image)
This is useful if you have an host instance running already and want to run the tests there instead of on a new instance.
```sh
make test-e2e-node REMOTE=true HOSTS="<comma-separated-list-of-hostnames>"
```
## Delete instance after tests run
This is useful if you want recreate the instance for each test run to trigger flakes related to starting the instance.
```sh
make test-e2e-node REMOTE=true DELETE_INSTANCES=true
```
## Keep instance, test binaries, and *processes* around after tests run
This is useful if you want to manually inspect or debug the kubelet process run as part of the tests.
```sh
make test-e2e-node REMOTE=true CLEANUP=false
```
## Run tests using an image in another project
This is useful if you want to create your own host image in another project and use it for testing.
```sh
make test-e2e-node REMOTE=true IMAGE_PROJECT="<name-of-project-with-images>" IMAGES="<image-name>"
```
Setting up your own host image may require additional steps such as installing etcd or docker. See
[setup_host.sh](../../test/e2e_node/environment/setup_host.sh) for common steps to setup hosts to run node tests.
## Create instances using a different instance name prefix
This is useful if you want to create instances using a different name so that you can run multiple copies of the
test in parallel against different instances of the same image.
```sh
make test-e2e-node REMOTE=true INSTANCE_PREFIX="my-prefix"
```
# Additional Test Options for both Remote and Local execution
## Only run a subset of the tests
To run tests matching a regex:
```sh
make test-e2e-node REMOTE=true FOCUS="<regex-to-match>"
```
To run tests NOT matching a regex:
```sh
make test-e2e-node REMOTE=true SKIP="<regex-to-match>"
```
## Run tests continually until they fail
This is useful if you are trying to debug a flaky test failure. This will cause ginkgo to continually
run the tests until they fail. **Note: this will only perform test setup once (e.g. creating the instance) and is
less useful for catching flakes related creating the instance from an image.**
```sh
make test-e2e-node REMOTE=true RUN_UNTIL_FAILURE=true
```
## Run tests in parallel
Running test in parallel can usually shorten the test duration. By default node
e2e test runs with`--nodes=8` (see ginkgo flag
[--nodes](https://onsi.github.io/ginkgo/#parallel-specs)). You can use the
`PARALLELISM` option to change the parallelism.
```sh
make test-e2e-node PARALLELISM=4 # run test with 4 parallel nodes
make test-e2e-node PARALLELISM=1 # run test sequentially
```
## Run tests with kubenet network plugin
[kubenet](http://kubernetes.io/docs/admin/network-plugins/#kubenet) is
the default network plugin used by kubelet since Kubernetes 1.3. The
plugin requires [CNI](https://github.com/containernetworking/cni) and
[nsenter](http://man7.org/linux/man-pages/man1/nsenter.1.html).
Currently, kubenet is enabled by default for Remote execution `REMOTE=true`,
but disabled for Local execution. **Note: kubenet is not supported for
local execution currently. This may cause network related test result to be
different for Local and Remote execution. So if you want to run network
related test, Remote execution is recommended.**
To enable/disable kubenet:
```sh
make test_e2e_node TEST_ARGS="--disable-kubenet=true" # enable kubenet
make test_e2e_node TEST_ARGS="--disable-kubenet=false" # disable kubenet
```
## Additional QoS Cgroups Hierarchy level testing
For testing with the QoS Cgroup Hierarchy enabled, you can pass --experimental-cgroups-per-qos flag as an argument into Ginkgo using TEST_ARGS
```sh
make test_e2e_node TEST_ARGS="--experimental-cgroups-per-qos=true"
```
# Notes on tests run by the Kubernetes project during pre-, post- submit.
The node e2e tests are run by the PR builder for each Pull Request and the results published at
the bottom of the comments section. To re-run just the node e2e tests from the PR builder add the comment
`@k8s-bot node e2e test this issue: #<Flake-Issue-Number or IGNORE>` and **include a link to the test
failure logs if caused by a flake.**
The PR builder runs tests against the images listed in [jenkins-pull.properties](../../test/e2e_node/jenkins/jenkins-pull.properties)
The post submit tests run against the images listed in [jenkins-ci.properties](../../test/e2e_node/jenkins/jenkins-ci.properties)
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/devel/e2e-node-tests.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
docs/devel/flaky-tests.md
View file @ bc8f7e2c
# Flaky tests This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/devel/flaky-tests.md](https://github.com/kubernetes/community/blob/master/contributors/devel/flaky-tests.md)
Any test that fails occasionally is "flaky". Since our merges only proceed when
all tests are green, and we have a number of different CI systems running the
tests in various combinations, even a small percentage of flakes results in a
lot of pain for people waiting for their PRs to merge.
Therefore, it's very important that we write tests defensively. Situations that
"almost never happen" happen with some regularity when run thousands of times in
resource-constrained environments. Since flakes can often be quite hard to
reproduce while still being common enough to block merges occasionally, it's
additionally important that the test logs be useful for narrowing down exactly
what caused the failure.
Note that flakes can occur in unit tests, integration tests, or end-to-end
tests, but probably occur most commonly in end-to-end tests.
## Filing issues for flaky tests
Because flakes may be rare, it's very important that all relevant logs be
discoverable from the issue.
1. Search for the test name. If you find an open issue and you're 90% sure the
flake is exactly the same, add a comment instead of making a new issue.
2. If you make a new issue, you should title it with the test name, prefixed by
"e2e/unit/integration flake:" (whichever is appropriate)
3. Reference any old issues you found in step one. Also, make a comment in the
old issue referencing your new issue, because people monitoring only their
email do not see the backlinks github adds. Alternatively, tag the person or
people who most recently worked on it.
4. Paste, in block quotes, the entire log of the individual failing test, not
just the failure line.
5. Link to durable storage with the rest of the logs. This means (for all the
tests that Google runs) the GCS link is mandatory! The Jenkins test result
link is nice but strictly optional: not only does it expire more quickly,
it's not accessible to non-Googlers.
## Finding filed flaky test cases
Find flaky tests issues on GitHub under the [kind/flake issue label][flake].
There are significant numbers of flaky tests reported on a regular basis and P2
flakes are under-investigated. Fixing flakes is a quick way to gain expertise
and community goodwill.
[flake]: https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Akind%2Fflake
## Expectations when a flaky test is assigned to you
Note that we won't randomly assign these issues to you unless you've opted in or
you're part of a group that has opted in. We are more than happy to accept help
from anyone in fixing these, but due to the severity of the problem when merges
are blocked, we need reasonably quick turn-around time on test flakes. Therefore
we have the following guidelines:
1. If a flaky test is assigned to you, it's more important than anything else
you're doing unless you can get a special dispensation (in which case it will
be reassigned). If you have too many flaky tests assigned to you, or you
have such a dispensation, then it's *still* your responsibility to find new
owners (this may just mean giving stuff back to the relevant Team or SIG Lead).
2. You should make a reasonable effort to reproduce it. Somewhere between an
hour and half a day of concentrated effort is "reasonable". It is perfectly
reasonable to ask for help!
3. If you can reproduce it (or it's obvious from the logs what happened), you
should then be able to fix it, or in the case where someone is clearly more
qualified to fix it, reassign it with very clear instructions.
4. PRs that fix or help debug flakes may have the P0 priority set to get them
through the merge queue as fast as possible.
5. Once you have made a change that you believe fixes a flake, it is conservative
to keep the issue for the flake open and see if it manifests again after the
change is merged.
6. If you can't reproduce a flake: __don't just close it!__ Every time a flake comes
back, at least 2 hours of merge time is wasted. So we need to make monotonic
progress towards narrowing it down every time a flake occurs. If you can't
figure it out from the logs, add log messages that would have help you figure
it out. If you make changes to make a flake more reproducible, please link
your pull request to the flake you're working on.
7. If a flake has been open, could not be reproduced, and has not manifested in
3 months, it is reasonable to close the flake issue with a note saying
why.
# Reproducing unit test flakes
Try the [stress command](https://godoc.org/golang.org/x/tools/cmd/stress).
Just
```
$ go install golang.org/x/tools/cmd/stress
```
Then build your test binary
```
$ go test -c -race
```
Then run it under stress
```
$ stress ./package.test -test.run=FlakyTest
```
It runs the command and writes output to `/tmp/gostress-*` files when it fails.
It periodically reports with run counts. Be careful with tests that use the
`net/http/httptest` package; they could exhaust the available ports on your
system!
# Hunting flaky unit tests in Kubernetes
Sometimes unit tests are flaky. This means that due to (usually) race
conditions, they will occasionally fail, even though most of the time they pass.
We have a goal of 99.9% flake free tests. This means that there is only one
flake in one thousand runs of a test.
Running a test 1000 times on your own machine can be tedious and time consuming.
Fortunately, there is a better way to achieve this using Kubernetes.
_Note: these instructions are mildly hacky for now, as we get run once semantics
and logging they will get better_
There is a testing image `brendanburns/flake` up on the docker hub. We will use
this image to test our fix.
Create a replication controller with the following config:
```yaml
apiVersion: v1
kind: ReplicationController
metadata:
name: flakecontroller
spec:
replicas: 24
template:
metadata:
labels:
name: flake
spec:
containers:
- name: flake
image: brendanburns/flake
env:
- name: TEST_PACKAGE
value: pkg/tools
- name: REPO_SPEC
value: https://github.com/kubernetes/kubernetes
```
Note that we omit the labels and the selector fields of the replication
controller, because they will be populated from the labels field of the pod
template by default.
```sh
kubectl create -f ./controller.yaml
```
This will spin up 24 instances of the test. They will run to completion, then
exit, and the kubelet will restart them, accumulating more and more runs of the
test.
You can examine the recent runs of the test by calling `docker ps -a` and
looking for tasks that exited with non-zero exit codes. Unfortunately, docker
ps -a only keeps around the exit status of the last 15-20 containers with the
same image, so you have to check them frequently.
You can use this script to automate checking for failures, assuming your cluster
is running on GCE and has four nodes:
```sh
echo "" > output.txt
for i in {1..4}; do
echo "Checking kubernetes-node-${i}"
echo "kubernetes-node-${i}:" >> output.txt
gcloud compute ssh "kubernetes-node-${i}" --command="sudo docker ps -a" >> output.txt
done
grep "Exited ([^0])" output.txt
```
Eventually you will have sufficient runs for your purposes. At that point you
can delete the replication controller by running:
```sh
kubectl delete replicationcontroller flakecontroller
```
If you do a final check for flakes with `docker ps -a`, ignore tasks that
exited -1, since that's what happens when you stop the replication controller.
Happy flake hunting!
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/devel/flaky-tests.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
docs/devel/generating-clientset.md
View file @ bc8f7e2c
# Generation and release cycle of clientset This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/devel/generating-clientset.md](https://github.com/kubernetes/community/blob/master/contributors/devel/generating-clientset.md)
Client-gen is an automatic tool that generates [clientset](../../docs/proposals/client-package-structure.md#high-level-client-sets) based on API types. This doc introduces the use the client-gen, and the release cycle of the generated clientsets.
## Using client-gen
The workflow includes three steps:
1. Marking API types with tags: in `pkg/apis/${GROUP}/${VERSION}/types.go`, mark the types (e.g., Pods) that you want to generate clients for with the `// +genclient=true` tag. If the resource associated with the type is not namespace scoped (e.g., PersistentVolume), you need to append the `nonNamespaced=true` tag as well.
2.
- a. If you are developing in the k8s.io/kubernetes repository, you just need to run hack/update-codegen.sh.
- b. If you are running client-gen outside of k8s.io/kubernetes, you need to use the command line argument `--input` to specify the groups and versions of the APIs you want to generate clients for, client-gen will then look into `pkg/apis/${GROUP}/${VERSION}/types.go` and generate clients for the types you have marked with the `genclient` tags. For example, to generated a clientset named "my_release" including clients for api/v1 objects and extensions/v1beta1 objects, you need to run:
```
$ client-gen --input="api/v1,extensions/v1beta1" --clientset-name="my_release"
```
3. ***Adding expansion methods***: client-gen only generates the common methods, such as CRUD. You can manually add additional methods through the expansion interface. For example, this [file](../../pkg/client/clientset_generated/release_1_5/typed/core/v1/pod_expansion.go) adds additional methods to Pod's client. As a convention, we put the expansion interface and its methods in file ${TYPE}_expansion.go. In most cases, you don't want to remove existing expansion files. So to make life easier, instead of creating a new clientset from scratch, ***you can copy and rename an existing clientset (so that all the expansion files are copied)***, and then run client-gen.
## Output of client-gen
- clientset: the clientset will be generated at `pkg/client/clientset_generated/` by default, and you can change the path via the `--clientset-path` command line argument.
- Individual typed clients and client for group: They will be generated at `pkg/client/clientset_generated/${clientset_name}/typed/generated/${GROUP}/${VERSION}/`
## Released clientsets
If you are contributing code to k8s.io/kubernetes, try to use the release_X_Y clientset in this [directory](../../pkg/client/clientset_generated/).
If you need a stable Go client to build your own project, please refer to the [client-go repository](https://github.com/kubernetes/client-go).
We are migrating k8s.io/kubernetes to use client-go as well, see issue [#35159](https://github.com/kubernetes/kubernetes/issues/35159).
<!-- BEGIN MUNGE: GENERATED_ANALYTICS --> [![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/devel/generating-clientset.md?pixel)]() <!-- END MUNGE: GENERATED_ANALYTICS -->
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/devel/generating-clientset.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
docs/devel/getting-builds.md
View file @ bc8f7e2c
# Getting Kubernetes Builds This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/devel/getting-builds.md](https://github.com/kubernetes/community/blob/master/contributors/devel/getting-builds.md)
You can use [hack/get-build.sh](http://releases.k8s.io/HEAD/hack/get-build.sh)
to get a build or to use as a reference on how to get the most recent builds
with curl. With `get-build.sh` you can grab the most recent stable build, the
most recent release candidate, or the most recent build to pass our ci and gce
e2e tests (essentially a nightly build).
Run `./hack/get-build.sh -h` for its usage.
To get a build at a specific version (v1.1.1) use:
```console
./hack/get-build.sh v1.1.1
```
To get the latest stable release:
```console
./hack/get-build.sh release/stable
```
Use the "-v" option to print the version number of a build without retrieving
it. For example, the following prints the version number for the latest ci
build:
```console
./hack/get-build.sh -v ci/latest
```
You can also use the gsutil tool to explore the Google Cloud Storage release
buckets. Here are some examples:
```sh
gsutil cat gs://kubernetes-release-dev/ci/latest.txt # output the latest ci version number
gsutil cat gs://kubernetes-release-dev/ci/latest-green.txt # output the latest ci version number that passed gce e2e
gsutil ls gs://kubernetes-release-dev/ci/v0.20.0-29-g29a55cc/ # list the contents of a ci release
gsutil ls gs://kubernetes-release/release # list all official releases and rcs
```
## Install `gsutil`
Example installation:
```console
$ curl -sSL https://storage.googleapis.com/pub/gsutil.tar.gz | sudo tar -xz -C /usr/local/src
$ sudo ln -s /usr/local/src/gsutil/gsutil /usr/bin/gsutil
```
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/devel/getting-builds.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
docs/devel/go-code.md
View file @ bc8f7e2c
# Kubernetes Go Tools and Tips This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/devel/go-code.md](https://github.com/kubernetes/community/blob/master/contributors/devel/go-code.md)
Kubernetes is one of the largest open source Go projects, so good tooling a solid understanding of
Go is critical to Kubernetes development. This document provides a collection of resources, tools
and tips that our developers have found useful.
## Recommended Reading
- [Kubernetes Go development environment](development.md#go-development-environment)
- [The Go Spec](https://golang.org/ref/spec) - The Go Programming Language
Specification.
- [Go Tour](https://tour.golang.org/welcome/2) - Official Go tutorial.
- [Effective Go](https://golang.org/doc/effective_go.html) - A good collection of Go advice.
- [Kubernetes Code conventions](coding-conventions.md) - Style guide for Kubernetes code.
- [Three Go Landmines](https://gist.github.com/lavalamp/4bd23295a9f32706a48f) - Surprising behavior in the Go language. These have caused real bugs!
## Recommended Tools
- [godep](https://github.com/tools/godep) - Used for Kubernetes dependency management. See also [Kubernetes godep and dependency management](development.md#godep-and-dependency-management)
- [Go Version Manager](https://github.com/moovweb/gvm) - A handy tool for managing Go versions.
- [godepq](https://github.com/google/godepq) - A tool for analyzing go import trees.
## Go Tips
- [Godoc bookmarklet](https://gist.github.com/timstclair/c891fb8aeb24d663026371d91dcdb3fc) - navigate from a github page to the corresponding godoc page.
- Consider making a separate Go tree for each project, which can make overlapping dependency management much easier. Remember to set the `$GOPATH` correctly! Consider [scripting](https://gist.github.com/timstclair/17ca792a20e0d83b06dddef7d77b1ea0) this.
- Emacs users - setup [go-mode](https://github.com/dominikh/go-mode.el)
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/devel/go-code.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
docs/devel/godep.md
View file @ bc8f7e2c
# Using godep to manage dependencies This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/devel/godep.md](https://github.com/kubernetes/community/blob/master/contributors/devel/godep.md)
This document is intended to show a way for managing `vendor/` tree dependencies
in Kubernetes. If you are not planning on managing `vendor` dependencies go here
[Godep dependency management](development.md#godep-dependency-management).
## Alternate GOPATH for installing and using godep
There are many ways to build and host Go binaries. Here is one way to get
utilities like `godep` installed:
Create a new GOPATH just for your go tools and install godep:
```sh
export GOPATH=$HOME/go-tools
mkdir -p $GOPATH
go get -u github.com/tools/godep
```
Add this $GOPATH/bin to your path. Typically you'd add this to your ~/.profile:
```sh
export GOPATH=$HOME/go-tools
export PATH=$PATH:$GOPATH/bin
```
## Using godep
Here's a quick walkthrough of one way to use godeps to add or update a
Kubernetes dependency into `vendor/`. For more details, please see the
instructions in [godep's documentation](https://github.com/tools/godep).
1) Devote a directory to this endeavor:
_Devoting a separate directory is not strictly required, but it is helpful to
separate dependency updates from other changes._
```sh
export KPATH=$HOME/code/kubernetes
mkdir -p $KPATH/src/k8s.io
cd $KPATH/src/k8s.io
git clone https://github.com/$YOUR_GITHUB_USERNAME/kubernetes.git # assumes your fork is 'kubernetes'
# Or copy your existing local repo here. IMPORTANT: making a symlink doesn't work.
```
2) Set up your GOPATH.
```sh
# This will *not* let your local builds see packages that exist elsewhere on your system.
export GOPATH=$KPATH
```
3) Populate your new GOPATH.
```sh
cd $KPATH/src/k8s.io/kubernetes
godep restore
```
4) Next, you can either add a new dependency or update an existing one.
To add a new dependency is simple (if a bit slow):
```sh
cd $KPATH/src/k8s.io/kubernetes
DEP=example.com/path/to/dependency
godep get $DEP/...
# Now change code in Kubernetes to use the dependency.
./hack/godep-save.sh
```
To update an existing dependency is a bit more complicated. Godep has an
`update` command, but none of us can figure out how to actually make it work.
Instead, this procedure seems to work reliably:
```sh
cd $KPATH/src/k8s.io/kubernetes
DEP=example.com/path/to/dependency
# NB: For the next step, $DEP is assumed be the repo root. If it is actually a
# subdir of the repo, use the repo root here. This is required to keep godep
# from getting angry because `godep restore` left the tree in a "detached head"
# state.
rm -rf $KPATH/src/$DEP # repo root
godep get $DEP/...
# Change code in Kubernetes, if necessary.
rm -rf Godeps
rm -rf vendor
./hack/godep-save.sh
git checkout -- $(git status -s | grep "^ D" | awk '{print $2}' | grep ^Godeps)
```
_If `go get -u path/to/dependency` fails with compilation errors, instead try
`go get -d -u path/to/dependency` to fetch the dependencies without compiling
them. This is unusual, but has been observed._
After all of this is done, `git status` should show you what files have been
modified and added/removed. Make sure to `git add` and `git rm` them. It is
commonly advised to make one `git commit` which includes just the dependency
update and Godeps files, and another `git commit` that includes changes to
Kubernetes code to use the new/updated dependency. These commits can go into a
single pull request.
5) Before sending your PR, it's a good idea to sanity check that your
Godeps.json file and the contents of `vendor/ `are ok by running `hack/verify-godeps.sh`
_If `hack/verify-godeps.sh` fails after a `godep update`, it is possible that a
transitive dependency was added or removed but not updated by godeps. It then
may be necessary to perform a `hack/godep-save.sh` to pick up the transitive
dependency changes._
It is sometimes expedient to manually fix the /Godeps/Godeps.json file to
minimize the changes. However without great care this can lead to failures
with `hack/verify-godeps.sh`. This must pass for every PR.
6) If you updated the Godeps, please also update `Godeps/LICENSES` by running
`hack/update-godep-licenses.sh`.
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/devel/godep.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
docs/devel/gubernator.md
View file @ bc8f7e2c
# Gubernator This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/devel/gubernator.md](https://github.com/kubernetes/community/blob/master/contributors/devel/gubernator.md)
*This document is oriented at developers who want to use Gubernator to debug while developing for Kubernetes.*
<!-- BEGIN MUNGE: GENERATED_TOC -->
- [Gubernator](#gubernator)
- [What is Gubernator?](#what-is-gubernator)
- [Gubernator Features](#gubernator-features)
- [Test Failures list](#test-failures-list)
- [Log Filtering](#log-filtering)
- [Gubernator for Local Tests](#gubernator-for-local-tests)
- [Future Work](#future-work)
<!-- END MUNGE: GENERATED_TOC -->
## What is Gubernator?
[Gubernator](https://k8s-gubernator.appspot.com/) is a webpage for viewing and filtering Kubernetes
test results.
Gubernator simplifies the debugging proccess and makes it easier to track down failures by automating many
steps commonly taken in searching through logs, and by offering tools to filter through logs to find relevant lines.
Gubernator automates the steps of finding the failed tests, displaying relevant logs, and determining the
failed pods and the corresponing pod UID, namespace, and container ID.
It also allows for filtering of the log files to display relevant lines based on selected keywords, and
allows for multiple logs to be woven together by timestamp.
Gubernator runs on Google App Engine and fetches logs stored on Google Cloud Storage.
## Gubernator Features
### Test Failures list
Issues made by k8s-merge-robot will post a link to a page listing the failed tests.
Each failed test comes with the corresponding error log from a junit file and a link
to filter logs for that test.
Based on the message logged in the junit file, the pod name may be displayed.
![alt text](gubernator-images/testfailures.png)
[Test Failures List Example](https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke/11721)
### Log Filtering
The log filtering page comes with checkboxes and textboxes to aid in filtering. Filtered keywords will be bolded
and lines including keywords will be highlighted. Up to four lines around the line of interest will also be displayed.
![alt text](gubernator-images/filterpage.png)
If less than 100 lines are skipped, the "... skipping xx lines ..." message can be clicked to expand and show
the hidden lines.
Before expansion:
![alt text](gubernator-images/skipping1.png)
After expansion:
![alt text](gubernator-images/skipping2.png)
If the pod name was displayed in the Test Failures list, it will automatically be included in the filters.
If it is not found in the error message, it can be manually entered into the textbox. Once a pod name
is entered, the Pod UID, Namespace, and ContainerID may be automatically filled in as well. These can be
altered as well. To apply the filter, check off the options corresponding to the filter.
![alt text](gubernator-images/filterpage1.png)
To add a filter, type the term to be filtered into the textbox labeled "Add filter:" and press enter.
Additional filters will be displayed as checkboxes under the textbox.
![alt text](gubernator-images/filterpage3.png)
To choose which logs to view check off the checkboxes corresponding to the logs of interest. If multiple logs are
included, the "Weave by timestamp" option can weave the selected logs together based on the timestamp in each line.
![alt text](gubernator-images/filterpage2.png)
[Log Filtering Example 1](https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubelet-gce-e2e-ci/5535/nodelog?pod=pod-configmaps-b5b876cb-3e1e-11e6-8956-42010af0001d&junit=junit_03.xml&wrap=on&logfiles=%2Fkubernetes-jenkins%2Flogs%2Fkubelet-gce-e2e-ci%2F5535%2Fartifacts%2Ftmp-node-e2e-7a5a3b40-e2e-node-coreos-stable20160622-image%2Fkube-apiserver.log&logfiles=%2Fkubernetes-jenkins%2Flogs%2Fkubelet-gce-e2e-ci%2F5535%2Fartifacts%2Ftmp-node-e2e-7a5a3b40-e2e-node-coreos-stable20160622-image%2Fkubelet.log&UID=on&poduid=b5b8a59e-3e1e-11e6-b358-42010af0001d&ns=e2e-tests-configmap-oi12h&cID=tmp-node-e2e-7a5a3b40-e2e-node-coreos-stable20160622-image)
[Log Filtering Example 2](https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke/11721/nodelog?pod=client-containers-a53f813c-503e-11e6-88dd-0242ac110003&junit=junit_19.xml&wrap=on)
### Gubernator for Local Tests
*Currently Gubernator can only be used with remote node e2e tests.*
**NOTE: Using Gubernator with local tests will publically upload your test logs to Google Cloud Storage**
To use Gubernator to view logs from local test runs, set the GUBERNATOR tag to true.
A URL link to view the test results will be printed to the console.
Please note that running with the Gubernator tag will bypass the user confirmation for uploading to GCS.
```console
$ make test-e2e-node REMOTE=true GUBERNATOR=true
...
================================================================
Running gubernator.sh
Gubernator linked below:
k8s-gubernator.appspot.com/build/yourusername-g8r-logs/logs/e2e-node/timestamp
```
The gubernator.sh script can be run after running a remote node e2e test for the same effect.
```console
$ ./test/e2e_node/gubernator.sh
Do you want to run gubernator.sh and upload logs publicly to GCS? [y/n]y
...
Gubernator linked below:
k8s-gubernator.appspot.com/build/yourusername-g8r-logs/logs/e2e-node/timestamp
```
## Future Work
Gubernator provides a framework for debugging failures and introduces useful features.
There is still a lot of room for more features and growth to make the debugging process more efficient.
How to contribute (see https://github.com/kubernetes/test-infra/blob/master/gubernator/README.md)
* Extend GUBERNATOR flag to all local tests
* More accurate identification of pod name, container ID, etc.
* Change content of logged strings for failures to include more information
* Better regex in Gubernator
* Automate discovery of more keywords
* Volume Name
* Disk Name
* Pod IP
* Clickable API objects in the displayed lines in order to add them as filters
* Construct story of pod's lifetime
* Have concise view of what a pod went through from when pod was started to failure
* Improve UI
* Have separate folders of logs in rows instead of in one long column
* Improve interface for adding additional features (maybe instead of textbox and checkbox, have chips)
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/devel/gubernator.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
docs/devel/how-to-doc.md
View file @ bc8f7e2c
# Document Conventions This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/devel/how-to-doc.md](https://github.com/kubernetes/community/blob/master/contributors/devel/how-to-doc.md)
Updated: 11/3/2015
*This document is oriented at users and developers who want to write documents
for Kubernetes.*
**Table of Contents**
<!-- BEGIN MUNGE: GENERATED_TOC -->
- [Document Conventions](#document-conventions)
- [General Concepts](#general-concepts)
- [How to Get a Table of Contents](#how-to-get-a-table-of-contents)
- [How to Write Links](#how-to-write-links)
- [How to Include an Example](#how-to-include-an-example)
- [Misc.](#misc)
- [Code formatting](#code-formatting)
- [Syntax Highlighting](#syntax-highlighting)
- [Headings](#headings)
- [What Are Mungers?](#what-are-mungers)
- [Auto-added Mungers](#auto-added-mungers)
- [Generate Analytics](#generate-analytics)
- [Generated documentation](#generated-documentation)
<!-- END MUNGE: GENERATED_TOC -->
## General Concepts
Each document needs to be munged to ensure its format is correct, links are
valid, etc. To munge a document, simply run `hack/update-munge-docs.sh`. We
verify that all documents have been munged using `hack/verify-munge-docs.sh`.
The scripts for munging documents are called mungers, see the
[mungers section](#what-are-mungers) below if you're curious about how mungers
are implemented or if you want to write one.
## How to Get a Table of Contents
Instead of writing table of contents by hand, insert the following code in your
md file:
```
<!-- BEGIN MUNGE: GENERATED_TOC -->
<!-- END MUNGE: GENERATED_TOC -->
```
After running `hack/update-munge-docs.sh`, you'll see a table of contents
generated for you, layered based on the headings.
## How to Write Links
It's important to follow the rules when writing links. It helps us correctly
versionize documents for each release.
Use inline links instead of urls at all times. When you add internal links to
`docs/` or `examples/`, use relative links; otherwise, use
`http://releases.k8s.io/HEAD/<path/to/link>`. For example, avoid using:
```
[GCE](https://github.com/kubernetes/kubernetes/blob/master/docs/getting-started-guides/gce.md) # note that it's under docs/
[Kubernetes package](../../pkg/) # note that it's under pkg/
http://kubernetes.io/ # external link
```
Instead, use:
```
[GCE](../getting-started-guides/gce.md) # note that it's under docs/
[Kubernetes package](http://releases.k8s.io/HEAD/pkg/) # note that it's under pkg/
[Kubernetes](http://kubernetes.io/) # external link
```
The above example generates the following links:
[GCE](../getting-started-guides/gce.md),
[Kubernetes package](http://releases.k8s.io/HEAD/pkg/), and
[Kubernetes](http://kubernetes.io/).
## How to Include an Example
While writing examples, you may want to show the content of certain example
files (e.g. [pod.yaml](../../test/fixtures/doc-yaml/user-guide/pod.yaml)). In this case, insert the
following code in the md file:
```
<!-- BEGIN MUNGE: EXAMPLE path/to/file -->
<!-- END MUNGE: EXAMPLE path/to/file -->
```
Note that you should replace `path/to/file` with the relative path to the
example file. Then `hack/update-munge-docs.sh` will generate a code block with
the content of the specified file, and a link to download it. This way, you save
the time to do the copy-and-paste; what's better, the content won't become
out-of-date every time you update the example file.
For example, the following:
```
<!-- BEGIN MUNGE: EXAMPLE ../../test/fixtures/doc-yaml/user-guide/pod.yaml -->
<!-- END MUNGE: EXAMPLE ../../test/fixtures/doc-yaml/user-guide/pod.yaml -->
```
generates the following after `hack/update-munge-docs.sh`:
<!-- BEGIN MUNGE: EXAMPLE ../../test/fixtures/doc-yaml/user-guide/pod.yaml -->
```yaml
apiVersion: v1
kind: Pod
metadata:
name: nginx
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx
ports:
- containerPort: 80
```
[Download example](../../test/fixtures/doc-yaml/user-guide/pod.yaml?raw=true)
<!-- END MUNGE: EXAMPLE ../../test/fixtures/doc-yaml/user-guide/pod.yaml -->
## Misc.
### Code formatting
Wrap a span of code with single backticks (`` ` ``). To format multiple lines of
code as its own code block, use triple backticks (```` ``` ````).
### Syntax Highlighting
Adding syntax highlighting to code blocks improves readability. To do so, in
your fenced block, add an optional language identifier. Some useful identifier
includes `yaml`, `console` (for console output), and `sh` (for shell quote
format). Note that in a console output, put `$ ` at the beginning of each
command and put nothing at the beginning of the output. Here's an example of
console code block:
```
```console
$ kubectl create -f test/fixtures/doc-yaml/user-guide/pod.yaml
pod "foo" created
```　
```
which renders as:
```console
$ kubectl create -f test/fixtures/doc-yaml/user-guide/pod.yaml
pod "foo" created
```
### Headings
Add a single `#` before the document title to create a title heading, and add
`##` to the next level of section title, and so on. Note that the number of `#`
will determine the size of the heading.
## What Are Mungers?
Mungers are like gofmt for md docs which we use to format documents. To use it,
simply place
```
<!-- BEGIN MUNGE: xxxx -->
<!-- END MUNGE: xxxx -->
```
in your md files. Note that xxxx is the placeholder for a specific munger.
Appropriate content will be generated and inserted between two brackets after
you run `hack/update-munge-docs.sh`. See
[munger document](http://releases.k8s.io/HEAD/cmd/mungedocs/) for more details.
## Auto-added Mungers
After running `hack/update-munge-docs.sh`, you may see some code / mungers in
your md file that are auto-added. You don't have to add them manually. It's
recommended to just read this section as a reference instead of messing up with
the following mungers.
### Generate Analytics
ANALYTICS munger inserts a Google Anaylytics link for this page.
```
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
<!-- END MUNGE: GENERATED_ANALYTICS -->
```
# Generated documentation
Some documents can be generated automatically. Run `hack/generate-docs.sh` to
populate your repository with these generated documents, and a list of the files
it generates is placed in `.generated_docs`. To reduce merge conflicts, we do
not want to check these documents in; however, to make the link checker in the
munger happy, we check in a placeholder. `hack/update-generated-docs.sh` puts a
placeholder in the location where each generated document would go, and
`hack/verify-generated-docs.sh` verifies that the placeholder is in place.
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/devel/how-to-doc.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
docs/devel/instrumentation.md
View file @ bc8f7e2c
## Instrumenting Kubernetes with a new metric This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/devel/instrumentation.md](https://github.com/kubernetes/community/blob/master/contributors/devel/instrumentation.md)
The following is a step-by-step guide for adding a new metric to the Kubernetes
code base.
We use the Prometheus monitoring system's golang client library for
instrumenting our code. Once you've picked out a file that you want to add a
metric to, you should:
1. Import "github.com/prometheus/client_golang/prometheus".
2. Create a top-level var to define the metric. For this, you have to:
1. Pick the type of metric. Use a Gauge for things you want to set to a
particular value, a Counter for things you want to increment, or a Histogram or
Summary for histograms/distributions of values (typically for latency).
Histograms are better if you're going to aggregate the values across jobs, while
summaries are better if you just want the job to give you a useful summary of
the values.
2. Give the metric a name and description.
3. Pick whether you want to distinguish different categories of things using
labels on the metric. If so, add "Vec" to the name of the type of metric you
want and add a slice of the label names to the definition.
https://github.com/kubernetes/kubernetes/blob/cd3299307d44665564e1a5c77d0daa0286603ff5/pkg/apiserver/apiserver.go#L53
https://github.com/kubernetes/kubernetes/blob/cd3299307d44665564e1a5c77d0daa0286603ff5/pkg/kubelet/metrics/metrics.go#L31
3. Register the metric so that prometheus will know to export it.
https://github.com/kubernetes/kubernetes/blob/cd3299307d44665564e1a5c77d0daa0286603ff5/pkg/kubelet/metrics/metrics.go#L74
https://github.com/kubernetes/kubernetes/blob/cd3299307d44665564e1a5c77d0daa0286603ff5/pkg/apiserver/apiserver.go#L78
4. Use the metric by calling the appropriate method for your metric type (Set,
Inc/Add, or Observe, respectively for Gauge, Counter, or Histogram/Summary),
first calling WithLabelValues if your metric has any labels
https://github.com/kubernetes/kubernetes/blob/3ce7fe8310ff081dbbd3d95490193e1d5250d2c9/pkg/kubelet/kubelet.go#L1384
https://github.com/kubernetes/kubernetes/blob/cd3299307d44665564e1a5c77d0daa0286603ff5/pkg/apiserver/apiserver.go#L87
These are the metric type definitions if you're curious to learn about them or
need more information:
https://github.com/prometheus/client_golang/blob/master/prometheus/gauge.go
https://github.com/prometheus/client_golang/blob/master/prometheus/counter.go
https://github.com/prometheus/client_golang/blob/master/prometheus/histogram.go
https://github.com/prometheus/client_golang/blob/master/prometheus/summary.go
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/devel/instrumentation.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
docs/devel/issues.md
View file @ bc8f7e2c
## GitHub Issues for the Kubernetes Project This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/devel/issues.md](https://github.com/kubernetes/community/blob/master/contributors/devel/issues.md)
A quick overview of how we will review and prioritize incoming issues at
https://github.com/kubernetes/kubernetes/issues
### Priorities
We use GitHub issue labels for prioritization. The absence of a priority label
means the bug has not been reviewed and prioritized yet.
We try to apply these priority labels consistently across the entire project,
but if you notice an issue that you believe to be incorrectly prioritized,
please do let us know and we will evaluate your counter-proposal.
- **priority/P0**: Must be actively worked on as someone's top priority right
now. Stuff is burning. If it's not being actively worked on, someone is expected
to drop what they're doing immediately to work on it. Team leaders are
responsible for making sure that all P0's in their area are being actively
worked on. Examples include user-visible bugs in core features, broken builds or
tests and critical security issues.
- **priority/P1**: Must be staffed and worked on either currently, or very soon,
ideally in time for the next release.
- **priority/P2**: There appears to be general agreement that this would be good
to have, but we may not have anyone available to work on it right now or in the
immediate future. Community contributions would be most welcome in the mean time
(although it might take a while to get them reviewed if reviewers are fully
occupied with higher priority issues, for example immediately before a release).
- **priority/P3**: Possibly useful, but not yet enough support to actually get
it done. These are mostly place-holders for potentially good ideas, so that they
don't get completely forgotten, and can be referenced/deduped every time they
come up.
### Milestones
We additionally use milestones, based on minor version, for determining if a bug
should be fixed for the next release. These milestones will be especially
scrutinized as we get to the weeks just before a release. We can release a new
version of Kubernetes once they are empty. We will have two milestones per minor
release.
- **vX.Y**: The list of bugs that will be merged for that milestone once ready.
- **vX.Y-candidate**: The list of bug that we might merge for that milestone. A
bug shouldn't be in this milestone for more than a day or two towards the end of
a milestone. It should be triaged either into vX.Y, or moved out of the release
milestones.
The above priority scheme still applies. P0 and P1 issues are work we feel must
get done before release. P2 and P3 issues are work we would merge into the
release if it gets done, but we wouldn't block the release on it. A few days
before release, we will probably move all P2 and P3 bugs out of that milestone
in bulk.
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/devel/issues.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
docs/devel/kubemark-guide.md
View file @ bc8f7e2c
# Kubemark User Guide This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/devel/kubemark-guide.md](https://github.com/kubernetes/community/blob/master/contributors/devel/kubemark-guide.md)
## Introduction
Kubemark is a performance testing tool which allows users to run experiments on
simulated clusters. The primary use case is scalability testing, as simulated
clusters can be much bigger than the real ones. The objective is to expose
problems with the master components (API server, controller manager or
scheduler) that appear only on bigger clusters (e.g. small memory leaks).
This document serves as a primer to understand what Kubemark is, what it is not,
and how to use it.
## Architecture
On a very high level Kubemark cluster consists of two parts: real master
components and a set of “Hollow” Nodes. The prefix “Hollow” means an
implementation/instantiation of a component with all “moving” parts mocked out.
The best example is HollowKubelet, which pretends to be an ordinary Kubelet, but
does not start anything, nor mount any volumes - it just lies it does. More
detailed design and implementation details are at the end of this document.
Currently master components run on a dedicated machine(s), and HollowNodes run
on an ‘external’ Kubernetes cluster. This design has a slight advantage, over
running master components on external cluster, of completely isolating master
resources from everything else.
## Requirements
To run Kubemark you need a Kubernetes cluster (called `external cluster`)
for running all your HollowNodes and a dedicated machine for a master.
Master machine has to be directly routable from HollowNodes. You also need an
access to some Docker repository.
Currently scripts are written to be easily usable by GCE, but it should be
relatively straightforward to port them to different providers or bare metal.
## Common use cases and helper scripts
Common workflow for Kubemark is:
- starting a Kubemark cluster (on GCE)
- running e2e tests on Kubemark cluster
- monitoring test execution and debugging problems
- turning down Kubemark cluster
Included in descriptions there will be comments helpful for anyone who’ll want to
port Kubemark to different providers.
### Starting a Kubemark cluster
To start a Kubemark cluster on GCE you need to create an external kubernetes
cluster (it can be GCE, GKE or anything else) by yourself, make sure that kubeconfig
points to it by default, build a kubernetes release (e.g. by running
`make quick-release`) and run `test/kubemark/start-kubemark.sh` script.
This script will create a VM for master components, Pods for HollowNodes
and do all the setup necessary to let them talk to each other. It will use the
configuration stored in `cluster/kubemark/config-default.sh` - you can tweak it
however you want, but note that some features may not be implemented yet, as
implementation of Hollow components/mocks will probably be lagging behind ‘real’
one. For performance tests interesting variables are `NUM_NODES` and
`MASTER_SIZE`. After start-kubemark script is finished you’ll have a ready
Kubemark cluster, a kubeconfig file for talking to the Kubemark cluster is
stored in `test/kubemark/kubeconfig.kubemark`.
Currently we're running HollowNode with limit of 0.05 a CPU core and ~60MB or
memory, which taking into account default cluster addons and fluentD running on
an 'external' cluster, allows running ~17.5 HollowNodes per core.
#### Behind the scene details:
Start-kubemark script does quite a lot of things:
- Creates a master machine called hollow-cluster-master and PD for it (*uses
gcloud, should be easy to do outside of GCE*)
- Creates a firewall rule which opens port 443\* on the master machine (*uses
gcloud, should be easy to do outside of GCE*)
- Builds a Docker image for HollowNode from the current repository and pushes it
to the Docker repository (*GCR for us, using scripts from
`cluster/gce/util.sh` - it may get tricky outside of GCE*)
- Generates certificates and kubeconfig files, writes a kubeconfig locally to
`test/kubemark/kubeconfig.kubemark` and creates a Secret which stores kubeconfig for
HollowKubelet/HollowProxy use (*used gcloud to transfer files to Master, should
be easy to do outside of GCE*).
- Creates a ReplicationController for HollowNodes and starts them up. (*will
work exactly the same everywhere as long as MASTER_IP will be populated
correctly, but you’ll need to update docker image address if you’re not using
GCR and default image name*)
- Waits until all HollowNodes are in the Running phase (*will work exactly the
same everywhere*)
<sub>\* Port 443 is a secured port on the master machine which is used for all
external communication with the API server. In the last sentence *external*
means all traffic coming from other machines, including all the Nodes, not only
from outside of the cluster. Currently local components, i.e. ControllerManager
and Scheduler talk with API server using insecure port 8080.</sub>
### Running e2e tests on Kubemark cluster
To run standard e2e test on your Kubemark cluster created in the previous step
you execute `test/kubemark/run-e2e-tests.sh` script. It will configure ginkgo to
use Kubemark cluster instead of something else and start an e2e test. This
script should not need any changes to work on other cloud providers.
By default (if nothing will be passed to it) the script will run a Density '30
test. If you want to run a different e2e test you just need to provide flags you want to be
passed to `hack/ginkgo-e2e.sh` script, e.g. `--ginkgo.focus="Load"` to run the
Load test.
By default, at the end of each test, it will delete namespaces and everything
under it (e.g. events, replication controllers) on Kubemark master, which takes
a lot of time. Such work aren't needed in most cases: if you delete your
Kubemark cluster after running `run-e2e-tests.sh`; you don't care about
namespace deletion performance, specifically related to etcd; etc. There is a
flag that enables you to avoid namespace deletion: `--delete-namespace=false`.
Adding the flag should let you see in logs: `Found DeleteNamespace=false,
skipping namespace deletion!`
### Monitoring test execution and debugging problems
Run-e2e-tests prints the same output on Kubemark as on ordinary e2e cluster, but
if you need to dig deeper you need to learn how to debug HollowNodes and how
Master machine (currently) differs from the ordinary one.
If you need to debug master machine you can do similar things as you do on your
ordinary master. The difference between Kubemark setup and ordinary setup is
that in Kubemark etcd is run as a plain docker container, and all master
components are run as normal processes. There’s no Kubelet overseeing them. Logs
are stored in exactly the same place, i.e. `/var/logs/` directory. Because
binaries are not supervised by anything they won't be restarted in the case of a
crash.
To help you with debugging from inside the cluster startup script puts a
`~/configure-kubectl.sh` script on the master. It downloads `gcloud` and
`kubectl` tool and configures kubectl to work on unsecured master port (useful
if there are problems with security). After the script is run you can use
kubectl command from the master machine to play with the cluster.
Debugging HollowNodes is a bit more tricky, as if you experience a problem on
one of them you need to learn which hollow-node pod corresponds to a given
HollowNode known by the Master. During self-registeration HollowNodes provide
their cluster IPs as Names, which means that if you need to find a HollowNode
named `10.2.4.5` you just need to find a Pod in external cluster with this
cluster IP. There’s a helper script
`test/kubemark/get-real-pod-for-hollow-node.sh` that does this for you.
When you have a Pod name you can use `kubectl logs` on external cluster to get
logs, or use a `kubectl describe pod` call to find an external Node on which
this particular HollowNode is running so you can ssh to it.
E.g. you want to see the logs of HollowKubelet on which pod `my-pod` is running.
To do so you can execute:
```
$ kubectl kubernetes/test/kubemark/kubeconfig.kubemark describe pod my-pod
```
Which outputs pod description and among it a line:
```
Node: 1.2.3.4/1.2.3.4
```
To learn the `hollow-node` pod corresponding to node `1.2.3.4` you use
aforementioned script:
```
$ kubernetes/test/kubemark/get-real-pod-for-hollow-node.sh 1.2.3.4
```
which will output the line:
```
hollow-node-1234
```
Now you just use ordinary kubectl command to get the logs:
```
kubectl --namespace=kubemark logs hollow-node-1234
```
All those things should work exactly the same on all cloud providers.
### Turning down Kubemark cluster
On GCE you just need to execute `test/kubemark/stop-kubemark.sh` script, which
will delete HollowNode ReplicationController and all the resources for you. On
other providers you’ll need to delete all this stuff by yourself.
## Some current implementation details
Kubemark master uses exactly the same binaries as ordinary Kubernetes does. This
means that it will never be out of date. On the other hand HollowNodes use
existing fake for Kubelet (called SimpleKubelet), which mocks its runtime
manager with `pkg/kubelet/dockertools/fake_manager.go`, where most logic sits.
Because there’s no easy way of mocking other managers (e.g. VolumeManager), they
are not supported in Kubemark (e.g. we can’t schedule Pods with volumes in them
yet).
As the time passes more fakes will probably be plugged into HollowNodes, but
it’s crucial to make it as simple as possible to allow running a big number of
Hollows on a single core.
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/devel/kubemark-guide.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
docs/devel/local-cluster/docker.md
View file @ bc8f7e2c
**Stop. This guide has been superseded by [Minikube](https://github.com/kubernetes/minikube) which is the recommended method of running Kubernetes on your local machine.** This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/devel/local-cluster/docker.md](https://github.com/kubernetes/community/blob/master/contributors/devel/local-cluster/docker.md)
The following instructions show you how to set up a simple, single node Kubernetes cluster using Docker.
Here's a diagram of what the final result will look like:
![Kubernetes Single Node on Docker](k8s-singlenode-docker.png)
## Prerequisites
**Note: These steps have not been tested with the [Docker For Mac or Docker For Windows beta programs](https://blog.docker.com/2016/03/docker-for-mac-windows-beta/).**
1. You need to have Docker version >= "1.10" installed on the machine.
2. Enable mount propagation. Hyperkube is running in a container which has to mount volumes for other containers, for example in case of persistent storage. The required steps depend on the init system.
In case of **systemd**, change MountFlags in the Docker unit file to shared.
```shell
DOCKER_CONF=$(systemctl cat docker | head -1 | awk '{print $2}')
sed -i.bak 's/^\(MountFlags=\).*/\1shared/' $DOCKER_CONF
systemctl daemon-reload
systemctl restart docker
```
**Otherwise**, manually set the mount point used by Hyperkube to be shared:
```shell
mkdir -p /var/lib/kubelet
mount --bind /var/lib/kubelet /var/lib/kubelet
mount --make-shared /var/lib/kubelet
```
### Run it
1. Decide which Kubernetes version to use. Set the `${K8S_VERSION}` variable to a version of Kubernetes >= "v1.2.0".
If you'd like to use the current **stable** version of Kubernetes, run the following:
```sh
export K8S_VERSION=$(curl -sS https://storage.googleapis.com/kubernetes-release/release/stable.txt)
```
and for the **latest** available version (including unstable releases):
```sh
export K8S_VERSION=$(curl -sS https://storage.googleapis.com/kubernetes-release/release/latest.txt)
```
2. Start Hyperkube
```shell
export ARCH=amd64
docker run -d \
--volume=/sys:/sys:rw \
--volume=/var/lib/docker/:/var/lib/docker:rw \
--volume=/var/lib/kubelet/:/var/lib/kubelet:rw,shared \
--volume=/var/run:/var/run:rw \
--net=host \
--pid=host \
--privileged \
--name=kubelet \
gcr.io/google_containers/hyperkube-${ARCH}:${K8S_VERSION} \
/hyperkube kubelet \
--hostname-override=127.0.0.1 \
--api-servers=http://localhost:8080 \
--config=/etc/kubernetes/manifests \
--cluster-dns=10.0.0.10 \
--cluster-domain=cluster.local \
--allow-privileged --v=2
```
> Note that `--cluster-dns` and `--cluster-domain` is used to deploy dns, feel free to discard them if dns is not needed.
> If you would like to mount an external device as a volume, add `--volume=/dev:/dev` to the command above. It may however, cause some problems described in [#18230](https://github.com/kubernetes/kubernetes/issues/18230)
> Architectures other than `amd64` are experimental and sometimes unstable, but feel free to try them out! Valid values: `arm`, `arm64` and `ppc64le`. ARM is available with Kubernetes version `v1.3.0-alpha.2` and higher. ARM 64-bit and PowerPC 64 little-endian are available with `v1.3.0-alpha.3` and higher. Track progress on multi-arch support [here](https://github.com/kubernetes/kubernetes/issues/17981)
> If you are behind a proxy, you need to pass the proxy setup to curl in the containers to pull the certificates. Create a .curlrc under /root folder (because the containers are running as root) with the following line:
```
proxy = <your_proxy_server>:<port>
```
This actually runs the kubelet, which in turn runs a [pod](http://kubernetes.io/docs/user-guide/pods/) that contains the other master components.
** **SECURITY WARNING** ** services exposed via Kubernetes using Hyperkube are available on the host node's public network interface / IP address. Because of this, this guide is not suitable for any host node/server that is directly internet accessible. Refer to [#21735](https://github.com/kubernetes/kubernetes/issues/21735) for additional info.
### Download `kubectl`
At this point you should have a running Kubernetes cluster. You can test it out
by downloading the kubectl binary for `${K8S_VERSION}` (in this example: `{{page.version}}.0`).
Downloads:
- `linux/amd64`: http://storage.googleapis.com/kubernetes-release/release/{{page.version}}.0/bin/linux/amd64/kubectl
- `linux/386`: http://storage.googleapis.com/kubernetes-release/release/{{page.version}}.0/bin/linux/386/kubectl
- `linux/arm`: http://storage.googleapis.com/kubernetes-release/release/{{page.version}}.0/bin/linux/arm/kubectl
- `linux/arm64`: http://storage.googleapis.com/kubernetes-release/release/{{page.version}}.0/bin/linux/arm64/kubectl
- `linux/ppc64le`: http://storage.googleapis.com/kubernetes-release/release/{{page.version}}.0/bin/linux/ppc64le/kubectl
- `OS X/amd64`: http://storage.googleapis.com/kubernetes-release/release/{{page.version}}.0/bin/darwin/amd64/kubectl
- `OS X/386`: http://storage.googleapis.com/kubernetes-release/release/{{page.version}}.0/bin/darwin/386/kubectl
- `windows/amd64`: http://storage.googleapis.com/kubernetes-release/release/{{page.version}}.0/bin/windows/amd64/kubectl.exe
- `windows/386`: http://storage.googleapis.com/kubernetes-release/release/{{page.version}}.0/bin/windows/386/kubectl.exe
The generic download path is:
```
http://storage.googleapis.com/kubernetes-release/release/${K8S_VERSION}/bin/${GOOS}/${GOARCH}/${K8S_BINARY}
```
An example install with `linux/amd64`:
```
curl -sSL "https://storage.googleapis.com/kubernetes-release/release/{{page.version}}.0/bin/linux/amd64/kubectl" > /usr/bin/kubectl
chmod +x /usr/bin/kubectl
```
On OS X, to make the API server accessible locally, setup a ssh tunnel.
```shell
docker-machine ssh `docker-machine active` -N -L 8080:localhost:8080
```
Setting up a ssh tunnel is applicable to remote docker hosts as well.
(Optional) Create kubernetes cluster configuration:
```shell
kubectl config set-cluster test-doc --server=http://localhost:8080
kubectl config set-context test-doc --cluster=test-doc
kubectl config use-context test-doc
```
### Test it out
List the nodes in your cluster by running:
```shell
kubectl get nodes
```
This should print:
```shell
NAME STATUS AGE
127.0.0.1 Ready 1h
```
### Run an application
```shell
kubectl run nginx --image=nginx --port=80
```
Now run `docker ps` you should see nginx running. You may need to wait a few minutes for the image to get pulled.
### Expose it as a service
```shell
kubectl expose deployment nginx --port=80
```
Run the following command to obtain the cluster local IP of this service we just created:
```shell{% raw %}
ip=$(kubectl get svc nginx --template={{.spec.clusterIP}})
echo $ip
{% endraw %}```
Hit the webserver with this IP:
```shell{% raw %}
curl $ip
{% endraw %}```
On OS X, since docker is running inside a VM, run the following command instead:
```shell
docker-machine ssh `docker-machine active` curl $ip
```
## Deploy a DNS
Read [documentation for manually deploying a DNS](http://kubernetes.io/docs/getting-started-guides/docker-multinode/#deploy-dns-manually-for-v12x) for instructions.
### Turning down your cluster
1. Delete the nginx service and deployment:
If you plan on re-creating your nginx deployment and service you will need to clean it up.
```shell
kubectl delete service,deployments nginx
```
2. Delete all the containers including the kubelet:
```shell
docker rm -f kubelet
docker rm -f `docker ps | grep k8s | awk '{print $1}'`
```
3. Cleanup the filesystem:
On OS X, first ssh into the docker VM:
```shell
docker-machine ssh `docker-machine active`
```
```shell
grep /var/lib/kubelet /proc/mounts | awk '{print $2}' | sudo xargs -n1 umount
sudo rm -rf /var/lib/kubelet
```
### Troubleshooting
#### Node is in `NotReady` state
If you see your node as `NotReady` it's possible that your OS does not have memcg enabled.
1. Your kernel should support memory accounting. Ensure that the
following configs are turned on in your linux kernel:
```shell
CONFIG_RESOURCE_COUNTERS=y
CONFIG_MEMCG=y
```
2. Enable the memory accounting in the kernel, at boot, as command line
parameters as follows:
```shell
GRUB_CMDLINE_LINUX="cgroup_enable=memory=1"
```
NOTE: The above is specifically for GRUB2.
You can check the command line parameters passed to your kernel by looking at the
output of /proc/cmdline:
```shell
$ cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-3.18.4-aufs root=/dev/sda5 ro cgroup_enable=memory=1
```
## Support Level
IaaS Provider | Config. Mgmt | OS | Networking | Conforms | Support Level
-------------------- | ------------ | ------ | ---------- | ---------| ----------------------------
Docker Single Node | custom | N/A | local | | Project ([@brendandburns](https://github.com/brendandburns))
## Further reading
Please see the [Kubernetes docs](http://kubernetes.io/docs) for more details on administering
and using a Kubernetes cluster.
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/devel/local-cluster/docker.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
docs/devel/local-cluster/local.md
View file @ bc8f7e2c
**Stop. This guide has been superseded by [Minikube](https://github.com/kubernetes/minikube) which is the recommended method of running Kubernetes on your local machine.** This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/devel/local-cluster/local.md](https://github.com/kubernetes/community/blob/master/contributors/devel/local-cluster/local.md)
### Requirements
#### Linux
Not running Linux? Consider running Linux in a local virtual machine with [vagrant](https://www.vagrantup.com/), or on a cloud provider like Google Compute Engine
#### Docker
At least [Docker](https://docs.docker.com/installation/#installation)
1.8.3+. Ensure the Docker daemon is running and can be contacted (try `docker
ps`). Some of the Kubernetes components need to run as root, which normally
works fine with docker.
#### etcd
You need an [etcd](https://github.com/coreos/etcd/releases) in your path, please make sure it is installed and in your ``$PATH``.
#### go
You need [go](https://golang.org/doc/install) at least 1.4+ in your path, please make sure it is installed and in your ``$PATH``.
### Starting the cluster
First, you need to [download Kubernetes](http://kubernetes.io/docs/getting-started-guides/binary_release/). Then open a separate tab of your terminal
and run the following (since one needs sudo access to start/stop Kubernetes daemons, it is easier to run the entire script as root):
```shell
cd kubernetes
hack/local-up-cluster.sh
```
This will build and start a lightweight local cluster, consisting of a master
and a single node. Type Control-C to shut it down.
You can use the cluster/kubectl.sh script to interact with the local cluster. hack/local-up-cluster.sh will
print the commands to run to point kubectl at the local cluster.
### Running a container
Your cluster is running, and you want to start running containers!
You can now use any of the cluster/kubectl.sh commands to interact with your local setup.
```shell
export KUBERNETES_PROVIDER=local
cluster/kubectl.sh get pods
cluster/kubectl.sh get services
cluster/kubectl.sh get deployments
cluster/kubectl.sh run my-nginx --image=nginx --replicas=2 --port=80
## begin wait for provision to complete, you can monitor the docker pull by opening a new terminal
sudo docker images
## you should see it pulling the nginx image, once the above command returns it
sudo docker ps
## you should see your container running!
exit
## end wait
## create a service for nginx, which serves on port 80
cluster/kubectl.sh expose deployment my-nginx --port=80 --name=my-nginx
## introspect Kubernetes!
cluster/kubectl.sh get pods
cluster/kubectl.sh get services
cluster/kubectl.sh get deployments
## Test the nginx service with the IP/port from "get services" command
curl http://10.X.X.X:80/
```
### Running a user defined pod
Note the difference between a [container](http://kubernetes.io/docs/user-guide/containers/)
and a [pod](http://kubernetes.io/docs/user-guide/pods/). Since you only asked for the former, Kubernetes will create a wrapper pod for you.
However you cannot view the nginx start page on localhost. To verify that nginx is running you need to run `curl` within the docker container (try `docker exec`).
You can control the specifications of a pod via a user defined manifest, and reach nginx through your browser on the port specified therein:
```shell
cluster/kubectl.sh create -f test/fixtures/doc-yaml/user-guide/pod.yaml
```
Congratulations!
### FAQs
#### I cannot reach service IPs on the network.
Some firewall software that uses iptables may not interact well with
kubernetes. If you have trouble around networking, try disabling any
firewall or other iptables-using systems, first. Also, you can check
if SELinux is blocking anything by running a command such as `journalctl --since yesterday | grep avc`.
By default the IP range for service cluster IPs is 10.0.*.* - depending on your
docker installation, this may conflict with IPs for containers. If you find
containers running with IPs in this range, edit hack/local-cluster-up.sh and
change the service-cluster-ip-range flag to something else.
#### I changed Kubernetes code, how do I run it?
```shell
cd kubernetes
hack/build-go.sh
hack/local-up-cluster.sh
```
#### kubectl claims to start a container but `get pods` and `docker ps` don't show it.
One or more of the Kubernetes daemons might've crashed. Tail the [logs](http://kubernetes.io/docs/admin/cluster-troubleshooting/#looking-at-logs) of each in /tmp.
```shell
$ ls /tmp/kube*.log
$ tail -f /tmp/kube-apiserver.log
```
#### The pods fail to connect to the services by host names
The local-up-cluster.sh script doesn't start a DNS service. Similar situation can be found [here](http://issue.k8s.io/6667). You can start a manually.
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/devel/local-cluster/local.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
docs/devel/logging.md
View file @ bc8f7e2c
## Logging Conventions This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/devel/logging.md](https://github.com/kubernetes/community/blob/master/contributors/devel/logging.md)
The following conventions for the glog levels to use.
[glog](http://godoc.org/github.com/golang/glog) is globally preferred to
[log](http://golang.org/pkg/log/) for better runtime control.
* glog.Errorf() - Always an error
* glog.Warningf() - Something unexpected, but probably not an error
* glog.Infof() has multiple levels:
* glog.V(0) - Generally useful for this to ALWAYS be visible to an operator
* Programmer errors
* Logging extra info about a panic
* CLI argument handling
* glog.V(1) - A reasonable default log level if you don't want verbosity.
* Information about config (listening on X, watching Y)
* Errors that repeat frequently that relate to conditions that can be corrected (pod detected as unhealthy)
* glog.V(2) - Useful steady state information about the service and important log messages that may correlate to significant changes in the system. This is the recommended default log level for most systems.
* Logging HTTP requests and their exit code
* System state changing (killing pod)
* Controller state change events (starting pods)
* Scheduler log messages
* glog.V(3) - Extended information about changes
* More info about system state changes
* glog.V(4) - Debug level verbosity (for now)
* Logging in particularly thorny parts of code where you may want to come back later and check it
As per the comments, the practical default level is V(2). Developers and QE
environments may wish to run at V(3) or V(4). If you wish to change the log
level, you can pass in `-v=X` where X is the desired maximum level to log.
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/devel/logging.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
docs/devel/node-performance-testing.md
View file @ bc8f7e2c
# Measuring Node Performance This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/devel/node-performance-testing.md](https://github.com/kubernetes/community/blob/master/contributors/devel/node-performance-testing.md)
This document outlines the issues and pitfalls of measuring Node performance, as
well as the tools available.
## Cluster Set-up
There are lots of factors which can affect node performance numbers, so care
must be taken in setting up the cluster to make the intended measurements. In
addition to taking the following steps into consideration, it is important to
document precisely which setup was used. For example, performance can vary
wildly from commit-to-commit, so it is very important to **document which commit
or version** of Kubernetes was used, which Docker version was used, etc.
### Addon pods
Be aware of which addon pods are running on which nodes. By default Kubernetes
runs 8 addon pods, plus another 2 per node (`fluentd-elasticsearch` and
`kube-proxy`) in the `kube-system` namespace. The addon pods can be disabled for
more consistent results, but doing so can also have performance implications.
For example, Heapster polls each node regularly to collect stats data. Disabling
Heapster will hide the performance cost of serving those stats in the Kubelet.
#### Disabling Add-ons
Disabling addons is simple. Just ssh into the Kubernetes master and move the
addon from `/etc/kubernetes/addons/` to a backup location. More details
[here](../../cluster/addons/).
### Which / how many pods?
Performance will vary a lot between a node with 0 pods and a node with 100 pods.
In many cases you'll want to make measurements with several different amounts of
pods. On a single node cluster scaling a replication controller makes this easy,
just make sure the system reaches a steady-state before starting the
measurement. E.g. `kubectl scale replicationcontroller pause --replicas=100`
In most cases pause pods will yield the most consistent measurements since the
system will not be affected by pod load. However, in some special cases
Kubernetes has been tuned to optimize pods that are not doing anything, such as
the cAdvisor housekeeping (stats gathering). In these cases, performing a very
light task (such as a simple network ping) can make a difference.
Finally, you should also consider which features yours pods should be using. For
example, if you want to measure performance with probing, you should obviously
use pods with liveness or readiness probes configured. Likewise for volumes,
number of containers, etc.
### Other Tips
**Number of nodes** - On the one hand, it can be easier to manage logs, pods,
environment etc. with a single node to worry about. On the other hand, having
multiple nodes will let you gather more data in parallel for more robust
sampling.
## E2E Performance Test
There is an end-to-end test for collecting overall resource usage of node
components: [kubelet_perf.go](../../test/e2e/kubelet_perf.go). To
run the test, simply make sure you have an e2e cluster running (`go run
hack/e2e.go -up`) and [set up](#cluster-set-up) correctly.
Run the test with `go run hack/e2e.go -v -test
--test_args="--ginkgo.focus=resource\susage\stracking"`. You may also wish to
customise the number of pods or other parameters of the test (remember to rerun
`make WHAT=test/e2e/e2e.test` after you do).
## Profiling
Kubelet installs the [go pprof handlers]
(https://golang.org/pkg/net/http/pprof/), which can be queried for CPU profiles:
```console
$ kubectl proxy &
Starting to serve on 127.0.0.1:8001
$ curl -G "http://localhost:8001/api/v1/proxy/nodes/${NODE}:10250/debug/pprof/profile?seconds=${DURATION_SECONDS}" > $OUTPUT
$ KUBELET_BIN=_output/dockerized/bin/linux/amd64/kubelet
$ go tool pprof -web $KUBELET_BIN $OUTPUT
```
`pprof` can also provide heap usage, from the `/debug/pprof/heap` endpoint
(e.g. `http://localhost:8001/api/v1/proxy/nodes/${NODE}:10250/debug/pprof/heap`).
More information on go profiling can be found
[here](http://blog.golang.org/profiling-go-programs).
## Benchmarks
Before jumping through all the hoops to measure a live Kubernetes node in a real
cluster, it is worth considering whether the data you need can be gathered
through a Benchmark test. Go provides a really simple benchmarking mechanism,
just add a unit test of the form:
```go
// In foo_test.go
func BenchmarkFoo(b *testing.B) {
b.StopTimer()
setupFoo() // Perform any global setup
b.StartTimer()
for i := 0; i < b.N; i++ {
foo() // Functionality to measure
}
}
```
Then:
```console
$ go test -bench=. -benchtime=${SECONDS}s foo_test.go
```
More details on benchmarking [here](https://golang.org/pkg/testing/).
## TODO
- (taotao) Measuring docker performance
- Expand cluster set-up section
- (vishh) Measuring disk usage
- (yujuhong) Measuring memory usage
- Add section on monitoring kubelet metrics (e.g. with prometheus)
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/devel/node-performance-testing.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
docs/devel/on-call-build-cop.md
View file @ bc8f7e2c
## Kubernetes "Github and Build-cop" Rotation This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/devel/on-call-build-cop.md](https://github.com/kubernetes/community/blob/master/contributors/devel/on-call-build-cop.md)
### Preqrequisites
* Ensure you have [write access to http://github.com/kubernetes/kubernetes](https://github.com/orgs/kubernetes/teams/kubernetes-maintainers)
* Test your admin access by e.g. adding a label to an issue.
### Traffic sources and responsibilities
* GitHub Kubernetes [issues](https://github.com/kubernetes/kubernetes/issues)
and [pulls](https://github.com/kubernetes/kubernetes/pulls): Your job is to be
the first responder to all new issues and PRs. If you are not equipped to do
this (which is fine!), it is your job to seek guidance!
* Support issues should be closed and redirected to Stackoverflow (see example
response below).
* All incoming issues should be tagged with a team label
(team/{api,ux,control-plane,node,cluster,csi,redhat,mesosphere,gke,release-infra,test-infra,none});
for issues that overlap teams, you can use multiple team labels
* There is a related concept of "Github teams" which allow you to @ mention
a set of people; feel free to @ mention a Github team if you wish, but this is
not a substitute for adding a team/* label, which is required
* [Google teams](https://github.com/orgs/kubernetes/teams?utf8=%E2%9C%93&query=goog-)
* [Redhat teams](https://github.com/orgs/kubernetes/teams?utf8=%E2%9C%93&query=rh-)
* [SIGs](https://github.com/orgs/kubernetes/teams?utf8=%E2%9C%93&query=sig-)
* If the issue is reporting broken builds, broken e2e tests, or other
obvious P0 issues, label the issue with priority/P0 and assign it to someone.
This is the only situation in which you should add a priority/* label
* non-P0 issues do not need a reviewer assigned initially
* Assign any issues related to Vagrant to @derekwaynecarr (and @mention him
in the issue)
* All incoming PRs should be assigned a reviewer.
* unless it is a WIP (Work in Progress), RFC (Request for Comments), or design proposal.
* An auto-assigner [should do this for you] (https://github.com/kubernetes/kubernetes/pull/12365/files)
* When in doubt, choose a TL or team maintainer of the most relevant team; they can delegate
* Keep in mind that you can @ mention people in an issue/PR to bring it to
their attention without assigning it to them. You can also @ mention github
teams, such as @kubernetes/goog-ux or @kubernetes/kubectl
* If you need help triaging an issue or PR, consult with (or assign it to)
@brendandburns, @thockin, @bgrant0607, @quinton-hoole, @davidopp, @dchen1107,
@lavalamp (all U.S. Pacific Time) or @fgrzadkowski (Central European Time).
* At the beginning of your shift, please add team/* labels to any issues that
have fallen through the cracks and don't have one. Likewise, be fair to the next
person in rotation: try to ensure that every issue that gets filed while you are
on duty is handled. The Github query to find issues with no team/* label is:
[here](https://github.com/kubernetes/kubernetes/issues?utf8=%E2%9C%93&q=is%3Aopen+is%3Aissue+-label%3Ateam%2Fcontrol-plane+-label%3Ateam%2Fmesosphere+-label%3Ateam%2Fredhat+-label%3Ateam%2Frelease-infra+-label%3Ateam%2Fnone+-label%3Ateam%2Fnode+-label%3Ateam%2Fcluster+-label%3Ateam%2Fux+-label%3Ateam%2Fapi+-label%3Ateam%2Ftest-infra+-label%3Ateam%2Fgke+-label%3A"team%2FCSI-API+Machinery+SIG"+-label%3Ateam%2Fhuawei+-label%3Ateam%2Fsig-aws).
Example response for support issues:
```code
Please re-post your question to [stackoverflow]
(http://stackoverflow.com/questions/tagged/kubernetes).
We are trying to consolidate the channels to which questions for help/support
are posted so that we can improve our efficiency in responding to your requests,
and to make it easier for you to find answers to frequently asked questions and
how to address common use cases.
We regularly see messages posted in multiple forums, with the full response
thread only in one place or, worse, spread across multiple forums. Also, the
large volume of support issues on github is making it difficult for us to use
issues to identify real bugs.
The Kubernetes team scans stackoverflow on a regular basis, and will try to
ensure your questions don't go unanswered.
Before posting a new question, please search stackoverflow for answers to
similar questions, and also familiarize yourself with:
* [user guide](http://kubernetes.io/docs/user-guide/)
* [troubleshooting guide](http://kubernetes.io/docs/admin/cluster-troubleshooting/)
Again, thanks for using Kubernetes.
The Kubernetes Team
```
### Build-copping
* The [merge-bot submit queue](http://submit-queue.k8s.io/)
([source](https://github.com/kubernetes/contrib/tree/master/mungegithub/mungers/submit-queue.go))
should auto-merge all eligible PRs for you once they've passed all the relevant
checks mentioned below and all [critical e2e tests]
(https://goto.google.com/k8s-test/view/Critical%20Builds/) are passing. If the
merge-bot been disabled for some reason, or tests are failing, you might need to
do some manual merging to get things back on track.
* Once a day or so, look at the [flaky test builds]
(https://goto.google.com/k8s-test/view/Flaky/); if they are timing out, clusters
are failing to start, or tests are consistently failing (instead of just
flaking), file an issue to get things back on track.
* Jobs that are not in [critical e2e tests](https://goto.google.com/k8s-test/view/Critical%20Builds/)
or [flaky test builds](https://goto.google.com/k8s-test/view/Flaky/) are not
your responsibility to monitor. The `Test owner:` in the job description will be
automatically emailed if the job is failing.
* If you are oncall, ensure that PRs confirming to the following
pre-requisites are being merged at a reasonable rate:
* [Have been LGTMd](https://github.com/kubernetes/kubernetes/labels/lgtm)
* Pass Travis and Jenkins per-PR tests.
* Author has signed CLA if applicable.
* Although the shift schedule shows you as being scheduled Monday to Monday,
working on the weekend is neither expected nor encouraged. Enjoy your time
off.
* When the build is broken, roll back the PRs responsible ASAP
* When E2E tests are unstable, a "merge freeze" may be instituted. During a
merge freeze:
* Oncall should slowly merge LGTMd changes throughout the day while monitoring
E2E to ensure stability.
* Ideally the E2E run should be green, but some tests are flaky and can fail
randomly (not as a result of a particular change).
* If a large number of tests fail, or tests that normally pass fail, that
is an indication that one or more of the PR(s) in that build might be
problematic (and should be reverted).
* Use the Test Results Analyzer to see individual test history over time.
* Flake mitigation
* Tests that flake (fail a small percentage of the time) need an issue filed
against them. Please read [this](flaky-tests.md#filing-issues-for-flaky-tests);
the build cop is expected to file issues for any flaky tests they encounter.
* It's reasonable to manually merge PRs that fix a flake or otherwise mitigate it.
### Contact information
[@k8s-oncall](https://github.com/k8s-oncall) will reach the current person on
call.
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/devel/on-call-build-cop.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
docs/devel/on-call-rotations.md
View file @ bc8f7e2c
## Kubernetes On-Call Rotations This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/devel/on-call-rotations.md](https://github.com/kubernetes/community/blob/master/contributors/devel/on-call-rotations.md)
### Kubernetes "first responder" rotations
Kubernetes has generated a lot of public traffic: email, pull-requests, bugs,
etc. So much traffic that it's becoming impossible to keep up with it all! This
is a fantastic problem to have. In order to be sure that SOMEONE, but not
EVERYONE on the team is paying attention to public traffic, we have instituted
two "first responder" rotations, listed below. Please read this page before
proceeding to the pages linked below, which are specific to each rotation.
Please also read our [notes on OSS collaboration](collab.md), particularly the
bits about hours. Specifically, each rotation is expected to be active primarily
during work hours, less so off hours.
During regular workday work hours of your shift, your primary responsibility is
to monitor the traffic sources specific to your rotation. You can check traffic
in the evenings if you feel so inclined, but it is not expected to be as highly
focused as work hours. For weekends, you should check traffic very occasionally
(e.g. once or twice a day). Again, it is not expected to be as highly focused as
workdays. It is assumed that over time, everyone will get weekday and weekend
shifts, so the workload will balance out.
If you can not serve your shift, and you know this ahead of time, it is your
responsibility to find someone to cover and to change the rotation. If you have
an emergency, your responsibilities fall on the primary of the other rotation,
who acts as your secondary. If you need help to cover all of the tasks, partners
with oncall rotations (e.g.,
[Redhat](https://github.com/orgs/kubernetes/teams/rh-oncall)).
If you are not on duty you DO NOT need to do these things. You are free to focus
on "real work".
Note that Kubernetes will occasionally enter code slush/freeze, prior to
milestones. When it does, there might be changes in the instructions (assigning
milestones, for instance).
* [Github and Build Cop Rotation](on-call-build-cop.md)
* [User Support Rotation](on-call-user-support.md)
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/devel/on-call-rotations.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
docs/devel/on-call-user-support.md
View file @ bc8f7e2c
## Kubernetes "User Support" Rotation This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/devel/on-call-user-support.md](https://github.com/kubernetes/community/blob/master/contributors/devel/on-call-user-support.md)
### Traffic sources and responsibilities
* [StackOverflow](http://stackoverflow.com/questions/tagged/kubernetes) and
[ServerFault](http://serverfault.com/questions/tagged/google-kubernetes):
Respond to any thread that has no responses and is more than 6 hours old (over
time we will lengthen this timeout to allow community responses). If you are not
equipped to respond, it is your job to redirect to someone who can.
* [Query for unanswered Kubernetes StackOverflow questions](http://stackoverflow.com/search?q=%5Bkubernetes%5D+answers%3A0)
* [Query for unanswered Kubernetes ServerFault questions](http://serverfault.com/questions/tagged/google-kubernetes?sort=unanswered&pageSize=15)
* Direct poorly formulated questions to [stackoverflow's tips about how to ask](http://stackoverflow.com/help/how-to-ask)
* Direct off-topic questions to [stackoverflow's policy](http://stackoverflow.com/help/on-topic)
* [Slack](https://kubernetes.slack.com) ([registration](http://slack.k8s.io)):
Your job is to be on Slack, watching for questions and answering or redirecting
as needed. Also check out the [Slack Archive](http://kubernetes.slackarchive.io/).
* [Email/Groups](https://groups.google.com/forum/#!forum/google-containers):
Respond to any thread that has no responses and is more than 6 hours old (over
time we will lengthen this timeout to allow community responses). If you are not
equipped to respond, it is your job to redirect to someone who can.
* [Legacy] [IRC](irc://irc.freenode.net/#google-containers)
(irc.freenode.net #google-containers): watch IRC for questions and try to
redirect users to Slack. Also check out the
[IRC logs](https://botbot.me/freenode/google-containers/).
In general, try to direct support questions to:
1. Documentation, such as the [user guide](../user-guide/README.md) and
[troubleshooting guide](http://kubernetes.io/docs/troubleshooting/)
2. Stackoverflow
If you see questions on a forum other than Stackoverflow, try to redirect them
to Stackoverflow. Example response:
```code
Please re-post your question to [stackoverflow]
(http://stackoverflow.com/questions/tagged/kubernetes).
We are trying to consolidate the channels to which questions for help/support
are posted so that we can improve our efficiency in responding to your requests,
and to make it easier for you to find answers to frequently asked questions and
how to address common use cases.
We regularly see messages posted in multiple forums, with the full response
thread only in one place or, worse, spread across multiple forums. Also, the
large volume of support issues on github is making it difficult for us to use
issues to identify real bugs.
The Kubernetes team scans stackoverflow on a regular basis, and will try to
ensure your questions don't go unanswered.
Before posting a new question, please search stackoverflow for answers to
similar questions, and also familiarize yourself with:
* [user guide](http://kubernetes.io/docs/user-guide/)
* [troubleshooting guide](http://kubernetes.io/docs/troubleshooting/)
Again, thanks for using Kubernetes.
The Kubernetes Team
```
If you answer a question (in any of the above forums) that you think might be
useful for someone else in the future, *please add it to one of the FAQs in the
wiki*:
* [User FAQ](https://github.com/kubernetes/kubernetes/wiki/User-FAQ)
* [Developer FAQ](https://github.com/kubernetes/kubernetes/wiki/Developer-FAQ)
* [Debugging FAQ](https://github.com/kubernetes/kubernetes/wiki/Debugging-FAQ).
Getting it into the FAQ is more important than polish. Please indicate the date
it was added, so people can judge the likelihood that it is out-of-date (and
please correct any FAQ entries that you see contain out-of-date information).
### Contact information
[@k8s-support-oncall](https://github.com/k8s-support-oncall) will reach the
current person on call.
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/devel/on-call-user-support.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
docs/devel/owners.md
View file @ bc8f7e2c
# Owners files This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/devel/owners.md](https://github.com/kubernetes/community/blob/master/contributors/devel/owners.md)
_Note_: This is a design for a feature that is not yet implemented. See the [contrib PR](https://github.com/kubernetes/contrib/issues/1389) for the current progress.
## Overview
We want to establish owners for different parts of the code in the Kubernetes codebase. These owners
will serve as the approvers for code to be submitted to these parts of the repository. Notably, owners
are not necessarily expected to do the first code review for all commits to these areas, but they are
required to approve changes before they can be merged.
**Note** The Kubernetes project has a hiatus on adding new approvers to OWNERS files. At this time we are [adding more reviewers](https://github.com/kubernetes/kubernetes/pulls?utf8=%E2%9C%93&q=is%3Apr%20%22Curating%20owners%3A%22%20) to take the load off of the current set of approvers and once we have had a chance to flush this out for a release we will begin adding new approvers again. Adding new approvers is planned for after the Kubernetes 1.6.0 release.
## High Level flow
### Step One: A PR is submitted
After a PR is submitted, the automated kubernetes PR robot will append a message to the PR indicating the owners
that are required for the PR to be submitted.
Subsequently, a user can also request the approval message from the robot by writing:
```
@k8s-bot approvers
```
into a comment.
In either case, the automation replies with an annotation that indicates
the owners required to approve. The annotation is a comment that is applied to the PR.
This comment will say:
```
Approval is required from <owner-a> OR <owner-b>, AND <owner-c> OR <owner-d>, AND ...
```
The set of required owners is drawn from the OWNERS files in the repository (see below). For each file
there should be multiple different OWNERS, these owners are listed in the `OR` clause(s). Because
it is possible that a PR may cover different directories, with disjoint sets of OWNERS, a PR may require
approval from more than one person, this is where the `AND` clauses come from.
`<owner-a>` should be the github user id of the owner _without_ a leading `@` symbol to prevent the owner
from being cc'd into the PR by email.
### Step Two: A PR is LGTM'd
Once a PR is reviewed and LGTM'd it is eligible for submission. However, for it to be submitted
an owner for all of the files changed in the PR have to 'approve' the PR. A user is an owner for a
file if they are included in the OWNERS hierarchy (see below) for that file.
Owner approval comes in two forms:
* An owner adds a comment to the PR saying "I approve" or "approved"
* An owner is the original author of the PR
In the case of a comment based approval, the same rules as for the 'lgtm' label apply. If the PR is
changed by pushing new commits to the PR, the previous approval is invalidated, and the owner(s) must
approve again. Because of this is recommended that PR authors squash their PRs prior to getting approval
from owners.
### Step Three: A PR is merged
Once a PR is LGTM'd and all required owners have approved, it is eligible for merge. The merge bot takes care of
the actual merging.
## Design details
We need to build new features into the existing github munger in order to accomplish this. Additionally
we need to add owners files to the repository.
### Approval Munger
We need to add a munger that adds comments to PRs indicating whose approval they require. This munger will
look for PRs that do not have approvers already present in the comments, or where approvers have been
requested, and add an appropriate comment to the PR.
### Status Munger
GitHub has a [status api](https://developer.github.com/v3/repos/statuses/), we will add a status munger that pushes a status onto a PR of approval status. This status will only be approved if the relevant
approvers have approved the PR.
### Requiring approval status
Github has the ability to [require status checks prior to merging](https://help.github.com/articles/enabling-required-status-checks/)
Once we have the status check munger described above implemented, we will add this required status check
to our main branch as well as any release branches.
### Adding owners files
In each directory in the repository we may add an OWNERS file. This file will contain the github OWNERS
for that directory. OWNERSHIP is hierarchical, so if a directory does not container an OWNERS file, its
parent's OWNERS file is used instead. There will be a top-level OWNERS file to back-stop the system.
Obviously changing the OWNERS file requires OWNERS permission.
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/devel/owners.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
docs/devel/profiling.md
View file @ bc8f7e2c
# Profiling Kubernetes This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/devel/profiling.md](https://github.com/kubernetes/community/blob/master/contributors/devel/profiling.md)
This document explain how to plug in profiler and how to profile Kubernetes services.
## Profiling library
Go comes with inbuilt 'net/http/pprof' profiling library and profiling web service. The way service works is binding debug/pprof/ subtree on a running webserver to the profiler. Reading from subpages of debug/pprof returns pprof-formatted profiles of the running binary. The output can be processed offline by the tool of choice, or used as an input to handy 'go tool pprof', which can graphically represent the result.
## Adding profiling to services to APIserver.
TL;DR: Add lines:
```go
m.mux.HandleFunc("/debug/pprof/", pprof.Index)
m.mux.HandleFunc("/debug/pprof/profile", pprof.Profile)
m.mux.HandleFunc("/debug/pprof/symbol", pprof.Symbol)
```
to the init(c *Config) method in 'pkg/master/master.go' and import 'net/http/pprof' package.
In most use cases to use profiler service it's enough to do 'import _ net/http/pprof', which automatically registers a handler in the default http.Server. Slight inconvenience is that APIserver uses default server for intra-cluster communication, so plugging profiler to it is not really useful. In 'pkg/kubelet/server/server.go' more servers are created and started as separate goroutines. The one that is usually serving external traffic is secureServer. The handler for this traffic is defined in 'pkg/master/master.go' and stored in Handler variable. It is created from HTTP multiplexer, so the only thing that needs to be done is adding profiler handler functions to this multiplexer. This is exactly what lines after TL;DR do.
## Connecting to the profiler
Even when running profiler I found not really straightforward to use 'go tool pprof' with it. The problem is that at least for dev purposes certificates generated for APIserver are not signed by anyone trusted and because secureServer serves only secure traffic it isn't straightforward to connect to the service. The best workaround I found is by creating an ssh tunnel from the kubernetes_master open unsecured port to some external server, and use this server as a proxy. To save everyone looking for correct ssh flags, it is done by running:
```sh
ssh kubernetes_master -L<local_port>:localhost:8080
```
or analogous one for you Cloud provider. Afterwards you can e.g. run
```sh
go tool pprof http://localhost:<local_port>/debug/pprof/profile
```
to get 30 sec. CPU profile.
## Contention profiling
To enable contention profiling you need to add line `rt.SetBlockProfileRate(1)` in addition to `m.mux.HandleFunc(...)` added before (`rt` stands for `runtime` in `master.go`). This enables 'debug/pprof/block' subpage, which can be used as an input to `go tool pprof`.
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/devel/profiling.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
docs/devel/pull-requests.md
View file @ bc8f7e2c
<!-- BEGIN MUNGE: GENERATED_TOC --> This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/devel/pull-requests.md](https://github.com/kubernetes/community/blob/master/contributors/devel/pull-requests.md)
- [Pull Request Process](#pull-request-process)
- [Life of a Pull Request](#life-of-a-pull-request)
- [Before sending a pull request](#before-sending-a-pull-request)
- [Release Notes](#release-notes)
- [Reviewing pre-release notes](#reviewing-pre-release-notes)
- [Visual overview](#visual-overview)
- [Other notes](#other-notes)
- [Automation](#automation)
<!-- END MUNGE: GENERATED_TOC -->
# Pull Request Process
An overview of how pull requests are managed for kubernetes. This document
assumes the reader has already followed the [development guide](development.md)
to set up their environment.
# Life of a Pull Request
Unless in the last few weeks of a milestone when we need to reduce churn and stabilize, we aim to be always accepting pull requests.
Either the [on call](on-call-rotations.md) manually or the [github "munger"](https://github.com/kubernetes/contrib/tree/master/mungegithub) submit-queue plugin automatically will manage merging PRs.
There are several requirements for the submit-queue to work:
* Author must have signed CLA ("cla: yes" label added to PR)
* No changes can be made since last lgtm label was applied
* k8s-bot must have reported the GCE E2E build and test steps passed (Jenkins unit/integration, Jenkins e2e)
Additionally, for infrequent or new contributors, we require the on call to apply the "ok-to-merge" label manually. This is gated by the [whitelist](https://github.com/kubernetes/contrib/blob/master/mungegithub/whitelist.txt).
## Before sending a pull request
The following will save time for both you and your reviewer:
* Enable [pre-commit hooks](development.md#committing-changes-to-your-fork) and verify they pass.
* Verify `make verify` passes.
* Verify `make test` passes.
* Verify `make test-integration` passes.
## Release Notes
This section applies only to pull requests on the master branch.
For cherry-pick PRs, see the [Cherrypick instructions](cherry-picks.md)
1. All pull requests are initiated with a `release-note-label-needed` label.
1. For a PR to be ready to merge, the `release-note-label-needed` label must be removed and one of the other `release-note-*` labels must be added.
1. `release-note-none` is a valid option if the PR does not need to be mentioned
at release time.
1. `release-note` labeled PRs generate a release note using the PR title by
default OR the release-note block in the PR template if filled in.
* See the [PR template](../../.github/PULL_REQUEST_TEMPLATE.md) for more
details.
* PR titles and body comments are mutable and can be modified at any time
prior to the release to reflect a release note friendly message.
The only exception to these rules is when a PR is not a cherry-pick and is
targeted directly to the non-master branch. In this case, a `release-note-*`
label is required for that non-master PR.
### Reviewing pre-release notes
At any time, you can see what the release notes will look like on any branch.
(NOTE: This only works on Linux for now)
```
$ git pull https://github.com/kubernetes/release
$ RELNOTES=$PWD/release/relnotes
$ cd /to/your/kubernetes/repo
$ $RELNOTES -man # for details on how to use the tool
# Show release notes from the last release on a branch to HEAD
$ $RELNOTES --branch=master
```
## Visual overview
![PR workflow](pr_workflow.png)
# Other notes
Pull requests that are purely support questions will be closed and
redirected to [stackoverflow](http://stackoverflow.com/questions/tagged/kubernetes).
We do this to consolidate help/support questions into a single channel,
improve efficiency in responding to requests and make FAQs easier
to find.
Pull requests older than 2 weeks will be closed. Exceptions can be made
for PRs that have active review comments, or that are awaiting other dependent PRs.
Closed pull requests are easy to recreate, and little work is lost by closing a pull
request that subsequently needs to be reopened. We want to limit the total number of PRs in flight to:
* Maintain a clean project
* Remove old PRs that would be difficult to rebase as the underlying code has changed over time
* Encourage code velocity
# Automation
We use a variety of automation to manage pull requests. This automation is described in detail
[elsewhere.](automation.md)
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/devel/pull-requests.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
docs/devel/running-locally.md
View file @ bc8f7e2c
Getting started locally This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/devel/running-locally.md](https://github.com/kubernetes/community/blob/master/contributors/devel/running-locally.md)
-----------------------
**Table of Contents**
- [Requirements](#requirements)
- [Linux](#linux)
- [Docker](#docker)
- [etcd](#etcd)
- [go](#go)
- [OpenSSL](#openssl)
- [Clone the repository](#clone-the-repository)
- [Starting the cluster](#starting-the-cluster)
- [Running a container](#running-a-container)
- [Running a user defined pod](#running-a-user-defined-pod)
- [Troubleshooting](#troubleshooting)
- [I cannot reach service IPs on the network.](#i-cannot-reach-service-ips-on-the-network)
- [I cannot create a replication controller with replica size greater than 1! What gives?](#i-cannot-create-a-replication-controller-with-replica-size-greater-than-1--what-gives)
- [I changed Kubernetes code, how do I run it?](#i-changed-kubernetes-code-how-do-i-run-it)
- [kubectl claims to start a container but `get pods` and `docker ps` don't show it.](#kubectl-claims-to-start-a-container-but-get-pods-and-docker-ps-dont-show-it)
- [The pods fail to connect to the services by host names](#the-pods-fail-to-connect-to-the-services-by-host-names)
### Requirements
#### Linux
Not running Linux? Consider running [Minikube](http://kubernetes.io/docs/getting-started-guides/minikube/), or on a cloud provider like [Google Compute Engine](../getting-started-guides/gce.md).
#### Docker
At least [Docker](https://docs.docker.com/installation/#installation)
1.3+. Ensure the Docker daemon is running and can be contacted (try `docker
ps`). Some of the Kubernetes components need to run as root, which normally
works fine with docker.
#### etcd
You need an [etcd](https://github.com/coreos/etcd/releases) in your path, please make sure it is installed and in your ``$PATH``.
#### go
You need [go](https://golang.org/doc/install) in your path (see [here](development.md#go-versions) for supported versions), please make sure it is installed and in your ``$PATH``.
#### OpenSSL
You need [OpenSSL](https://www.openssl.org/) installed. If you do not have the `openssl` command available, you may see the following error in `/tmp/kube-apiserver.log`:
```
server.go:333] Invalid Authentication Config: open /tmp/kube-serviceaccount.key: no such file or directory
```
### Clone the repository
In order to run kubernetes you must have the kubernetes code on the local machine. Cloning this repository is sufficient.
```$ git clone --depth=1 https://github.com/kubernetes/kubernetes.git```
The `--depth=1` parameter is optional and will ensure a smaller download.
### Starting the cluster
In a separate tab of your terminal, run the following (since one needs sudo access to start/stop Kubernetes daemons, it is easier to run the entire script as root):
```sh
cd kubernetes
hack/local-up-cluster.sh
```
This will build and start a lightweight local cluster, consisting of a master
and a single node. Type Control-C to shut it down.
If you've already compiled the Kubernetes components, then you can avoid rebuilding them with this script by using the `-O` flag.
```sh
./hack/local-up-cluster.sh -O
```
You can use the cluster/kubectl.sh script to interact with the local cluster. hack/local-up-cluster.sh will
print the commands to run to point kubectl at the local cluster.
### Running a container
Your cluster is running, and you want to start running containers!
You can now use any of the cluster/kubectl.sh commands to interact with your local setup.
```sh
cluster/kubectl.sh get pods
cluster/kubectl.sh get services
cluster/kubectl.sh get replicationcontrollers
cluster/kubectl.sh run my-nginx --image=nginx --replicas=2 --port=80
## begin wait for provision to complete, you can monitor the docker pull by opening a new terminal
sudo docker images
## you should see it pulling the nginx image, once the above command returns it
sudo docker ps
## you should see your container running!
exit
## end wait
## introspect Kubernetes!
cluster/kubectl.sh get pods
cluster/kubectl.sh get services
cluster/kubectl.sh get replicationcontrollers
```
### Running a user defined pod
Note the difference between a [container](../user-guide/containers.md)
and a [pod](../user-guide/pods.md). Since you only asked for the former, Kubernetes will create a wrapper pod for you.
However you cannot view the nginx start page on localhost. To verify that nginx is running you need to run `curl` within the docker container (try `docker exec`).
You can control the specifications of a pod via a user defined manifest, and reach nginx through your browser on the port specified therein:
```sh
cluster/kubectl.sh create -f test/fixtures/doc-yaml/user-guide/pod.yaml
```
Congratulations!
### Troubleshooting
#### I cannot reach service IPs on the network.
Some firewall software that uses iptables may not interact well with
kubernetes. If you have trouble around networking, try disabling any
firewall or other iptables-using systems, first. Also, you can check
if SELinux is blocking anything by running a command such as `journalctl --since yesterday | grep avc`.
By default the IP range for service cluster IPs is 10.0.*.* - depending on your
docker installation, this may conflict with IPs for containers. If you find
containers running with IPs in this range, edit hack/local-cluster-up.sh and
change the service-cluster-ip-range flag to something else.
#### I cannot create a replication controller with replica size greater than 1! What gives?
You are running a single node setup. This has the limitation of only supporting a single replica of a given pod. If you are interested in running with larger replica sizes, we encourage you to try the local vagrant setup or one of the cloud providers.
#### I changed Kubernetes code, how do I run it?
```sh
cd kubernetes
make
hack/local-up-cluster.sh
```
#### kubectl claims to start a container but `get pods` and `docker ps` don't show it.
One or more of the Kubernetes daemons might've crashed. Tail the logs of each in /tmp.
#### The pods fail to connect to the services by host names
To start the DNS service, you need to set the following variables:
```sh
KUBE_ENABLE_CLUSTER_DNS=true
KUBE_DNS_SERVER_IP="10.0.0.10"
KUBE_DNS_DOMAIN="cluster.local"
KUBE_DNS_REPLICAS=1
```
To know more on DNS service you can look [here](http://issue.k8s.io/6667). Related documents can be found [here](../../build-tools/kube-dns/#how-do-i-configure-it)
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/devel/running-locally.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
docs/devel/scheduler.md
View file @ bc8f7e2c
# The Kubernetes Scheduler This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/devel/scheduler.md](https://github.com/kubernetes/community/blob/master/contributors/devel/scheduler.md)
The Kubernetes scheduler runs as a process alongside the other master
components such as the API server. Its interface to the API server is to watch
for Pods with an empty PodSpec.NodeName, and for each Pod, it posts a Binding
indicating where the Pod should be scheduled.
## The scheduling process
```
+-------+
+---------------+ node 1|
| +-------+
|
+----> | Apply pred. filters
| |
| | +-------+
| +----+---------->+node 2 |
| | +--+----+
| watch | |
| | | +------+
| +---------------------->+node 3|
+--+---------------+ | +--+---+
| Pods in apiserver| | |
+------------------+ | |
| |
| |
+------------V------v--------+
| Priority function |
+-------------+--------------+
|
| node 1: p=2
| node 2: p=5
v
select max{node priority} = node 2
```
The Scheduler tries to find a node for each Pod, one at a time.
- First it applies a set of "predicates" to filter out inappropriate nodes. For example, if the PodSpec specifies resource requests, then the scheduler will filter out nodes that don't have at least that much resources available (computed as the capacity of the node minus the sum of the resource requests of the containers that are already running on the node).
- Second, it applies a set of "priority functions"
that rank the nodes that weren't filtered out by the predicate check. For example, it tries to spread Pods across nodes and zones while at the same time favoring the least (theoretically) loaded nodes (where "load" - in theory - is measured as the sum of the resource requests of the containers running on the node, divided by the node's capacity).
- Finally, the node with the highest priority is chosen (or, if there are multiple such nodes, then one of them is chosen at random). The code for this main scheduling loop is in the function `Schedule()` in [plugin/pkg/scheduler/generic_scheduler.go](http://releases.k8s.io/HEAD/plugin/pkg/scheduler/generic_scheduler.go)
## Scheduler extensibility
The scheduler is extensible: the cluster administrator can choose which of the pre-defined
scheduling policies to apply, and can add new ones.
### Policies (Predicates and Priorities)
The built-in predicates and priorities are
defined in [plugin/pkg/scheduler/algorithm/predicates/predicates.go](http://releases.k8s.io/HEAD/plugin/pkg/scheduler/algorithm/predicates/predicates.go) and
[plugin/pkg/scheduler/algorithm/priorities/priorities.go](http://releases.k8s.io/HEAD/plugin/pkg/scheduler/algorithm/priorities/priorities.go), respectively.
### Modifying policies
The policies that are applied when scheduling can be chosen in one of two ways. Normally,
the policies used are selected by the functions `defaultPredicates()` and `defaultPriorities()` in
[plugin/pkg/scheduler/algorithmprovider/defaults/defaults.go](http://releases.k8s.io/HEAD/plugin/pkg/scheduler/algorithmprovider/defaults/defaults.go).
However, the choice of policies can be overridden by passing the command-line flag `--policy-config-file` to the scheduler, pointing to a JSON file specifying which scheduling policies to use. See [examples/scheduler-policy-config.json](../../examples/scheduler-policy-config.json) for an example
config file. (Note that the config file format is versioned; the API is defined in [plugin/pkg/scheduler/api](http://releases.k8s.io/HEAD/plugin/pkg/scheduler/api/)).
Thus to add a new scheduling policy, you should modify predicates.go or priorities.go, and either register the policy in `defaultPredicates()` or `defaultPriorities()`, or use a policy config file.
## Exploring the code
If you want to get a global picture of how the scheduler works, you can start in
[plugin/cmd/kube-scheduler/app/server.go](http://releases.k8s.io/HEAD/plugin/cmd/kube-scheduler/app/server.go)
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/devel/scheduler.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
docs/devel/scheduler_algorithm.md
View file @ bc8f7e2c
# Scheduler Algorithm in Kubernetes This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/devel/scheduler_algorithm.md](https://github.com/kubernetes/community/blob/master/contributors/devel/scheduler_algorithm.md)
For each unscheduled Pod, the Kubernetes scheduler tries to find a node across the cluster according to a set of rules. A general introduction to the Kubernetes scheduler can be found at [scheduler.md](scheduler.md). In this document, the algorithm of how to select a node for the Pod is explained. There are two steps before a destination node of a Pod is chosen. The first step is filtering all the nodes and the second is ranking the remaining nodes to find a best fit for the Pod.
## Filtering the nodes
The purpose of filtering the nodes is to filter out the nodes that do not meet certain requirements of the Pod. For example, if the free resource on a node (measured by the capacity minus the sum of the resource requests of all the Pods that already run on the node) is less than the Pod's required resource, the node should not be considered in the ranking phase so it is filtered out. Currently, there are several "predicates" implementing different filtering policies, including:
- `NoDiskConflict`: Evaluate if a pod can fit due to the volumes it requests, and those that are already mounted. Currently supported volumes are: AWS EBS, GCE PD, and Ceph RBD. Only Persistent Volume Claims for those supported types are checked. Persistent Volumes added directly to pods are not evaluated and are not constrained by this policy.
- `NoVolumeZoneConflict`: Evaluate if the volumes a pod requests are available on the node, given the Zone restrictions.
- `PodFitsResources`: Check if the free resource (CPU and Memory) meets the requirement of the Pod. The free resource is measured by the capacity minus the sum of requests of all Pods on the node. To learn more about the resource QoS in Kubernetes, please check [QoS proposal](../design/resource-qos.md).
- `PodFitsHostPorts`: Check if any HostPort required by the Pod is already occupied on the node.
- `HostName`: Filter out all nodes except the one specified in the PodSpec's NodeName field.
- `MatchNodeSelector`: Check if the labels of the node match the labels specified in the Pod's `nodeSelector` field and, as of Kubernetes v1.2, also match the `scheduler.alpha.kubernetes.io/affinity` pod annotation if present. See [here](../user-guide/node-selection/) for more details on both.
- `MaxEBSVolumeCount`: Ensure that the number of attached ElasticBlockStore volumes does not exceed a maximum value (by default, 39, since Amazon recommends a maximum of 40 with one of those 40 reserved for the root volume -- see [Amazon's documentation](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/volume_limits.html#linux-specific-volume-limits)). The maximum value can be controlled by setting the `KUBE_MAX_PD_VOLS` environment variable.
- `MaxGCEPDVolumeCount`: Ensure that the number of attached GCE PersistentDisk volumes does not exceed a maximum value (by default, 16, which is the maximum GCE allows -- see [GCE's documentation](https://cloud.google.com/compute/docs/disks/persistent-disks#limits_for_predefined_machine_types)). The maximum value can be controlled by setting the `KUBE_MAX_PD_VOLS` environment variable.
- `CheckNodeMemoryPressure`: Check if a pod can be scheduled on a node reporting memory pressure condition. Currently, no ``BestEffort`` should be placed on a node under memory pressure as it gets automatically evicted by kubelet.
- `CheckNodeDiskPressure`: Check if a pod can be scheduled on a node reporting disk pressure condition. Currently, no pods should be placed on a node under disk pressure as it gets automatically evicted by kubelet.
The details of the above predicates can be found in [plugin/pkg/scheduler/algorithm/predicates/predicates.go](http://releases.k8s.io/HEAD/plugin/pkg/scheduler/algorithm/predicates/predicates.go). All predicates mentioned above can be used in combination to perform a sophisticated filtering policy. Kubernetes uses some, but not all, of these predicates by default. You can see which ones are used by default in [plugin/pkg/scheduler/algorithmprovider/defaults/defaults.go](http://releases.k8s.io/HEAD/plugin/pkg/scheduler/algorithmprovider/defaults/defaults.go).
## Ranking the nodes
The filtered nodes are considered suitable to host the Pod, and it is often that there are more than one nodes remaining. Kubernetes prioritizes the remaining nodes to find the "best" one for the Pod. The prioritization is performed by a set of priority functions. For each remaining node, a priority function gives a score which scales from 0-10 with 10 representing for "most preferred" and 0 for "least preferred". Each priority function is weighted by a positive number and the final score of each node is calculated by adding up all the weighted scores. For example, suppose there are two priority functions, `priorityFunc1` and `priorityFunc2` with weighting factors `weight1` and `weight2` respectively, the final score of some NodeA is:
finalScoreNodeA = (weight1 * priorityFunc1) + (weight2 * priorityFunc2)
After the scores of all nodes are calculated, the node with highest score is chosen as the host of the Pod. If there are more than one nodes with equal highest scores, a random one among them is chosen.
Currently, Kubernetes scheduler provides some practical priority functions, including:
- `LeastRequestedPriority`: The node is prioritized based on the fraction of the node that would be free if the new Pod were scheduled onto the node. (In other words, (capacity - sum of requests of all Pods already on the node - request of Pod that is being scheduled) / capacity). CPU and memory are equally weighted. The node with the highest free fraction is the most preferred. Note that this priority function has the effect of spreading Pods across the nodes with respect to resource consumption.
- `BalancedResourceAllocation`: This priority function tries to put the Pod on a node such that the CPU and Memory utilization rate is balanced after the Pod is deployed.
- `SelectorSpreadPriority`: Spread Pods by minimizing the number of Pods belonging to the same service, replication controller, or replica set on the same node. If zone information is present on the nodes, the priority will be adjusted so that pods are spread across zones and nodes.
- `CalculateAntiAffinityPriority`: Spread Pods by minimizing the number of Pods belonging to the same service on nodes with the same value for a particular label.
- `ImageLocalityPriority`: Nodes are prioritized based on locality of images requested by a pod. Nodes with larger size of already-installed packages required by the pod will be preferred over nodes with no already-installed packages required by the pod or a small total size of already-installed packages required by the pod.
- `NodeAffinityPriority`: (Kubernetes v1.2) Implements `preferredDuringSchedulingIgnoredDuringExecution` node affinity; see [here](../user-guide/node-selection/) for more details.
The details of the above priority functions can be found in [plugin/pkg/scheduler/algorithm/priorities](http://releases.k8s.io/HEAD/plugin/pkg/scheduler/algorithm/priorities/). Kubernetes uses some, but not all, of these priority functions by default. You can see which ones are used by default in [plugin/pkg/scheduler/algorithmprovider/defaults/defaults.go](http://releases.k8s.io/HEAD/plugin/pkg/scheduler/algorithmprovider/defaults/defaults.go). Similar as predicates, you can combine the above priority functions and assign weight factors (positive number) to them as you want (check [scheduler.md](scheduler.md) for how to customize).
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/devel/scheduler_algorithm.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
docs/devel/testing.md
View file @ bc8f7e2c
# Testing guide This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/devel/testing.md](https://github.com/kubernetes/community/blob/master/contributors/devel/testing.md)
Updated: 5/21/2016
**Table of Contents**
<!-- BEGIN MUNGE: GENERATED_TOC -->
- [Testing guide](#testing-guide)
- [Unit tests](#unit-tests)
- [Run all unit tests](#run-all-unit-tests)
- [Set go flags during unit tests](#set-go-flags-during-unit-tests)
- [Run unit tests from certain packages](#run-unit-tests-from-certain-packages)
- [Run specific unit test cases in a package](#run-specific-unit-test-cases-in-a-package)
- [Stress running unit tests](#stress-running-unit-tests)
- [Unit test coverage](#unit-test-coverage)
- [Benchmark unit tests](#benchmark-unit-tests)
- [Integration tests](#integration-tests)
- [Install etcd dependency](#install-etcd-dependency)
- [Etcd test data](#etcd-test-data)
- [Run integration tests](#run-integration-tests)
- [Run a specific integration test](#run-a-specific-integration-test)
- [End-to-End tests](#end-to-end-tests)
<!-- END MUNGE: GENERATED_TOC -->
This assumes you already read the [development guide](development.md) to
install go, godeps, and configure your git client. All command examples are
relative to the `kubernetes` root directory.
Before sending pull requests you should at least make sure your changes have
passed both unit and integration tests.
Kubernetes only merges pull requests when unit, integration, and e2e tests are
passing, so it is often a good idea to make sure the e2e tests work as well.
## Unit tests
* Unit tests should be fully hermetic
- Only access resources in the test binary.
* All packages and any significant files require unit tests.
* The preferred method of testing multiple scenarios or input is
[table driven testing](https://github.com/golang/go/wiki/TableDrivenTests)
- Example: [TestNamespaceAuthorization](../../test/integration/auth/auth_test.go)
* Unit tests must pass on OS X and Windows platforms.
- Tests using linux-specific features must be skipped or compiled out.
- Skipped is better, compiled out is required when it won't compile.
* Concurrent unit test runs must pass.
* See [coding conventions](coding-conventions.md).
### Run all unit tests
`make test` is the entrypoint for running the unit tests that ensures that
`GOPATH` is set up correctly. If you have `GOPATH` set up correctly, you can
also just use `go test` directly.
```sh
cd kubernetes
make test # Run all unit tests.
```
### Set go flags during unit tests
You can set [go flags](https://golang.org/cmd/go/) by setting the
`KUBE_GOFLAGS` environment variable.
### Run unit tests from certain packages
`make test` accepts packages as arguments; the `k8s.io/kubernetes` prefix is
added automatically to these:
```sh
make test WHAT=pkg/api # run tests for pkg/api
```
To run multiple targets you need quotes:
```sh
make test WHAT="pkg/api pkg/kubelet" # run tests for pkg/api and pkg/kubelet
```
In a shell, it's often handy to use brace expansion:
```sh
make test WHAT=pkg/{api,kubelet} # run tests for pkg/api and pkg/kubelet
```
### Run specific unit test cases in a package
You can set the test args using the `KUBE_TEST_ARGS` environment variable.
You can use this to pass the `-run` argument to `go test`, which accepts a
regular expression for the name of the test that should be run.
```sh
# Runs TestValidatePod in pkg/api/validation with the verbose flag set
make test WHAT=pkg/api/validation KUBE_GOFLAGS="-v" KUBE_TEST_ARGS='-run ^TestValidatePod$'
# Runs tests that match the regex ValidatePod|ValidateConfigMap in pkg/api/validation
make test WHAT=pkg/api/validation KUBE_GOFLAGS="-v" KUBE_TEST_ARGS="-run ValidatePod\|ValidateConfigMap$"
```
For other supported test flags, see the [golang
documentation](https://golang.org/cmd/go/#hdr-Description_of_testing_flags).
### Stress running unit tests
Running the same tests repeatedly is one way to root out flakes.
You can do this efficiently.
```sh
# Have 2 workers run all tests 5 times each (10 total iterations).
make test PARALLEL=2 ITERATION=5
```
For more advanced ideas please see [flaky-tests.md](flaky-tests.md).
### Unit test coverage
Currently, collecting coverage is only supported for the Go unit tests.
To run all unit tests and generate an HTML coverage report, run the following:
```sh
make test KUBE_COVER=y
```
At the end of the run, an HTML report will be generated with the path
printed to stdout.
To run tests and collect coverage in only one package, pass its relative path
under the `kubernetes` directory as an argument, for example:
```sh
make test WHAT=pkg/kubectl KUBE_COVER=y
```
Multiple arguments can be passed, in which case the coverage results will be
combined for all tests run.
### Benchmark unit tests
To run benchmark tests, you'll typically use something like:
```sh
go test ./pkg/apiserver -benchmem -run=XXX -bench=BenchmarkWatch
```
This will do the following:
1. `-run=XXX` is a regular expression filter on the name of test cases to run
2. `-bench=BenchmarkWatch` will run test methods with BenchmarkWatch in the name
* See `grep -nr BenchmarkWatch .` for examples
3. `-benchmem` enables memory allocation stats
See `go help test` and `go help testflag` for additional info.
## Integration tests
* Integration tests should only access other resources on the local machine
- Most commonly etcd or a service listening on localhost.
* All significant features require integration tests.
- This includes kubectl commands
* The preferred method of testing multiple scenarios or inputs
is [table driven testing](https://github.com/golang/go/wiki/TableDrivenTests)
- Example: [TestNamespaceAuthorization](../../test/integration/auth/auth_test.go)
* Each test should create its own master, httpserver and config.
- Example: [TestPodUpdateActiveDeadlineSeconds](../../test/integration/pods/pods_test.go)
* See [coding conventions](coding-conventions.md).
### Install etcd dependency
Kubernetes integration tests require your `PATH` to include an
[etcd](https://github.com/coreos/etcd/releases) installation. Kubernetes
includes a script to help install etcd on your machine.
```sh
# Install etcd and add to PATH
# Option a) install inside kubernetes root
hack/install-etcd.sh # Installs in ./third_party/etcd
echo export PATH="\$PATH:$(pwd)/third_party/etcd" >> ~/.profile # Add to PATH
# Option b) install manually
grep -E "image.*etcd" cluster/saltbase/etcd/etcd.manifest # Find version
# Install that version using yum/apt-get/etc
echo export PATH="\$PATH:<LOCATION>" >> ~/.profile # Add to PATH
```
### Etcd test data
Many tests start an etcd server internally, storing test data in the operating system's temporary directory.
If you see test failures because the temporary directory does not have sufficient space,
or is on a volume with unpredictable write latency, you can override the test data directory
for those internal etcd instances with the `TEST_ETCD_DIR` environment variable.
### Run integration tests
The integration tests are run using `make test-integration`.
The Kubernetes integration tests are writting using the normal golang testing
package but expect to have a running etcd instance to connect to. The `test-
integration.sh` script wraps `make test` and sets up an etcd instance
for the integration tests to use.
```sh
make test-integration # Run all integration tests.
```
This script runs the golang tests in package
[`test/integration`](../../test/integration/).
### Run a specific integration test
You can use also use the `KUBE_TEST_ARGS` environment variable with the `hack
/test-integration.sh` script to run a specific integration test case:
```sh
# Run integration test TestPodUpdateActiveDeadlineSeconds with the verbose flag set.
make test-integration KUBE_GOFLAGS="-v" KUBE_TEST_ARGS="-run ^TestPodUpdateActiveDeadlineSeconds$"
```
If you set `KUBE_TEST_ARGS`, the test case will be run with only the `v1` API
version and the watch cache test is skipped.
## End-to-End tests
Please refer to [End-to-End Testing in Kubernetes](e2e-tests.md).
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/devel/testing.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
docs/devel/update-release-docs.md
View file @ bc8f7e2c
# Table of Contents This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/devel/update-release-docs.md](https://github.com/kubernetes/community/blob/master/contributors/devel/update-release-docs.md)
<!-- BEGIN MUNGE: GENERATED_TOC -->
- [Table of Contents](#table-of-contents)
- [Overview](#overview)
- [Adding a new docs collection for a release](#adding-a-new-docs-collection-for-a-release)
- [Updating docs in an existing collection](#updating-docs-in-an-existing-collection)
- [Updating docs on HEAD](#updating-docs-on-head)
- [Updating docs in release branch](#updating-docs-in-release-branch)
- [Updating docs in gh-pages branch](#updating-docs-in-gh-pages-branch)
<!-- END MUNGE: GENERATED_TOC -->
# Overview
This document explains how to update kubernetes release docs hosted at http://kubernetes.io/docs/.
http://kubernetes.io is served using the [gh-pages
branch](https://github.com/kubernetes/kubernetes/tree/gh-pages) of kubernetes repo on github.
Updating docs in that branch will update http://kubernetes.io
There are 2 scenarios which require updating docs:
* Adding a new docs collection for a release.
* Updating docs in an existing collection.
# Adding a new docs collection for a release
Whenever a new release series (`release-X.Y`) is cut from `master`, we push the
corresponding set of docs to `http://kubernetes.io/vX.Y/docs`. The steps are as follows:
* Create a `_vX.Y` folder in `gh-pages` branch.
* Add `vX.Y` as a valid collection in [_config.yml](https://github.com/kubernetes/kubernetes/blob/gh-pages/_config.yml)
* Create a new `_includes/nav_vX.Y.html` file with the navigation menu. This can
be a copy of `_includes/nav_vX.Y-1.html` with links to new docs added and links
to deleted docs removed. Update [_layouts/docwithnav.html]
(https://github.com/kubernetes/kubernetes/blob/gh-pages/_layouts/docwithnav.html)
to include this new navigation html file. Example PR: [#16143](https://github.com/kubernetes/kubernetes/pull/16143).
* [Pull docs from release branch](#updating-docs-in-gh-pages-branch) in `_vX.Y`
folder.
Once these changes have been submitted, you should be able to reach the docs at
`http://kubernetes.io/vX.Y/docs/` where you can test them.
To make `X.Y` the default version of docs:
* Update [_config.yml](https://github.com/kubernetes/kubernetes/blob/gh-pages/_config.yml)
and [/kubernetes/kubernetes/blob/gh-pages/_docs/index.md](https://github.com/kubernetes/kubernetes/blob/gh-pages/_docs/index.md)
to point to the new version. Example PR: [#16416](https://github.com/kubernetes/kubernetes/pull/16416).
* Update [_includes/docversionselector.html](https://github.com/kubernetes/kubernetes/blob/gh-pages/_includes/docversionselector.html)
to make `vX.Y` the default version.
* Add "Disallow: /vX.Y-1/" to existing [robots.txt](https://github.com/kubernetes/kubernetes/blob/gh-pages/robots.txt)
file to hide old content from web crawlers and focus SEO on new docs. Example PR:
[#16388](https://github.com/kubernetes/kubernetes/pull/16388).
* Regenerate [sitemaps.xml](https://github.com/kubernetes/kubernetes/blob/gh-pages/sitemap.xml)
so that it now contains `vX.Y` links. Sitemap can be regenerated using
https://www.xml-sitemaps.com. Example PR: [#17126](https://github.com/kubernetes/kubernetes/pull/17126).
* Resubmit the updated sitemaps file to [Google
webmasters](https://www.google.com/webmasters/tools/sitemap-list?siteUrl=http://kubernetes.io/) for google to index the new links.
* Update [_layouts/docwithnav.html] (https://github.com/kubernetes/kubernetes/blob/gh-pages/_layouts/docwithnav.html)
to include [_includes/archivedocnotice.html](https://github.com/kubernetes/kubernetes/blob/gh-pages/_includes/archivedocnotice.html)
for `vX.Y-1` docs which need to be archived.
* Ping @thockin to update docs.k8s.io to redirect to `http://kubernetes.io/vX.Y/`. [#18788](https://github.com/kubernetes/kubernetes/issues/18788).
http://kubernetes.io/docs/ should now be redirecting to `http://kubernetes.io/vX.Y/`.
# Updating docs in an existing collection
The high level steps to update docs in an existing collection are:
1. Update docs on `HEAD` (master branch)
2. Cherryick the change in relevant release branch.
3. Update docs on `gh-pages`.
## Updating docs on HEAD
[Development guide](development.md) provides general instructions on how to contribute to kubernetes github repo.
[Docs how to guide](how-to-doc.md) provides conventions to follow while writing docs.
## Updating docs in release branch
Once docs have been updated in the master branch, the changes need to be
cherrypicked in the latest release branch.
[Cherrypick guide](cherry-picks.md) has more details on how to cherrypick your change.
## Updating docs in gh-pages branch
Once release branch has all the relevant changes, we can pull in the latest docs
in `gh-pages` branch.
Run the following 2 commands in `gh-pages` branch to update docs for release `X.Y`:
```
_tools/import_docs vX.Y _vX.Y release-X.Y release-X.Y
```
For ex: to pull in docs for release 1.1, run:
```
_tools/import_docs v1.1 _v1.1 release-1.1 release-1.1
```
Apart from copying over the docs, `_tools/release_docs` also does some post processing
(like updating the links to docs to point to http://kubernetes.io/docs/ instead of pointing to github repo).
Note that we always pull in the docs from release branch and not from master (pulling docs
from master requires some extra processing like versionizing the links and removing unversioned warnings).
We delete all existing docs before pulling in new ones to ensure that deleted
docs go away.
If the change added or deleted a doc, then update the corresponding `_includes/nav_vX.Y.html` file as well.
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/devel/update-release-docs.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
docs/devel/updating-docs-for-feature-changes.md
View file @ bc8f7e2c
# How to update docs for new kubernetes features This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/devel/updating-docs-for-feature-changes.md](https://github.com/kubernetes/community/blob/master/contributors/devel/updating-docs-for-feature-changes.md)
This document describes things to consider when updating Kubernetes docs for new features or changes to existing features (including removing features).
## Who should read this doc?
Anyone making user facing changes to kubernetes. This is especially important for Api changes or anything impacting the getting started experience.
## What docs changes are needed when adding or updating a feature in kubernetes?
### When making Api changes
*e.g. adding Deployments*
* Always make sure docs for downstream effects are updated *(StatefulSet -> PVC, Deployment -> ReplicationController)*
* Add or update the corresponding *[Glossary](http://kubernetes.io/docs/reference/)* item
* Verify the guides / walkthroughs do not require any changes:
* **If your change will be recommended over the approaches shown in these guides, then they must be updated to reflect your change**
* [Hello Node](http://kubernetes.io/docs/hellonode/)
* [K8s101](http://kubernetes.io/docs/user-guide/walkthrough/)
* [K8S201](http://kubernetes.io/docs/user-guide/walkthrough/k8s201/)
* [Guest-book](https://github.com/kubernetes/kubernetes/tree/release-1.2/examples/guestbook)
* [Thorough-walkthrough](http://kubernetes.io/docs/user-guide/)
* Verify the [landing page examples](http://kubernetes.io/docs/samples/) do not require any changes (those under "Recently updated samples")
* **If your change will be recommended over the approaches shown in the "Updated" examples, then they must be updated to reflect your change**
* If you are aware that your change will be recommended over the approaches shown in non-"Updated" examples, create an Issue
* Verify the collection of docs under the "Guides" section do not require updates (may need to use grep for this until are docs are more organized)
### When making Tools changes
*e.g. updating kube-dash or kubectl*
* If changing kubectl, verify the guides / walkthroughs do not require any changes:
* **If your change will be recommended over the approaches shown in these guides, then they must be updated to reflect your change**
* [Hello Node](http://kubernetes.io/docs/hellonode/)
* [K8s101](http://kubernetes.io/docs/user-guide/walkthrough/)
* [K8S201](http://kubernetes.io/docs/user-guide/walkthrough/k8s201/)
* [Guest-book](https://github.com/kubernetes/kubernetes/tree/release-1.2/examples/guestbook)
* [Thorough-walkthrough](http://kubernetes.io/docs/user-guide/)
* If updating an existing tool
* Search for any docs about the tool and update them
* If adding a new tool for end users
* Add a new page under [Guides](http://kubernetes.io/docs/)
* **If removing a tool (kube-ui), make sure documentation that references it is updated appropriately!**
### When making cluster setup changes
*e.g. adding Multi-AZ support*
* Update the relevant [Administering Clusters](http://kubernetes.io/docs/) pages
### When making Kubernetes binary changes
*e.g. adding a flag, changing Pod GC behavior, etc*
* Add or update a page under [Configuring Kubernetes](http://kubernetes.io/docs/)
## Where do the docs live?
1. Most external user facing docs live in the [kubernetes/docs](https://github.com/kubernetes/kubernetes.github.io) repo
* Also see the *[general instructions](http://kubernetes.io/editdocs/)* for making changes to the docs website
2. Internal design and development docs live in the [kubernetes/kubernetes](https://github.com/kubernetes/kubernetes) repo
## Who should help review docs changes?
* cc *@kubernetes/docs*
* Changes to [kubernetes/docs](https://github.com/kubernetes/kubernetes.github.io) repo must have both a Technical Review and a Docs Review
## Tips for writing new docs
* Try to keep new docs small and focused
* Document pre-requisites (if they exist)
* Document what concepts will be covered in the document
* Include screen shots or pictures in documents for GUIs
* *TODO once we have a standard widget set we are happy with* - include diagrams to help describe complex ideas (not required yet)
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/devel/updating-docs-for-feature-changes.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
docs/devel/writing-a-getting-started-guide.md
View file @ bc8f7e2c
# Writing a Getting Started Guide This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/devel/writing-a-getting-started-guide.md](https://github.com/kubernetes/community/blob/master/contributors/devel/writing-a-getting-started-guide.md)
This page gives some advice for anyone planning to write or update a Getting Started Guide for Kubernetes.
It also gives some guidelines which reviewers should follow when reviewing a pull request for a
guide.
A Getting Started Guide is instructions on how to create a Kubernetes cluster on top of a particular
type(s) of infrastructure. Infrastructure includes: the IaaS provider for VMs;
the node OS; inter-node networking; and node Configuration Management system.
A guide refers to scripts, Configuration Management files, and/or binary assets such as RPMs. We call
the combination of all these things needed to run on a particular type of infrastructure a
**distro**.
[The Matrix](../../docs/getting-started-guides/README.md) lists the distros. If there is already a guide
which is similar to the one you have planned, consider improving that one.
Distros fall into two categories:
- **versioned distros** are tested to work with a particular binary release of Kubernetes. These
come in a wide variety, reflecting a wide range of ideas and preferences in how to run a cluster.
- **development distros** are tested work with the latest Kubernetes source code. But, there are
relatively few of these and the bar is much higher for creating one. They must support
fully automated cluster creation, deletion, and upgrade.
There are different guidelines for each.
## Versioned Distro Guidelines
These guidelines say *what* to do. See the Rationale section for *why*.
- Send us a PR.
- Put the instructions in `docs/getting-started-guides/...`. Scripts go there too. This helps devs easily
search for uses of flags by guides.
- We may ask that you host binary assets or large amounts of code in our `contrib` directory or on your
own repo.
- Add or update a row in [The Matrix](../../docs/getting-started-guides/README.md).
- State the binary version of Kubernetes that you tested clearly in your Guide doc.
- Setup a cluster and run the [conformance tests](e2e-tests.md#conformance-tests) against it, and report the
results in your PR.
- Versioned distros should typically not modify or add code in `cluster/`. That is just scripts for developer
distros.
- When a new major or minor release of Kubernetes comes out, we may also release a new
conformance test, and require a new conformance test run to earn a conformance checkmark.
If you have a cluster partially working, but doing all the above steps seems like too much work,
we still want to hear from you. We suggest you write a blog post or a Gist, and we will link to it on our wiki page.
Just file an issue or chat us on [Slack](http://slack.kubernetes.io) and one of the committers will link to it from the wiki.
## Development Distro Guidelines
These guidelines say *what* to do. See the Rationale section for *why*.
- the main reason to add a new development distro is to support a new IaaS provider (VM and
network management). This means implementing a new `pkg/cloudprovider/providers/$IAAS_NAME`.
- Development distros should use Saltstack for Configuration Management.
- development distros need to support automated cluster creation, deletion, upgrading, etc.
This mean writing scripts in `cluster/$IAAS_NAME`.
- all commits to the tip of this repo need to not break any of the development distros
- the author of the change is responsible for making changes necessary on all the cloud-providers if the
change affects any of them, and reverting the change if it breaks any of the CIs.
- a development distro needs to have an organization which owns it. This organization needs to:
- Setting up and maintaining Continuous Integration that runs e2e frequently (multiple times per day) against the
Distro at head, and which notifies all devs of breakage.
- being reasonably available for questions and assisting with
refactoring and feature additions that affect code for their IaaS.
## Rationale
- We want people to create Kubernetes clusters with whatever IaaS, Node OS,
configuration management tools, and so on, which they are familiar with. The
guidelines for **versioned distros** are designed for flexibility.
- We want developers to be able to work without understanding all the permutations of
IaaS, NodeOS, and configuration management. The guidelines for **developer distros** are designed
for consistency.
- We want users to have a uniform experience with Kubernetes whenever they follow instructions anywhere
in our Github repository. So, we ask that versioned distros pass a **conformance test** to make sure
really work.
- We want to **limit the number of development distros** for several reasons. Developers should
only have to change a limited number of places to add a new feature. Also, since we will
gate commits on passing CI for all distros, and since end-to-end tests are typically somewhat
flaky, it would be highly likely for there to be false positives and CI backlogs with many CI pipelines.
- We do not require versioned distros to do **CI** for several reasons. It is a steep
learning curve to understand our automated testing scripts. And it is considerable effort
to fully automate setup and teardown of a cluster, which is needed for CI. And, not everyone
has the time and money to run CI. We do not want to
discourage people from writing and sharing guides because of this.
- Versioned distro authors are free to run their own CI and let us know if there is breakage, but we
will not include them as commit hooks -- there cannot be so many commit checks that it is impossible
to pass them all.
- We prefer a single Configuration Management tool for development distros. If there were more
than one, the core developers would have to learn multiple tools and update config in multiple
places. **Saltstack** happens to be the one we picked when we started the project. We
welcome versioned distros that use any tool; there are already examples of
CoreOS Fleet, Ansible, and others.
- You can still run code from head or your own branch
if you use another Configuration Management tool -- you just have to do some manual steps
during testing and deployment.
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/devel/writing-a-getting-started-guide.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment