Automatic merge from submit-queue

Node E2E: Remove fatal error in e2e_node_suite_test.go

Addresses https://github.com/kubernetes/kubernetes/issues/30779#issuecomment-240532190.

Currently we run node e2e test in parallel, and ginkgo makes sure that we only initialize test framework in the first test node.
However, because we throw out some fatal error during the initialization. Once there is an fatal error, the first test node will die immediately without reporting any error, and the other nodes will exit because the first node is gone with meaningless error.

If kubelet start fails, we'll get something like:
```
------------------------------
Failure [132.485 seconds]
[BeforeSuite] BeforeSuite 
/usr/local/google/home/lantaol/workspace/src/k8s.io/kubernetes/test/e2e_node/e2e_node_suite_test.go:138

  BeforeSuite on Node 1 failed

  /usr/local/google/home/lantaol/workspace/src/k8s.io/kubernetes/test/e2e_node/e2e_node_suite_test.go:138
------------------------------
......
------------------------------
Failure [132.465 seconds]
[BeforeSuite] BeforeSuite 
/usr/local/google/home/lantaol/workspace/src/k8s.io/kubernetes/test/e2e_node/e2e_node_suite_test.go:138

  BeforeSuite on Node 1 failed

  /usr/local/google/home/lantaol/workspace/src/k8s.io/kubernetes/test/e2e_node/e2e_node_suite_test.go:138
```

This PR replaces these fatal errors with gomega assertion, with this PR, we'll get:
```
Failure [132.482 seconds]
[BeforeSuite] BeforeSuite 
/usr/local/google/home/lantaol/workspace/src/k8s.io/kubernetes/test/e2e_node/e2e_node_suite_test.go:138

  should be able to start node services.
  Expected success, but got an error:
      <*errors.errorString | 0xc8203351b0>: {
          s: "failed to run server start command \"/tmp/ginkgo869068712/e2e_node.test --run-services-mode --server-start-timeout 2m0s --report-dir  --node-name lantaol0.mtv.corp.google.com --disable-kubenet=true --cgroups-per-qos=false --manifest-path /tmp/node-e2e-pod221291440 --eviction-hard memory.available<250Mi\": exit status 255",
      }
      failed to run server start command "/tmp/ginkgo869068712/e2e_node.test --run-services-mode --server-start-timeout 2m0s --report-dir  --node-name lantaol0.mtv.corp.google.com --disable-kubenet=true --cgroups-per-qos=false --manifest-path /tmp/node-e2e-pod221291440 --eviction-hard memory.available<250Mi": exit status 255

  /usr/local/google/home/lantaol/workspace/src/k8s.io/kubernetes/test/e2e_node/e2e_node_suite_test.go:117
------------------------------
Failure [132.485 seconds]
[BeforeSuite] BeforeSuite 
/usr/local/google/home/lantaol/workspace/src/k8s.io/kubernetes/test/e2e_node/e2e_node_suite_test.go:138

  BeforeSuite on Node 1 failed

  /usr/local/google/home/lantaol/workspace/src/k8s.io/kubernetes/test/e2e_node/e2e_node_suite_test.go:138
------------------------------
......
------------------------------
Failure [132.465 seconds]
[BeforeSuite] BeforeSuite 
/usr/local/google/home/lantaol/workspace/src/k8s.io/kubernetes/test/e2e_node/e2e_node_suite_test.go:138

  BeforeSuite on Node 1 failed

  /usr/local/google/home/lantaol/workspace/src/k8s.io/kubernetes/test/e2e_node/e2e_node_suite_test.go:138
```

This is much more informative.

/cc @kubernetes/sig-node

Automatic merge from submit-queue

Add Get() to cache listers

@kargakis @deads2k  PTAL

Automatic merge from submit-queue

Remove implicit Prometheus metrics from client

**What this PR does / why we need it**: This PR starts to cut away at dependencies that the client has.

**Release note**:
<!--  Steps to write your release note:
1. Use the release-note-* labels to set the release note state (if you have access) 
2. Enter your extended release note in the below block; leaving it blank means using the PR title as the release note. If no release note is required, just write `NONE`. 
-->
```release-note
The implicit registration of Prometheus metrics for request count and latency have been removed, and a plug-able interface was added. If you were using our client libraries in your own binaries and want these metrics, add the following to your imports in the main package: "k8s.io/pkg/client/metrics/prometheus". 
```

cc: @kubernetes/sig-api-machinery @kubernetes/sig-instrumentation @fgrzadkowski  @wojtek-t

Automatic merge from submit-queue

Don't bind pre-bound pvc & pv if size request not satisfied

as discussed briefly here https://github.com/kubernetes/kubernetes/pull/30522 , volume size ought to be verified before binding a pv & pvc regardless of what's in the pv's claimRef. @thockin

Automatic merge from submit-queue

Support for creation/removal of master replicas.

HA master: initial support for creation/removal of masters replicas by
kube-up/kube-down scripts for GCE on gci (other distributions, including debian, are not supported yet).

Automatic merge from submit-queue

Fix data race in PVC Run/Stop methods

Fixes #30947
Fixes #30775

Automatic merge from submit-queue

kubectl run add pull-policy flag to control image pull policy

```release-note
Add support for --image-pull-policy to 'kubectl run'
```

Fix #30493 
@pwittrock @thockin ptal

Automatic merge from submit-queue

Object meta fixes in federation controllers

This PR adds helper functions for handling object meta structs and applies them to the existing controllers.

cc: @quinton-hoole @jianhuiz @kubernetes/sig-cluster-federation

Automatic merge from submit-queue

x509 authenticator: get groups from subject's organization field

Note that the current X509 tests provide a bunch of certs but no private keys or commands to reproduce the testdata, so the new test case isn't added to the certificate chain.

Closes  #30260

cc @treed @gtank @mikedanese @deads2k  @kubernetes/sig-auth

Automatic merge from submit-queue

Node E2E: Wait for node ready before the node e2e test started.

Fixes https://github.com/kubernetes/kubernetes/issues/30252.

This PR makes node e2e test wait for exactly one node ready before running other test.

@ronnielai @mtaufen

Automatic merge from submit-queue

Kubelet: add --container-runtime-endpoint and --image-service-endpoint

Flag `--container-runtime-endpoint` (overrides `--container-runtime`) is introduced to identify the unix socket file of the remote runtime service. And flag `--image-service-endpoint` is introduced to identify the unix socket file of the image service.

This PR is part of #28789 Milestone 0. 

CC @yujuhong @Random-Liu

Automatic merge from submit-queue

Add initial support for TokenFile to to the client config file.

@smarterclayton @deads2k @cjcullen @krousey 

Part of the fix for #28260

Automatic merge from submit-queue

fix path handling in hack/lib/init.sh

Jenkinsfile pipeline jobs get cloned into "\<project\> (\<branch\>)". As a result, I can't use certain things in `hack/lib/init.sh`.

This is a small fix for that problem.

**Release note**:
<!--  Steps to write your release note:
1. Use the release-note-* labels to set the release note state (if you have access) 
2. Enter your extended release note in the below block; leaving it blank means using the PR title as the release note. If no release note is required, just write `NONE`. 
-->
```release-note
NONE
```

Automatic merge from submit-queue

pkg/genericapiserver/options: don't import pkg/apiserver

Refactor the authorization options for the API server so
pkg/apiserver isn't directly imported by the options package.

Closes #28544

cc @smarterclayton

@madhusudancs, @nikhiljindal I've updated `federation/cmd/federation-apiserver/app/server.go` to include the RBAC options with this change. I don't know if this was intentionally left out in the first place but would like your feedback.

Automatic merge from submit-queue

update kubectl help output for better organization

## Pull Request Guidelines

1. Please read our [contributor guidelines](https://github.com/kubernetes/kubernetes/blob/master/CONTRIBUTING.md).
1. See our [developer guide](https://github.com/kubernetes/kubernetes/blob/master/docs/devel/development.md).
1. Follow the instructions for [labeling and writing a release note for this PR](https://github.com/kubernetes/kubernetes/blob/master/docs/devel/pull-requests.md#release-notes) in the block below.

```release-note
* Use the release-note-* labels to set the release note state 
* Clear this block to use the PR title as the release note 
-OR-
* Enter your extended release note here
```


[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/.github/PULL_REQUEST_TEMPLATE.md?pixel)]()

Automatic merge from submit-queue

Add benchmark to jenkins

This PR contains the following changes:

1. Add more tests in density benchmark test;
2. Add the peak value (100%) in latency and CPU usage statistic data;
3. Move the Ginkgo focus flag from e2e_remote.go to run_e2e.go;
4. Support running benchmark in run_e2e.go. The benchmark configuration file is an extension of image configuration. Each item requires additional GCE machine type (e.g. n1-standard-1, default value will be used if empty) and test names (Ginkgo focus regex strings). A test item is regarded as benchmark if the tests field is non-empty.

Automatic merge from submit-queue

Node Conformance Test: Statically link apiserver

For #30122, #30174.
Based on #30116, #30198, #30200.

**Please only review the 4th PR.**

This PR is part of our roadmap to package node conformance test.
The 1st commit is from #30116, which started e2e services in a separate process.
The 2nd commit is from #30198, it statically linked etcd into the node e2e framework.
The 3rd commit is from #30200, it moved namespace controller into e2e services.

The 4th commit is new, it statically linked apiserver into the node e2e framework.

@dchen1107 @vishh 
/cc @kubernetes/sig-node @kubernetes/sig-testing

Automatic merge from submit-queue

Kubelet rkt CRI stubs & fakes

Adds stubs for CRI interfaces and creates a fake for `ContainerRuntime` interface

Automatic merge from submit-queue

daemonset controller should respect taints

cc @dchen1107 @davidopp

Automatic merge from submit-queue

Move init-container feature from alpha to beta.

```release-note
Moved init-container feature from alpha to beta.

In 1.3, an init container is specified with this annotation key
on the pod or pod template: `pods.alpha.kubernetes.io/init-containers`.

In 1.4, either that key or this key: pods.beta.kubernetes.io/init-containers`,
can be used.

When you GET an object, you will see both annotation keys with the same values.

You can safely roll back from 1.4 to 1.3, and things with init-containers
will still work (pods, deployments, etc).

If you are running 1.3, only use the alpha annotation, or it may be lost when
rolling forward.

The status has moved from annotation key
`pods.beta.kubernetes.io/init-container-statuses` to
`pods.beta.kubernetes.io/init-container-statuses`.
Any code that inspects this annotation should be changed to use the new key.

State of Initialization will continue to be reported in both pods.alpha.kubernetes.io/initialized
and in `podStatus.conditions.{status: "True", type: Initialized}`
```

Mini-design for this change:

Goals:
1. A user can create an object with the beta annotation
  on 1.4, and it works.  The fact that the annotation has beta
  in it communicates to the user that the feature is beta,
  and so the user should have confidence in using it.  Preferably,
  when the user gets the annotation back, he see the beta
  annotation.
1) If someone had an existing alpha object in their apiserver,
  such as a RS with a pod template with an init-containers
  annotation on it, it should continue to work (init containers
  run) when stack upgraded to 1.4.
2) If someone is using a chart or blog post that has alpha
  annotation on it and they create it on a 1.4 cluster, it should
  work.
3) If someone had something with an init container in 1.4
  and they roll back stack to 1.3, it should not silently stop
  working (init containers don't run anymore).

To meet all these, we mirror an absent beta label from the alpha
key and vice versa.  If they are out of sync, we use the alpha
one. We do this in conversion since there was already logic there.

In 1.3 code, all annotations are preserved across a round trip
(v1 -> api -> v1), and the alpha annotation turns into the internal
field that kubelet uses.

In 1.4 code, the alpha annotation is always preserved across
a round trip, and a beta annotation is always set equal to
the alpha one, after a round trip.

Currently, the kubelet always sees the object after a round trip
when it GETs it.  But, we don't want to rely on that behavior,
since it will break when fastpath is implemented.

So, we rely on this:
all objects either are created with an alpha annotation (1.3 or 1.4
code) or are created with a beta annotation under 1.4.  In the later
case, they are round tripped at creation time, and so get both
annotations.  So all subsequent GETs see both labels.

Automatic merge from submit-queue

godep: bump etcd to 3.0.6

What?
Bump etcd godep dependency to v3.0.6

Why?
ref: #30843, https://github.com/coreos/etcd/pull/6222 
We have some fix to do secure client connection in unit tests.

Automatic merge from submit-queue

Implement TLS bootstrap for kubelet using `--experimental-bootstrap-kubeconfig`  (2nd take)

Ref kubernetes/features#43 (comment)

cc @gtank @philips @mikedanese @aaronlevy @liggitt @deads2k @errordeveloper @justinsb 


Continue on the older PR https://github.com/kubernetes/kubernetes/pull/30094 as there are too many comments on that one and it's not loadable now.

Automatic merge from submit-queue

rkt: Support subPath volume mounts feature

So that at most one volume object will be created for every unique
host path. Also the volume's name is random generated UUID to avoid
collision since the mount point's name passed by kubelet is not
guaranteed to be unique when 'subpath' is specified.

Should partially fix https://github.com/kubernetes/kubernetes/issues/26986 
The non-existing host path creation issue is not touched here.

cc @kubernetes/sig-rktnetes 
also cc @kubernetes/sig-node for the Mount name comments I added.

Automatic merge from submit-queue

pkg/kubelet/rkt: skip empty lines in getOSReleaseInfo

Follow-up of #31022

Automatic merge from submit-queue

GlusterFS dynamic provisioner and deleter interface based on storageclass claims

This PR depends on PR#29006

Automatic merge from submit-queue

Return container command exit codes in kubectl run/exec

Fixes https://github.com/kubernetes/kubernetes/issues/26424
Based on https://github.com/kubernetes/kubernetes/pull/25273.

TODO:
- [x] add e2e tests
- [x] investigate `kubectl run` exit code for `--restart=Never` (compare issue #24533 and PR #25253)
- [x] document exit codes

Automatic merge from submit-queue

kubelet status manager: Fix nil in error message due to var shadowing

Variable shadowing can cause this log message to print a nil:

```go
glog.Warningf("Failed to update status for pod %q: %v", format.Pod(pod), err)
```

@kubernetes/rh-cluster-infra

Automatic merge from submit-queue

Load Balancer Health Check responder library for ESIPP

This is an independent component that is needed for the Load Balancer health traffic steering functionality (part of the 1.4 ESIPP work)

Automatic merge from submit-queue

Followup fixes for disruption controller.

Part of #12611.
- Record an event when a pod does not have exactly 1 controller.
- Add TODO comment suggesting we simplify the two cases: integer and percentage.

Automatic merge from submit-queue

ResourceNames() should prohibit unnamed arguments

Automatic merge from submit-queue

Add support for the standard objectmeta field selectors

certificates API was missing standard field selectors (noticed in https://github.com/kubernetes/kubernetes/pull/30922/files/4120179db5ce6c3fadc872a268347f5be038f98f#r75413160)

also silences client-side warnings when using field selectors that don't have a registered client-side transformation (no functional change, since we were already returning the original field/value). if we want to trend toward embedding less conversion logic in our clients, I don't see us fully duplicating field selector conversions client-side to make that warning disappear.

Automatic merge from submit-queue

Return the current kubeconfig as the starting config in ClientConfigGetter.

This fixes issue #30790.

cc @kubernetes/sig-cluster-federation