Automatic merge from submit-queue

Update New Relic sample docs with dynamic hostname info

This PR shows how to set extra environment variables for the New Relic DaemonSet that can be generated at runtime. I'm using this technique to prepend cluster names to host names as they are reported to New Relic.

Automatic merge from submit-queue (batch tested with PRs 39022, 39331, 39070, 39344)

Add a build rule for the boilerplate unit test.

We recently added unit tests that just run whenever you run `hack/verify_boilerplate.sh`, which really isn't the right time to do that. This adds a build rule instead.

Automatic merge from submit-queue (batch tested with PRs 39022, 39331, 39070, 39344)

remove unused parameter

Automatic merge from submit-queue (batch tested with PRs 39022, 39331, 39070, 39344)

Makefile speedup on generated code

This has been languishing in a  branch for a long time.  It makes the build more consistent wrt GOPATH (I still hope to enforce GOPATH at some point) and it removes a `go list` from each codegen.

I verified manually that the files that this emits as part of the make are only change in safe ways (ordering, _test files removed, etc).

Automatic merge from submit-queue

genericapiserver: extract CA cert from server cert and SNI cert chains

Without this PR a matching server cert or SNI cert is directly used as CA cert in the loopback client config. This fails if the cert is no CA cert.

With this PR the loopback client setup code walks through the chains of the server cert and the SNI certs to find a `CA:TRUE` cert. This is then used as the CA in the loopback client config.

Automatic merge from submit-queue (batch tested with PRs 39092, 39126, 37380, 37093, 39237)

Add unit test function to test DeletePod

Automatic merge from submit-queue (batch tested with PRs 39092, 39126, 37380, 37093, 39237)

Endpoints with TolerateUnready annotation, should list Pods in state terminating

**What this PR does / why we need it**:

We are using preStop lifecycle hooks to gracefully remove a node from a cluster. This hook is potentially long running and after the preStop hook is fired, the DNS resolution of the soon to be stopped Pod is failing, which causes a failure there.

**Special notes for your reviewer**:

Would be great to backport that to 1.4, 1.3 

**Release note**:

```release-note
Endpoints, that tolerate unready Pods, are now listing Pods in state Terminating as well
```

@bprashanth

Automatic merge from submit-queue (batch tested with PRs 39092, 39126, 37380, 37093, 39237)

Improve error reporting in Ceph RBD provisioner.

- We should report an error when user references a secret that cannot be found
- We should report output of rbd create/delete commands, logging "exit code 1"
  is not enough.


Before:
```
Events:
  FirstSeen     LastSeen        Count   From                            SubobjectPath   Type            Reason                  Message
  ---------     --------        -----   ----                            -------------   --------        ------                  -------
  33m           33m             1       {persistentvolume-controller }                  Warning         ProvisioningFailed      Failed to provision volume with StorageClass "cephrbdprovisioner": rbd: create volume failed, err: exit status 1
```

After:

```
Events:
  FirstSeen     LastSeen        Count   From                            SubobjectPath   Type            Reason                  Message
  ---------     --------        -----   ----                            -------------   --------        ------                  -------
  33m           33m             1       {persistentvolume-controller }                  Warning         ProvisioningFailed      Failed to provision volume with StorageClass "cephrbdprovisioner": failed to create rbd image: exit status 1, command output: rbd: couldn't connect to the cluster
```


@rootfs, PTAL

Automatic merge from submit-queue (batch tested with PRs 39092, 39126, 37380, 37093, 39237)

update prompt in create rolebinding/clusterrolebinding

Automatic merge from submit-queue

remove abac

Remove the abac authorizer as an authorizer for e2e.

Automatic merge from submit-queue

retry RBAC initialization for up to 30 seconds, kill server on failure

RBAC initialization needs to complete in order to bootstrap a cluster.  When the bootstrapping fails (etcd not ready has happened in e2e runs), things fail badly and we don't even kill the API server to force it to retry.  This retries for up to 30 seconds and kills the server if it never succeeds.

Fixes https://github.com/kubernetes/kubernetes/issues/39108

Automatic merge from submit-queue (batch tested with PRs 39075, 39350, 39353)

Move pkg/api.{Context,RequestContextMapper} into pkg/genericapiserver/api/request

**Based on #39350**

Automatic merge from submit-queue (batch tested with PRs 39075, 39350, 39353)

Move pkg/apiserver to pkg/genericapiserver/api

Automatic merge from submit-queue

no need to sleep for last retry

break for needless sleep

Automatic merge from submit-queue

Try parse golang logs by default

Glog by default logs to stderr, so Stackdriver Logging shows them all as errors. This PR makes fluentd try to parse messages using glog format and if succeeded, set timestamp and severity accordingly.

CC @piosz @fgrzadkowski

* Otherwise it prevents long running task in a preStop hook to succeed,
that require DNS resolution

Automatic merge from submit-queue

Ensure that assumed pod won't be expired until the end of binding

In case if api server is overloaded and will reply with 429 too many requests error, binding may take longer than ttl of scheduler cache for assumed pods https://github.com/kubernetes/kubernetes/blob/1199d42210c8ff6b232a80c79490c6a3c38a5de6/pkg/client/restclient/request.go#L787-L850 

This problem was mitigated by using this fix https://github.com/kubernetes/kubernetes/commit/e4d215d5084078c561659ab18ed6397e2e2f0d84 and increased rate limit for api server. But it is possible that it will occur again.

Automatic merge from submit-queue

fix swapped new/old job obj for validating object meta update

@nikhiljindal

In such cases when api server is overloaded and returns a lot of
429 (too many requests) errors - binding may take a lot of time
to succeed due to retry policy implemented in rest client.
In such events cache ttl for assumed pods wasn't big enough.

In order to minimize probability of such errors ttl for assumed pods
will be counted from the time when binding for particular pod is finished
(either with error or success)

Change-Id: Ib0122f8a76dc57c82f2c7c52497aad1bdd8be411

Automatic merge from submit-queue

DaemonSet ObservedGeneration

Extracting ObserverdGeneration part from #31693. It also implements #7328 for DaemonSets.

cc @kargakis

Automatic merge from submit-queue (batch tested with PRs 39351, 39322)

Remove the deprecated --reconcile-cidr flag from kubelet

**What this PR does / why we need it**:

Removes the `--reconcile-cidr` flag that has been deprecated since v1.5

**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #

**Special notes for your reviewer**:

**Release note**:

```release-note
The `--reconcile-cidr` kubelet flag was removed since it had been deprecated since v1.5
```
@thockin

Automatic merge from submit-queue

Update Google API dependencies

In order to experiment with alpha or beta GCP APIs (that are new), we need to make the client compatible with v1 client. And there were some cosmetic changes that make them incompatible with the version that we were using before (e.g. passing pointers vs non-pointers in gensupport functions).

Add unit test function to test DeletePod() of podManager

Automatic merge from submit-queue (batch tested with PRs 39150, 38615)

Add work queues to PV controller

PV controller should not use Controller.Requeue, as as it is not available in
shared informers. We need to implement our own work queues instead, where we
can enqueue volumes/claims as we want.

Automatic merge from submit-queue

kubectl: ignore deleted pods in the rolling updater

Fixes https://github.com/kubernetes/kubernetes/issues/37955

The Deployment controller already ignores deleted pods or more precisely the ReplicaSet controller ignores deleted pods ([pod classification](https://github.com/kubernetes/kubernetes/blob/a30b6e2d163ce5363daf1106350f46ea99982a64/pkg/controller/replicaset/replica_set.go#L600) ignores [deleted pods](https://github.com/kubernetes/kubernetes/blob/a30b6e2d163ce5363daf1106350f46ea99982a64/pkg/controller/controller_ref_manager.go#L63)) and the Deployment controller reuses what's in the ReplicaSet status.

@kubernetes/sig-apps-misc @kubernetes/sig-cli-misc

Automatic merge from submit-queue

Add Persistent Volume E2E in the context of a disrupted kubelet

This PR adds a test suite for persistent volumes affected by a disrupted kubelet.  Two cases are presented:

1.  A volume mounted via PVC remains accessible after a kubelet restart.
2.  When a pod is deleted while the kubelet is down, the mounted volume is unmounted successfully.

PV controller should not use Controller.Requeue, as as it is not available in
shared informers. We need to implement our own work queues instead where we
can enqueue volumes/claims as we want.

Automatic merge from submit-queue

Add default imagePullPolicy for initContainers.

fixes #38542

Merge pull request #36221 from pospispa/86-5-add-checks-and-documentation-about-template-pods-for-recycling

Automatic merge from submit-queue (batch tested with PRs 37959, 36221)

Recycle Pod Template Check

The kube-controller-manager has two command line arguments (--pv-recycler-pod-template-filepath-hostpath and --pv-recycler-pod-template-filepath-nfs) that specify a recycle pod template. The recycle pod template may not contain the volume that shall be recycled.

A check is added to make sure that the recycle pod template contains at least a volume.

cc: @jsafrane