Automatic merge from submit-queue (batch tested with PRs 41116, 41804, 42104, 42111, 42120)

Add support for attacher/detacher interface in Flex volume

Add support for attacher/detacher interface in Flex volume
This change breaks backward compatibility and requires to be release noted.

```release-note
Flex volume plugin is updated to support attach/detach interfaces. It broke backward compatibility. Please update your drivers and implement the new callouts. 
```

Automatic merge from submit-queue (batch tested with PRs 41116, 41804, 42104, 42111, 42120)

DaemonSet updates - take 2

It implements https://github.com/kubernetes/community/blob/master/contributors/design-proposals/daemonset-update.md
Feature kubernetes/features#124

```release-note
Implement the update feature for DaemonSet.
```

cc @kargakis @janetkuo @mikedanese

Automatic merge from submit-queue (batch tested with PRs 42058, 41160, 42065, 42076, 39338)

[Federation] Create configmap for the cluster kube-dns when cluster joins and remove when it unjoins

This PR implements the functionality as needed in https://github.com/kubernetes/kubernetes/issues/38400

cc @kubernetes/sig-cluster-federation @nikhiljindal @madhusudancs 

**Release note**:

```
kubefed join can now automatically create a configmap or add information to already existing one, storing this federation name vs zone name information in the joining cluster.
Further kubefed unjoin can remove this configmap or only this information from the configmap, if the unjoin cluster is registered with multiple federations.
The name of the configmap is kube-dns and the information in it is consumed by the in-cluster dns server.
```

Automatic merge from submit-queue (batch tested with PRs 42058, 41160, 42065, 42076, 39338)

add OWNER file to kubelet/network

Automatic merge from submit-queue (batch tested with PRs 42058, 41160, 42065, 42076, 39338)

Bump up dns-horizontal-autoscaler to 1.1.1

cluster-proportional-autoscaler 1.1.1 is releasing by kubernetes-incubator/cluster-proportional-autoscaler#26, also bump it up for dns-horizontal-autoscaler to introduce below features:
- Add PreventSinglePointFailure option in linear mode.
- Use protobufs for communication with apiserver.
- Support switching control mode on-the-fly.

Note:
The new entry `"preventSinglePointFailure":true` ensures kube-dns to have at least 2 replicas when there is more than one node. Mitigate the issue mentioned in #40063.

@bowei @thockin 

**Release note**:

```release-note
NONE
```

Automatic merge from submit-queue (batch tested with PRs 42058, 41160, 42065, 42076, 39338)

New command for stand-alone GKE certificates controller

New stand-alone certificates controller for GKE. Rather than requiring the CA's private key on disk, this allows making external calls to GKE in order to sign cluster certificates.
    
**Which issue this PR fixes**: fixes #39761

**Release note**:

```release-note
New GKE certificates controller.
```

CC @mikedanese @jcbsmpsn

Automatic merge from submit-queue (batch tested with PRs 42058, 41160, 42065, 42076, 39338)

Juju: Fix shebangs in charm actions to use python3

**What this PR does / why we need it**:

This fixes the microbot and create-rbd-pv actions by reverting them back to python3. We accidentally switched them to python2 to match the boilerplate checker's expectations for python files.

It looks like hack/verify-boilerplate.sh does not check these since they don't have the .py extension, so we should be good with no changes there.

**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*:

https://github.com/juju-solutions/bundle-canonical-kubernetes/issues/212

**Special notes for your reviewer**:

**Release note**:

```release-note
Juju: Fix shebangs in charm actions to use python3
```

Automatic merge from submit-queue (batch tested with PRs 41962, 42055, 42062, 42019, 42054)

Update flag to --check-version-skew instead of --check_version_skew

https://github.com/kubernetes/test-infra/issues/2012

Also add a `--` to send the flags to kubetest without triggering a warning.

Automatic merge from submit-queue (batch tested with PRs 41962, 42055, 42062, 42019, 42054)

PV upgrade test

**What this PR does / why we need it**:
This PR adds a PV upgrade test to the new upgrade test framework.  Before, this test had to be done manually.  Currently the upgrade test framework only works on the GCE environment, so I plan to add support for other providers later.  In order to write the test, I had to modify and refactor some volume test util libraries.  I reran the impacted tests to make sure they still passed.

**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #

**Special notes for your reviewer**:
It's probably easier to review the two commits separately.  I split it up into the refactor changes, and the upgrade test changes.

**Release note**:

NONE


cc @saad-ali @krousey

Automatic merge from submit-queue (batch tested with PRs 41962, 42055, 42062, 42019, 42054)

Don't restrict etcd on self host installs b/c a clipped etcd can have

**What this PR does / why we need it**:
Remove cpu-clipping from self hosted etcd install from the masters.

**Special notes for your reviewer**:

**Release note**:

```
NONE
```

Automatic merge from submit-queue (batch tested with PRs 41962, 42055, 42062, 42019, 42054)

dockershim puts pause container in pod cgroup

**What this PR does / why we need it**:
The CRI was not launching the pause container in the pod level cgroup.  The non-CRI code path was.

Automatic merge from submit-queue (batch tested with PRs 41962, 42055, 42062, 42019, 42054)

add kubeadm completion command

reference to `kubectl completion`

```
#kubeadm <tab><tab>
alpha       completion  init        join        reset       version
```

It implements https://github.com/kubernetes/community/blob/master/contributors/design-proposals/daemonset-update.md
Feature https://github.com/kubernetes/features/issues/124

Automatic merge from submit-queue (batch tested with PRs 42044, 41694, 41927, 42050, 41987)

Simplify and fix hack/{verify,update}-staging-{client-go,godeps}.sh

- merge `hack/{verify,update}-staging-client-go.sh`
- pin godep with shared code
- remove godep-restore completely from the process and replace with a simple check that godeps are restored
- add safety check in `staging/copy.sh` that there is no lingering `k8s.io/apimachinery` in the GOPATH which would lead to inconsistent client-go builds (!)
- check that all these scripts only operate in a clean working dir.

Automatic merge from submit-queue (batch tested with PRs 42044, 41694, 41927, 42050, 41987)

Use existing nginx image in the node

**What this PR does / why we need it**:
 Fixes a test flake: `[k8s.io] Garbage collector should orphan pods created by rc if delete options say so`

**Which issue this PR fixes** 

fixes #35771

**Special notes for your reviewer**:

**Release note**:
```release-note NONE
```

Automatic merge from submit-queue (batch tested with PRs 42044, 41694, 41927, 42050, 41987)

federation: use generated listers

**What this PR does / why we need it**: switch federation code to use generated listers

**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #

**Special notes for your reviewer**:

**Release note**:

```release-note
```

cc @smarterclayton @derekwaynecarr @marun @gmarek @wojtek-t @deads2k @liggitt @sttts @timothysc @nikhiljindal @jianhuiz @kubernetes/sig-federation-pr-reviews @kubernetes/sig-scalability-pr-reviews

Automatic merge from submit-queue (batch tested with PRs 42044, 41694, 41927, 42050, 41987)

Add apply set-last-applied subcommand 

implement part of https://github.com/kubernetes/community/pull/287, will rebase after https://github.com/kubernetes/kubernetes/pull/41699 got merged, EDIT: since bug output format has been confirmed, will update the behavior of output format soon  
cc @kubernetes/sig-cli-pr-reviews @AdoHe @pwittrock 

```release-note
Support kubectl apply set-last-applied command to update the applied-applied-configuration annotation
```

Automatic merge from submit-queue

daemon not deamon

Automatic merge from submit-queue

Use docker log rotation mechanism instead of logrotate

This is a solution for https://github.com/kubernetes/kubernetes/issues/38495.

Instead of rotating logs using logrotate tool, which is configured quite rigidly, this PR makes docker responsible for the rotation and makes it possible to configure docker logging parameters. It solves the following problems:

* Logging agent will stop loosing lines upon rotation
* Container's logs size will be more strictly constrained. Instead of checking the size hourly, size will be checked upon write, preventing https://github.com/kubernetes/kubernetes/issues/27754

It's still far from ideal, for example setting logging options per pod, as suggested in https://github.com/kubernetes/kubernetes/issues/15478 would be much more flexible, but latter approach requires deep changes, including changes in API, which may be in vain because of CRI and long-term vision for logging.

Changes include:

* Change in salt. It's possible to configure docker log parameters, using variables in pillar. They're exported from env variables on `gce`, but for different cloud provider they have to be exported first.
* Change in `configure-helper.sh` scripts for those os on `gce` that don't use salt + default values exposed via env variables

This change may be problematic for kubelet logs functionality with CRI enabled, that will be tackled in the follow-up PR, if confirmed.

CC @piosz @Random-Liu @yujuhong @dashpole @dchen1107 @vishh @kubernetes/sig-node-pr-reviews

```release-note
On GCI by default logrotate is disabled for application containers in favor of rotation mechanism provided by docker logging driver.
```

Automatic merge from submit-queue

Cleanup fluentd-gcp image, rebase on debian-base

**Why we need this PR**:

There are several problems with our current fluentd-gcp image:

- It pulls in lots of unused packages, which expose unnecessary risk and create noise in CVE scans (and scare customers). The most notable example is the fluent-ui, which pulls in rails.
- `curl | sh ` is not a good practice for a Dockerfile. First, the script is not checked in the same source control branch, so builds are not reproducible. Second, the actions it is taking are opaque. Third, in this case, using non-standard packages means they're harder to manage with CVE scans & upstream fixes.

**What is changed by this PR?**

- Rather than relying on td-agent (which includes fluent-ui), use standard upstream packages. This is largely based off the [official fluentd debian-based image](https://github.com/fluent/fluentd-docker-image/blob/master/v0.12/debian/Dockerfile).
- Rebases the image on debian-base (depends on https://github.com/kubernetes/kubernetes/pull/41915). We would like to move towards a single full-distro base image we can maintain. This change should be relatively minor.

As a result of these changes, the image size is reduced from 360.6 MB to 185.8 MB (nearly half). Many packages were removed, and the full diff (focus on the unversioned files) is listed here: https://github.com/timstclair/kubernetes/commit/3fb704f977875bb4cbb64687b2f6836fbe101cf6

**Which issue this PR fixes** https://github.com/kubernetes/kubernetes/issues/40248

**Special notes for your reviewer**:

This change both addresses security concerns, and is expected to greatly reduce the maintenance burden of the fluentd-gcp image. I'd *really* like to get this into 1.6, so please prioritize this review if possible.

I tested this by running the default e2e suite on a private e2e cluster using the new image. If there are other tests you'd like me to run, please let me know ASAP.

**Release note**:
```release-note
Cleanup fluentd-gcp image: rebase on debian-base, switch to upstream packages, remove fluent-ui & rails
```

Automatic merge from submit-queue (batch tested with PRs 35408, 41915, 41992, 41964, 41925)

bump version numbers for heapster/influxdb/grafana images

**What this PR does / why we need it**:
It updates version of monitoring components (heapster/influxdb/grafana) to the latest one used by heapster

**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #

**Special notes for your reviewer**:
[e2e/monitoring.go](https://github.com/kubernetes/kubernetes/blob/master/test/e2e/monitoring.go) test seems to be passing without modifications

**Release note**:

```release-note
```

Automatic merge from submit-queue (batch tested with PRs 35408, 41915, 41992, 41964, 41925)

e2e/upgrade: add sysctls

Add sysctl upgrade tests.

How can these be effectively tested?

Automatic merge from submit-queue (batch tested with PRs 35408, 41915, 41992, 41964, 41925)

azure: document config file (+ remove unused field)

**What this PR does / why we need it**:
* documents the config file used by the Azure cloudprovider
* removes an unused field that shouldn't have been added

```release-note
NONE
```

Automatic merge from submit-queue (batch tested with PRs 35408, 41915, 41992, 41964, 41925)

Standard Debian base image

**What this PR does / why we need it**:

The goal of this new image is to provide a standard base image for Kubernetes system images that require substantial external dependencies (e.g. kube-proxy and fluentd). The image is significantly reduced from the standard `debian:jessie-slim` image (40 MB vs 80 MB), and removes a number of unnecessary dependencies such as e2fsprogs, systemd, and sysv-rc. In the future we may consider further reducing it to the bare minimum to run the package manager, with the requirement that images based on it add all the dependencies they need.

I tested this by rebasing kube-proxy on this image and running the e2e tests. I'm targeting 1.6 for rebasing kube-proxy & fluentd on this.

For the rational behind basing on Debian, see https://github.com/kubernetes/kubernetes/issues/40248#issuecomment-280781931

Largely based off [debian-iptables](https://github.com/kubernetes/kubernetes/tree/master/build/debian-iptables/) and [ubuntu-slim](https://github.com/kubernetes/ingress/tree/master/images/ubuntu-slim).

**Which issue this PR fixes**
https://github.com/kubernetes/kubernetes/issues/40248

**Special notes for your reviewer**:
@luxas Please review the qemu cross-build logic in the Makefile. It's copied from [debian-iptables](https://github.com/kubernetes/kubernetes/blob/master/build/debian-iptables/Makefile), but I'm not sure exactly what it's doing.

/cc @jessfraz @dlorenc

Automatic merge from submit-queue (batch tested with PRs 35408, 41915, 41992, 41964, 41925)

add secret option to flag

To resolve the issue of security(pr #35030 ),

> @smarterclayton commented 5 days ago
> This is unfortunately not all flags that could be secrets. The best option would be to add support in spf13/pflag to tag a flag as a secret, and then use that bit to determine the list.
> 
> Also, Command() could be used in contexts that need exact parameters (for subshell execution), so we would need to add a new method or extend the signature here to allow exact flags to be retrieved.

we could add a secret option to the flags.

Automatic merge from submit-queue

make iscsi portals optional

**What this PR does / why we need it**: Make iSCSI portals optional

**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #

**Special notes for your reviewer**:

**Release note**:

```release-note
NONE
```

Automatic merge from submit-queue (batch tested with PRs 41954, 40528, 41875, 41165, 41877)

preflight check external etcd version when kubeadm init

**What this PR does / why we need it**:
1. preflight check if verson of external etcd server meets the demand of kubeadm, currently requires >= 3.0.14
2. support mixed http endpoints and https endpoints

**Which issue this PR fixes** : fixes  https://github.com/kubernetes/kubeadm/issues/174

**Special notes for your reviewer**:
i have tested against single endpoint including http etcd server , https etcd server,  but multiple endpoints not tested yet. i'll do it tomorrow

Automatic merge from submit-queue (batch tested with PRs 41954, 40528, 41875, 41165, 41877)

Updating apiserver to return 202 when resource is being deleted asynchronously via cascading deletion

As per https://github.com/kubernetes/kubernetes/issues/33196#issuecomment-278440622.

cc @kubernetes/sig-api-machinery-pr-reviews @smarterclayton @caesarxuchao @bgrant0607 @kubernetes/api-reviewers 

```release-note
Updating apiserver to return http status code 202 for a delete request when the resource is not immediately deleted because of user requesting cascading deletion using DeleteOptions.OrphanDependents=false.
```

Automatic merge from submit-queue (batch tested with PRs 41954, 40528, 41875, 41165, 41877)

Move the scheduler Fake* to testing folder

Address this issue: https://github.com/kubernetes/kubernetes/issues/41662
@timothysc I moved the interface to the types.go, and the Fake* to the testing package, not sure whether you like or not.

Automatic merge from submit-queue (batch tested with PRs 41954, 40528, 41875, 41165, 41877)

[Federation][kubefed] Support configuring dns-provider

**What this PR does / why we need it**:
Some environments might need to configure the dns-provider using custom configurations for deploying federation control plane. This PR will facilitate such scenarios. please refer to #40620 

Now we can pass dns provider configuration using `dns-provider-config` flag to `kubefed init`

**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #40620

**Release note**:

```
[Federation] Introduced a new flag --dns-provider-config to kubefed to configure dns provider via a config file in local file system.
```

cc @kubernetes/sig-federation-misc @madhusudancs @irfanurrehman @marun

Automatic merge from submit-queue

bump(github.com/onsi/gingko):bb93381d543b0e5725244abe752214a110791d01

Adds support for extracting stdout and stderr from failed tests runs
into junit, which makes junit more effective for debugging failures.

After this I'll submit a change to gubernator to show the stdout/stderr

Automatic merge from submit-queue (batch tested with PRs 41701, 41818, 41897, 41119, 41562)

Updated configmap_controller_test.go

**What this PR does / why we need it**: A few style decisions to make this test a bit harder to break. Including a fix to help it fail and not panic when channel produces a nil.

**Release note**:

```
NONE
```

Automatic merge from submit-queue (batch tested with PRs 41701, 41818, 41897, 41119, 41562)

Optimization of on-wire information sent to scheduler extender interfaces that are stateful

The changes are to address the issue described in https://github.com/kubernetes/kubernetes/issues/40991

```release-note
Added support to minimize sending verbose node information to scheduler extender by sending only node names and expecting extenders to cache the rest of the node information
```

cc @ravigadde

Automatic merge from submit-queue (batch tested with PRs 41701, 41818, 41897, 41119, 41562)

kubeadm: Secure the control plane communication and add the kubeconfig phase command

**What this PR does / why we need it**:

This generates kubeconfig files for the controller-manager and the scheduler, ref: https://github.com/kubernetes/kubeadm/issues/172

The second commit adds the `kubeadm alpha phase kubeconfig` command as described in the design doc: https://github.com/kubernetes/kubeadm/pull/156

**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #

**Special notes for your reviewer**:

@dmmcquay What kind of tests would you like for the kubeconfig phase command?

**Release note**:

```release-note
```
@jbeda @mikedanese @dmmcquay @pires @liggitt @deads2k @errordeveloper

Automatic merge from submit-queue (batch tested with PRs 41701, 41818, 41897, 41119, 41562)

Allow updates to pod tolerations.

Opening this PR to continue discussion for pod spec tolerations updates when a pod has been scheduled already. This PR is built on top of https://github.com/kubernetes/kubernetes/pull/38957.

@kubernetes/sig-scheduling-pr-reviews @liggitt @davidopp @derekwaynecarr @kubernetes/rh-cluster-infra