Automatic merge from submit-queue

Adding vishh to cluster/gce owners

Automatic merge from submit-queue

Re-writing of the resolv.conf file generated by docker

Fixes #17406 

Docker 1.12 will contain feature "The option --dns and --net=host should not be mutually exclusive" (docker/docker#22408)
This patch adds optional support for this ability in kubelet (for now in case of "hostNetwork: true" set all dns settings are ignored if any).
To enable feature use newly added kubelet flag: --allow-dns-for-hostnet=true

Automatic merge from submit-queue

Adding OWNERS file for test/integration/federation

Ref https://github.com/kubernetes/kubernetes/issues/40705

File copied from federation/OWNERS.

Existing pending PRs for this diectory:
https://github.com/kubernetes/kubernetes/pull/42278 and https://github.com/kubernetes/kubernetes/pull/42225

cc @kubernetes/sig-federation-pr-reviews

Automatic merge from submit-queue

HPA Controller: Use Custom Metrics API

This commit switches over the HPA controller to use the custom metrics
API.  It also converts the HPA controller to use the generated client
in k8s.io/metrics for the resource metrics API.

In order to enable support, you must enable
`--horizontal-pod-autoscaler-use-rest-clients` on the
controller-manager, which will switch the HPA controller's MetricsClient
implementation over to use the standard rest clients for both custom
metrics and resource metrics.  This requires that at the least resource
metrics API is registered with kube-aggregator, and that the controller
manager is pointed at kube-aggregator.  For this to work, Heapster
must be serving the new-style API server (`--api-server=true`).

Before this merges, this will need kubernetes/metrics#2 to merge, and a godeps update to pull that in.
It's also semi-dependent on kubernetes/heapster#1537, but that is not required in order for this to merge.

**Release note**:
```release-note
Allow the Horizontal Pod Autoscaler controller to talk to the metrics API and custom metrics API as standard APIs.
```

Automatic merge from submit-queue

statefulset: wait for pvc cache sync

#42056 switched the statefulset controller to use the pvc shared informer/lister, but accidentally left out waiting for its cache to sync.

cc @kubernetes/sig-apps-pr-reviews @kargakis @foxish @kow3ns @smarterclayton @deads2k

Automatic merge from submit-queue

fix rsListerSynced and podListerSynced for DeploymentController

**What this PR does / why we need it**:

There is a mistake when initializing `DeploymentController`'s `rsListerSynced` and `podListerSynced` in `NewDeploymentController`, they are all initialized to `Deployment`'s `Informer`, so the `DeploymentController` maybe running before the `ReplicaSet` cache and `Pod` cache has been synced.

**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #

**Special notes for your reviewer**:

Indeed according unit test is neccessary, but this bug fix is simple, and if the tests is neccessary I will submit another PR later.

**Release note**:

```release-note
```

Automatic merge from submit-queue

Implement support for mount options in PVs

**What this PR does / why we need it**:

This PR implements support for mount options in PersistentVolume via `volume.beta.kubernetes.io/mount-options` annotation.

**Which issue this PR fixes** 

Fixes https://github.com/kubernetes/features/issues/168

**Release note**:
```
Enable additional, custom mount options to be passed to PersistentVolume objects via volume.beta.kubernetes.io/mount-options annotation.
```

Automatic merge from submit-queue

show patch error as string not byte slice

Debugging for https://github.com/kubernetes/kubernetes/issues/39471 shows

```
client_test.go:309: Failed updating patchpod with patch type application/json-patch+json: Operation cannot be fulfilled on pods "patchpod": there is a meaningful conflict:
	 diff1=[123 34 109 101 116 97 100 97 116 97 34 58 123 34 108 97 98 101 108 115 34 58 123 34 102 111 111 34 58 110 117 108 108 125 44 34 114 101 115 111 117 114 99 101 86 101 114 115 105 111 110 34 58 34 49 52 50 50 34 125 125]
	, diff2=[123 34 109 101 116 97 100 97 116 97 34 58 123 34 108 97 98 101 108 115 34 58 110 117 108 108 125 125]
```

I don't speak byte slice, so this makes it a string for next time.

@sttts

Automatic merge from submit-queue

Extensible Userspace Proxy

This PR refactors the userspace proxy to allow for custom proxy socket implementations.
It changes the the ProxySocket interface to ensure that other packages can properly implement it (making sure all arguments are publicly exposed types, etc), and adds in a mechanism for an implementation to create an instance of the userspace proxy with a non-standard ProxySocket.
Custom ProxySockets are useful to inject additional logic into the actual proxying.  For example, our idling proxier uses a custom proxy socket to hold connections and notify the cluster that idled scalable resources need to be woken up.

Also-Authored-By: Ben Bennett bbennett@redhat.com

Introduced chages:
   1. Re-writing of the resolv.conf file generated by docker.
      Cluster dns settings aren't passed anymore to docker api in all cases, not only for pods with host network:
      the resolver conf will be overwritten after infra-container creation to override docker's behaviour.

   2. Added new one dnsPolicy - 'ClusterFirstWithHostNet', so now there are:
      - ClusterFirstWithHostNet - use dns settings in all cases, i.e. with hostNet=true as well
      - ClusterFirst - use dns settings unless hostNetwork is true
      - Default

Fixes #17406

Add support for mount options via annotations on PVs

Automatic merge from submit-queue

Add apps/v1beta1 deployments with new defaults

This pull introduces deployments under `apps/v1beta1` and fixes #23597 and #23304.

TODO:

* [x] - create new type `apps/v1beta1.Deployment`
* [x] - update kubectl (stop, scale)
* [ ] - ~~new `kubectl run` generator~~ - this will only duplicate half of generator code, I suggest replacing current to use new endpoint
* [ ] - ~~create extended tests~~ - I've added integration and cmd tests verifying new endpoints
* [ ] - ~~create `hack/test-update-storage-objects.sh`~~ - see above

This is currently blocked by https://github.com/kubernetes/kubernetes/pull/38071, due to conflicting name `v1beta1.Deployment`. 

```release-note
Introduce apps/v1beta1.Deployments resource with modified defaults compared to extensions/v1beta1.Deployments.
```

@kargakis @mfojtik @kubernetes/sig-apps-misc

Automatic merge from submit-queue (batch tested with PRs 42316, 41618, 42201, 42113, 42191)

Make 'docker-email' optional on dockercfg secrets

It is not required for most username/password registries.

Fixes #41727

Automatic merge from submit-queue (batch tested with PRs 42316, 41618, 42201, 42113, 42191)

Fixing a couple of things in the example, and bumping docker version

**What this PR does / why we need it**:

Updates the Cassandra version, and fixes a couple of issues in the example documentation.

fixes #40435

Automatic merge from submit-queue (batch tested with PRs 42316, 41618, 42201, 42113, 42191)

[Kubemark] Add init container in hollow node for setting inotify limit of node to 200

Fixes #41713 

Along with adding the init container, I also changed the manifest to a yaml as otherwise the entire init container annotation would have to be in a single line (with escaped characters), as json doesn't allow multi-line strings.

cc @kubernetes/sig-scalability-misc @wojtek-t @gmarek @Random-Liu

Automatic merge from submit-queue (batch tested with PRs 42316, 41618, 42201, 42113, 42191)

Support unqualified and partially qualified domain name in DNS query in Windows kube-proxy

**What this PR does / why we need it**:
In Windows container networking, --dns-search is not currently supported on Windows Docker. Besides, even with --dns-suffix, inside Windows container DNS suffix is not appended to DNS query names. That makes unqualified domain name or partially qualified domain name in DNS query not able to resolve.

This PR provides a solution to resolve unqualified domain name or partially qualified domain name in DNS query for Windows container in Windows kube-proxy. It uses well-known Kubernetes DNS suffix as well host DNS suffix search list to append to the name in DNS query. DNS packet in kube-proxy UDP stream is modified as appropriate.

This PR affects the Windows kube-proxy only.

**Special notes for your reviewer**:
This PR is based on top of Anthony Howe's commit 48647fb9, 0e37f0a8 and 7e2c71f6 which is already included in the PR 41487. Please only review commit b9dfb69d.

**Release note**:

```release-note
Add DNS suffix search list support in Windows kube-proxy.
```

This commit adds the staging repos to the GOPATH in
hack/verify-godeps.sh.  This allows vendored librarys
to depend on staging repos and not break verify-godeps.

This also adds `hack/godep-restore.sh`, which acts like
`godep restore`, but sets the GOPATH appropriate as well.

This commit introduces a dependency on k8s.io/metrics, for
use with the horizontal pod autoscaler.

This commit switches over the HPA controller to use the custom metrics
API.  It also converts the HPA controller to use the generated client
in k8s.io/metrics for the resource metrics API.

In order to enable support, you must enable
`--horizontal-pod-autoscaler-use-rest-clients` on the
controller-manager, which will switch the HPA controller's MetricsClient
implementation over to use the standard rest clients for both custom
metrics and resource metrics.  This requires that at the least resource
metrics API is registered with kube-aggregator, and that the controller
manager is pointed at kube-aggregator.  For this to work, Heapster
must be serving the new-style API server (`--api-server=true`).

Automatic merge from submit-queue

This PR adds a new environmental variable ENABLE_CRI for customizing CRI

**What this PR does / why we need it**:

This PR adds a new environmental variable `ENABLE_CRI` for customizing CRI (e.g. switching between dockershim and dockertools) and sets `--enable-cri=true` by default.

**Which issue this PR fixes** 

Fixes #42315.

**Release note**:

```release-note
NONE
```

Maybe we should also cherry-pick this to 1.6 branch.

cc @yujuhong @Random-Liu

Automatic merge from submit-queue

Fix update stage

This PR addressed following problems:

1. Removes comments from all staging godeps and makes sure that they are not placed there. This saves false positive during update when `Comment` field is cut by one digit.
2. Moves updating staging to a separate script (`hack/update-all-staging.sh`), `hack/update-all.sh` only runs verification of staging deps.  

@sttts ptal

Automatic merge from submit-queue

Prevent caching empty discovery docs

If kubectl makes requests before a server fully initialized, it could cache bad discovery responses (results of a partially forbidden discovery call, etc)

This changes to only cache responses that included at least one group or resource so that total failure doesn't get cached for 5 minutes

Automatic merge from submit-queue

Extend experimental support to multiple Nvidia GPUs

Extended from #28216

```release-note
`--experimental-nvidia-gpus` flag is **replaced** by `Accelerators` alpha feature gate along with  support for multiple Nvidia GPUs. 
To use GPUs, pass `Accelerators=true` as part of `--feature-gates` flag.
Works only with Docker runtime.
```

1. Automated testing for this PR is not possible since creation of clusters with GPUs isn't supported yet in GCP.
1. To test this PR locally, use the node e2e.
```shell
TEST_ARGS='--feature-gates=DynamicKubeletConfig=true' FOCUS=GPU SKIP="" make test-e2e-node
```

TODO:

- [x] Run manual tests
- [x] Add node e2e
- [x] Add unit tests for GPU manager (< 100% coverage)
- [ ] Add unit tests in kubelet package

Automatic merge from submit-queue (batch tested with PRs 41921, 41695, 42139, 42090, 41949)

Collect npd log in cluster e2e test.

Collect node problem detector log in cluster e2e.

@dchen1107 @kubernetes/node-problem-detector-reviewers

Automatic merge from submit-queue (batch tested with PRs 41921, 41695, 42139, 42090, 41949)

Rebase kube-proxy and debian-iptables on debian-base

**What this PR does / why we need it**:

Slimmer images are generally preferred, but it's a minor optimization. The larger advantage to this change is the reduced attack surface from removing unnecessary packages, and easier maintenance from sharing a common base image.

Size comparison:
```
gcr.io/google-containers/debian-iptables-amd64:v6         127.9 MB
gcr.io/google-containers/debian-iptables-amd64:v7          45.1 MB
```

**Which issue this PR fixes** https://github.com/kubernetes/kubernetes/issues/40248

**Special notes for your reviewer**:
Tested by deploying to a private test cluster and running the e2es. This will fail the jenkins builds until I push the `gcr.io/google-containers/debian-iptables-amd64:v7` image, which I will do once I have an LGTM.

**Release note**:
```release-note
Clean up the kube-proxy container image by removing unnecessary packages and files.
```

/cc @luxas @ixdy

Automatic merge from submit-queue (batch tested with PRs 41921, 41695, 42139, 42090, 41949)

Unify fake runtime helper in kuberuntime, rkt and dockertools.

Addresses https://github.com/kubernetes/kubernetes/pull/42081#issuecomment-282429775.

Add `pkg/kubelet/container/testing/fake_runtime_helper.go`, and change `kuberuntime`, `rkt` and `dockertools` to use it.

@yujuhong This is a small unit test refactoring PR. Could you help me review it?

Automatic merge from submit-queue (batch tested with PRs 41921, 41695, 42139, 42090, 41949)

AWS: Support shared tag `kubernetes.io/cluster/<clusterid>`

We recognize an additional cluster tag:

kubernetes.io/cluster/<clusterid>

This now allows us to share resources, in particular subnets.

In addition, the value is used to track ownership/lifecycle.  When we
create objects, we record the value as "owned".

We also refactor out tags into its own file & class, as we are touching
most of these functions anyway.

```release-note
AWS: Support shared tag `kubernetes.io/cluster/<clusterid>`
```

Automatic merge from submit-queue (batch tested with PRs 41921, 41695, 42139, 42090, 41949)

kubeadm: join ux changes

**What this PR does / why we need it**: Update `kubeadm join` UX according to https://github.com/kubernetes/community/pull/381

**Which issue this PR fixes**: fixes # https://github.com/kubernetes/kubeadm/issues/176

**Special notes for your reviewer**: /cc @luxas @jbeda 

**Release note**:
```release-note
NONE
```

Automatic merge from submit-queue

kubeadm: Turn off insecure apiserver access on localhost:8080

**What this PR does / why we need it**:

ref: https://github.com/kubernetes/kubeadm/issues/181
depends on: https://github.com/kubernetes/kubernetes/pull/41897

**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #

**Special notes for your reviewer**:

**Release note**:

```release-note
Insecure access to the API Server at localhost:8080 will be turned off in v1.6 when using kubeadm
```
@jbeda @liggitt @deads2k @pires @lukemarsden @mikedanese @errordeveloper

Automatic merge from submit-queue (batch tested with PRs 38676, 41765, 42103, 41833, 41702)

AWS: Skip instances that are taggged as a master

We recognize a few AWS tags, and skip over masters when finding zones
for dynamic volumes.  This will fix #34583.

This is not perfect, in that really the scheduler is the only component
that can correctly choose the zone, but should address the common
problem.

```release-note
AWS: Do not consider master instance zones for dynamic volume creation
```

Automatic merge from submit-queue (batch tested with PRs 38676, 41765, 42103, 41833, 41702)

kubelet: cm: refactor QoS logic into seperate interface

This commit has no functional change.  It refactors the QoS cgroup logic into a new `QOSContainerManager` interface to allow for better isolation for QoS cgroup features coming down the pike.

This is a breakout of the refactoring component of my QoS memory limits PR https://github.com/kubernetes/kubernetes/pull/41149 which will need to be rebased on top of this.

@vishh @derekwaynecarr