Automatic merge from submit-queue (batch tested with PRs 36071, 32752, 37998, 38350, 38401)

Pass addressable values to DeepCopy

Extracted from https://github.com/kubernetes/kubernetes/pull/35728

These are the places we are currently calling DeepCopy incorrectly, and we need to fix, even if we don't pick up the changes to DeepCopy in #35728:
* creating a new cloner means we have no generated functions registered
* passing non-addressable values doesn't pick up generated deep copy functions, and forces us into reflective mode

Automatic merge from submit-queue (batch tested with PRs 36071, 32752, 37998, 38350, 38401)

Eradicate ExpectNoError from test/e2e.

```
$ cd test/e2e
$ sed -i "s/\tExpectNoError/\tframework.ExpectNoError/g" *.go
```

Automatic merge from submit-queue (batch tested with PRs 36071, 32752, 37998, 38350, 38401)

Add test for concurrent evictions requests

This is a followup PR after #37668.
Add a test case to make sure concurrent eviction requests can be handled.

@davidopp @lavalamp

Automatic merge from submit-queue (batch tested with PRs 36071, 32752, 37998, 38350, 38401)

Allow a selector when retrieving logs

#19873

initial commit to see if I am headed in the right direction.
Its missing all the test cases, but the selector path works.

Automatic merge from submit-queue

fix local resource output when `-f` not specified

**Release note**:
```release-note
release-note-none
```

`kubectl set image` does not have a `--dry-run` option. Although it offers a
`--local` flag, it does not support server request, limiting input to that of stdin
or that of a local file.

This patch adds a `--dry-run` option to the `kubectl set image` command,
allowing for resources from the server to be selected, without making any
mutations.

cc @ncdc 

Related PR: https://github.com/kubernetes/kubernetes/pull/36174

Automatic merge from submit-queue

Fix unmountDevice issue caused by shared mount in GCI

This is a fix on top #38124. In this fix, we move the logic to filter
out shared mount references into operation_executor's UnmountDevice
function to avoid this part is being used by other types volumes such as
rdb, azure etc. This filter function should be only needed during
unmount device for GCI image.

Automatic merge from submit-queue

[Federation] Implement dry run support in kubefed init

This one implements one of the TODO items pending in the previous set of kubefed PRs.
This one is done on top of another todo PR https://github.com/kubernetes/kubernetes/pull/36310 which is being reviewed separately.
Please review only the last 2 commits in this one.
The design doc PR for kubefed is at https://github.com/kubernetes/kubernetes/pull/34484.

cc  @kubernetes/sig-cluster-federation  @madhusudancs @nikhiljindal



**Release note**:
<!--  Steps to write your release note:
1. Use the release-note-* labels to set the release note state (if you have access) 
2. Enter your extended release note in the below block; leaving it blank means using the PR title as the release note. If no release note is required, just write `NONE`. 
-->
``` 
[Federation] `kubefed init` now supports dry run mode. 
```

Automatic merge from submit-queue (batch tested with PRs 36310, 37349, 38319, 38402, 38338)

Fix space issue in volumePath with vSphere Cloud Provider

I tried to create a kubernetes deployment with vSphere volume with volume path
"[datastore] kubevols/redis-master".
In this case the cloud provider queries the getDeviceNameFromMount() to return the path of the volume mounted. Since getDeviceNameFromMount() queries the filesystem to get the mount references, it returns a volume path "[datastore]\\040kubevols/redis-master". Later the kubelet searches for this volume path in both the actual and desired states. Th actual and desired states contains volume with path "[datastore] kubevols/redis-master". So, it couldn't find such volume path and therefore kubernetes stalls unable to make any progress further similar to one described in #37022.

This PR will fix the space issue in volume path by replacing \\040 to empty space. This fixes #37712.
Also fixes #38148
@kerneltime @pdhamdhere

Automatic merge from submit-queue (batch tested with PRs 36310, 37349, 38319, 38402, 38338)

Use the `serviceShard` variable in the service shard block, not the `service` variable.

cc @kubernetes/sig-federation

Automatic merge from submit-queue (batch tested with PRs 36310, 37349, 38319, 38402, 38338)

add summarizing discovery controller and handlers

Requires https://github.com/kubernetes/kubernetes/pull/38304 .

This adds discovery support to the new `kubernetes-discovery` that runs based on a controller wired up to the `APIService.apiregistration.k8s.io`.  It also adds in plumbing for `local-up-cluster.sh` to register the "normal" kube resources.

@kubernetes/sig-api-machinery 
@sttts

Automatic merge from submit-queue

replace HTTP status code with  HTTP status code const

<!--  Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, read our contributor guidelines https://github.com/kubernetes/kubernetes/blob/master/CONTRIBUTING.md and developer guide https://github.com/kubernetes/kubernetes/blob/master/docs/devel/development.md
2. If you want *faster* PR reviews, read how: https://github.com/kubernetes/kubernetes/blob/master/docs/devel/faster_reviews.md
3. Follow the instructions for writing a release note: https://github.com/kubernetes/kubernetes/blob/master/docs/devel/pull-requests.md#release-notes
-->

**What this PR does / why we need it**:

replace HTTP status code with  HTTP status code const

Thanks!

**Special notes for your reviewer**:

**Release note**:
<!--  Steps to write your release note:
1. Use the release-note-* labels to set the release note state (if you have access) 
2. Enter your extended release note in the below block; leaving it blank means using the PR title as the release note. If no release note is required, just write `NONE`. 
-->
```release-note
```

This is a fix on top #38124. In this fix, we move the logic to filter
out shared mount references into operation_executor's UnmountDevice
function to avoid this part is being used by other types volumes such as
rdb, azure etc. This filter function should be only needed during
unmount device for GCI image.

Automatic merge from submit-queue

[Federation] Make federation etcd PVC size configurable

This one implements one of the many TODO items pending in the previous set of kubefed PRs.
The design doc PR is at https://github.com/kubernetes/kubernetes/pull/34484

cc  @kubernetes/sig-cluster-federation  @madhusudancs



**Release note**:
<!--  Steps to write your release note:
1. Use the release-note-* labels to set the release note state (if you have access) 
2. Enter your extended release note in the below block; leaving it blank means using the PR title as the release note. If no release note is required, just write `NONE`. 
-->
``` 
[Federation] kubefed init now has a new flag, --etcd-pv-capacity, which can be used to configure the persistent volume capacity for etcd.  
```

This patch adds a `--dry-run` option to `kubectl set image...`

Automatic merge from submit-queue

Customizable vagrant rsync args and excludes

<!--  Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, read our contributor guidelines https://github.com/kubernetes/kubernetes/blob/master/CONTRIBUTING.md and developer guide https://github.com/kubernetes/kubernetes/blob/master/docs/devel/development.md
2. If you want *faster* PR reviews, read how: https://github.com/kubernetes/kubernetes/blob/master/docs/devel/faster_reviews.md
3. Follow the instructions for writing a release note: https://github.com/kubernetes/kubernetes/blob/master/docs/devel/pull-requests.md#release-notes
-->

**What this PR does / why we need it**:

There are some annoyances with Vagrant+rsync currently.

When using rsync file synchronization with Vagrant, the whole kubernetes
repo directory is copied over into the virtual machine. This includes
the _output directory, which tends to be gigabytes in size, while often
just _output/release-tars (a few hundred MB) would be enough.

Furthermore, if the some of the directories contains a recursive
symlink, rsync with the default args will keep descending and copying
files endlessly until filling up the VM disk.

**The improvement**:

Making the rsync args and excluded dirs/files customizable via
KUBERNETES_VAGRANT_RSYNC_ARGS and KUBERNETES_VAGRANT_RSYNC_EXLUDE,
respectively, allows the developer to prevent the issues mentioned
above. A new KUBERNETES_VAGRANT_USE_RSYNC variable is also added to
control whether Vagrant should force usage of rsync as the file
synchronization backend. The args/exclude customizations only take
effect when KUBERNETES_VAGRANT_USE_RSYNC is 'true'.

**Release note**:
<!--  Steps to write your release note:
1. Use the release-note-* labels to set the release note state (if you have access) 
2. Enter your extended release note in the below block; leaving it blank means using the PR title as the release note. If no release note is required, just write `NONE`. 
-->
```release-note
NONE
```

Automatic merge from submit-queue

kubedns: use initial resource listing as ready signal

Fix #35140.

Set up the ready signal after the first resource listing finished for both endpoints and services instead of listen on kubernetes service.

@bprashanth @bowei @thockin

**Release note**:
```
```

Automatic merge from submit-queue (batch tested with PRs 38377, 36365, 36648, 37691, 38339)

Exponential back off when volume delete fails

**What this PR does / why we need it**:

This PR implements ability in pv_controller to back off when deleting a volume fails from plugin API. 

**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: 

Partly fixes #38295 , but I think volume delete is most problematic thing happening in pv_controller without any sort of backoff. 

After this change the attempts of volume deletion look like:

```
controller : I1208 00:18:35.532061   16388 aws_util.go:55] Error deleting EBS Disk volume aws://us-east-1d/vol-abcdefg: VolumeInUse: Volume vol-abcdefg is currently attached to i-1234567
controller : I1208 00:20:50.578325   16388 aws_util.go:55] Error deleting EBS Disk volume aws://us-east-1d/vol-abcdefg: VolumeInUse: Volume vol-abcdefg is currently attached to i-1234567
controller : I1208 00:23:05.563488   16388 aws_util.go:55] Error deleting EBS Disk volume aws://us-east-1d/vol-abcdefg: VolumeInUse: Volume vol-abcdefg is currently attached to i-1234567
controller : I1208 00:25:20.599158   16388 aws_util.go:55] Error deleting EBS Disk volume aws://us-east-1d/vol-abcdefg: VolumeInUse: Volume vol-abcdefg is currently attached to i-1234567
controller : I1208 00:27:35.560009   16388 aws_util.go:55] Error deleting EBS Disk volume aws://us-east-1d/vol-abcdefg: VolumeInUse: Volume vol-abcdefg is currently attached to i-1234567
controller : I1208 00:29:50.594967   16388 aws_util.go:55] Error deleting EBS Disk volume aws://us-east-1d/vol-abcdefg: VolumeInUse: Volume vol-abcdefg is currently attached to i-1234567
controller : I1208 00:32:05.539168   16388 aws_util.go:55] Error deleting EBS Disk volume aws://us-east-1d/vol-abcdefg: VolumeInUse: Volume vol-abcdefg is currently attached to i-1234567
controller : I1208 00:34:20.581665   16388 aws_util.go:55] Error deleting EBS Disk volume aws://us-east-1d/vol-abcdefg: VolumeInUse: Volume vol-abcdefg is currently attached to i-1234567
```

Automatic merge from submit-queue (batch tested with PRs 38377, 36365, 36648, 37691, 38339)

Do not create selector and namespaces in a loop where possible

With 1000 nodes and 5000 pods (5 pods per node) with anti-affinity a lot of CPU wasted on creating LabelSelector and sets.String (map). 

With this change we are able to deploy that number of pods in ~25 minutes. Without - it takes 30 minutes to deploy 500 pods with anti-affinity configured.

Automatic merge from submit-queue (batch tested with PRs 38377, 36365, 36648, 37691, 38339)

controller: sync stuck deployments in a secondary queue

@kubernetes/deployment this makes Deployments not depend on a tight resync interval in order to estimate progress.

Automatic merge from submit-queue (batch tested with PRs 38377, 36365, 36648, 37691, 38339)

Backoff correctly when adopting replica sets/pods

@kubernetes/deployment ptal

Fixes https://github.com/kubernetes/kubernetes/issues/34534

Automatic merge from submit-queue (batch tested with PRs 38377, 36365, 36648, 37691, 38339)

HPA e2e tests: fixed problem w/blocking channel.

Automatic merge from submit-queue

Fix local-cluster-up on Mac and discovery on all systems

- use cfssl on the host, not from Docker. Solves a number of permission problem with selinux and Mac
- fix discovery startup

Automatic merge from submit-queue

Add secrets to Density and Load tests

cc @jeremyeder @timstclair @sjug

Automatic merge from submit-queue (batch tested with PRs 35939, 38381, 37825, 38306, 38110)

split easy controllers from giant start func

continues work from https://github.com/kubernetes/kubernetes/pull/37976 to separate controller initialization.

Automatic merge from submit-queue (batch tested with PRs 35939, 38381, 37825, 38306, 38110)

Add test for multi-threaded use of ratelimiter

Adds a test to help prevent #38273 from occurring again

Automatic merge from submit-queue (batch tested with PRs 35939, 38381, 37825, 38306, 38110)

Moved start-kubemark-master.sh from test/kubemark/ to test/kubemark/r…

Automatic merge from submit-queue

apiserver(s): do not create self-signed certs if port is zero

Automatic merge from submit-queue

Minor cleanup: fix typos

Fix some typos.

re-run godep license