Automatic merge from submit-queue

Fix for Premature iSCSI logout #39202.

**What this PR does / why we need it**:

Modifies the iSCSI volume plugin code to prevent premature iSCSI logouts and the establishment of multiple iSCSI connections to the same target in certain cases.

**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #39202, fixes #41041, fixes #40941

**Special notes for your reviewer**:

The existing iSCSI connections are now rescanned on every AttachDisk call to discover newly created LUNs.

The disk mount points now contain an additional directory in the path corresponding to the disk iface that is later used for iSCSI logout.

The device prefixes that are used to count the existing references to the portal-target pair now contain the whole path including the mount point until the lun index.

**Release note**:
```release-note
Fixed issues #39202, #41041 and #40941 that caused the iSCSI connections to be prematurely closed when deleting a pod with an iSCSI persistent volume attached and that prevented the use of newly created LUNs on targets with preestablished connections.
```

Automatic merge from submit-queue (batch tested with PRs 40297, 41285, 41211, 41243, 39735)

fix variables in openstack.go to keep camel casing and remove unused var

Automatic merge from submit-queue (batch tested with PRs 40297, 41285, 41211, 41243, 39735)

Add LegacyHostIP as a fallback to federation api-server nodeport service

**What this PR does / why we need it**:
while deploying federation using kubefed and using NodePort type service for api-server, if the cluster does not have NodeExternalIP for nodes, then it leads to incorrect endpoint being written to kubeconfig.
So falling back to use LegacyHostIP in such cases.

**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #
Partly fixes an issue in this [thread](https://github.com/kubernetes/kubernetes/issues/41127#issuecomment-278888658)

**Special notes for your reviewer**:

**Release note**:
`NONE`

```release-note
```
cc @kubernetes/sig-federation-bugs @madhusudancs

Automatic merge from submit-queue (batch tested with PRs 40297, 41285, 41211, 41243, 39735)

cluster/gce: Add env var to enable apiserver basic audit log.

For now, this is focused on a fixed set of flags that makes the audit
log show up under /var/log/kube-apiserver-audit.log and behave similarly
to /var/log/kube-apiserver.log. Allowing other customization would
require significantly more complex changes.

Audit log rotation is handled the same as for `kube-apiserver.log`.

**What this PR does / why we need it**:

Add a knob to enable [basic audit logging](https://kubernetes.io/docs/admin/audit/) in GCE.

**Which issue this PR fixes**:

**Special notes for your reviewer**:

We would like to cherrypick/port this to release-1.5 also.

**Release note**:
```release-note
The kube-apiserver [basic audit log](https://kubernetes.io/docs/admin/audit/) can be enabled in GCE by exporting the environment variable `ENABLE_APISERVER_BASIC_AUDIT=true` before running `cluster/kube-up.sh`. This will log to `/var/log/kube-apiserver-audit.log` and use the same `logrotate` settings as `/var/log/kube-apiserver.log`.
```

Automatic merge from submit-queue (batch tested with PRs 40297, 41285, 41211, 41243, 39735)

Secure kube-scheduler

This PR:
* Adds a bootstrap `system:kube-scheduler` clusterrole
* Adds a bootstrap clusterrolebinding to the `system:kube-scheduler` user
* Sets up a kubeconfig for kube-scheduler on GCE (following the controller-manager pattern)
* Switches kube-scheduler to running with kubeconfig against secured port (salt changes, beware)
* Removes superuser permissions from kube-scheduler in local-up-cluster.sh
* Adds detailed RBAC deny logging

```release-note
On kube-up.sh clusters on GCE, kube-scheduler now contacts the API on the secured port.
```

Automatic merge from submit-queue (batch tested with PRs 40297, 41285, 41211, 41243, 39735)

Use fnv.New32a() in hash instead adler32

Ref: https://github.com/kubernetes/kubernetes/issues/40140

Benchmark results: https://github.com/kubernetes/kubernetes/pull/39527

NOTE: I leave  `GetPodTemplateSpecHash` as it is since we have unit test to test its un-normal behaviour.

Disable CRI in hollow-node

Some parts of the new dockershim are not properly mocked out in the
hollow-node. Disabling CRI in the hollow node until a proper fix is put
in.

Automatic merge from submit-queue

Update rbac data to v1beta1

Update RBAC fixtures to v1beta1

Automatic merge from submit-queue

Fix bug in status manager TerminatePod

In TerminatePod, we previously pass pod.Status to updateStatusInternal.  This is a bug, since it is the original status that we are given.  Not only does it skip updates made to container statuses, but in some cases it reverted the pod's status to an earlier version, since it was being passed a stale status initially.

This was the case in #40239 and #41095.  As shown in #40239, the pod's status is set to running after it is set to failed, occasionally causing very long delays in pod deletion since we have to wait for this to be corrected.

This PR fixes the bug, adds some helpful debugging statements, and adds a unit test for TerminatePod (which for some reason didnt exist before?).

@kubernetes/sig-node-bugs @vish @Random-Liu

Automatic merge from submit-queue

Fix comment for method SetForTests

cc @dchen1107

Automatic merge from submit-queue (batch tested with PRs 41378, 41413, 40743, 41155, 41385)

Expose the constants in pkg/controller/bootstrap and add a validate token function

**What this PR does / why we need it**: In order to hook up #36101 against kubeadm, we have to expose the constants and add a function to validate the token

**Special notes for your reviewer**:

**Release note**:

```release-note
NONE
```
cc @jbeda @mikedanese @pires @dmmcquay

Automatic merge from submit-queue (batch tested with PRs 41378, 41413, 40743, 41155, 41385)

Reconcile bootstrap clusterroles on server start

Currently, on server start, bootstrap roles and bindings are only created if there are no existing roles or rolebindings.

Instead, we should look at each bootstrap role and rolebinding, and ensure it exists and has required permissions and subjects at server start. This allows seamless upgrades to new versions that define roles for new controllers, or add permissions to existing roles.

```release-note
Default RBAC ClusterRole and ClusterRoleBinding objects are automatically updated at server start to add missing permissions and subjects (extra permissions and subjects are left in place). To prevent autoupdating a particular role or rolebinding, annotate it with `rbac.authorization.kubernetes.io/autoupdate=false`.
```

Automatic merge from submit-queue (batch tested with PRs 41378, 41413, 40743, 41155, 41385)

'core' package to prevent dependency creep and isolate core functiona…

**What this PR does / why we need it**:

Solves these two problems: 

- Top level Scheduler root directory has several files in it that are needed really by the factory and algorithm implementations.  Thus they should be subpackages of scheduler.
- In addition scheduler.go and generic_scheduler.go don't naturally differentiate themselves when they are in the same package.  scheduler.go is eseentially the daemon entry point and so it should be isolated from the core 

*No release note needed*

Automatic merge from submit-queue (batch tested with PRs 41378, 41413, 40743, 41155, 41385)

move kube-aggregator to staging

Straight move of kube-aggregator to staging.  The scripts are the interesting bits.

@sttts @lavalamp

Automatic merge from submit-queue

Make EnableCRI default to true

This change makes kubelet to use the CRI implementation by default,
unless the users opt out explicitly by using --enable-cri=false.
For the rkt integration, the --enable-cri flag will have no effect
since rktnetes does not use CRI.

Also, mark the original --experimental-cri flag hidden and deprecated,
so that we can remove it in the next release. If both flags are specified,
the --enable-cri flag overrides the --experimental-cri flag.

Automatic merge from submit-queue

Make kubectl edit work with unstructured objects

Fixes https://github.com/kubernetes/kubernetes/issues/35993


1. First (before any other changes), added several test cases for complex edit scenarios:
   - [x] ensure the edit loop bails out if given the same result that already caused errors
   - [x] ensure an edited file with a syntax error is reopened preserving the input
   - [x] ensure objects with existing "caused-by" annotations get updated with the current command

2. Refactored the edit code to prep for switching to unstructured:
   - [x] made editFn operate on a slice of resource.Info objects passed as an arg, regardless of edit mode
   - [x] simplified short-circuiting logic when re-editing a file containing an error
   - [x] refactored how we build the various visitors (namespace enforcement, annotation application, patching, creating) so we could easily switch to just using a single visitor over a set of resource infos read from the updated input for all of them

3. Switched to using a resource builder to parse the stream of the user's edited output
   - [x] improve the error message you get on syntax errors
   - [x] preserve the user's input more faithfully (see how the captured testcase requests to the server changed to reflect exactly what the user edited)
   - [x] stopped doing client-side conversion (means deprecating `--output-version`)

4. Switched edit to work with generic objects
   - [x] use unstructured objects
   - [x] fall back to generic json merge patch for unrecognized group/version/kinds

5. Added new test cases
   - [x] schemaless objects falls back to generic json merge (covers TPR scenario)
   - [x] edit unknown version of known kind (version "v0" of storageclass) falls back to generic json merge

```release-note
`kubectl edit` now edits objects exactly as they were retrieved from the API. This allows using `kubectl edit` with third-party resources and extension API servers. Because client-side conversion is no longer done, the `--output-version` option is deprecated for `kubectl edit`. To edit using a particular API version, fully-qualify the resource, version, and group used to fetch the object (for example, `job.v1.batch/myjob`)
```

Automatic merge from submit-queue

delete cadvisor pod after test

tracing looks at events for pod deletion and volume teardown.  SInce the cadvisor pod has more than 1 volume, this can make results harder to analyze.
This PR moves the deletion of the cadvisor pod to after the logPodCreateThroughput call, since that marks the "end" of the test.

cc: @dchen1107 @Random-Liu

Automatic merge from submit-queue

kubeadm: add dmmcquay as a reviewer

**What this PR does / why we need it**: add dmmcquay (myself) as a reviewer to kubeadm. 

**Special notes for your reviewer**: /cc @luxas 

**Release note**:
```release-note
NONE
```

Automatic merge from submit-queue

fix comment 

**What this PR does / why we need it**:

fix comment 

Thanks.

**Special notes for your reviewer**:

**Release note**:

```release-note
```

This change makes kubelet to use the CRI implementation by default,
unless the users opt out explicitly by using --enable-cri=false.
For the rkt integration, the --enable-cri flag will have no effect
since rktnetes does not use CRI.

Also, mark the original --experimental-cri flag hidden and deprecated,
so that we can remove it in the next release.

Automatic merge from submit-queue (batch tested with PRs 41216, 41362, 41275, 41277, 41412)

Fix statefulset e2e test

...by removing the liveness/readiness probes from the cockroachdb
manifests, as explained in
https://github.com/kubernetes/test-infra/issues/1740#issuecomment-279555187

@kow3ns @spxtr

Automatic merge from submit-queue (batch tested with PRs 41216, 41362, 41275, 41277, 41412)

kubeadm: test-cmd: join{config, discovery}

**What this PR does / why we need it**: Adding test-cmds for kubeadm join {config, discovery}. Will followup with more test-cmds for other flags as soon as validation for the flags works. 

Adding tests is a WIP from #34136

**Special notes for your reviewer**: /cc @luxas @pires 

**Release note**:
```release-note
NONE
```

Automatic merge from submit-queue (batch tested with PRs 41216, 41362, 41275, 41277, 41412)

kubeadm: test-cmd: init{config, version, api-port}

**What this PR does / why we need it**: Adding test-cmds for kubeadm init {config, version, api-port}. Will followup with more test-cmds for other flags as soon as validation for the flags works. 

Adding tests is a WIP from #34136

**Special notes for your reviewer**: /cc @luxas @pires 

**Release note**:
```release-note
NONE
```

Automatic merge from submit-queue (batch tested with PRs 41216, 41362, 41275, 41277, 41412)

kubeadm: added unit tests for discovery pkg

**What this PR does / why we need it**: added tests to discovery pkg and raised coverage from ~25% to ~71%.  

Adding unit tests is a WIP from #34136

**Special notes for your reviewer**: /cc @luxas @pires 

**Release note**:
```release-note
NONE
```

Automatic merge from submit-queue

[Federation][kubefed]: Bind flag values automatically

Update ``kubefed`` to take advantage of cobra's ability to declare variable bindings at the same time as declaring flags, so that flag values are bound automatically.

Targets #41209 

cc: @kubernetes/sig-federation-pr-reviews @irfanurrehman @madhusudancs

For now, this is focused on a fixed set of flags that makes the audit
log show up under /var/log/kube-apiserver-audit.log and behave similarly
to /var/log/kube-apiserver.log. Allowing other customization would
require significantly more complex changes.

Audit log rotation is handled externally by the wildcard /var/log/*.log
already configured in configure-helper.sh.

Automatic merge from submit-queue

kubeadm: test-cmd: token{delete}

**What this PR does / why we need it**: Adding test-cmds for kubeadm ex token delete. Will followup with more test-cmds for other flags as soon as validation for the flags works. 

Adding tests is a WIP from #34136

**Special notes for your reviewer**: /cc @luxas @pires 

**Release note**:
```release-note
NONE
```

Automatic merge from submit-queue (batch tested with PRs 41382, 41407, 41409, 41296, 39636)

Update to use proxy subresource consistently

Proxy subresources have been in place since 1.2.0 and improve the ability to put policy in place around proxy access.

This PR updates the last few clients to use proxy subresources rather than the root proxy

Automatic merge from submit-queue (batch tested with PRs 41382, 41407, 41409, 41296, 39636)

implement configmap upgrade test

**What this PR does / why we need it**: Add an automated ConfigMap upgrade test to the e2e upgrade test suite. The test creates a ConfigMap and verifies it can be consumed by pods before/after upgrade. See PR #39953 and #40747 for context.

**Special notes for your reviewer**:
@krousey please review for consistency with new upgrade test interface. I copied heavily from secrets test for this. I will implement similar tests for DaemonSets and Jobs next. Note that I have not run this test locally.

**Release note**:
`NONE`

Automatic merge from submit-queue (batch tested with PRs 41382, 41407, 41409, 41296, 39636)

Update the DNS spec from upstream

**What this PR does / why we need it**:
Updates kubeadm to use the latest DNS spec.

**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #

**Special notes for your reviewer**:

@bowei @MrHohn @thockin  In the future, kubedns changes should be applied to this kubeadm file as well

**Release note**:

```release-note
NONE
```

@pires @errordeveloper @dmmcquay @mikedanese

Automatic merge from submit-queue (batch tested with PRs 41382, 41407, 41409, 41296, 39636)

Fix copying systemd libraries upon fluentd-gcp startup

Fix https://github.com/kubernetes/kubernetes/issues/40936
Revert https://github.com/kubernetes/kubernetes/pull/40017

Implemented suggestions for #39202 fix to facilitate kubelet upgrade. The detachDisk behavior is now preserved for pods that were created before the kubelet upgrade.