Automatic merge from submit-queue (batch tested with PRs 45766, 46223)

Scheduler should use a shared informer, and fix broken watch behavior for cached watches

Can be used either from a true shared informer or a local shared
informer created just for the scheduler.

Fixes a bug in the cache watcher where we were returning the "current" object from a watch event, not the historic event.  This means that we broke behavior when introducing the watch cache.  This may have API implications for filtering watch consumers - but on the other hand, it prevents clients filtering from seeing objects outside of their watch correctly, which can lead to other subtle bugs.

```release-note
The behavior of some watch calls to the server when filtering on fields was incorrect.  If watching objects with a filter, when an update was made that no longer matched the filter a DELETE event was correctly sent.  However, the object that was returned by that delete was not the (correct) version before the update, but instead, the newer version.  That meant the new object was not matched by the filter.  This was a regression from behavior between cached watches on the server side and uncached watches, and thus broke downstream API clients.
```

Automatic merge from submit-queue (batch tested with PRs 45766, 46223)

Audit: fill audit.Event in handler chain

Related:
- external API types https://github.com/kubernetes/kubernetes/pull/45315
- policy checker https://github.com/kubernetes/kubernetes/pull/46009

Decisions:
- ~~[ ] decide whether we want to send an event before `WriteHeader` https://github.com/kubernetes/kubernetes/pull/45766#pullrequestreview-38664161~~ Follow-up described in https://github.com/kubernetes/kubernetes/pull/46065/files#r117438531
- [ ] decide how to handle `AuditID`s and the IP chain https://github.com/kubernetes/kubernetes/pull/45766#pullrequestreview-38659371. Is the variant in the proposal (https://github.com/kubernetes/community/pull/625) final? Then we need the API type update.
- ~~[ ] decide how to mark intermediate/incomplete events? set a special reason in `ResponseStatus.Reason` vs. having extra fields for that `Event.NonFinal`
 https://github.com/kubernetes/kubernetes/pull/45766#discussion_r116795888~~ Follow-up of #46065
- [ ] decide whether and how to protect the `Audit-Level` header https://github.com/kubernetes/kubernetes/pull/45766#pullrequestreview-38937691

TODOs:
- ~~[ ] move `AuditIDHeader`, `AuditLevelHeader` to types https://github.com/kubernetes/kubernetes/pull/45766#discussion_r117064094, @timstclair for the type PR~~ Follow-up of https://github.com/kubernetes/kubernetes/pull/46065
- [x] add SourceIP/ForwardedFor support https://github.com/kubernetes/kubernetes/pull/45766#discussion_r116778101
- [x] adapt ObjectReference.Resource to API PR https://github.com/kubernetes/kubernetes/pull/45766#pullrequestreview-38656828

Automatic merge from submit-queue

tighten and simplify owners in some staging repos

With the move to staging, we can have much cleaner owners across the related packages.  This pares down the list of OWNERS to better match for code and activity.  It should help get PRs directed to people more active and familiar with the areas for quicker review.

@kubernetes/sig-api-machinery-misc 
@lavalamp @smarterclayton ptal.

Automatic merge from submit-queue

remove --api-version

Automatic merge from submit-queue (batch tested with PRs 46060, 46234)

Speedup generating selflinks for list and watch requests

I've seen profiles, where GenerateSelflink was 8-9% of whole cpu usage of apiserver (profiles over 30s). Most of this where spent in getting RequestInfo from the context and creating the context.

This PR changes the API of the GenerateLink method of the namer which results in computing the context and requestInfo only once per LIST/WATCH request (instead of computing it for every single returned element of LIST/WATCH).

@smarterclayton @deads2k - can one of you please take a look?

Automatic merge from submit-queue (batch tested with PRs 46060, 46234)

Randomize test nodePort to prevent collision

Fix #37982.

/assign @bowei 

**Release note**:

```release-note
NONE
```

Automatic merge from submit-queue

Add `auto_unmount` mount option for glusterfs fuse mount.

libfuse has an auto_unmount option which, if enabled, ensures that
the file system is unmounted at FUSE server termination by running a
separate monitor process that performs the unmount when that occurs.
(This feature would probably better be called "robust auto-unmount",
as FUSE servers usually do try to unmount their file systems upon
termination, it's just this mechanism is not crash resilient.)
This change implements that option and behavior for glusterfs.

This option will be only supported for clients with version >3.11.
Signed-off-by: Humble Chirammal <hchiramm@redhat.com>

Automatic merge from submit-queue

vSphere storage policy support for dynamic volume provisioning

Till now, vSphere cloud provider provides support to configure persistent volume with VSAN storage capabilities - kubernetes#42974. Right now this only works with VSAN.

Also there might be other use cases:

- The user might need a way to configure a policy on other datastores like VMFS, NFS etc.
- Use Storage IO control, VMCrypt policies for a persistent disk.

We can achieve about 2 use cases by using existing storage policies which are already created on vCenter using the Storage Policy Based Management service. The user will specify the SPBM policy ID as part of dynamic provisioning 

- resultant persistent volume will have the policy configured with it. 
- The persistent volume will be created on the compatible datastore that satisfies the storage policy requirements. 
- If there are multiple compatible datastores, the datastore with the max free space would be chosen by default.
- If the user specifies the datastore along with the storage policy ID, the volume will created on this datastore if its compatible. In case if the user specified datastore is incompatible, it would error out the reasons for incompatibility to the user.
- Also, the user will be able to see the associations of persistent volume object with the policy on the vCenter once the volume is attached to the node.

For instance in the below example, the volume will created on a compatible datastore with max free space that satisfies the "Gold" storage policy requirements.

```
kind: StorageClass
apiVersion: storage.k8s.io/v1beta1
metadata:
       name: fast
provisioner: kubernetes.io/vsphere-volume
parameters:
      diskformat: zeroedthick
      storagepolicyName: Gold
```

For instance in the below example, the vSphere CP checks if "VSANDatastore" is compatible with "Gold" storage policy requirements. If yes, volume will be provisioned on "VSANDatastore" else it will error that "VSANDatastore" is not compatible with the exact reason for failure.

```
kind: StorageClass
apiVersion: storage.k8s.io/v1beta1
metadata:
       name: fast
provisioner: kubernetes.io/vsphere-volume
parameters:
      diskformat: zeroedthick
      storagepolicyName: Gold
      datastore: VSANDatastore
```

As a part of this change, 4 commits have been added to this PR.

1. Vendor changes for vmware/govmomi
2. Changes to the VsphereVirtualDiskVolumeSource in the Kubernetes API. Added 2 additional fields StoragePolicyName, StoragePolicyID
3. Swagger and Open spec API changes.
4. vSphere Cloud Provider changes to implement the storage policy support.

**Release note**:


```release-note
vSphere cloud provider: vSphere Storage policy Support for dynamic volume provisioning
```

Automatic merge from submit-queue

Recording openstack metrics

add openstack operation metrics


**Release note**:
```release-note
Add support for emitting metrics from openstack cloudprovider about storage operations.
```

/assign @gnufied

Automatic merge from submit-queue (batch tested with PRs 46201, 45952, 45427, 46247, 46062)

kubectl: fix deprecation warning bug

**What this PR does / why we need it**:

Some kubectl commands were deprecated but would fail to print the
correct warning message when a flag was given before the command name.

	# Correctly prints the warning that "resize" is deprecated and
	# "scale" is now preferred.
	kubectl resize [...]

	# Should print the same warning but no warning is printed.
	kubectl --v=1 resize [...]

This was due to a fragile check on os.Args[1].

This commit implements a new function deprecatedCmd() that is used to
construct new "passthrough" commands which are marked as deprecated and
hidden.

Note that there is an existing "filters" system that may be preferable
to the system created in this commit. I'm not sure why the "filters"
array was not used for all deprecated commands in the first place.

**Release note**:

```release-note
NONE
```

Automatic merge from submit-queue (batch tested with PRs 46201, 45952, 45427, 46247, 46062)

[Federation][kubefed]: Add support for etcd image override

This PR adds support for overriding the default etcd image used by ``kubefed init`` by providing an argument to ``--etcd-image``.  This is primarily intended to allow consumers like openshift to provide a different default, but as a nice side-effect supports code-free validation of non-default etcd images. 

**Release note**:

```release-note
'kubefed init' now supports overriding the default etcd image name with the --etcd-image parameter.
```
cc: @kubernetes/sig-federation-pr-reviews

Automatic merge from submit-queue (batch tested with PRs 46201, 45952, 45427, 46247, 46062)

Use shared informers in gc controller if possible

Modify the garbage collector controller to try to use shared informers for resources, if possible, to reduce the number of unique reflectors listing and watching the same thing.

cc @kubernetes/sig-api-machinery-pr-reviews @caesarxuchao @deads2k @liggitt @sttts @smarterclayton @timothysc @soltysh @kargakis @kubernetes/rh-cluster-infra @derekwaynecarr @wojtek-t @gmarek

Automatic merge from submit-queue (batch tested with PRs 46201, 45952, 45427, 46247, 46062)

remove the elasticsearch template

**What this PR does / why we need it**:

**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #

**Special notes for your reviewer**:

**Release note**:

```
NONE
```
Loading file-based index template has been disabled since 2.0.0-beta1 version of Elasticsearch.  https://www.elastic.co/guide/en/elasticsearch/reference/2.0/breaking_20_index_api_changes.html#_file_based_index_templates 

So the `template-k8s-logstash.json` is not longer useful.

On the other hand, as https://github.com/kubernetes/kubernetes/issues/25127 indicated, we might better curl the elasticsearch API to load this template.

Automatic merge from submit-queue

Address remaining TODOs in kube-proxy.

Followup PR from the previous two.

Automatic merge from submit-queue (batch tested with PRs 46022, 46055, 45308, 46209, 43590)

Eviction does not evict unless the previous pod has been cleaned up

Addresses #43166
This PR makes two main changes:
First, it makes the eviction loop re-trigger immediately if there may still be pressure.  This way, if we already waited 10 seconds to delete a pod, we dont need to wait another 10 seconds for the next synchronize call.
Second, it waits for the pod to be cleaned up (including volumes, cgroups, etc), before moving on to the next synchronize call.  It has a timeout for this operation currently set to 30 seconds.

Automatic merge from submit-queue (batch tested with PRs 46022, 46055, 45308, 46209, 43590)

Remove Save() from iptables interface

This is what @thockin requested in one of the reviews.

Automatic merge from submit-queue (batch tested with PRs 46022, 46055, 45308, 46209, 43590)

More cli sanity verifications

Adds some more `kubectl` command sanity checks to improve consistency and avoid the need of code reviews for some of our CLI style and standards.

**Release note**:

```release-note
NONE
```
@kubernetes/sig-cli-pr-reviews

Automatic merge from submit-queue (batch tested with PRs 46022, 46055, 45308, 46209, 43590)

embed kube-apiextensions inside of kube-apiserver

To reduce operation complexity, we decided to include the kube-apiextensions-server inside of kube-apiserver (https://github.com/kubernetes/community/blob/master/sig-api-machinery/api-extensions-position-statement.md#q-should-kube-aggregator-be-a-separate-binaryprocess-than-kube-apiserver).  With the API reasonably well established and a finalizer about merge, I think its time to add ourselves.

This pull wires kube-apiextensions-server ahead of the TPRs so that one will replace the other if both are added by accident (CRDs should have priority) and wires a controller for automatic aggregation.

WIP because I still need tests: unit test for controller, test-cmd test to mirror the TPR test.


```release-note
Adds the `CustomResourceDefinition` (crd) types to the `kube-apiserver`.  These are the successors to `ThirdPartyResource`.  See https://github.com/kubernetes/community/blob/master/contributors/design-proposals/thirdpartyresources.md for more details.
```

Automatic merge from submit-queue

add rolebinding/clusterrolebinding to describe.go

**What this PR does / why we need it**:

```
./cluster/kubectl.sh describe clusterrolebinding system:kube-dns
Name:		system:kube-dns
Labels:		kubernetes.io/bootstrapping=rbac-defaults
Annotations:	rbac.authorization.kubernetes.io/autoupdate=true
Role:
  Kind:	ClusterRole
  Name:	system:kube-dns
Subjects:
  Kind			Name		Namespace
  ----			----		---------
  ServiceAccount	kube-dns	kube-system
```

**Which issue this PR fixes**: 

**Special notes for your reviewer**:

**Release note**:

```release-note
NONE
```

Automatic merge from submit-queue (batch tested with PRs 38990, 45781, 46225, 44899, 43663)

Fix command exec -- COMMAND can not contain spaces

Fixes https://github.com/kubernetes/kubernetes/issues/7688
the problem is when you execute command:
 `cluster/kubectl.sh exec -p client-blue-8yw37 -c client -i -t -- 'ls -t /usr'`
the args is 
[`client-blue-8yw37` , `ls -t /usr`] 
**instead of** 
[`client-blue-8yw37`, `ls`, `-t`, `/usr`]
@kubernetes/sig-cli-pr-reviews, so I add a warning, wdyt ?
cc @ymqytw @adohe @fabianofranz

Automatic merge from submit-queue (batch tested with PRs 38990, 45781, 46225, 44899, 43663)

Support parallel scaling on StatefulSets

Fixes #41255

```release-note
StatefulSets now include an alpha scaling feature accessible by setting the `spec.podManagementPolicy` field to `Parallel`.  The controller will not wait for pods to be ready before adding the other pods, and will replace deleted pods as needed.  Since parallel scaling creates pods out of order, you cannot depend on predictable membership changes within your set.
```

Automatic merge from submit-queue (batch tested with PRs 38990, 45781, 46225, 44899, 43663)

Fix missing close file

None

Automatic merge from submit-queue (batch tested with PRs 38990, 45781, 46225, 44899, 43663)

kubectl: fixes issues #45736 and #45737

**What this PR does / why we need it**:
This PR improves the help displayed by the `kubectl proxy` command.

**Which issue this PR fixes**: 
fixes #45736, fixes #45737 

**Special notes for your reviewer**:
NONE

**Release note**:
NONE

```release-note
```

Automatic merge from submit-queue (batch tested with PRs 38990, 45781, 46225, 44899, 43663)

migrate set generation to go_genrule

Depends on https://github.com/kubernetes/release/pull/238

Automatic merge from submit-queue

Fix kubelet event recording

**What this PR does / why we need it**:
There are numerous areas where the kubelet was not properly recording events due to an incorrect type.

To keep this small, I updated all references to `RefManager` that result in throwing an event to ensure it does a conversion.

**Which issue this PR fixes**
Fixes https://github.com/kubernetes/kubernetes/issues/46241
Fixes #44348
Fixes #44652

**Special notes for your reviewer**:
I updated all references I could find to the existing RefManager in kubelet.

**Release note**:
```release-note
fix kubelet event recording for selected events.
```

Adds long description from #45902

Automatic merge from submit-queue (batch tested with PRs 46133, 46211, 46224, 46205, 45910)

Correct client-go TPR example type registration

**What this PR does / why we need it**:

Eliminate duplicate registration of third-party resource types in the _client-go_ example, precluding any number of readers from copying the mistake into their own applications as they adapt the example to their own needs.

**Special notes for your reviewer**:

See [the preceding discussion](https://github.com/kubernetes/kubernetes/commit/a6c97715ed7e24581dd9a02a256d57e56b70ac44#commitcomment-22146536) about a6c97715, committed as part of #45463 but only noticed after the merge.

It's possible that we could take a few of the changes proposed here and apply them more broadly throughout the rest of the code, such as not exporting the `AddToScheme` var in favor of an actual function declaration. Given the flux in #44784, I'd prefer that we don't hold up these small touch-ups on a broader unification.

People I expect will want to weigh in: @sttts, @caesarxuchao, and @nilebox.

Automatic merge from submit-queue (batch tested with PRs 46133, 46211, 46224, 46205, 45910)

test/images/network-tester:bump rc/pod image version to 1.9

Current image version is 1.9，update the image version of the associated json file to 1.9
```release-note
NONE
```

Automatic merge from submit-queue (batch tested with PRs 46133, 46211, 46224, 46205, 45910)

Make CPU request for heapster in kubemark scale with the number of Nodes

Automatic merge from submit-queue (batch tested with PRs 46133, 46211, 46224, 46205, 45910)

Add more logs to kubelet_stats

Ref. #46198

Automatic merge from submit-queue (batch tested with PRs 46133, 46211, 46224, 46205, 45910)

clientgo/examples/in-cluster: add instructions to run the example