Automatic merge from submit-queue (batch tested with PRs 38277, 36361, 38452)

local-up: resolve sudo -E'd paths better

This problem won't affect everyone, but for people who have `--with-secure-path` configured for their sudo binaries, `sudo -E` won't actually preserve PATH, which means the sub-bash won't necessarily be able to find the binary which `test` found to exist.
This fixes that.

Ubuntu [used to](https://bugs.launchpad.net/ubuntu/+source/sudo/+bug/50797) configure it that way, and they might still.
It's configured as such on my (Gentoo) machine.

Automatic merge from submit-queue (batch tested with PRs 38277, 36361, 38452)

Add options for build container rsync optimization

KUBE_RSYNC_COMPRESS env var sets rsync compression level.
KUBE_RSYNC_GENERATED_TO_BUILD_CONTAINER env var disables rsyncing
generated files to build containers.

Why KUBE_RSYNC_COMPRESS is needed -- from rsync manual on `--compress` option (implied by non-zero `--compress-level`):
> Note that this option typically achieves better compression ratios than can be achieved by using a compressing remote shell or a compressing transport because it takes advantage of the implicit information in the matching data blocks that are not explicitly sent over  the connection.

Use case for `KUBE_RSYNC_GENERATED_TO_BUILD_CONTAINER`: when you sometimes build stuff locally (e.g. `make WHAT=cmd/kubectl`) and sometimes do it on remote docker (`build-tools/run.sh make WHAT=cmd/hyperkube`), local builds touch generated files which causes them to be rsynced to the build data container, which may slow down the builds. Still, I'm not sure whether local->remote rsync of generated files is useful (e.g. someone may want to edit generated files for debugging purposes?), so I made not rsyncing these files an option instead of forcing such behavior.

Automatic merge from submit-queue

fix set advertise address in local cluster

Automatic merge from submit-queue

Keeps addon manager yamls in sync

From #38437.

We should have kept all addon manager YAML files in sync. This does not fix the release scripts issue, but we should still have this.

@mikedanese @ixdy

Automatic merge from submit-queue (batch tested with PRs 38058, 38523)

Renames kube-dns configure files from skydns* to kubedns*

`skydns-` prefix and `-rc` suffix are confusing and misleading. Renaming it to `kubedns` in existing yaml files and scripts.

@bowei @thockin

Automatic merge from submit-queue (batch tested with PRs 38058, 38523)

Move unstructured types under `meta/v1` to indicate they are typed

Add a few abstractions that make them simpler to use from generic code that does not need accessors. Move OwnerReference to meta/v1 and remote metatypes.go

Part of #37530

Automatic merge from submit-queue

Don't require docker or other tools when running `make clean`

**What this PR does / why we need it**: we don't require `docker` for a local build, so we shouldn't require it for `make clean` either.

**Which issue this PR fixes**: fixes #37690

cc @sebgoa

Also ensure meta/v1 swagger docs are generated.

Avoid directly accessing an unstructured type if it is not required.

OwnerReference is common.

It is a versioned type.

Automatic merge from submit-queue

fix duplicate validation/field/errors

**Release note**:

``` release-note
release-note-none
```

Related PR: https://github.com/kubernetes/kubernetes/pull/30313

PR #30313 fixed duplicate errors for invalid aggregate errors in
https://github.com/kubernetes/kubernetes/blob/master/pkg/kubectl/cmd/util/helpers.go

However, duplicate aggregate errors that went through
https://github.com/kubernetes/kubernetes/blob/master/pkg/util/validation/field/errors.go
were not affected by that patch.

This patch adds duplicate aggregate error checking to
`pkg/util/validation/field/errors.go`
##### Before

`$ kubectl set env rc/idling-echo-1 test-abc=1234`

```
error: ReplicationController "idling-echo-1" is invalid:
[spec.template.spec.containers[0].env[0].name: Invalid value:
"test-abc": must be a C identifier (matching regex
[A-Za-z_][A-Za-z0-9_]*): e.g. "my_name" or "MyName",
spec.template.spec.containers[1].env[0].name: Invalid value: "test-abc":
must be a C identifier (matching regex [A-Za-z_][A-Za-z0-9_]*): e.g.
"my_name" or "MyName", spec.template.spec.containers[0].env[0].name:
Invalid value: "test-abc": must be a C identifier (matching regex
[A-Za-z_][A-Za-z0-9_]*): e.g. "my_name" or "MyName",
spec.template.spec.containers[1].env[0].name: Invalid value: "test-abc":
must be a C identifier (matching regex [A-Za-z_][A-Za-z0-9_]*): e.g.
"my_name" or "MyName"]
```

`$ kubectl set env rc/node-1 test-abc=1234`

```
error: ReplicationController "idling-echo-1" is invalid:
[spec.template.spec.containers[0].env[0].name: Invalid value:
"test-abc": must be a C identifier (matching regex
[A-Za-z_][A-Za-z0-9_]*): e.g. "my_name" or "MyName",
spec.template.spec.containers[1].env[0].name: Invalid value: "test-abc":
must be a C identifier (matching regex [A-Za-z_][A-Za-z0-9_]*): e.g.
"my_name" or "MyName"]
```
##### After

`$ kubectl set env rc/idling-echo-1 test-abc=1234`

```
error: ReplicationController "idling-echo-1" is invalid:
[spec.template.spec.containers[0].env[0].name: Invalid value:
"test-abc": must be a C identifier (matching regex
[A-Za-z_][A-Za-z0-9_]*): e.g. "my_name" or "MyName",
spec.template.spec.containers[1].env[0].name: Invalid value: "test-abc":
must be a C identifier (matching regex [A-Za-z_][A-Za-z0-9_]*): e.g.
"my_name" or "MyName"]
```

`$ kubectl set env rc/node-1 test-abc=1234`

```
error: ReplicationController "node-1" is invalid:
spec.template.spec.containers[0].env[0].name: Invalid value: "test-abc":
must be a C identifier (matching regex [A-Za-z_][A-Za-z0-9_]*): e.g.
"my_name" or "MyName"
```

Automatic merge from submit-queue

Remove json serialization annotations from internal types

fixes #3933

Internal types should never be serialized, and including json serialization tags on them makes it possible to accidentally do that without realizing it.

fixes in this PR:

* types
  * [x] remove json tags from internal types
  * [x] fix references from serialized types to internal ObjectMeta
* generation
  * [x] remove generated json codecs for internal types (they should never be used)
* kubectl
  * [x] fix `apply` to operate on versioned object
  * [x] fix sorting by field to operate on versioned object
  * [x] fix `--record` to build annotation patch using versioned object
* hpa
  * [x] fix unmarshaling to internal CustomMetricTargetList in validation
* thirdpartyresources
  * [x] fix encoding API responses using internal ObjectMeta
* tests
  * [x] fix tests to use versioned objects when checking encoded content
  * [x] fix tests passing internal objects to generic printers

follow ups (will open tracking issues or additional PRs):
- [ ] remove json tags from internal kubeconfig types (`kubectl config set` pathfinding needs to work against external type)
- [ ] HPA should version CustomMetricTargetList serialization in annotations
- [ ] revisit how TPR resthandlers encoding objects
- [ ] audit and add tests for printer use (human-readable printer requires internal versions, generic printers require external versions)
- [ ] add static analysis tests preventing new internal types from adding tags
- [ ] add static analysis tests requiring json tags on external types (and enforcing lower-case first letter)
- [ ] add more tests for `kubectl get` exercising known and unknown types with all output options

Automatic merge from submit-queue

providers: Remove long-deprecated Instances.List()

Fix the cross-build

This method has been unused by k8s for some time, and yet is the last
piece of the cloud provider API that encourages provider names to be
human-friendly strings (this method applies a regex to instance names).

Actually removing this deprecated method is part of a long effort to
migrate from instance names to instance IDs in at least the OpenStack
provider plugin.

Automatic merge from submit-queue

Add support for Azure Container Registry, update Azure dependencies

@colemickens

Automatic merge from submit-queue (batch tested with PRs 34002, 38535, 37330, 38522, 38423)

Fix panic in vSphere cloud provider

Currently vSphere Cloud Provider triggers panic in controller-manager pod kubernetes. This is because it queries for the cluster name from the VC. We have eliminated that code from the vSphere cloud provider. 

Fixes #36295

Automatic merge from submit-queue (batch tested with PRs 34002, 38535, 37330, 38522, 38423)

Node E2E: `make test-e2e-node` runs the same test with pr builder by default.

This PR makes `make test-e2e-node` run non-serial, non-flaky, non-slow test by default.
This will make it easier to use.

/cc @timstclair

Automatic merge from submit-queue (batch tested with PRs 34002, 38535, 37330, 38522, 38423)

Rename pet to stateful pods in statefulset e2e tests logs

Follow up https://github.com/kubernetes/kubernetes/pull/37243#discussion_r89193790

Note that only mentions of pet in logs are renamed, but not variable name, or file name / import path 

@foxish 

```release-note
NONE
```

Automatic merge from submit-queue

Remove hack/verify-munge-docs.sh from pre-commit hooks

Follow up of #38309

Automatic merge from submit-queue

fix broken cluster/centos and enhance the style

replace the PR #33818

Automatic merge from submit-queue

fix typo in pet_set

fix typo in pet_set.

Automatic merge from submit-queue (batch tested with PRs 37677, 38519)

Better deal with failures under golang 1.8beta1

If there is any error in the initial parsing then we should just
try adding the scheme.

url.Parse(base) has changed in 1.8. Please see the following change
https://github.com/golang/go/commit/c5ccbdd22bdbdc43d541b7e7d4ed66ceb559030e

Fixes #38380

Automatic merge from submit-queue

Proper fix for non-receiving watchers

Follow up on #37195

Automatic merge from submit-queue (batch tested with PRs 37270, 38309, 37568, 34554)

Ability to quota storage by storage class

Adds the ability to quota storage by storage class.
1. `<storage-class>.storageclass.storage.k8s.io/persistentvolumeclaims` - quota the number of claims with a specific storage class
2. `<storage-class>.storageclass.storage.k8s.io/requests.storage` - quota the cumulative request for storage in a particular storage class.

For example:

```
$ cat quota.yaml
apiVersion: v1
kind: ResourceQuota
metadata:
  name: storage-quota
spec:
  hard:
    requests.storage: 100Gi
    persistentvolumeclaims: 100
    gold.storageclass.storage.k8s.io/requests.storage: 50Gi
    gold.storageclass.storage.k8s.io/persistentvolumeclaims: 5
    silver.storageclass.storage.k8s.io/requests.storage: 75Gi
    silver.storageclass.storage.k8s.io/persistentvolumeclaims: 10
    bronze.storageclass.storage.k8s.io.kubernetes.io/requests.storage: 100Gi
    bronze.storageclass.storage.k8s.io/persistentvolumeclaims: 15
$ kubectl create -f quota.yaml
$ cat pvc-bronze.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  generateName: pvc-bronze-
  annotations:
    volume.beta.kubernetes.io/storage-class: "bronze"
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 8Gi
$ kubectl create -f pvc-bronze.yaml
$ kubectl get quota storage-quota -o yaml
apiVersion: v1
kind: ResourceQuota
...
status:
  hard:
    bronze.storageclass.storage.k8s.io/persistentvolumeclaims: "15"
    bronze.storageclass.storage.k8s.io/requests.storage: 100Gi
    gold.storageclass.storage.k8s.io/persistentvolumeclaims: "5"
    gold.storageclass.storage.k8s.io/requests.storage: 50Gi
    persistentvolumeclaims: "100"
    requests.storage: 100Gi
    silver.storageclass.storage.k8s.io/persistentvolumeclaims: "10"
    silver.storageclass.storage.k8s.io/requests.storage: 75Gi
  used:
    bronze.storageclass.storage.k8s.io/persistentvolumeclaims: "1"
    bronze.storageclass.storage.k8s.io/requests.storage: 8Gi
    gold.storageclass.storage.k8s.io/persistentvolumeclaims: "0"
    gold.storageclass.storage.k8s.io/requests.storage: "0"
    persistentvolumeclaims: "1"
    requests.storage: 8Gi
    silver.storageclass.storage.k8s.io/persistentvolumeclaims: "0"
    silver.storageclass.storage.k8s.io/requests.storage: "0"
```