Automatic merge from submit-queue

refactor conformance test

@Random-Liu  Rebase the testing code.
Signed-off-by: liang chenye <liangchenye@huawei.com>

Automatic merge from submit-queue

Store node information in NodeInfo

This is significantly improving scheduler throughput.

On 1000-node cluster:
- empty cluster: ~70pods/s
- full cluster: ~45pods/s
Drop in throughput is mostly related to priority functions, which I will be looking into next (I already have some PR #24095, but we need for more things before).

This is roughly ~40% increase.
However, we still need better understanding of predicate function, because in my opinion it should be even faster as it is now. I'm going to look into it next week.

@gmarek @hongchaodeng @xiang90 

Automatic merge from submit-queue

Uncomment the code that caused by #19254

Fix https://github.com/kubernetes/kubernetes/issues/24546.

@lavalamp 

Signed-off-by: liang chenye <liangchenye@huawei.com>

Automatic merge from submit-queue

Do not update cache with so much effort

Fixes: #24298
1. Remove automatic update
2. Every time we check if we can get valid value from cache, if not, get the value directly from api

cc @Random-Liu 

Automatic merge from submit-queue

Add flag -t as shorthand for --tty

`-t` was deprecated in #12813 (Aug. 2015, about 6+ months ago). 

Now remove `--template`'s shorthand `-t` and create a shorthand `-t` for `--tty` in `kubectl run`. 

@kubernetes/kubectl 

Automatic merge from submit-queue

Add protobuf documentation

First commit is in another PR, @wojtek-t review

Automatic merge from submit-queue

Add support for running clusters on GCI

Google Container-VM Image (GCI) is the next revision of Container-VM. See documentation at https://cloud.google.com/compute/docs/containers/vm-image/. This change adds support for starting a Kubernetes cluster using GCI.

With this change, users can start a kubernetes cluster using the latest kubelet and kubectl release binary built in the GCI image by running:

	$ KUBE_OS_DISTRIBUTION="gci" cluster/kube-up.sh

Or run a testing cluster on GCI by running:

	$ KUBE_OS_DISTRIBUTION="gci" go run hack/e2e.go -v --up

The commands above will choose the latest GCI image by default.

Automatic merge from submit-queue

Generated clients can return their RESTClients, RESTClient can return its RateLimiter

cc @lavalamp @krousey @wojtek-t @smarterclayton @timothysc 

Ref. #22421

Automatic merge from submit-queue

Rename PULL_REQUEST_TEMPLATE to PULL_REQUEST_TEMPLATE.md.

## Pull Request Guidelines

1. Please read our [contributor guidelines](https://github.com/kubernetes/kubernetes/blob/master/CONTRIBUTING.md).
1. See our [developer guide](https://github.com/kubernetes/kubernetes/blob/master/docs/devel/development.md).
1. Follow the instructions for [labeling and writing a release note for this PR](https://github.com/kubernetes/kubernetes/blob/master/docs/devel/pull-requests.md#release-notes) in the block below.

```release-note
* Use the release-note-* labels to set the release note state 
* Clear this block to use the PR title as the release note 
-OR-
* Enter your extended release note here
```

Automatic merge from submit-queue

Ignore all _output* directories.

In the new release tooling we build into multiple _output directories and this caught us on a recent build of 1.2.4-beta.0.

ref: #24837 #23839

Automatic merge from submit-queue

Add blocking suites to daily summary

Add a section for blocking suites, and refactor some of the code.

![blocking](https://cloud.githubusercontent.com/assets/940341/14578274/40415830-0338-11e6-8d8e-9434c87a7288.png)


/cc @rmmh 

Automatic merge from submit-queue

Use zone from the test context for static IPs.

Automatic merge from submit-queue

Switch to ABAC authorization from AllowAll

Switch from AllowAll to ABAC. All existing identities (that are created by deployment scripts) are given full permissions through ABAC. Manually created identities will need policies added to the `policy.jsonl` file on the master.

Automatic merge from submit-queue

don't source the kube-env in addon-manager

This was added in https://github.com/kubernetes/kubernetes/commit/2feb658ed7a6a9f9726b04ac4c890b5dee14c0a8 which became unused after #23603 but wasn't removed

Automatic merge from submit-queue

Move install of version handler to genericapiserver

This is to satisfy kbuectl verification

Please review only the last commit.

#19313 #23653
@nikhiljindal @quinton-hoole, @deepak-vij, @XiaoningDing, @alfred-huangjian @mfanjie @huangyuqi @colhom

Automatic merge from submit-queue

rkt: Add post-start hook support.

This adds a poll-and-timeout procedure after the pod is
started, to make sure the post-start hooks execute when the
container is actually running.

This is a temporal workaround for implementing post-hooks,
a long term solution is to use lifecycle event to trigger
those hooks, see https://github.com/kubernetes/kubernetes/issues/23084.

Also this fixes a bug of getting container ID for a non-running
container when running pre-stop hook.


cc @sjpotter @euank @kubernetes/sig-node 

Automatic merge from submit-queue

Trusty: Add debug supports for docker and kubelet

This PR adds debug support in two aspects: (1) For a test cluster, docker command will have "--debug" flag. Recently we noticed that this is very helpful in debug e2e test failures; (2) The kubelet command line will be put in /etc/default/kubelet. If a developer wants to test kubelet flags without recreating a cluster, she/he only needs to revise this file and then run "initctl restart kubelet". In addition, this PR fixes a couple of small things like comments and alignment.

Test result:
(1) Manually verified changing /etc/default/kubelet and run "initctl restart kubelet";
(2) Verified docker command line flag "--debug";
(3) e2e on pure trusty cluster and hybrid cluster all passed.

@roberthbailey @dchen1107 @zmerlynn please review it.

cc/ @yujuhong @fabioy @wonderfly FYI.

Automatic merge from submit-queue

Add kubelet flags for eviction threshold configuration

This PR just adds the flags for kubelet eviction and the associated generated code.

I am happy to tweak text, but we can also do that later at this point in the release.

Since this causes codegen, I wanted to stage this first.

/cc @vishh @kubernetes/sig-node 

Automatic merge from submit-queue

Rename cleanerTuple to cleaner

Rename cleanerTuple to cleaner.
This is a follow up to address: https://github.com/kubernetes/kubernetes/pull/19503#discussion_r49538769

@saad-ali 

Automatic merge from submit-queue

Remove flaky tag from e2e net granular

The tests are running reliably on every origin merge and can be verified
similarly reliable on kube.

As per @jayunit100 on slack/sig-testing:

    ./cluster/kubectl.sh 2>&1 | grep Passed | grep Failed ; done
    SUCCESS! -- 2 Passed | 0 Failed | 0 Pending | 275 Skipped PASS
    SUCCESS! -- 2 Passed | 0 Failed | 0 Pending | 275 Skipped PASS
    SUCCESS! -- 2 Passed | 0 Failed | 0 Pending | 275 Skipped PASS
    SUCCESS! -- 2 Passed | 0 Failed | 0 Pending | 275 Skipped PASS
    SUCCESS! -- 2 Passed | 0 Failed | 0 Pending | 275 Skipped PASS
    SUCCESS! -- 2 Passed | 0 Failed | 0 Pending | 275 Skipped PASS
    SUCCESS! -- 2 Passed | 0 Failed | 0 Pending | 275 Skipped PASS
    SUCCESS! -- 2 Passed | 0 Failed | 0 Pending | 275 Skipped PASS

Automatic merge from submit-queue

Enable resource gatherer in Kubemark tests

Depends on #24291

Ref. #23820

cc @jlowdermilk 

Automatic merge from submit-queue

apiserver latency reductions

Combined effect of these two commits on the latency observed by the 1000-node kubemark benchmark:

```
name                               old ms/op  new ms/op   delta
LIST_nodes_p50                      127 ±16%    121 ± 9%   -4.58%  (p=0.000 n=29+27)
LIST_nodes_p90                      326 ±12%    266 ±12%  -18.48%  (p=0.000 n=29+27)
LIST_nodes_p99                      453 ±11%    400 ±14%  -11.79%  (p=0.000 n=29+28)
LIST_replicationcontrollers_p50    29.4 ±49%   26.2 ±54%     ~     (p=0.085 n=30+29)
LIST_replicationcontrollers_p90    83.0 ±78%   68.6 ±59%  -17.33%  (p=0.013 n=30+28)
LIST_replicationcontrollers_p99     216 ±43%    177 ±49%  -17.68%  (p=0.000 n=29+29)
DELETE_pods_p50                    24.5 ±14%   24.3 ±13%     ~     (p=0.562 n=30+29)
DELETE_pods_p90                    30.7 ± 1%   30.7 ± 1%   -0.30%  (p=0.011 n=29+29)
DELETE_pods_p99                    77.2 ±34%   54.2 ±23%  -29.76%  (p=0.000 n=30+27)
PUT_replicationcontrollers_p50     5.86 ±26%   5.94 ±32%     ~     (p=0.734 n=29+29)
PUT_replicationcontrollers_p90     15.8 ± 7%   15.5 ± 6%   -2.06%  (p=0.010 n=29+29)
PUT_replicationcontrollers_p99     57.8 ±35%   39.5 ±55%  -31.60%  (p=0.000 n=29+29)
PUT_nodes_p50                      14.9 ± 2%   14.8 ± 2%   -0.68%  (p=0.012 n=30+27)
PUT_nodes_p90                      16.5 ± 1%   16.3 ± 2%   -0.90%  (p=0.000 n=27+28)
PUT_nodes_p99                      57.9 ±47%   41.3 ±35%  -28.61%  (p=0.000 n=30+28)
POST_replicationcontrollers_p50    6.35 ±29%   6.34 ±20%     ~     (p=0.944 n=30+28)
POST_replicationcontrollers_p90    15.4 ± 5%   15.0 ± 5%   -2.18%  (p=0.001 n=29+29)
POST_replicationcontrollers_p99    52.2 ±71%   32.9 ±46%  -36.99%  (p=0.000 n=29+27)
POST_pods_p50                      8.99 ±13%   8.95 ±16%     ~     (p=0.903 n=30+29)
POST_pods_p90                      16.2 ± 4%   16.1 ± 4%     ~     (p=0.287 n=29+29)
POST_pods_p99                      30.9 ±21%   26.4 ±12%  -14.73%  (p=0.000 n=28+28)
POST_bindings_p50                  9.34 ±12%   8.92 ±15%   -4.54%  (p=0.013 n=30+28)
POST_bindings_p90                  16.6 ± 1%   16.5 ± 3%   -0.73%  (p=0.017 n=28+29)
POST_bindings_p99                  23.5 ± 9%   21.1 ± 4%  -10.09%  (p=0.000 n=27+28)
PUT_pods_p50                       10.8 ±11%   10.2 ± 5%   -5.47%  (p=0.000 n=30+27)
PUT_pods_p90                       16.1 ± 1%   16.0 ± 1%   -0.64%  (p=0.000 n=29+28)
PUT_pods_p99                       23.4 ± 9%   20.9 ± 9%  -10.93%  (p=0.000 n=28+27)
DELETE_replicationcontrollers_p50  2.42 ±16%   2.50 ±13%     ~     (p=0.054 n=29+28)
DELETE_replicationcontrollers_p90  11.5 ±12%   11.8 ±13%     ~     (p=0.141 n=30+28)
DELETE_replicationcontrollers_p99  19.5 ±21%   19.1 ±21%     ~     (p=0.397 n=29+29)
GET_nodes_p50                      0.77 ±10%   0.76 ±10%     ~     (p=0.317 n=28+28)
GET_nodes_p90                      1.20 ±16%   1.14 ±24%   -4.66%  (p=0.036 n=28+29)
GET_nodes_p99                      11.4 ±48%    7.5 ±46%  -34.28%  (p=0.000 n=28+29)
GET_replicationcontrollers_p50     0.74 ±17%   0.73 ±17%     ~     (p=0.222 n=30+28)
GET_replicationcontrollers_p90     1.04 ±25%   1.01 ±27%     ~     (p=0.231 n=30+29)
GET_replicationcontrollers_p99     12.1 ±81%  10.0 ±145%     ~     (p=0.063 n=28+29)
GET_pods_p50                       0.78 ±12%   0.77 ±10%     ~     (p=0.178 n=30+28)
GET_pods_p90                       1.06 ±19%   1.02 ±19%     ~     (p=0.120 n=29+28)
GET_pods_p99                       3.92 ±43%   2.45 ±38%  -37.55%  (p=0.000 n=27+25)
LIST_services_p50                  0.20 ±13%   0.20 ±16%     ~     (p=0.854 n=28+29)
LIST_services_p90                  0.28 ±15%   0.27 ±14%     ~     (p=0.219 n=29+28)
LIST_services_p99                  0.49 ±20%   0.47 ±24%     ~     (p=0.140 n=29+29)
LIST_endpoints_p50                 0.19 ±14%   0.19 ±15%     ~     (p=0.709 n=29+29)
LIST_endpoints_p90                 0.26 ±16%   0.26 ±13%     ~     (p=0.274 n=29+28)
LIST_endpoints_p99                 0.46 ±24%   0.44 ±21%     ~     (p=0.111 n=29+29)
LIST_horizontalpodautoscalers_p50  0.16 ±15%   0.15 ±13%     ~     (p=0.253 n=30+27)
LIST_horizontalpodautoscalers_p90  0.22 ±24%   0.21 ±16%     ~     (p=0.152 n=30+28)
LIST_horizontalpodautoscalers_p99  0.31 ±33%   0.31 ±38%     ~     (p=0.817 n=28+29)
LIST_daemonsets_p50                0.16 ±20%   0.15 ±11%     ~     (p=0.135 n=30+27)
LIST_daemonsets_p90                0.22 ±18%   0.21 ±25%     ~     (p=0.135 n=29+28)
LIST_daemonsets_p99                0.29 ±28%   0.29 ±32%     ~     (p=0.606 n=28+28)
LIST_jobs_p50                      0.16 ±16%   0.15 ±12%     ~     (p=0.375 n=29+28)
LIST_jobs_p90                      0.22 ±18%   0.21 ±16%     ~     (p=0.090 n=29+26)
LIST_jobs_p99                      0.31 ±28%   0.28 ±35%  -10.29%  (p=0.005 n=29+27)
LIST_deployments_p50               0.15 ±16%   0.15 ±13%     ~     (p=0.565 n=29+28)
LIST_deployments_p90               0.22 ±22%   0.21 ±19%     ~     (p=0.107 n=30+28)
LIST_deployments_p99               0.31 ±27%   0.29 ±34%     ~     (p=0.068 n=29+28)
LIST_namespaces_p50                0.21 ±25%   0.21 ±26%     ~     (p=0.768 n=29+27)
LIST_namespaces_p90                0.28 ±29%   0.26 ±25%     ~     (p=0.101 n=30+28)
LIST_namespaces_p99                0.30 ±48%   0.29 ±42%     ~     (p=0.339 n=30+29)
LIST_replicasets_p50               0.15 ±18%   0.15 ±16%     ~     (p=0.612 n=30+28)
LIST_replicasets_p90               0.22 ±19%   0.21 ±18%   -5.13%  (p=0.011 n=28+27)
LIST_replicasets_p99               0.31 ±39%   0.28 ±29%     ~     (p=0.066 n=29+28)
LIST_persistentvolumes_p50         0.16 ±23%   0.15 ±21%     ~     (p=0.124 n=30+29)
LIST_persistentvolumes_p90         0.21 ±23%   0.20 ±23%     ~     (p=0.092 n=30+25)
LIST_persistentvolumes_p99         0.21 ±24%   0.20 ±23%     ~     (p=0.053 n=30+25)
LIST_resourcequotas_p50            0.16 ±12%   0.16 ±13%     ~     (p=0.175 n=27+28)
LIST_resourcequotas_p90            0.20 ±22%   0.20 ±24%     ~     (p=0.388 n=30+28)
LIST_resourcequotas_p99            0.22 ±24%   0.22 ±23%     ~     (p=0.575 n=30+28)
LIST_persistentvolumeclaims_p50    0.15 ±21%   0.15 ±29%     ~     (p=0.079 n=30+28)
LIST_persistentvolumeclaims_p90    0.19 ±26%   0.18 ±34%     ~     (p=0.446 n=29+29)
LIST_persistentvolumeclaims_p99    0.19 ±26%   0.18 ±34%     ~     (p=0.446 n=29+29)
LIST_pods_p50                      68.0 ±16%   56.3 ± 9%  -17.19%  (p=0.000 n=29+28)
LIST_pods_p90                       119 ±19%     93 ± 8%  -21.88%  (p=0.000 n=28+28)
LIST_pods_p99                       230 ±18%    202 ±14%  -12.13%  (p=0.000 n=27+28)
```

Automatic merge from submit-queue

Add traces to scheduler

This will simplify performance debugging of scheduler.

@davidopp @hongchaodeng @xiang90 

Automatic merge from submit-queue

Quota ignores pod compute resources on updates

Scenario:

1. define a quota Q that tracks memory and cpu
2. create pod P that uses memory=100Mi, cpu=100m
3. update pod P to use memory=50Mi,cpu=10m

Expected Results:
Step 3 should fail with validation error.
Quota Q should not have changed.

Actual Results:
Step 3 fails validation, but quota Q is decremented to have memory usage down 50Mi and cpu usage down 40m.  This is because the quota was getting updated even though the pod was going to fail validation.

Fix:
Quota should only support modifying pod compute resources when pods themselves support modifying their compute resources.

This also fixes https://github.com/kubernetes/kubernetes/issues/24352

/cc @smarterclayton - this is what we discussed.

fyi: @kubernetes/rh-cluster-infra 

Automatic merge from submit-queue

Reduce LOC in third party controller test

Extract method refactor to make this test a little more readable

@kubernetes/sig-api-machinery 

Automatic merge from submit-queue

Improve error messages in jwt_test.go

Automatic merge from submit-queue

only close new ports upon iptables-restore failure

fixes: #24357

Make DrainOptions.getPodsForDeletion public to be reused in ClusterAutoscaler

Automatic merge from submit-queue

Fix DeletingLoadBalancer event generation.

release-note:
```release-note
Fix DeletingLoadBalancer event generation.
```

Also patch up a log statement to log the service's namespace and name
rather than potentially just logging `kind:"" apiVersion:""`.

It doesn't work as written because `service` is always nil on this line. If it wasn't, then we wouldn't have taken this branch. `deltaService` is what contains the info that we have about the service.

Example of how the event generation fails as currently written:
```
E0426 22:18:41.726835      10 event.go:257] Could not construct reference to: '&api.Service{TypeMeta:unversioned.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:api.ObjectMeta{Name:"", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:unversioned.Time{Time:time.Time{sec:0, nsec:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*unversioned.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil)}, Spec:api.ServiceSpec{Type:"", Ports:[]api.ServicePort(nil), Selector:map[string]string(nil), ClusterIP:"", ExternalIPs:[]string(nil), LoadBalancerIP:"", SessionAffinity:""}, Status:api.ServiceStatus{LoadBalancer:api.LoadBalancerStatus{Ingress:[]api.LoadBalancerIngress(nil)}}}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Normal' 'DeletingLoadBalancer' 'Deleting load balancer'
```

Automatic merge from submit-queue

Allow KUBE_ROOT to be defined externally

Automatic merge from submit-queue

Initial kube-up support for VMware's Photon Controller

This is for: https://github.com/kubernetes/kubernetes/issues/24121

Photon Controller is an open-source cloud management platform. More
information is available at:
http://vmware.github.io/photon-controller/

This commit provides initial support for Photon Controller. The
following features are tested and working:
- kube-up and kube-down
- Basic pod and service management
- Networking within the Kubernetes cluster
- UI and DNS addons

It has been tested with a Kubernetes cluster of up to 10
nodes. Further work on scaling is planned for the near future.

Internally we have implemented continuous integration testing and will
run it multiple times per day against the Kubernetes master branch
once this is integrated so we can quickly react to problems.

A few things have not yet been implemented, but are planned:
- Support for kube-push
- Support for test-build-release, test-setup, test-teardown

Assuming this is accepted for inclusion, we will write documentation
for the kubernetes.io site.

We have included a script to help users configure Photon Controller
for use with Kubernetes. While not required, it will help some
users get started more quickly. It will be documented.

We are aware of the kube-deploy efforts and will track them and
support them as appropriate.