Automatic merge from submit-queue

Define interfaces for kubelet pod admission and eviction

There is too much code and logic in `kubelet.go` that makes it hard to test functions in discrete pieces.

I propose an interface that an internal module can implement that will let it make an admission decision for a pod.  If folks are ok with the pattern, I want to move the a) predicate checking, b) out of disk, c) eviction preventing best-effort pods being admitted into their own dedicated handlers that would be easier for us to mock test.  We can then just write tests to ensure that the `Kubelet` calls a call-out, and we can write easier unit tests to ensure that dedicated handlers do the right thing.

The second interface I propose was a `PodEvictor` that is invoked in the main kubelet sync loop to know if pods should be pro-actively evicted from the machine.  The current active deadline check should move into a simple evictor implementation, and I want to plug the out of resource killer code path as an implementation of the same interface.

 @vishh @timothysc - if you guys can ack on this, I will add some unit testing to ensure we do the call-outs.

/cc @kubernetes/sig-node @kubernetes/rh-cluster-infra 

Automatic merge from submit-queue

Avoid allocations and a reflect.Call in conversion

reflect.Call is fairly expensive, performing 8 allocations and having to
set up a call stack. Using a fairly straightforward to generate switch
statement, we can bypass that early in conversion (as long as the
function takes responsibility for invocation). We may also be able to
avoid an allocation for the conversion scope, but not positive yet.

```
benchmark                    old ns/op     new ns/op     delta
BenchmarkPodConversion-8     14713         12173         -17.26%

benchmark                    old allocs     new allocs     delta
BenchmarkPodConversion-8     80             72             -10.00%

benchmark                    old bytes     new bytes     delta
BenchmarkPodConversion-8     9133          8712          -4.61%
```

@wojtek-t related to #20309

Automatic merge from submit-queue

kubenet: fix up CNI bridge TX queue length if needed

CNI's bridge plugin mis-handles the TxQLen when creating the bridge,
leading to a zero-length TX queue.  This doesn't typically cause
problems (since virtual interfaces don't have hard queue limits)
but when adding traffic shaping, some qdiscs pull their packet
limits from the TX queue length, leading to a packet limit of 0
in some cases.  Until we can depend on a new enough version of
CNI, fix up the TX queue length internally.

Closes: https://github.com/kubernetes/kubernetes/issues/25092

Automatic merge from submit-queue

e2e: Enable persistent volume test

The test is already there and all packages should be already available on all test machines.
    
It tests:

- binding
- using bound claim in a pod
- recycling NFS volume

(we should see shortly if all nfs packages are really installed as Jenkins tests it...)

Automatic merge from submit-queue

Update e2e-runner.sh so it can fetch multiple types of Trusty images

Trusty beta images now work with k8s 1.2.

@spxtr, @andyzheng0831 Can you review this?

Automatic merge from submit-queue

Remove nodeName from predicate signature.

With this approach, I'm getting the initial throughput (in empty cluster) in 1000-node cluster of ~95pods/s.
Which is ~30% improvement.

@kubernetes/sig-scalability 

Automatic merge from submit-queue

Kubelet: Cleanup with new engine api

Finish step 2 of #23563

This PR:
1) Cleanup go-dockerclient reference in the code.
2) Bump up the engine-api version.
3) Cleanup the code with new engine-api.

Fixes #24076.
Fixes #23809.

/cc @yujuhong 

Automatic merge from submit-queue

Add quantity benchmarks

Fix proto configs

Inter pod topological affinity and anti-affinity implementation

Fix codecgen sort

Automatic merge from submit-queue

Jenkins: Clean up even if we can't bring cluster up

We're gathering all the cluster logs, go ahead and clean up the
resources.

Automatic merge from submit-queue

fix #24937:  flake pod not found

https://github.com/kubernetes/kubernetes/issues/24937
Split from #24191
Signed-off-by: liang chenye <liangchenye@huawei.com>

Automatic merge from submit-queue

Delete pod with uid as precondition.

Addressed https://github.com/kubernetes/kubernetes/issues/25169#issuecomment-217033202.

Fix #25169 
Fix #24937

This PR change status manager to delete pods with uid as a precondition, so that kubelet won't delete pods with different uid but the same name and namespace accidentally.

/cc @yujuhong 

Automatic merge from submit-queue

start etcd compactor in background

ref: #22448

What's in this PR?
- StartCompactor starts a compactor in the background in order to compact keys older than fixed time. We need to compact keys because we can't let on disk data grow forever. We save the most recent 10 minutes data. It should be enough for slow watchers and to tolerate burst. We might keep a longer history (12h) in the future once storage API can take advantage of multi-version key.
- Have only one compaction job for each cluster. Use endpoints from user input to differentiate clusters.

Automatic merge from submit-queue

API changes for Cascading deletion 

This PR includes the necessary API changes to implement cascading deletion with finalizers as proposed is in #23656. Comments are welcome.

@lavalamp @derekwaynecarr @bgrant0607 @rata @hongchaodeng 

Automatic merge from submit-queue

Allow etcd to store protobuf objects

Split storage serialization from client negotiation, and allow API server to take flag controlling serialization.

TODO:

* [x] API server still doesn't start - range allocation object doesn't seem to round trip correctly to etcd
* [ ] Verify that third party resources are ignoring protobuf (add a test)
* [ ] Add integration tests that verify storage is correctly protobuf
* [ ] Add a global default for which storage format to prefer?

Automatic merge from submit-queue

fix log message for self-signed cert generation

Automatic merge from submit-queue

Add a better error message to run.sh

I ran build/run.sh w/o any args (by mistake) and it just said
   `Invalid input.`
after several other steps. I had no idea whether I was doing something
wrong or if my env was busted. Clearly, I just forget to include the
command that run.sh was to invoke in the Docker container.  But it took
me time to go track down where this error came from and why. So to help
others I just tweaked the error message to be:
   `Invalid input - please specify a command to run.`

Very minor thing,I know, but if it helps others...
Signed-off-by: Doug Davis <dug@us.ibm.com>

Automatic merge from submit-queue

Reduce kubelet LOC: extract getters

Step 1 of #25028 as discussed in @kubernetes/sig-node meeting

Automatic merge from submit-queue

Map secret files into dockerized e2e

Rather than copying the GCE and AWS private keys/credentials to each Jenkins VM, we can put them in credentials and map them through.

This is one half of the change; if the relevant environment variables are set, we'll mount the files in.

cc @fejta @rmmh @apelisse 

Don't surround node-tags list with square brackets

That's not how yaml list parsing works...

Actually populate nodeTags field in GCE provider struct

I ran build/run.sh w/o any args (by mistake) and it just said
   `Invalid input.`
after several other steps. I had no idea whether I was doing something
wrong or if my env was busted. Clearly, I just forget to include the
command that run.sh was to invoke in the Docker container.  But it took
me time to go track down where this error came from and why. So to help
others I just tweaked the error message to be:
   `Invalid input - please specify a command to run.`

Very minor thing,I know, but if it helps others...
Signed-off-by: Doug Davis <dug@us.ibm.com>

Automatic merge from submit-queue

Fix v1.3.0-alpha.3 CHANGELOG entry

We got a double entry in #24942; my bad.  Another example of why automation > humans.  👍 

Automatic merge from submit-queue

GCI: Add two GCI specific metadata pairs

This PR adds two GCI specific metadata pairs when using GCI image.

(1) "gci-update-strategy": by default the GCI in-place updater is enabled. It means that when a new image is released, the instance on the old image will be upgraded to the new image. In this change, we turn it off;

(2) "gci-ensure-gke-docker": GCI is built with two versions of docker. When this metadata is set to "true", the version satisfying kubernetes qualification will be used. Setting this metadata prevents from using incorrect docker version.

Automatic merge from submit-queue

e2e/framework/util.StartPods: don't wait for pods that are not created

When running ``[k8s.io] SchedulerPredicates [Serial] validates resource limits of pods that are allowed to run [Conformance]`` pods can be created in a way in which additional pods have to be create to fully saturate node's capacity CPU in a cluster. Additional pods are created by calling ``framework.StartPods``. The function creates pods with a given label and waits for them (if ``waitForRunning`` is ``true``). This is fine as long as the number of pods to created is non-zero. If there are zero pods to be created and ``waitForRunning`` is ``true``, the function waits forever as there is not going to be any pods with requested label. Thus, resulting in ``Error waiting for 0 pods to be running - probably a timeout``. Causing the e2e test to fail even if it should not.

Adding condition to return from the function immediately if there is not pod to create.

The codec factory should support two distinct interfaces - negotiating
for a serializer with a client, vs reading or writing data to a storage
form (etcd, disk, etc). Make the EncodeForVersion and DecodeToVersion
methods only take Encoder and Decoder, and slight refactoring elsewhere.

In the storage factory, use a content type to control what serializer to
pick, and use the universal deserializer. This ensures that storage can
read JSON (which might be from older objects) while only writing
protobuf. Add exceptions for those resources that may not be able to
write to protobuf (specifically third party resources, but potentially
others in the future).