Skip to content

K
k3s
Commit 06c610e2 authored by Kubernetes Submit Queue's avatar Kubernetes Submit Queue Committed by GitHub
Browse files
Options

Merge pull request #39949 from Crassirostris/fluentd-liveness-probe-fix

Automatic merge from submit-queue (batch tested with PRs 38592, 39949, 39946, 39882) Remove fluentd buffers if fluentd is stuck Fluentd now stores its buffers on disk for the resiliency. However, if buffer is corrupted, fluentd will be restarting forever. Following change will make fluentd liveness probe delete buffers if fluentd is stuck for more than X minutes (15 by default).
parents 4811ad02 edf1ffc0
Hide whitespace changes
Inline Side-by-side
Showing with 15 additions and 2 deletions
cluster/addons/fluentd-gcp/fluentd-gcp-ds.yaml
View file @ 06c610e2
......@@ -57,10 +57,23 @@ spec:
- '/bin/sh'
- '-c'
- >
LIVENESS_THRESHOLD_SECONDS=${LIVENESS_THRESHOLD_SECONDS:-600};
LIVENESS_THRESHOLD_SECONDS=${LIVENESS_THRESHOLD_SECONDS:-300};
STUCK_THRESHOLD_SECONDS=${LIVENESS_THRESHOLD_SECONDS:-900};
if [ ! -e /var/log/fluentd-buffers ];
then
exit 1;
fi;
LAST_MODIFIED_DATE=`stat /var/log/fluentd-buffers | grep Modify | sed -r "s/Modify: (.*)/\1/"`;
LAST_MODIFIED_TIMESTAMP=`date -d "$LAST_MODIFIED_DATE" +%s`;
if [ `date +%s` -gt `expr $LAST_MODIFIED_TIMESTAMP + $LIVENESS_THRESHOLD_SECONDS` ]; then exit 1; fi;
if [ `date +%s` -gt `expr $LAST_MODIFIED_TIMESTAMP + $STUCK_THRESHOLD_SECONDS` ];
then
rm -rf /var/log/fluentd-buffers;
exit 1;
fi;
if [ `date +%s` -gt `expr $LAST_MODIFIED_TIMESTAMP + $LIVENESS_THRESHOLD_SECONDS` ];
then
exit 1;
fi;
nodeSelector:
alpha.kubernetes.io/fluentd-ds-ready: "true"
terminationGracePeriodSeconds: 30
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment