• Kubernetes Submit Queue's avatar
    Merge pull request #39199 from deimosfr/cassandra_exemple_fix_shutdown · c08758e9
    Kubernetes Submit Queue authored
    Automatic merge from submit-queue (batch tested with PRs 39199, 37273, 29183, 39638, 40199)
    
    fixing Cassandra shutdown example to avoid data corruption
    
    Hi,
    
    I was playing with Cassandra example stored in the Kubernetes project and I encountered issues on shutdown (not anytime). After checking it looks like the shutdown of a node is brutal and data corruption may occur during a flush on disk. To avoid that, I'm suggesting a hook to gracefully shutdown Cassandra before stopping the container.
    
    Here are logs of corruption after a pod delete:
    ```
    /10.2.76.4:[-8699848499000118463, -8567123670484406873, -8496767951391579058, -8426990834929543369, -7697118318683556771, -6942779781591907873, -6795880495022459877, -6496399078175245235, -5450122121479522544, -5002551029990001224, -4914532712178218138, -4884518674849288097, -3667338763252443465, -3316742521554936832, -2844544359955291760, -1291351295404368159, -794348397160283083, -705240847455001090, -652995206518489298, -284127251294286231, 173240967232234690, 616476682204879844, 826670457841382100, 1815369334084765465, 4431706613761077084, 4743606016174161647, 5637469692783959686, 5802957011124852712, 6759688243703331970, 7679657413128857702, 7713766696628426028, 9098158217036036188]
    
    ERROR 16:23:06 Exception in thread Thread[CompactionExecutor:2,1,main]
    org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted: /cassandra_data/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/mc-2-big-Data.db
    	at org.apache.cassandra.io.sstable.format.big.BigTableScanner$KeyScanningIterator.computeNext(BigTableScanner.java:351) ~[apache-cassandra-3.9.jar:3.9]
    	at org.apache.cassandra.io.sstable.format.big.BigTableScanner$KeyScanningIterator.computeNext(BigTableScanner.java:265) ~[apache-cassandra-3.9.jar:3.9]
    	at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.9.jar:3.9]
    	at org.apache.cassandra.io.sstable.format.big.BigTableScanner.hasNext(BigTableScanner.java:245) ~[apache-cassandra-3.9.jar:3.9]
    	at org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:374) ~[apache-cassandra-3.9.jar:3.9]
    	at org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:186) ~[apache-cassandra-3.9.jar:3.9]
    	at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:155) ~[apache-cassandra-3.9.jar:3.9]
    	at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.9.jar:3.9]
    	at org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$2.hasNext(UnfilteredPartitionIterators.java:150) ~[apache-cassandra-3.9.jar:3.9]
    	at org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:92) ~[apache-cassandra-3.9.jar:3.9]
    	at org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:232) ~[apache-cassandra-3.9.jar:3.9]
    	at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:184) ~[apache-cassandra-3.9.jar:3.9]
    	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-3.9.jar:3.9]
    	at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:82) ~[apache-cassandra-3.9.jar:3.9]
    	at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60) ~[apache-cassandra-3.9.jar:3.9]
    	at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:264) ~[apache-cassandra-3.9.jar:3.9]
    
    ```
    
    It works well for me now and do not have data corruption anymore.
    c08758e9
Name
Last commit
Last update
..
apiserver Loading commit data...
cluster-dns Loading commit data...
cockroachdb Loading commit data...
elasticsearch Loading commit data...
explorer Loading commit data...
guestbook Loading commit data...
guestbook-go Loading commit data...
https-nginx Loading commit data...
javaee Loading commit data...
javaweb-tomcat-sidecar Loading commit data...
job Loading commit data...
k8petstore Loading commit data...
kubectl-container Loading commit data...
meteor Loading commit data...
mysql-cinder-pd Loading commit data...
mysql-wordpress-pd Loading commit data...
newrelic Loading commit data...
nodesjs-mongodb Loading commit data...
openshift-origin Loading commit data...
persistent-volume-provisioning Loading commit data...
phabricator Loading commit data...
podsecuritypolicy/rbac Loading commit data...
runtime-constraints Loading commit data...
selenium Loading commit data...
sharing-clusters Loading commit data...
spark Loading commit data...
storage Loading commit data...
storm Loading commit data...
sysdig-cloud Loading commit data...
volumes Loading commit data...
BUILD Loading commit data...
OWNERS Loading commit data...
README.md Loading commit data...
doc.go Loading commit data...
examples_test.go Loading commit data...
guidelines.md Loading commit data...
pod Loading commit data...
scheduler-policy-config-with-extender.json Loading commit data...
scheduler-policy-config.json Loading commit data...
simple-nginx.md Loading commit data...