stage2: tunable squashfs compression

The existing implementation would handle kernel differences just fine but a bit too automatically: if it sees xz support, that's what will end up being used (and if there's -Xbcj binary compression filter available for the target platform, it will be applied unequivocally either). It's perfectly suitabe for getting fine-tuned release images but is also a bit too resource-consuming while developing the image configuration which has no business with its compression. The one and only knob is SQUASHFS (see doc/variables.txt); to give an idea of the differences, here are some numbers for a mostly-binary (43% as per 99-elf-stats) webkiosk livecd and a rather less so (18%) flightgear one on a dual quad-core X5570 node (each mksquashfs run used up all the cores): SQUASHFS | live-webkiosk.iso | live-flightgear.iso ---------+-------------------+--------------------- fast | 3:30 / 130M | 5:11 / 852M normal * | 3:37 / 100M | 5:35 / 688M tight | 3:50 / 98M | 6:47 / 683M Thus if the knob isn't fiddled with, the defaults will allow for a reasonably fast build of a pretty slim image; if one is building a release or if a particular image is very sensitive being close to the media capacity then just add SQUASHFS=tight and see it a percent or two down on size. Please note that lzo/gzip-compressed images are also quicker to uncompress thus further helping with test iterations. Thanks to led@ and glebfm@ for helpful hints and questions.

stage2: tunable squashfs compression
fe58c46e · Michael Shigorin · 0cc20d10 · fe58c46e · fe58c46e · fe58c46e
Commit fe58c46e authored Apr 08, 2012 by Michael Shigorin
Hide whitespace changes
Inline Side-by-side

Showing with 44 additions and 17 deletions

variables.txt doc/variables.txt +8 -0

03-test-kernel features.in/stage2/stage1/scripts.d/03-test-kernel +33 -17

Makefile sub.in/stage1/Makefile +3 -0

No files found.
--- a/doc/variables.txt
+++ b/doc/variables.txt
@@ -81,6 +81,14 @@
  + значение: пусто (по умолчанию) либо любая строка
  + см. ../image.in/Makefile

+- SQUASHFS
+  + определяет характер сжатия squashfs для stage2
+  + значение:
+    - пусто (по умолчанию) либо normal: xz
+    - tight: xz с -Xbcj по платформе (лучше, но дольше -- подбор в два прохода)
+    - fast: gzip/lzo (быстрее запаковывается и распаковывается, меньше степень)
+  + см. ../features.in/stage2/stage1/scripts.d/03-test-kernel
+
 пример
 ~~~~~~
 make DEBUG=1 CLEAN=1 distro/syslinux.iso
--- a/features.in/stage2/stage1/scripts.d/03-test-kernel
+++ b/features.in/stage2/stage1/scripts.d/03-test-kernel
 #!/bin/sh
+# NB: -f prohibited in this shebang
 # check relevant kernel features availability

+CONFIGS=/boot/config-*
 GZ_OPTS="-comp gzip"
 XZ_OPTS="-comp xz -b 262144 -noI"
+LZO_OPTS="-comp lzo"

-# test for installer-required filesystems support
-for opt in CONFIG_SQUASHFS CONFIG_AUFS_FS; do
-	if grep -q "^$opt=[my]$" /boot/config-*; then
-		[ -n "$GLOBAL_VERBOSE" ] && echo "** $opt available"
+verbose() { [ -n "$GLOBAL_VERBOSE" ] && echo "** $@" >&2; }
+fatal() { echo "** error: $@" >&2; exit 1; }
+
+configured()
+{
+	local option="CONFIG_$1"
+	if grep -q "^$option" $CONFIGS; then
+		verbose "${option%=*} available"
+		return 0
 	else
-		if [ "$?" = 1 ]; then	# a file exists but lacks the pattern
-			echo "** error: stage1 kernel must have $opt support" >&2
-			exit 1
-		fi
+		return 1
 	fi
+}
+
+ls $CONFIGS >&/dev/null || fatal "no stage1 kernel config found in /boot"
+
+# test for installer-required filesystems support
+for opt in SQUASHFS AUFS_FS; do
+	configured "$opt=[my]" || fatal "stage1 kernel must have $opt support"
 done

+verbose "GLOBAL_SQUASHFS: $GLOBAL_SQUASHFS"
+
 # squashfs options: not really neccessary but better than none
 # NB: this config file should be carried over into install2
-if grep -q '^CONFIG_SQUASHFS_XZ=y$' /boot/config-*; then
-	# TODO: figure out if it's generally worth it even on x86:
-	#       if binaries account for less than ~70% of blocks,
-	#       the decompression filter overhead might hurt
+if [ "$GLOBAL_SQUASHFS" != "fast" ] && configured "SQUASHFS_XZ=y"; then
+	# NB: the decompression filter overhead might hurt
 	# NB: there are arm, powerpc and some other filters too
-	if grep -q "^CONFIG_X86" /boot/config-*; then
+	if [ "$GLOBAL_SQUASHFS" = "tight" ] && configured "X86"; then
 		XZ_OPTS="$XZ_OPTS -Xbcj x86"
 	fi
-	echo "PACK_SQUASHFS_OPTS=$XZ_OPTS" > /.image/squashcfg.mk
-else
-	echo "PACK_SQUASHFS_OPTS=$GZ_OPTS" > /.image/squashcfg.mk
-fi
+	echo "PACK_SQUASHFS_OPTS=$XZ_OPTS"
+else	# go fast, or even faster if possible
+	if configured "SQUASHFS_LZO=y"; then
+		echo "PACK_SQUASHFS_OPTS=$LZO_OPTS"
+	else	# safe default
+		echo "PACK_SQUASHFS_OPTS=$GZ_OPTS"
+	fi
+fi > /.image/squashcfg.mk
--- a/sub.in/stage1/Makefile
+++ b/sub.in/stage1/Makefile
@@ -38,6 +38,9 @@ endif
 CHROOT_PACKAGES_REGEXP += $(STAGE1_PACKAGES_REGEXP)
 CHROOT_PACKAGES = $(STAGE1_PACKAGES) $(SYSTEM_PACKAGES)

+# pass for use/stage2
+GLOBAL_SQUASHFS := $(SQUASHFS)
+
 # scripts prepare bootloader configuration, too
 # NB: we pass tested squashfs options for ../stage2/Makefile to include
 all: | debug prepare-workdir copy-tree run-scripts $(BUILD_PROPAGATOR) \