- massive squashfs tuning

sub.in/stage2/image-scripts.d/99cleanup-boot,
livecd-install need /boot files too

glebfm@ asked what to do with new package lists: whether these
belong to features, or to distributions themselves.  This question
is actually open and up for discussion but there are guidelines
that can and should be written down already; and so they were.

Added pkgdups utility reference as well.

Multiple ARCHES won't just magically work without
the ability to figure out the correct apt.conf;
fortunately there's just the right example handy
in profiles.mk.sample already.

Thanks glebfm@ for feedback.

Looks like the 128k default block size is pretty well chosen:
it saves ~6% of image size compared to 64k, and subsequent
differences are ~3% per doubling the block size up to 1M
(thanks led@ for carrying out the tests).

So we'll stick with 256k for "normal" xz compression (inodes
uncompressed) and get 512k back for "tight" one (compressed).
The runtime performance issues are to be examined yet when
bootchart or the like is deployed, nothing drastic though.

With "fast" (gzip/lzo) squash compression inodes go unmolested.

For the record, tight live-webkiosk builds as 95M image in 3:40,
and tight live-flightgear.iso builds as 669M image in 6:34.  Nice.

There's no much sense going for 1M block size: e.g. live-webkiosk
would drop to 93M (3:46) but its load time would increase up to
2:07 as compared to 1:48 for -b 524288 and 1:42 for -b 262144 -noI
on a Duron 500/512M system given the very same DVD+RW media.

If there's no dot(1) then its input file should just be stored;
thanks glebfm@ for reminding about this.

Should be removed when console autologin is there
(mind the live-builder).

The existing implementation would handle kernel differences
just fine but a bit too automatically: if it sees xz support,
that's what will end up being used (and if there's -Xbcj binary
compression filter available for the target platform, it will
be applied unequivocally either).

It's perfectly suitabe for getting fine-tuned release images
but is also a bit too resource-consuming while developing the
image configuration which has no business with its compression.

The one and only knob is SQUASHFS (see doc/variables.txt);
to give an idea of the differences, here are some numbers
for a mostly-binary (43% as per 99-elf-stats) webkiosk livecd
and a rather less so (18%) flightgear one on a dual quad-core
X5570 node (each mksquashfs run used up all the cores):

SQUASHFS | live-webkiosk.iso | live-flightgear.iso
---------+-------------------+---------------------
fast     | 3:30 / 130M       | 5:11 / 852M
normal * | 3:37 / 100M       | 5:35 / 688M
tight    | 3:50 / 98M        | 6:47 / 683M

Thus if the knob isn't fiddled with, the defaults will allow
for a reasonably fast build of a pretty slim image; if one is
building a release or if a particular image is very sensitive
being close to the media capacity then just add SQUASHFS=tight
and see it a percent or two down on size.

Please note that lzo/gzip-compressed images are also quicker
to uncompress thus further helping with test iterations.

Thanks to led@ and glebfm@ for helpful hints and questions.

APM enabled notebooks would usually hibernate to
a partition of special type and special format;
thus to make use of this APM BIOS feature folks
might need a corresponding formatter.

This kind of test was proposed by led@ to gather statistics
on chroot's contents going to become squashfs (the script
optimizations lowering added overhead from ~10 sec down
to a subsecond range were also proposed by him).

Intentionally not documented in doc/variables.txt due to
the rather lowlevel nature of the probe (at least so far).

The knobs involved are SQUASHFS (the additional effort kicks
in only for "tight" case) and GLOBAL_SQUASHFS_SORT (must be
non-empty for this extra overhead to occur).

Additional experimentation is needed to find out whether
the difference in squashfs size and performance is worth
the trouble (seems the impact is non-zero but pretty minor).

gns@ has 512k, led@ suggests that it's too much a bit;
let's try 256k.

bin/ prefix tried to become a permatemp(tm).  No way.

Thanks Chernetsky Andrey for testing and suggesting it.
(NB: currently only t6 build is proper, see input-utils bugs)

There is at least one known deficiency for mkimage-profiles:
build.log will be truncated if verbose mode is enabled and
hasher version is lower than 1.3.22.

The check is done here since it's where the logging is arranged,
and doing it in image.in/Makefile would result in the warning
about log-truncating software being truncated by the said software.

Thanks Max Kosmach for reporting this inobviousity.

- better live-webkiosk and initial live-flightgear
- cleanup, syslinux, xorg feature tweaks

Status updates and readability tweaks all over the place.

A very basic server image with sshd onboard for testing
and very basic installs; thanks legion@ for the query.

The output was still somewhat ragged in 80x24 terminal window
with fmt(1) which wasn't anticipating the word length difference
subsequent column(1) would have to cope with later on.

Thanks Loic Cattani for his shell columnizer implementation:
https://github.com/Arko/Columnize

Thanks snejok@ for spotting the missing, I didn't get around
to tests with headphones...

Also fixed nouveau getting in after target shuffling,
and tweaked firefox homepage to be useful in this context.

There are three more variables to adjust the redirect page:
HOMEPAGE, HOMENAME and HOMEWAIT; e.g. live-webkiosk is now
quite a bit more whitelabel.

- incompatible change (to fix the rather broken early style):
  use/syslinux/ui-% is now use/syslinux/ui/%;

- default timeout changed to 9 seconds (long enough and keeps
  the countdown in a single figure);

- added totaltimeout of 300 seconds;

- provided live kiosk images with almost-instant boot by default;

...and some other assorted tweaks here and there, sorry.

Thanks to a reviewer who came with useful feedback and a goal:
http://www.opennet.ru/openforum/vsluhforumID3/83728.html#136
the live-webkiosk image got forked into a separate one:

- dropped DRI, virtualbox GA, mc & co, docs, rpmdb;
- added Russian keyboard layout (ctrl+shift to toggle);
- rebased live-webkiosk onto live-webkiosk-mini ;-)

Maybe vbox guest additions will get back but rpmdb is a bit
impractical on a kiosk squashfs image, even in presence of
aufs rw overlay.

Now is the time for all fonts to be pulled in when needed and not
along with the X server and hardware drivers; tablet support is
moved to a (preexisting) specific target either.

There's no need now to arch-discriminate a few older drivers too.

There's much reason for reuse instead of duplication
among the different stage2-based subprofiles.

In particular, the rather monolithic driver cleanup script
of the ancient is better done in several clear pieces with
the final depmod run.

Scripts dropping apt/rpm databases will dump pkglist first.

A script purging /boot/* will honour live-install if present.

Minor inno^Wfixups all over the map too.

Folks asked to include aptitude in desktop user distros:
http://www.opennet.ru/openforum/vsluhforumID3/83728.html#117

Checking the image size is one of the usual things
after building it, and there's some media related
testing to be ported from m-p-d as well.

Removing /boot stuff from the live root was a nice idea
to spare some space except that live-install depends on it...

This one should help (erm... hope not the other way around!)
testing both 3D setup and FlightGear packages I happen to
maintain in a known clean environment.

The previous configuration would result in intel-only
3D being available since nouveau and radeon kernel modules
are packaged separately with most kernel-images; getting
NVIDIA/AMD drivers in is more tricky due to availability
of both proprietary and free implementations with the choice
being rather a tradeoff in each case (somewhat less so with
ATI/AMD drivers).

So this is a first shot at the problem: FlightGear would
freeze on me with today's nouveau.

This is actually a band-aid until decent l10n is there...

The <html> tag wasn't closed but actually duplicated;
thanks Aleksey Ilchenko for spotting.

127 is also returned by dot(1), and we're toasted
if unable to create $(BUILDDIR).

- ISO9660 metadata support
- initial alien VE image

Thanks someone for review and feedback:
http://www.opennet.ru/openforum/vsluhforumID3/83728.html#136

Better cleanup and i18n controls are due but in the meantime,
let's at least build something actually useful.

"Alien" rather means "CentOS-based" than "xenophilic" here;
in case you still wonder: http://www.altlinux.org/Hasher/CentOS

As was noted by Alexey Shabalin in libosinfo context,
current ALT Linux images tend to lack ISO9660 metadata
-- which they did have back in the day of Master 2.4.

Please note that the data collection occurs this way
due to mkimage's config.mk resetting the values to be
empty; this was worked around by using another config
file, $(BUILDDIR)lib/iso.mk, and including it later
but that would require a separate target with per-target
CONFIG variable which isn't elegant at all given the need
to actually build up the metadata set.

So the variables were changed (to be more readable anyways)
and then proxied back to BOOT_*.  This might be cleaned up
some day after the inclusion order is tweaked or mkimage
defaults get set-if-unset-yet (?=).

openssh-server is in need indeed on almost any server instance;
thanks Aleksey Cheusov for reporting the shortage.

This might be amended in the future but is reasonable right now.