- simply better (tm)

There will be two types of Simply: DVD and Live versions.

Following m-p-d, a more involved default output directory
structure is feasible now:

  ~/out/name/date/name-date-arch.type

instead of plain

  ~/out/name-date-arch.type

This particular behaviour can be achieved by passing
SORTDIR='$(IMAGE_NAME)/$(DATE)'; note the single quotes.

Reports are also saved in this resulting structure
albeit the place is still highly debatable.

REPORT_PATH comes from a makefile;
let's use corresponding notation.

Trying to extract reports for a failed build is a bad idea.

That's what command-line variables serve for
(without causing a combinatorial explosion) :)

- xfce-utils removed from Sisyphus
- apt-indicator needs gksu
- add slinux misc packages:
  + pm-utils
  + strace
- image-scripts.d/50services is a script which starts/stop some services

use/slinux-live: in p6 slinux had install-dvd version too
lists/slinux/misc-dvd: user 3d-proprietary comes from use/x11/3d-proprietary
lists/slinux/misc-dvd:restore compiz
slinux: use/syslinux/localboot.cfg

Sometimes it's desirable to provide the kernel supporting
maximal amount of RAM on the system; bad news is that x86
has a kludge named PAE, good news is that x86_64 doesn't
need it at all; but now we must be able to choose between
those.

BIGRAM will hold the flavour needed.

Toplevel Makefile is a bit optimistic regarding
calling reports.mk...

This one extracts the exact time-ordered list of script hooks
run while building the image.

Some commons refactoring is also due.

This could harm the debug...

The missing part so far is passing "lang=ab_CD" to the kernel;
it's done with gfxboot and is somewhat doable without it...

This script specifies the (excessive) lists of services
to be enabled and disabled explicitly; these are mostly based
on profiles/live/image-scripts.d/init3-services from m-p-d.

There might be systemd related pecularities though...

Thanks slava@ for the packages and inspiration,
and aris@ for the useful advice and fixups.

livecd-install support wasn't added properly;
fixed (thanks glebfm@ yet again).

Something like fine-grained KEEP_* might be due...

The early version considered ISO and KOI encoding families
as obsolete; the current one is a bit more wise and knows
these are just /rare/.  Thanks glebfm@ for #27168 research
and cinnamon by slava@ for ISO-related noises at startup.

There's no real reason to keep bcmwl and ndiswrapper
around exclusively as the currently available support
vastly takes over the early attempts at the task.

(it's not about bare firmware though, and some day
something like use/hardware/wireless should get in)

Initial SPICE support has been added for kvm/libvirt installation
and boot-up using qxl and spice by default as proposed by shaba@.

VirtualBox part is shifted a level deeper correspondingly
but otherwise stays the same.

It is actually an effort by glebfm@ to create an experimental
systemd-based Simply Linux LiveCD; I merely reviewed the original
diff, moved kernel related bits to firmware (see preceding commits)
and introduced a dedicated pkglist namespace by creating a directory.

Calling ../../bin/pkgdups with a bunch of pkglists was messy,
so is done algorithmically from now on while in pkg.in/lists:

  make pkgdups

THE_PACKAGES_REGEXP is in place, let's rebase firmware packages
so these would be available in LiveCDs either.

The news for systems being installed is that MAIN_* is optional
while THE_* is included in base system; firmware packages tend
to be pretty tiny and harmless.

kernel-wifi pkglist has absolutely no sense by now, hence purged;
firmware-rt* and firmware-i2400m are merged into firmware-linux.

There were STAGE1_PACKAGES_REGEXP and MAIN_PACKAGES_REGEXP
but adding more of those was postponed to avoid bloat and
bitrot; THE_PACKAGES_REGEXP is needed for use/firmware now
and looks like BASE_PACKAGES_REGEXP and LIVE_PACKAGES_REGEXP
will be useful before too long either.

Docs updated to include stage-specific package related vatiables.

A pretty common issue breaking the image build is inter-package
file conflict resulting in hsh-install failure down there.
Let's bring that back to attention conveniently.

- massive squashfs tuning

sub.in/stage2/image-scripts.d/99cleanup-boot,
livecd-install need /boot files too

glebfm@ asked what to do with new package lists: whether these
belong to features, or to distributions themselves.  This question
is actually open and up for discussion but there are guidelines
that can and should be written down already; and so they were.

Added pkgdups utility reference as well.

Multiple ARCHES won't just magically work without
the ability to figure out the correct apt.conf;
fortunately there's just the right example handy
in profiles.mk.sample already.

Thanks glebfm@ for feedback.

Looks like the 128k default block size is pretty well chosen:
it saves ~6% of image size compared to 64k, and subsequent
differences are ~3% per doubling the block size up to 1M
(thanks led@ for carrying out the tests).

So we'll stick with 256k for "normal" xz compression (inodes
uncompressed) and get 512k back for "tight" one (compressed).
The runtime performance issues are to be examined yet when
bootchart or the like is deployed, nothing drastic though.

With "fast" (gzip/lzo) squash compression inodes go unmolested.

For the record, tight live-webkiosk builds as 95M image in 3:40,
and tight live-flightgear.iso builds as 669M image in 6:34.  Nice.

There's no much sense going for 1M block size: e.g. live-webkiosk
would drop to 93M (3:46) but its load time would increase up to
2:07 as compared to 1:48 for -b 524288 and 1:42 for -b 262144 -noI
on a Duron 500/512M system given the very same DVD+RW media.

If there's no dot(1) then its input file should just be stored;
thanks glebfm@ for reminding about this.

Should be removed when console autologin is there
(mind the live-builder).

The existing implementation would handle kernel differences
just fine but a bit too automatically: if it sees xz support,
that's what will end up being used (and if there's -Xbcj binary
compression filter available for the target platform, it will
be applied unequivocally either).

It's perfectly suitabe for getting fine-tuned release images
but is also a bit too resource-consuming while developing the
image configuration which has no business with its compression.

The one and only knob is SQUASHFS (see doc/variables.txt);
to give an idea of the differences, here are some numbers
for a mostly-binary (43% as per 99-elf-stats) webkiosk livecd
and a rather less so (18%) flightgear one on a dual quad-core
X5570 node (each mksquashfs run used up all the cores):

SQUASHFS | live-webkiosk.iso | live-flightgear.iso
---------+-------------------+---------------------
fast     | 3:30 / 130M       | 5:11 / 852M
normal * | 3:37 / 100M       | 5:35 / 688M
tight    | 3:50 / 98M        | 6:47 / 683M

Thus if the knob isn't fiddled with, the defaults will allow
for a reasonably fast build of a pretty slim image; if one is
building a release or if a particular image is very sensitive
being close to the media capacity then just add SQUASHFS=tight
and see it a percent or two down on size.

Please note that lzo/gzip-compressed images are also quicker
to uncompress thus further helping with test iterations.

Thanks to led@ and glebfm@ for helpful hints and questions.

APM enabled notebooks would usually hibernate to
a partition of special type and special format;
thus to make use of this APM BIOS feature folks
might need a corresponding formatter.

This kind of test was proposed by led@ to gather statistics
on chroot's contents going to become squashfs (the script
optimizations lowering added overhead from ~10 sec down
to a subsecond range were also proposed by him).

Intentionally not documented in doc/variables.txt due to
the rather lowlevel nature of the probe (at least so far).

The knobs involved are SQUASHFS (the additional effort kicks
in only for "tight" case) and GLOBAL_SQUASHFS_SORT (must be
non-empty for this extra overhead to occur).

Additional experimentation is needed to find out whether
the difference in squashfs size and performance is worth
the trouble (seems the impact is non-zero but pretty minor).

gns@ has 512k, led@ suggests that it's too much a bit;
let's try 256k.

bin/ prefix tried to become a permatemp(tm).  No way.

Thanks Chernetsky Andrey for testing and suggesting it.
(NB: currently only t6 build is proper, see input-utils bugs)

There is at least one known deficiency for mkimage-profiles:
build.log will be truncated if verbose mode is enabled and
hasher version is lower than 1.3.22.

The check is done here since it's where the logging is arranged,
and doing it in image.in/Makefile would result in the warning
about log-truncating software being truncated by the said software.

Thanks Max Kosmach for reporting this inobviousity.