mksquashfs: rewrite default queue size code so it is based on physical memory
For many years Mksquashfs has used fixed size default queues.
This has made choosing a *usable* default set of queue sizes impossible
because, obviously, I have no idea what size physical memory
people have in their machines, or what what amount of memory they
want Mksquashfs to use by default.
Choose a too large set of queues, and this means Mksquashfs won't
be able to be used on machines with smaller RAM, and/or it
will go into swap and suffer extremely poor performance.
Choose a too small set of queues, and we don't maximise Mksquashfs
performance by using the available RAM on machines with lots
of memory.
My original decision was to choose an extremely conservative set of
queues, only 64Mbytes for Read-Queue and Fragment-Queue, and 512Mbytes
for the Write-Queue which because the writer thread is never the
bottleneck is not expected to grow beyond ~20% of the maximum. So
a total queue size of less than 256Mbytes, which was conservative
when I made my estimate of the "usable RAM in a typical machine" in
2006 where I anticipated the typical machine might have 2Gbytes of RAM.
It is now rediculously small.
My expectation was most people would see that the defaults were
(deliberately) made small, and use the -XXX-queue options to increase
the queue sizes to something more usable. This AFAIK has not
happened. People just run Mksquashfs, and as if by magic, they expect
Mksquashfs to just use the right amount of memory.
One reason for this I suspect, is that I made a major mistake in
presenting 3 raw queue-size options to the user. Most people have
no idea what the queues are, and have no idea what the right ratio is
between the 3 queue sizes... and obviously think changing them
"ramdomly" will break Mksquashfs and just leave well alone.
The moral of the story is if you expect/hope people will increase the
memory used by Mksquashfs, then make sure you give them options
they will understand.... and if they don't change the options make
sure your default memory size is at least sensible for their
machine. I failed twice last time.
This commit is step 2. Base the default queue sizes on the amount of
physical memory. In this I have choosen to use a 1/4 of physical memory.
Any more than this and we can't guarantee we'll hit memory pressure on a
loaded machine, anything less than this, and we get back to the silly
amount of memory usage issue.
Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
2 files changed
