[GSOC] HAMMER2 compression feature week7 report

[Daniel Flores <daniel5555@xxxxxxxxx>]

--089e0160b5e470ae3304e324937d
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: quoted-printable

Hello everyone,
here is my report for week 7.

This week I had to create a new VM for DragonFly. This new VM has different
settings and works faster than the previous one. So, since all my work and
tests will be done on that new VM, it won't be possible to directly compare
new results with the results obtained in previous tests on old VM.

Now, as for work done this week, the code was cleaned up significantly and
optimized a bit too. This affected mostly write path, since most of new
code was there. More specifically, the write path now looks like this:

We have hammer2_write_file() function that contains all the code that is
shared among 3 possible options for write path =96 no compression,
zero-cheking and LZ4 compression. At the end of the function where paths
start to differ depending on selected option, it simply determines the
option and calls one of 3 functions: hammer2_compress_and_write()
(corresponds to LZ4 compression), hammer2_zero_check_and_write()
(corresponds to zero-checking option) and hammer2_just_write() (no
compression or zero-checking). Those functions do everything necessary to
finish the write path.

hammer2_just_write() mostly contains the code that was previously in the
end of hammer2_write_file() function.

hammer2_zero_check_and_write() is a very simple function that checks if the
block to be written contains only zeros with a specific function called
not_zero_filled_block() and calls, if necessary, another function called
zero_check() that deals with the zero-filled block. If the block is not
zero-filled, the function calls hammer2_just_write().

hammer2_compress_and_write() is the most complex function that performs the
compression and then writes the block, the compressed version if the
compression was successful and the original version if it wasn't. It also
uses not_zero_filled_block() and zero_check() for zero-filled block case.

There are also small improvements, such as that now we use obcache_create()
instead of obcache_create_simple().

What I'll do now is exhaustively test the code to ensure that it is stable.
Right now it is not, because we still have a certain bug that provokes file
corruption while reading and the system crash under certain circumstances.
I'll be working on fixing that next week. Also there are a couple of
enhancements for write path such as detecting the incompressible files and
not trying to compress them on which I'll be working as well. I also expect
that, probably, some other bugs will be found in process of testing.

Now a bit on tests... Earlier this week I was asked to test the performance
on small files. The testing methodology was exactly the same as the one I
employed in tests from previous week's report. For testing I used 5 files
in total:

1 .jpg (incompressible) =96 roughly 62KB In size.
1 small log file (perfectly compressible) =96 62KB in size.
1 .png (incompressible) =96 roughly 2KB in size.
1 very small log file (perfectly compressible) =96 2 KB in size.
1 even smaller log file =96 512B in size. I didn't use an incompressible
file, because all files of that size or smaller are embedded directly into
an inode, so their path is the same.

For the group test, the same files were copied 20 times per test.

The results are summarized in this table [1].

Basically, it looks like for such small files there is no detectable
difference in performance. It should be noted that the average seek time on
modern hard drives is about 0.009 s, so at this rate other factors are more
important for performance than the path used. It also should be noted that
currently the write path with compression tries to compress the whole
logical block even if the file is smaller than this block, but this doesn't
seem to affect the performance on a scale of single file.

On other hand, when the total size is large enough, like 2.5MB in this
case, it appears that the difference starts to be perceivable.

My code is available, as usually, in my leaf, branch =93hammer2_LZ4=94 [2].
I'll appreciate any comments, feedback and criticism.


Daniel

[1]
http://leaf.dragonflybsd.org/~iostream/performance_table_small_files.html
[2] git://leaf.dragonflybsd.org/~iostream/dragonfly.git

--089e0160b5e470ae3304e324937d
Content-Type: text/html; charset=windows-1252
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div>Hello everyone,<br></div><div>here is my report for w=
eek 7.</div><div><br></div><div>This week I had to create a new VM for Drag=
onFly. This new VM has different settings and works faster than the previou=
s one. So, since all my work and tests will be done on that new VM, it won&=
#39;t be possible to directly compare new results with the results obtained=
 in previous tests on old VM.</div>
<div><br></div><div>Now, as for work done this week, the code was cleaned u=
p significantly and optimized a bit too. This affected mostly write path, s=
ince most of new code was there. More specifically, the write path now look=
s like this:</div>
<div><br></div><div>We have hammer2_write_file() function that contains all=
 the code that is shared among 3 possible options for write path =96 no com=
pression, zero-cheking and LZ4 compression. At the end of the function wher=
e paths start to differ depending on selected option, it simply determines =
the option and calls one of 3 functions: hammer2_compress_and_write() (corr=
esponds to LZ4 compression), hammer2_zero_check_and_write() (corresponds to=
 zero-checking option) and hammer2_just_write() (no compression or zero-che=
cking). Those functions do everything necessary to finish the write path.</=
div>
<div><br></div><div>hammer2_just_write() mostly contains the code that was =
previously in the end of hammer2_write_file() function.</div><div><br></div=
><div>hammer2_zero_check_and_write() is a very simple function that checks =
if the block to be written contains only zeros with a specific function cal=
led not_zero_filled_block() and calls, if necessary, another function calle=
d zero_check() that deals with the zero-filled block. If the block is not z=
ero-filled, the function calls hammer2_just_write().</div>
<div><br></div><div>hammer2_compress_and_write() is the most complex functi=
on that performs the compression and then writes the block, the compressed =
version if the compression was successful and the original version if it wa=
sn&#39;t. It also uses not_zero_filled_block() and zero_check() for zero-fi=
lled block case.</div>
<div><br></div><div>There are also small improvements, such as that now we =
use obcache_create() instead of obcache_create_simple().</div><div><br></di=
v><div>What I&#39;ll do now is exhaustively test the code to ensure that it=
 is stable. Right now it is not, because we still have a certain bug that p=
rovokes file corruption while reading and the system crash under certain ci=
rcumstances. I&#39;ll be working on fixing that next week. Also there are a=
 couple of enhancements for write path such as detecting the incompressible=
 files and not trying to compress them on which I&#39;ll be working as well=
. I also expect that, probably, some other bugs will be found in process of=
 testing.</div>
<div><br></div><div>Now a bit on tests... Earlier this week I was asked to =
test the performance on small files. The testing methodology was exactly th=
e same as the one I employed in tests from previous week&#39;s report. For =
testing I used 5 files in total:</div>
<div><br></div><div>1 .jpg (incompressible) =96 roughly 62KB In size.</div>=
<div>1 small log file (perfectly compressible) =96 62KB in size.</div><div>=
1 .png (incompressible) =96 roughly 2KB in size.</div><div>1 very small log=
 file (perfectly compressible) =96 2 KB in size.</div>
<div>1 even smaller log file =96 512B in size. I didn&#39;t use an incompre=
ssible file, because all files of that size or smaller are embedded directl=
y into an inode, so their path is the same.</div><div><br></div><div>For th=
e group test, the same files were copied 20 times per test.</div>
<div><br></div><div>The results are summarized in this table [1].</div><div=
><br></div><div>Basically, it looks like for such small files there is no d=
etectable difference in performance. It should be noted that the average se=
ek time on modern hard drives is about 0.009 s, so at this rate other facto=
rs are more important for performance than the path used. It also should be=
 noted that currently the write path with compression tries to compress the=
 whole logical block even if the file is smaller than this block, but this =
doesn&#39;t seem to affect the performance on a scale of single file.</div>
<div><br></div><div>On other hand, when the total size is large enough, lik=
e 2.5MB in this case, it appears that the difference starts to be perceivab=
le.</div><div><br></div><div>My code is available, as usually, in my leaf, =
branch =93hammer2_LZ4=94 [2]. I&#39;ll appreciate any comments, feedback an=
d criticism.</div>
<div><br></div><div><br></div><div>Daniel</div><div><br></div><div>[1] <a h=
ref=3D"http://leaf.dragonflybsd.org/~iostream/performance_table_small_files=
.html">http://leaf.dragonflybsd.org/~iostream/performance_table_small_files=
.html</a></div>
<div>[2] git://<a href=3D"http://leaf.dragonflybsd.org/~iostream/dragonfly.=
git">leaf.dragonflybsd.org/~iostream/dragonfly.git</a></div><div><br></div>=
</div>

--089e0160b5e470ae3304e324937d--