DragonFly kernel List (threaded) for 2013-06
[
Date Prev][
Date Next]
[
Thread Prev][
Thread Next]
[
Date Index][
Thread Index]
Re: [GSOC] Introduction to HAMMER2 compression feature
------DK29URFGMFOYY765IRDKEA3J4SYL8Y
Content-Type: text/plain;
charset=UTF-8
Content-Transfer-Encoding: 8bit
Im looking forward to use this feature.
What about a compression like gzip e.g. for log file compression or LZJB?
Will you make it possible to let the user choose the compression algorithm or do you just looking for the "best" one?
Daniel Flores <daniel5555@gmail.com> wrote:
>Hello everyone,
>
>My name is Daniel Flores and my proposal called â??HAMMER2 compression
>featureâ?? was accepted for this yearâ??s Google Summer of Code. I already
>posted the draft of my proposal [1] in this kernel list, so I will not
>repeat much of it, but instead I want to focus on some design decisions
>that Iâ??ve already made. Since Iâ??m an inexperienced developer at this
>point,
>Iâ??d be happy to receive your suggestions and criticism.
>
>The feature itself consists in that it attempts to compress a HAMMER2
>block, which is of 64KB in size. The result should be a block of 32KB,
>16KB, 8KB, etc. in size.
>
>Currently Iâ??m searching for the algorithms that are the most
>appropriate
>for a file system. Particularly Iâ??m searching for algorithms that are
>very
>fast; donâ??t require a lot of memory and processing power and offer
>fairly
>good compression rate (but not necessarily the best compression rate
>out of
>all). Right now I have two candidates â?? DEFLATE [2] and LZO [3]. Both
>of
>them have some available implementations which I intend to use, as Alex
>suggested in his review of my proposal.
>
>DEFLATE seems to be a good choice, because it works with small amounts
>of
>data and has a sliding window of 32KB â?? just nice for a 64KB block. It
>is
>based on another algorithm â?? LZ77, which is successfully used in
>compression feature for NTFS, so hopefully DEFLATE would be good as
>well.
>
>LZO seems to be a good choice, because, similarly, it works on small
>amounts of data, it is as fast as DEFLATE and was specifically designed
>to
>have a very fast decompression speed.
>
>My initial proposal only covers the implementation of one algorithm,
>but if
>the time allows, I would try to add both of them and maybe add some
>more.
>The design will be modular, so it will not be difficult to change the
>algorithm.
>
>My initial strategy, as I already mentioned in my proposal, is to
>develop,
>first, a prototype application to test those algorithms. The
>application
>would take a file, break it in 64KB blocks, compress them one by one,
>and
>then decompress them. After that the resulting uncompressed blocks
>would be
>compared with the original blocks and if they are identical, then it
>would
>mean that the algorithm works correctly.
>
>I will try to compare the algorithms in terms of efficiency and speed,
>and
>choose the best one. I also expect that Iâ??ll have to modify a bit the
>implementations, because most likely they wonâ??t produce a block of an
>exact
>size as required, so I would have to add some padding to it.
>
>After the exhaustive testing with the prototype application, I will add
>the
>compression feature in HAMMER2 itself and do real-life testing. The
>utility
>to set up the compression feature would also be done at this point. By
>the
>end of summer, the compression feature should be fully and correctly
>implemented and ready for use.
>
>
>Daniel
>
>[1] â??
>http://lists.dragonflybsd.org/pipermail/kernel/2013-April/031228.html
>[2] â?? http://en.wikipedia.org/wiki/DEFLATE
>[3] â?? http://en.wikipedia.org/wiki/LZO
--
Sent from my Android phone with K-9 Mail. Please excuse my brevity.
------DK29URFGMFOYY765IRDKEA3J4SYL8Y
Content-Type: text/html;
charset=utf-8
Content-Transfer-Encoding: 8bit
<html><head/><body><html><head></head><body>Im looking forward to use this feature.<br>
What about a compression like gzip e.g. for log file compression or LZJB?<br>
<br>
Will you make it possible to let the user choose the compression algorithm or do you just looking for the "best" one?<br><br><div class="gmail_quote">Daniel Flores <daniel5555@gmail.com> wrote:<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<div dir="ltr">Hello everyone,<br /><br />My name is Daniel Flores and my proposal called â??HAMMER2 compression featureâ?? was accepted for this yearâ??s Google Summer of Code. I already posted the draft of my proposal [1] in this kernel list, so I will not repeat much of it, but instead I want to focus on some design decisions that Iâ??ve already made. Since Iâ??m an inexperienced developer at this point, Iâ??d be happy to receive your suggestions and criticism.<br />
<br />The feature itself consists in that it attempts to compress a HAMMER2 block, which is of 64KB in size. The result should be a block of 32KB, 16KB, 8KB, etc. in size.<br /><br />Currently Iâ??m searching for the algorithms that are the most appropriate for a file system. Particularly Iâ??m searching for algorithms that are very fast; donâ??t require a lot of memory and processing power and offer fairly good compression rate (but not necessarily the best compression rate out of all). Right now I have two candidates â?? DEFLATE [2] and LZO [3]. Both of them have some available implementations which I intend to use, as Alex suggested in his review of my proposal.<br />
<br />DEFLATE seems to be a good choice, because it works with small amounts of data and has a sliding window of 32KB â?? just nice for a 64KB block. It is based on another algorithm â?? LZ77, which is successfully used in compression feature for NTFS, so hopefully DEFLATE would be good as well.<br />
<br />LZO seems to be a good choice, because, similarly, it works on small amounts of data, it is as fast as DEFLATE and was specifically designed to have a very fast decompression speed.<br /><br />My initial proposal only covers the implementation of one algorithm, but if the time allows, I would try to add both of them and maybe add some more. The design will be modular, so it will not be difficult to change the algorithm.<br />
<br />My initial strategy, as I already mentioned in my proposal, is to develop, first, a prototype application to test those algorithms. The application would take a file, break it in 64KB blocks, compress them one by one, and then decompress them. After that the resulting uncompressed blocks would be compared with the original blocks and if they are identical, then it would mean that the algorithm works correctly.<br />
<br />I will try to compare the algorithms in terms of efficiency and speed, and choose the best one. I also expect that Iâ??ll have to modify a bit the implementations, because most likely they wonâ??t produce a block of an exact size as required, so I would have to add some padding to it.<br />
<br />After the exhaustive testing with the prototype application, I will add the compression feature in HAMMER2 itself and do real-life testing. The utility to set up the compression feature would also be done at this point. By the end of summer, the compression feature should be fully and correctly implemented and ready for use.<br />
<br /><br />Daniel<br /><br />[1] â?? <a href="http://lists.dragonflybsd.org/pipermail/kernel/2013-April/031228.html";>http://lists.dragonflybsd.org/pipermail/kernel/2013-April/031228.html</a><br />[2] â?? <a href="http://en.wikipedia.org/wiki/DEFLATE";>http://en.wikipedia.org/wiki/DEFLATE</a><br />
[3] â?? <a href="http://en.wikipedia.org/wiki/LZO";>http://en.wikipedia.org/wiki/LZO</a><br /><br /></div>
</blockquote></div><br>
-- <br>
Sent from my Android phone with K-9 Mail. Please excuse my brevity.</body></html></body></html>
------DK29URFGMFOYY765IRDKEA3J4SYL8Y--
[
Date Prev][
Date Next]
[
Thread Prev][
Thread Next]
[
Date Index][
Thread Index]
