Re: [GSoC] HAMMER2 block compression feature

[Daniel Flores <daniel5555@xxxxxxxxx>]

--001a11c38e9ac7c72104db361abf
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: quoted-printable

Thank you for your answers.

About git: I don=92t have experience using it, but I have used SVN while
working on some team projects. I=92m not sure how similar they are, but in
any case I=92ll try to learn how to use it as soon as possible.

About HAMMER2: I was looking at the code located in github [1]. I can=92t s=
ay
that I understand how it works in-depth, but I think I understand the
basics and it=92s only a question of time for me to get fully familiarized
with it.

I think that the code related to compression and encryption functions would
be located in separate files, like hammer2_compression.c and
hammer2_encryption.c, so there won=92t be many changes in other code. I
assume that one change would be that before writing something it would
check attributes of a directory and if compression/encryption is enabled,
it would pass the data to compression/encryption functions. In case of
compression function, it would probably return the resulting size of new
block along with the already compressed data. So, the main code would write
it then just like it does now, perhaps taking into account the new size of
block.

Hopefully, this is a plausible approach=85 For now, I don=92t have any spec=
ific
questions, but probably I=92ll have them in the future if I=92ll get to mor=
e
specific details of implementation.

One thing I would like to ask is if it=92s OK to send several proposals
related to HAMMER2? The reason I=92m asking this is that, generally, I=92m
equally interested in both compression feature and encryption feature, even
though compression feature would be my first choice. But I would gladly do
any of those.

I am confident that I can succeed with encryption feature too because
overall it is a similar proposal and I have completed a course on
cryptography, so it=92s not unfamiliar to me. I would choose an encryption
algorithm and would implement it using a prototype approach first and then
gradually integrating it in HAMMER2 code. In fact, I assume that
AES128-encryption would be OK and I=92m relatively familiar with that
algorithm.

I plan to send an official proposal regarding compression feature through
Google Summer of Code page by this weekend and, if it=92s OK, another
proposal regarding encryption feature by the same time or a bit later.

Thank you.

[1] https://github.com/DragonFlyBSD/DragonFlyBSD/tree/master/sys/vfs/hammer=
2


Daniel


On Thu, Apr 25, 2013 at 4:34 AM, Justin Sherrill
<justin@shiningsilence.com>wrote:

> Daniel - One additional item that may be useful is learning something
> about git, the revision control tool DragonFly (and many other
> organizations) use.  You may already have encountered it or something
> similar, but I don't see specific mention in your email.  It will be
> the preferred way to store your code and to make it accessible for
> others.
>
> I can't tell you if there's a strong candidate for block compression -
> the application period just started, so there's nothing yet.  Rather
> than second-guess what's available, apply with a project proposal that
> has what you most want to do.  That will do you the most good.
>
> On Tue, Apr 23, 2013 at 3:14 PM, Daniel Flores <daniel5555@gmail.com>
> wrote:
> > Hello.
> >
> > My name is Daniel Flores and I=92m a student from Spain. I study in
> Barcelona
> > School of Informatics, which is a faculty of Polytechnic University of
> > Catalonia, and it=92s my last year before I graduate.
> >
> > I sent a mail with this proposal to Matthew 4 days ago, but I didn't ge=
t
> a
> > reply yet, so I decided to send this to the mailing list, as it is
> suggested
> > on Dragonfly's GSoC page. Before submitting any official application, I
> > would like to ask for your opinion and criticism regarding my proposal
> and
> > whether I have the required skill and adequate planning to successfully
> > finish this project.
> >
> > I=92m interested in implementing the block compression feature in HAMME=
R2,
> > which is described on Dragonfly=92s GSoC page [1]. The feature consists=
 in
> > that any written block should be compressed, if possible, to 50% of its
> size
> > or less (in powers of 2 =96 32 KB, 16 KB and so on, the initial size of
> block
> > being 64 KB). It should be possible to set that compression mode on any
> > directory, so that new contents written in it and its subdirectories
> would
> > be compressed.
> >
> > I believe that I can succeed because I have programming experience in
> C/C++,
> > since we use those languages for most practices in our university. I al=
so
> > completed courses centered on algorithms and their implementation in C =
or
> > C++. I also have an interest in operating systems, their concepts and
> their
> > development. I completed a course on Operating Systems development wher=
e
> I
> > and a partner implemented a simple linux-based kernel called =93ZeOS=94
> (using C
> > and, at times, x86 assembler). It=92s a very simple kernel that has a t=
ask
> > scheduler, system calls, exceptions, interruptions, memory management
> and a
> > file system. We were able to successfully run it on bochs emulator and =
it
> > passed all tests needed to successfully finish the course.
> >
> > If you wish, you may take a look at it [2].
> >
> > Sadly, it=92s documented in Spanish, but I just want to provide it as a=
n
> > example of code that I can produce.
> >
> > In spite of it being a kernel implementation, I think that my skills an=
d
> > knowledge aren=92t yet the best suitable for kernel development project=
s,
> so
> > instead I prefer to do something moderately difficult, yet related to
> > operating systems, like HAMMER2 block compression feature.
> >
> > My initial outline for project planning is the following:
> >
> > 1. (Starting May 27 or before =96 June 17) During community bonding per=
iod
> I
> > would get myself familiarized with the community, Dragonfly BSD
> system/code
> > and =93hammer2=94 branch code. I would also refresh my knowledge on
> compression
> > algorithms and would complete a design for the future implementation,
> like
> > choosing the appropriate algorithm and having defined everything needed
> to
> > be done. I have read Matthew's design document for HAMMER2 [3] and, as =
I
> > understand, the algorithm #2 is not yet determined, so I believe this
> should
> > be decided before the actual coding period will start. Also, more
> detailed
> > planning for coding period would be done, with clear milestones and
> > deliverables.
> >
> > The deliverables for this phase would be in the form of documents that
> would
> > describe the design, the chosen algorithm(s) and the calendar with
> > milestones.
> >
> > 2. (June 17 =96 July 29) I would start by coding a prototype applicatio=
n
> that
> > would implement the defined design, consisting of algorithms #0, #1 and
> #2.
> > I would need it to insure that the algorithm is implemented correctly
> and to
> > do all necessary debugging and refining before integrating it in
> HAMMER2. I
> > would create a series of tests to prove that the algorithm works well. =
If
> > time allows, I would also try to study its behavior statistically and t=
o
> > improve it, if possible.
> >
> > The deliverables would be code of the prototype as well as any supporti=
ng
> > applications (to generate tests, for example) and tests themselves.
> >
> > 3. (July 29 =96 September 16) After confirming that the prototype works
> well,
> > I would start actually integrating it into HAMMER2 filesystem. At this
> > point, the task would be gradually adding prototype=92s code without an=
y
> > conceptual modification into HAMMER2 and testing if it works correctly =
in
> > real-life. A series of testing applications with appropriate system cal=
ls
> > would be created and the results checked for correctness. Also, the
> > appropriate hammer2 utility command to set the compression feature to
> > directories would be created and tested.
> >
> > The deliverables would be the code of implemented feature and applicati=
on
> > tests for it.
> >
> > 4. (September 16 =96 End of GSoC 2013) Hopefully, all known bugs would =
be
> > corrected at this point, so I would take the time to do final
> refinements to
> > documentation, to write a detailed description of what was implemented
> for
> > future developers and to finish everything needed to start submitting m=
y
> > code to Google.
> >
> > Generally, my philosophy is to start =93as soon as possible=94, so if s=
ome
> phase
> > is done before the deadline, I would immediately go to the next phase. =
If
> > project goes smoothly and there is some time left, I would be happy to =
do
> > small refinements or bugs fixes on general system or HAMMER2.
> >
> > During the coding period (June 17 =96 September 16) I don=92t have any
> > commitments (like courses or trips) at all, so I should be able to easi=
ly
> > dedicate 40 hours per week or more to this project. Prior to June 17 I
> have
> > my studies and some exams, but generally it should not prevent me from
> > getting familiarized with code and getting the design/algorithm decisio=
ns
> > done, because my workload is not very big.
> >
> > I don=92t have any past experience with Open Source or generally big
> projects,
> > but I believe that I can accomplish this task since it is generally
> related
> > to C programming and algorithms, and I believe that I have sufficient
> > knowledge of those fields from my university courses.
> >
> > Currently I am getting myself familiarized with Dragonfly 3.2 (for i386=
)
> > running it on Oracle VM Virtual Box 4.2.12, but I=92m able to install i=
t on
> > physical machine, if necessary. I=92m also trying to learn as much as
> possible
> > about BSD in general and particularly Dragonfly.
> >
> > If you think that my set of skills is more appropriate to some other
> project
> > related to HAMMER2 or Dragonfly, I would be grateful if you pointed at
> them.
> > I also would consider working on HAMMER2 block encryption feature, sinc=
e
> it
> > is similar in requirements to my current proposal, if there is already =
a
> > strong candidate to implement the block compression feature.
> >
> > Thank you.
> >
> > [1] http://www.dragonflybsd.org/docs/developer/gsocprojectspage/
> > [2] https://dl.dropboxusercontent.com/u/56646280/zeos.zip
> > [3] http://leaf.dragonflybsd.org/mailarchive/users/2012-02/msg00020.htm=
l
> >
> >
> > Daniel
>

--001a11c38e9ac7c72104db361abf
Content-Type: text/html; charset=windows-1252
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div>Thank you for your answers.<br><br>About git: I don=
=92t have experience using it, but I have used SVN while working on some te=
am projects. I=92m not sure how similar they are, but in any case I=92ll tr=
y to learn how to use it as soon as possible.<br>
<br>About HAMMER2: I was looking at the code located in github [1]. I can=
=92t say that I understand how it works in-depth, but I think I understand =
the basics and it=92s only a question of time for me to get fully familiari=
zed with it.<br>
<br>I think that the code related to compression and encryption functions w=
ould be located in separate files, like hammer2_compression.c and hammer2_e=
ncryption.c, so there won=92t be many changes in other code. I assume that =
one change would be that before writing something it would check attributes=
 of a directory and if compression/encryption is enabled, it would pass the=
 data to compression/encryption functions. In case of compression function,=
 it would probably return the resulting size of new block along with the al=
ready compressed data. So, the main code would write it then just like it d=
oes now, perhaps taking into account the new size of block.<br>
<br>Hopefully, this is a plausible approach=85 For now, I don=92t have any =
specific questions, but probably I=92ll have them in the future if I=92ll g=
et to more specific details of implementation.<br><br>One thing I would lik=
e to ask is if it=92s OK to send several proposals related to HAMMER2? The =
reason I=92m asking this is that, generally, I=92m equally interested in bo=
th compression feature and encryption feature, even though compression feat=
ure would be my first choice. But I would gladly do any of those.<br>
<br>I am confident that I can succeed with encryption feature too because o=
verall it is a similar proposal and I have completed a course on cryptograp=
hy, so it=92s not unfamiliar to me. I would choose an encryption algorithm =
and would implement it using a prototype approach first and then gradually =
integrating it in HAMMER2 code. In fact, I assume that AES128-encryption wo=
uld be OK and I=92m relatively familiar with that algorithm.<br>
<br>I plan to send an official proposal regarding compression feature throu=
gh Google Summer of Code page by this weekend and, if it=92s OK, another pr=
oposal regarding encryption feature by the same time or a bit later.<br><br=
>
Thank you.<br><br>[1] <a href=3D"https://github.com/DragonFlyBSD/DragonFlyB=
SD/tree/master/sys/vfs/hammer2">https://github.com/DragonFlyBSD/DragonFlyBS=
D/tree/master/sys/vfs/hammer2</a><br><br><br></div>Daniel<br></div><div cla=
ss=3D"gmail_extra">
<br><br><div class=3D"gmail_quote">On Thu, Apr 25, 2013 at 4:34 AM, Justin =
Sherrill <span dir=3D"ltr">&lt;<a href=3D"mailto:justin@shiningsilence.com"=
 target=3D"_blank">justin@shiningsilence.com</a>&gt;</span> wrote:<br><bloc=
kquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #cc=
c solid;padding-left:1ex">
Daniel - One additional item that may be useful is learning something<br>
about git, the revision control tool DragonFly (and many other<br>
organizations) use. =A0You may already have encountered it or something<br>
similar, but I don&#39;t see specific mention in your email. =A0It will be<=
br>
the preferred way to store your code and to make it accessible for<br>
others.<br>
<br>
I can&#39;t tell you if there&#39;s a strong candidate for block compressio=
n -<br>
the application period just started, so there&#39;s nothing yet. =A0Rather<=
br>
than second-guess what&#39;s available, apply with a project proposal that<=
br>
has what you most want to do. =A0That will do you the most good.<br>
<div class=3D"HOEnZb"><div class=3D"h5"><br>
On Tue, Apr 23, 2013 at 3:14 PM, Daniel Flores &lt;<a href=3D"mailto:daniel=
5555@gmail.com">daniel5555@gmail.com</a>&gt; wrote:<br>
&gt; Hello.<br>
&gt;<br>
&gt; My name is Daniel Flores and I=92m a student from Spain. I study in Ba=
rcelona<br>
&gt; School of Informatics, which is a faculty of Polytechnic University of=
<br>
&gt; Catalonia, and it=92s my last year before I graduate.<br>
&gt;<br>
&gt; I sent a mail with this proposal to Matthew 4 days ago, but I didn&#39=
;t get a<br>
&gt; reply yet, so I decided to send this to the mailing list, as it is sug=
gested<br>
&gt; on Dragonfly&#39;s GSoC page. Before submitting any official applicati=
on, I<br>
&gt; would like to ask for your opinion and criticism regarding my proposal=
 and<br>
&gt; whether I have the required skill and adequate planning to successfull=
y<br>
&gt; finish this project.<br>
&gt;<br>
&gt; I=92m interested in implementing the block compression feature in HAMM=
ER2,<br>
&gt; which is described on Dragonfly=92s GSoC page [1]. The feature consist=
s in<br>
&gt; that any written block should be compressed, if possible, to 50% of it=
s size<br>
&gt; or less (in powers of 2 =96 32 KB, 16 KB and so on, the initial size o=
f block<br>
&gt; being 64 KB). It should be possible to set that compression mode on an=
y<br>
&gt; directory, so that new contents written in it and its subdirectories w=
ould<br>
&gt; be compressed.<br>
&gt;<br>
&gt; I believe that I can succeed because I have programming experience in =
C/C++,<br>
&gt; since we use those languages for most practices in our university. I a=
lso<br>
&gt; completed courses centered on algorithms and their implementation in C=
 or<br>
&gt; C++. I also have an interest in operating systems, their concepts and =
their<br>
&gt; development. I completed a course on Operating Systems development whe=
re I<br>
&gt; and a partner implemented a simple linux-based kernel called =93ZeOS=
=94 (using C<br>
&gt; and, at times, x86 assembler). It=92s a very simple kernel that has a =
task<br>
&gt; scheduler, system calls, exceptions, interruptions, memory management =
and a<br>
&gt; file system. We were able to successfully run it on bochs emulator and=
 it<br>
&gt; passed all tests needed to successfully finish the course.<br>
&gt;<br>
&gt; If you wish, you may take a look at it [2].<br>
&gt;<br>
&gt; Sadly, it=92s documented in Spanish, but I just want to provide it as =
an<br>
&gt; example of code that I can produce.<br>
&gt;<br>
&gt; In spite of it being a kernel implementation, I think that my skills a=
nd<br>
&gt; knowledge aren=92t yet the best suitable for kernel development projec=
ts, so<br>
&gt; instead I prefer to do something moderately difficult, yet related to<=
br>
&gt; operating systems, like HAMMER2 block compression feature.<br>
&gt;<br>
&gt; My initial outline for project planning is the following:<br>
&gt;<br>
&gt; 1. (Starting May 27 or before =96 June 17) During community bonding pe=
riod I<br>
&gt; would get myself familiarized with the community, Dragonfly BSD system=
/code<br>
&gt; and =93hammer2=94 branch code. I would also refresh my knowledge on co=
mpression<br>
&gt; algorithms and would complete a design for the future implementation, =
like<br>
&gt; choosing the appropriate algorithm and having defined everything neede=
d to<br>
&gt; be done. I have read Matthew&#39;s design document for HAMMER2 [3] and=
, as I<br>
&gt; understand, the algorithm #2 is not yet determined, so I believe this =
should<br>
&gt; be decided before the actual coding period will start. Also, more deta=
iled<br>
&gt; planning for coding period would be done, with clear milestones and<br=
>
&gt; deliverables.<br>
&gt;<br>
&gt; The deliverables for this phase would be in the form of documents that=
 would<br>
&gt; describe the design, the chosen algorithm(s) and the calendar with<br>
&gt; milestones.<br>
&gt;<br>
&gt; 2. (June 17 =96 July 29) I would start by coding a prototype applicati=
on that<br>
&gt; would implement the defined design, consisting of algorithms #0, #1 an=
d #2.<br>
&gt; I would need it to insure that the algorithm is implemented correctly =
and to<br>
&gt; do all necessary debugging and refining before integrating it in HAMME=
R2. I<br>
&gt; would create a series of tests to prove that the algorithm works well.=
 If<br>
&gt; time allows, I would also try to study its behavior statistically and =
to<br>
&gt; improve it, if possible.<br>
&gt;<br>
&gt; The deliverables would be code of the prototype as well as any support=
ing<br>
&gt; applications (to generate tests, for example) and tests themselves.<br=
>
&gt;<br>
&gt; 3. (July 29 =96 September 16) After confirming that the prototype work=
s well,<br>
&gt; I would start actually integrating it into HAMMER2 filesystem. At this=
<br>
&gt; point, the task would be gradually adding prototype=92s code without a=
ny<br>
&gt; conceptual modification into HAMMER2 and testing if it works correctly=
 in<br>
&gt; real-life. A series of testing applications with appropriate system ca=
lls<br>
&gt; would be created and the results checked for correctness. Also, the<br=
>
&gt; appropriate hammer2 utility command to set the compression feature to<=
br>
&gt; directories would be created and tested.<br>
&gt;<br>
&gt; The deliverables would be the code of implemented feature and applicat=
ion<br>
&gt; tests for it.<br>
&gt;<br>
&gt; 4. (September 16 =96 End of GSoC 2013) Hopefully, all known bugs would=
 be<br>
&gt; corrected at this point, so I would take the time to do final refineme=
nts to<br>
&gt; documentation, to write a detailed description of what was implemented=
 for<br>
&gt; future developers and to finish everything needed to start submitting =
my<br>
&gt; code to Google.<br>
&gt;<br>
&gt; Generally, my philosophy is to start =93as soon as possible=94, so if =
some phase<br>
&gt; is done before the deadline, I would immediately go to the next phase.=
 If<br>
&gt; project goes smoothly and there is some time left, I would be happy to=
 do<br>
&gt; small refinements or bugs fixes on general system or HAMMER2.<br>
&gt;<br>
&gt; During the coding period (June 17 =96 September 16) I don=92t have any=
<br>
&gt; commitments (like courses or trips) at all, so I should be able to eas=
ily<br>
&gt; dedicate 40 hours per week or more to this project. Prior to June 17 I=
 have<br>
&gt; my studies and some exams, but generally it should not prevent me from=
<br>
&gt; getting familiarized with code and getting the design/algorithm decisi=
ons<br>
&gt; done, because my workload is not very big.<br>
&gt;<br>
&gt; I don=92t have any past experience with Open Source or generally big p=
rojects,<br>
&gt; but I believe that I can accomplish this task since it is generally re=
lated<br>
&gt; to C programming and algorithms, and I believe that I have sufficient<=
br>
&gt; knowledge of those fields from my university courses.<br>
&gt;<br>
&gt; Currently I am getting myself familiarized with Dragonfly 3.2 (for i38=
6)<br>
&gt; running it on Oracle VM Virtual Box 4.2.12, but I=92m able to install =
it on<br>
&gt; physical machine, if necessary. I=92m also trying to learn as much as =
possible<br>
&gt; about BSD in general and particularly Dragonfly.<br>
&gt;<br>
&gt; If you think that my set of skills is more appropriate to some other p=
roject<br>
&gt; related to HAMMER2 or Dragonfly, I would be grateful if you pointed at=
 them.<br>
&gt; I also would consider working on HAMMER2 block encryption feature, sin=
ce it<br>
&gt; is similar in requirements to my current proposal, if there is already=
 a<br>
&gt; strong candidate to implement the block compression feature.<br>
&gt;<br>
&gt; Thank you.<br>
&gt;<br>
&gt; [1] <a href=3D"http://www.dragonflybsd.org/docs/developer/gsocprojects=
page/" target=3D"_blank">http://www.dragonflybsd.org/docs/developer/gsocpro=
jectspage/</a><br>
&gt; [2] <a href=3D"https://dl.dropboxusercontent.com/u/56646280/zeos.zip"; =
target=3D"_blank">https://dl.dropboxusercontent.com/u/56646280/zeos.zip</a>=
<br>
&gt; [3] <a href=3D"http://leaf.dragonflybsd.org/mailarchive/users/2012-02/=
msg00020.html" target=3D"_blank">http://leaf.dragonflybsd.org/mailarchive/u=
sers/2012-02/msg00020.html</a><br>
&gt;<br>
&gt;<br>
&gt; Daniel<br>
</div></div></blockquote></div><br></div>

--001a11c38e9ac7c72104db361abf--