Re: Porting sbcl to DragonFly

[vasily postnicov <shamaz.mazum@xxxxxxxxx>]

--089e0158b3aa2279c404d93ab3ad
Content-Type: text/plain; charset=ISO-8859-1

2013/3/30 Chris Turner <c.turner@199technologies.com>

> On 03/30/13 00:16, vasily postnicov wrote:
>
>  So I have some questions:
>> 1. src/runtime/Config.x86-64-**freebsd states that we must use 1:1
>> threading not m:n threading and uses lthr for that purpose. It seems that
>> there isn't lthr in dragonfly, only lpthread is present. Should I use it?
>> Is 1:1 threading supported? (It seems to work with lpthread anyway).
>> 2. SBCL uses GCC TLS with (and only with) FreeBSD. I heard something
>> about lack of "native" TLS in FreeBSD. Is it true? Is TLS present on
>> DragonFly?
>>
>
> I'm pretty certain that both of these are out of date statements w/r/t
> FreeBSD (relating to libc_r and KSE in the 4,5,6,7 era)
>
> To also clarify, our libpthread uses 1:1 model.
>
> If you get frustrated or finished with SBCL and want some other 'Low
> Hanging LISP Fruit',
> fixing up boehmgc should make ECL work properly, and will also help with
> other fun and interesting
> interpreters - see also:
>
> http://bugs.dragonflybsd.org/**issues/1525<http://bugs.dragonflybsd.org/issues/1525>
>
> ECL is a bit more straightforward of a LISP since it isn't doing as
> much code generation/low level stuff IIRC, instead relying on more
> 'standard' posixy stuff.
>
> Cheers and good luck!
>
> - Chris
>

Thanks for helping. I still have problems with sb-concurrency on x86-64.
I've also implemented x86 support (but without threads) just because it is
simple "make sbcl to do like in FreeBSD" job. Thread support is rather
difficult for me to implement. To be specific, look at this piece of code:

int arch_os_thread_init(struct thread *thread) {

#ifdef LISP_FEATURE_SB_THREAD
    int n;

    struct segment_descriptor ldt_entry = { 0, 0, SDT_MEMRW, SEL_UPL, 1,
                                            0, 0, 1, 0, 0 };

    set_data_desc_addr(&ldt_entry, thread);
    set_data_desc_size(&ldt_entry, dynamic_values_bytes);

    n = i386_set_ldt(LDT_AUTO_ALLOC, (union descriptor*) &ldt_entry, 1);
    if (n < 0) {
        perror("i386_set_ldt");
        lose("unexpected i386_set_ldt(..) failure\n");
    }
    FSHOW_SIGNAL((stderr, "/ TLS: Allocated LDT %x\n", n));
    thread->tls_cookie=n;
    arch_os_load_ldt(thread);

....... /* Skipped few lines */
}

int arch_os_thread_cleanup(struct thread *thread) {

#if defined(LISP_FEATURE_SB_THREAD)
    int n = thread->tls_cookie;

    /* Set the %%fs register back to 0 and free the ldt by setting it
     * to NULL.
     */
    FSHOW_SIGNAL((stderr, "/ TLS: Freeing LDT %x\n", n));

    __asm__ __volatile__ ("mov %0, %%fs" : : "r"(0));
    i386_set_ldt(n, NULL, 1);
#endif

    return 1;                  /* success */
}

The problem is in i386_set_ldt. LDT_AUTO_ALLOC as a first argument says to
this FreeBSD version of i386_set_ldt to add new segment descriptor to LDT
and return its selector. In DragonFly, first argument is itself a selector
number and there is no such thing LDT_AUTO_ALLOC, so I have completely no
idea what I must do here.

BTW, arch_os_thread_init, of course, is called from new thread, not from
parent thread.

Also, I wonder why LDT is used at all. Wikipedia says, that descriptor
tables were used in i286 as a memory addressing mechanism prior to paging
(AFAIK, paging first appeared in i386). With descriptor tables it was
possible to translate "logical" addresses to real physical addresses
(sorry, if I use wrong terminology). With LDT it was possible to translate
same "logical" addresses to different physical for different processes, as
I understand it. Wikipedia says: "Therefore, modern
32-bit<http://en.wikipedia.org/wiki/32-bit>x86 operating systems use
the LDT very little, primarily to run legacy
16-bit <http://en.wikipedia.org/wiki/16-bit> code." I do not know, what LDT
has to do with TLS. Can you explain this?

So, all in all, things to do:
1. Add threads to x86 (Or x86 is dead now?)
2. Fix sb-concurrency (and possibly other pitfalls) in x86-64

--089e0158b3aa2279c404d93ab3ad
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div><div><div><br><div class=3D"gmail_extra"><br><br><div=
 class=3D"gmail_quote">2013/3/30 Chris Turner <span dir=3D"ltr">&lt;<a href=
=3D"mailto:c.turner@199technologies.com"; target=3D"_blank">c.turner@199tech=
nologies.com</a>&gt;</span><br>
<blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-=
left:1px solid rgb(204,204,204);padding-left:1ex"><div class=3D"im">On 03/3=
0/13 00:16, vasily postnicov wrote:<br>
<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-=
left:1px solid rgb(204,204,204);padding-left:1ex">
So I have some questions:<br>
1. src/runtime/Config.x86-64-<u></u>freebsd states that we must use 1:1 thr=
eading not m:n threading and uses lthr for that purpose. It seems that ther=
e isn&#39;t lthr in dragonfly, only lpthread is present. Should I use it? I=
s 1:1 threading supported? (It seems to work with lpthread anyway).<br>

2. SBCL uses GCC TLS with (and only with) FreeBSD. I heard something about =
lack of &quot;native&quot; TLS in FreeBSD. Is it true? Is TLS present on Dr=
agonFly?<br>
</blockquote>
<br></div>
I&#39;m pretty certain that both of these are out of date statements w/r/t =
FreeBSD (relating to libc_r and KSE in the 4,5,6,7 era)<br>
<br>
To also clarify, our libpthread uses 1:1 model.<br>
<br>
If you get frustrated or finished with SBCL and want some other &#39;Low Ha=
nging LISP Fruit&#39;,<br>
fixing up boehmgc should make ECL work properly, and will also help with ot=
her fun and interesting<br>
interpreters - see also:<br>
<br>
<a href=3D"http://bugs.dragonflybsd.org/issues/1525"; target=3D"_blank">http=
://bugs.dragonflybsd.org/<u></u>issues/1525</a><br>
<br>
ECL is a bit more straightforward of a LISP since it isn&#39;t doing as<br>
much code generation/low level stuff IIRC, instead relying on more &#39;sta=
ndard&#39; posixy stuff.<br>
<br>
Cheers and good luck!<span class=3D""><font color=3D"#888888"><br>
<br>
- Chris<br>
</font></span></blockquote></div><br></div><div class=3D"gmail_extra">Thank=
s for helping. I still have problems with sb-concurrency on x86-64. I&#39;v=
e also implemented x86 support (but without threads) just because it is sim=
ple &quot;make sbcl to do like in FreeBSD&quot; job. Thread support is rath=
er difficult for me to implement. To be specific, look at this piece of cod=
e:<br>
<br>int arch_os_thread_init(struct thread *thread) {<br><br>#ifdef LISP_FEA=
TURE_SB_THREAD<br>=A0=A0=A0 int n;<br><br>=A0=A0=A0 struct segment_descript=
or ldt_entry =3D { 0, 0, SDT_MEMRW, SEL_UPL, 1,<br>=A0=A0=A0=A0=A0=A0=A0=A0=
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 0, 0, 1, 0, 0 };<br>
<br>=A0=A0=A0 set_data_desc_addr(&amp;ldt_entry, thread);<br>=A0=A0=A0 set_=
data_desc_size(&amp;ldt_entry, dynamic_values_bytes);<br><br>=A0=A0=A0 n =
=3D i386_set_ldt(LDT_AUTO_ALLOC, (union descriptor*) &amp;ldt_entry, 1);<br=
>=A0=A0=A0 if (n &lt; 0) {<br>
=A0=A0=A0=A0=A0=A0=A0 perror(&quot;i386_set_ldt&quot;);<br>=A0=A0=A0=A0=A0=
=A0=A0 lose(&quot;unexpected i386_set_ldt(..) failure\n&quot;);<br>=A0=A0=
=A0 }<br>=A0=A0=A0 FSHOW_SIGNAL((stderr, &quot;/ TLS: Allocated LDT %x\n&qu=
ot;, n));<br>=A0=A0=A0 thread-&gt;tls_cookie=3Dn;<br>
=A0=A0=A0 arch_os_load_ldt(thread);<br><br></div><div class=3D"gmail_extra"=
>....... /* Skipped few lines */<br></div><div class=3D"gmail_extra">}<br><=
br>int arch_os_thread_cleanup(struct thread *thread) {<br><br>#if defined(L=
ISP_FEATURE_SB_THREAD)<br>
=A0=A0=A0 int n =3D thread-&gt;tls_cookie;<br><br>=A0=A0=A0 /* Set the %%fs=
 register back to 0 and free the ldt by setting it<br>=A0=A0=A0=A0 * to NUL=
L.<br>=A0=A0=A0=A0 */<br>=A0=A0=A0 FSHOW_SIGNAL((stderr, &quot;/ TLS: Freei=
ng LDT %x\n&quot;, n));<br><br>=A0=A0=A0 __asm__ __volatile__ (&quot;mov %0=
, %%fs&quot; : : &quot;r&quot;(0));<br>
=A0=A0=A0 i386_set_ldt(n, NULL, 1);<br>#endif<br><br>=A0=A0=A0 return 1;=A0=
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 /* success */<br>}<br><br>=
</div><div class=3D"gmail_extra">The problem is in i386_set_ldt. LDT_AUTO_A=
LLOC as a first argument says to this FreeBSD version of i386_set_ldt to ad=
d new segment descriptor to LDT and return its selector. In DragonFly, firs=
t argument is itself a selector number and there is no such thing LDT_AUTO_=
ALLOC, so I have completely no idea what I must do here.<br>
<br></div><div class=3D"gmail_extra">BTW, arch_os_thread_init, of course, i=
s called from new thread, not from parent thread. <br><br></div><div class=
=3D"gmail_extra">Also, I wonder why LDT is used at all. Wikipedia says, tha=
t descriptor tables were used in i286 as a memory addressing mechanism prio=
r to paging (AFAIK, paging first appeared in i386). With descriptor tables =
it was possible to translate &quot;logical&quot; addresses to real physical=
 addresses (sorry, if I use wrong terminology). With LDT it was possible to=
 translate same &quot;logical&quot; addresses to different physical for dif=
ferent processes, as I understand it. Wikipedia says: &quot;Therefore, mode=
rn <a href=3D"http://en.wikipedia.org/wiki/32-bit"; title=3D"32-bit">32-bit<=
/a> x86 operating systems use the LDT very little, primarily to run legacy =
<a href=3D"http://en.wikipedia.org/wiki/16-bit"; title=3D"16-bit">16-bit</a>=
 code.&quot; I do not know, what LDT has to do with TLS. Can you explain th=
is?</div>
<br></div>So, all in all, things to do:<br></div>1. Add threads to x86 (Or =
x86 is dead now?)<br></div>2. Fix sb-concurrency (and possibly other pitfal=
ls) in x86-64<br></div>

--089e0158b3aa2279c404d93ab3ad--