Re: GSoC: Add SMT/HT awareness to DragonFlyBSD scheduler

["Samuel J. Greear" <sjg@xxxxxxxxxxxx>]

--14dae934041bdf2c1604c390f55d
Content-Type: text/plain; charset=ISO-8859-1

Mihai,

Dillon tried this out on monster, the 48-core Opteron and it seems like it
got a little confused, output in the link below:

http://apollo.backplane.com/DFlyMisc/smt.txt

Sam

On Sun, Jun 3, 2012 at 5:00 AM, Mihai Carabas <mihai.carabas@gmail.com>wrote:

> Hello,
>
> This week, I managed to *dynamicly detect *the CPU topology of a
> platform. There are 2 components: platform depedent code which determines
> the mask_size within APICID for logical and core ids. For now, is available
> only for 64bit, intel and amd (I will make code available for 32bit soon).
> The code was written here [1], from line 1107.
>
> Another component, is the platform independent. I used the
> logical_CPU_bits and core_bits to determine the number of entities per
> level and build a topology tree (I tried to make all as general as
> possible). For this code I added a new file [2]. The init function is
> registered through SYSINIT to run before the initialization of the
> USCHED. All the topology information is available through sysctl.
>
> Here are outputs for 3 different topologies. Unfortunately, I don't have
> an AMD platform to test on. All tests are done on INTEL platforms.
>
> *1) two physical CPUs (emulated through VMWARE - yes vmware gives this
> posibility)*
>
> hw.cpu_topology.tree:
> \-PACKAGE MEMBERS: cpu0 cpu1
>   |-CHIP ID 0: cpu0
>   \-CHIP ID 1: cpu1
> hw.cpu_topology.members: cpu0 cpu1
>
> hw.cpu_topology.cpu0.physical_id: 0
> hw.cpu_topology.cpu0.physical_sibings: cpu0
>
> hw.cpu_topology.cpu1.physical_id: 1
> hw.cpu_topology.cpu1.physical_sibings: cpu1
>
>
> *2) two cores on one physical CPU*
>
> hw.cpu_topology.tree:
> \-PACKAGE MEMBERS: cpu0 cpu1
>   \-CHIP ID 0: cpu0 cpu1
>     |-CORE ID 0: cpu0
>     \-CORE ID 1: cpu1
> hw.cpu_topology.members: cpu0 cpu1
>
> hw.cpu_topology.cpu0.physical_id: 0
> hw.cpu_topology.cpu0.physical_sibings: cpu0 cpu1
> hw.cpu_topology.cpu0.core_id: 0
> hw.cpu_topology.cpu0.core_siblings: cpu0
>
> hw.cpu_topology.cpu1.physical_id: 0
> hw.cpu_topology.cpu1.physical_sibings: cpu0 cpu1
> hw.cpu_topology.cpu1.core_id: 1
> hw.cpu_topology.cpu1.core_siblings: cpu1
>
>
> *3) two cores on one physical CPU and two threads on each core.*
>
> hw.cpu_topology.tree:
> \-PACKAGE MEMBERS: cpu0 cpu1 cpu2 cpu3
>   \-CHIP ID 0: cpu0 cpu1 cpu2 cpu3
>     |-CORE ID 0: cpu0 cpu2
>     | |-THREAD ID 0: cpu0
>     | \-THREAD ID 1: cpu2
>     \-CORE ID 1: cpu1 cpu3
>       |-THREAD ID 0: cpu1
>       \-THREAD ID 1: cpu3
> hw.cpu_topology.members: cpu0 cpu1 cpu2 cpu3
>
> hw.cpu_topology.cpu0.physical_id: 0
> hw.cpu_topology.cpu0.physical_sibings: cpu0 cpu1 cpu2 cpu3
> hw.cpu_topology.cpu0.core_id: 0
> hw.cpu_topology.cpu0.core_siblings: cpu0 cpu2
>
> hw.cpu_topology.cpu1.physical_id: 0
> hw.cpu_topology.cpu1.physical_sibings: cpu0 cpu1 cpu2 cpu3
> hw.cpu_topology.cpu1.core_id: 1
> hw.cpu_topology.cpu1.core_siblings: cpu1 cpu3
>
> hw.cpu_topology.cpu2.physical_id: 0
> hw.cpu_topology.cpu2.physical_sibings: cpu0 cpu1 cpu2 cpu3
> hw.cpu_topology.cpu2.core_id: 0
> hw.cpu_topology.cpu2.core_siblings: cpu0 cpu2
>
> hw.cpu_topology.cpu3.physical_id: 0
> hw.cpu_topology.cpu3.physical_sibings: cpu0 cpu1 cpu2 cpu3
> hw.cpu_topology.cpu3.core_id: 1
> hw.cpu_topology.cpu3.core_siblings: cpu1 cpu3
>
> Now, every CPU, have in its own structure, the knowledge of the whole
> topology. The next step is to develop some meaningful tests and begin
> playing with scheduling decisions. Scheduling will take into account the
> load of the target's siblings cpu, too.
>
> For those who want to test this, they have to clone the repo [3]. May be
> next week, if you want, I will make a full patch with my code till now.
>
> Any feedback is welcome!
>
> Thanks,
> Mihai Carabas
>
> [1]
> https://github.com/mihaicarabas/dragonfly/blob/master/sys/platform/pc64/x86_64/mp_machdep.c
> [2]
> https://github.com/mihaicarabas/dragonfly/blob/master/sys/kern/cpu_topology.c
> [3] https://github.com/mihaicarabas/dragonfly
>

--14dae934041bdf2c1604c390f55d
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Mihai,<div><br></div><div>Dillon tried this out on monster, the 48-core Opt=
eron and it seems like it got a little confused, output in the link below:<=
/div><div><br></div><div><a href=3D"http://apollo.backplane.com/DFlyMisc/sm=
t.txt">http://apollo.backplane.com/DFlyMisc/smt.txt</a>=A0</div>
<div><br></div><div>Sam<br><br><div class=3D"gmail_quote">On Sun, Jun 3, 20=
12 at 5:00 AM, Mihai Carabas <span dir=3D"ltr">&lt;<a href=3D"mailto:mihai.=
carabas@gmail.com" target=3D"_blank">mihai.carabas@gmail.com</a>&gt;</span>=
 wrote:<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex">Hello,<div><br></div><div>This week, I manag=
ed to <i style=3D"font-weight:bold">dynamicly detect </i>the CPU topology o=
f a platform.=A0<span>There are 2 components: platform depedent code which =
determines the mask_size within APICID for logical and core ids. For now, i=
s available only for 64bit, intel and amd (I will make code available for 3=
2bit soon). The code was written here [1], from line 1107.</span></div>

<div><div><br></div><div>Another component, is the platform independent. I =
used the logical_CPU_bits and core_bits to determine the number of entities=
 per level and build a topology tree (I tried to make all as general as pos=
sible). For this code I added a new file [2]. The init function is register=
ed through SYSINIT to run before the initialization of the USCHED.=A0All th=
e topology information is available through sysctl.</div>

<div><br></div><div>Here are outputs for 3 different topologies. Unfortunat=
ely, I don&#39;t have an AMD platform to test on. All tests are done on INT=
EL platforms.</div><div><br></div><div><b><i>1) two physical CPUs (emulated=
 through VMWARE - yes vmware gives this posibility)</i></b></div>

<div><div><br></div><div>hw.cpu_topology.tree:</div><div>\-PACKAGE MEMBERS:=
 cpu0 cpu1</div><div>=A0 |-CHIP ID 0: cpu0</div><div>=A0 \-CHIP ID 1: cpu1<=
/div><div>hw.cpu_topology.members: cpu0 cpu1</div><div><br></div><div>
hw.cpu_topology.cpu0.physical_id: 0</div><div>hw.cpu_topology.cpu0.physical=
_sibings: cpu0</div><div><br></div><div>hw.cpu_topology.cpu1.physical_id: 1=
</div><div>hw.cpu_topology.cpu1.physical_sibings: cpu1</div><div><br></div>

</div><div><br></div><div><b><i>2) two cores on one physical CPU</i></b></d=
iv><div><div><br></div><div>hw.cpu_topology.tree:</div><div>\-PACKAGE MEMBE=
RS: cpu0 cpu1</div><div>=A0 \-CHIP ID 0: cpu0 cpu1</div>
<div>=A0 =A0 |-CORE ID 0: cpu0</div><div>=A0 =A0 \-CORE ID 1: cpu1</div><di=
v>hw.cpu_topology.members: cpu0 cpu1</div><div><br></div><div>hw.cpu_topolo=
gy.cpu0.physical_id: 0</div><div>hw.cpu_topology.cpu0.physical_sibings: cpu=
0 cpu1</div>

<div>hw.cpu_topology.cpu0.core_id: 0</div><div>hw.cpu_topology.cpu0.core_si=
blings: cpu0</div><div><br></div><div>hw.cpu_topology.cpu1.physical_id: 0</=
div><div>hw.cpu_topology.cpu1.physical_sibings: cpu0 cpu1</div><div>hw.cpu_=
topology.cpu1.core_id: 1</div>

<div>hw.cpu_topology.cpu1.core_siblings: cpu1</div></div><div><br></div><di=
v><br></div><div><i><b>3) two cores on one physical CPU and two threads on =
each core.</b></i></div><div><div><br></div><div>
hw.cpu_topology.tree:</div><div>\-PACKAGE MEMBERS: cpu0 cpu1 cpu2 cpu3</div=
><div>=A0 \-CHIP ID 0: cpu0 cpu1 cpu2 cpu3</div><div>=A0 =A0 |-CORE ID 0: c=
pu0 cpu2</div><div>=A0 =A0 | |-THREAD ID 0: cpu0</div><div>=A0 =A0 | \-THRE=
AD ID 1: cpu2</div>

<div>=A0 =A0 \-CORE ID 1: cpu1 cpu3</div><div>=A0 =A0 =A0 |-THREAD ID 0: cp=
u1</div><div>=A0 =A0 =A0 \-THREAD ID 1: cpu3</div><div>hw.cpu_topology.memb=
ers: cpu0 cpu1 cpu2 cpu3</div><div><br></div><div>hw.cpu_topology.cpu0.phys=
ical_id: 0</div>

<div>hw.cpu_topology.cpu0.physical_sibings: cpu0 cpu1 cpu2 cpu3</div><div>h=
w.cpu_topology.cpu0.core_id: 0</div><div>hw.cpu_topology.cpu0.core_siblings=
: cpu0 cpu2</div><div><br></div><div>hw.cpu_topology.cpu1.physical_id: 0</d=
iv>

<div>hw.cpu_topology.cpu1.physical_sibings: cpu0 cpu1 cpu2 cpu3</div><div>h=
w.cpu_topology.cpu1.core_id: 1</div><div>hw.cpu_topology.cpu1.core_siblings=
: cpu1 cpu3</div><div><br></div><div>hw.cpu_topology.cpu2.physical_id: 0</d=
iv>

<div>hw.cpu_topology.cpu2.physical_sibings: cpu0 cpu1 cpu2 cpu3</div><div>h=
w.cpu_topology.cpu2.core_id: 0</div><div>hw.cpu_topology.cpu2.core_siblings=
: cpu0 cpu2</div><div><br></div><div>hw.cpu_topology.cpu3.physical_id: 0</d=
iv>

<div>hw.cpu_topology.cpu3.physical_sibings: cpu0 cpu1 cpu2 cpu3</div><div>h=
w.cpu_topology.cpu3.core_id: 1</div><div>hw.cpu_topology.cpu3.core_siblings=
: cpu1 cpu3</div></div><div><br></div><div>Now, every CPU, have in its own =
structure, the knowledge of the whole topology. The next step is to develop=
 some meaningful tests and begin playing with scheduling decisions. Schedul=
ing will take into account the load of the target&#39;s siblings cpu, too.<=
/div>

<div class=3D"gmail_quote"><br></div></div><div class=3D"gmail_quote">For t=
hose who want to test this, they have to clone the repo [3]. May be next we=
ek, if you want, I will make a full patch with my code till now.</div><div =
class=3D"gmail_quote">

<div><br></div><div>Any feedback is welcome!</div><div><br></div><div>Thank=
s,</div><div>Mihai Carabas</div><div><br></div><div>[1]=A0<a href=3D"https:=
//github.com/mihaicarabas/dragonfly/blob/master/sys/platform/pc64/x86_64/mp=
_machdep.c" style=3D"color:rgb(17,85,204)" target=3D"_blank">https://github=
.com/mihaicarabas/dragonfly/blob/master/sys/platform/pc64/x86_64/mp_machdep=
.c</a></div>

<div>[2]=A0<a href=3D"https://github.com/mihaicarabas/dragonfly/blob/master=
/sys/kern/cpu_topology.c" style=3D"color:rgb(17,85,204)" target=3D"_blank">=
https://github.com/mihaicarabas/dragonfly/blob/master/sys/kern/cpu_topology=
.c</a></div>

</div><div class=3D"gmail_quote">[3]=A0<a href=3D"https://github.com/mihaic=
arabas/dragonfly" target=3D"_blank">https://github.com/mihaicarabas/dragonf=
ly</a></div>
</blockquote></div><br></div>

--14dae934041bdf2c1604c390f55d--