Re: rt remote free backtrace (#2)

["Sepherosa Ziehau" <sepherosa@xxxxxxxxx>]

On Sun, Nov 2, 2008 at 11:59 AM, Sepherosa Ziehau <sepherosa@gmail.com> wrote:
> On Sun, Nov 2, 2008 at 6:25 AM, Peter Avalos <pavalos@theshell.com> wrote:
>> On Thu, Oct 30, 2008 at 10:04:10PM +0800, Sepherosa Ziehau wrote:
>>> On Thu, Oct 30, 2008 at 10:06 AM, Sepherosa Ziehau <sepherosa@gmail.com> wrote:
>>> > On Thu, Oct 30, 2008 at 12:12 AM, Peter Avalos <pavalos@theshell.com> wrote:
>>> >> On Thu, Sep 11, 2008 at 07:36:02PM +0800, Sepherosa Ziehau wrote:
>>> >>> Hi all,
>>> >>>
>>> >>> I have committed a patch to HEAD, which will print backtrace if
>>> >>> rtfree_remote() is called.  If you see this kind of messages on your
>>> >>> console, please post them to bugs@.  If you think your box could
>>> >>> afford panicking (mainly to get coredumps), you could set
>>> >>> net.route.remote_free_panic to 1, and post the URL to the coredump so
>>> >>> we could fetch and debug it.
>>> >>>
>>> >>
>>> >> Here's a different one I got twice yesterday:
>>> >>
>>> >> rt remote free rt_cpuid 1, mycpuid 0
>>> >> Trace beginning at frame 0xda6f5c90
>>> >> rtfree_remote(c0371380,1,0,da6f5cf4,c023dc8d) at rtfree_remote+0x74
>>> >> rtfree_remote(f059e120,1) at rtfree_remote+0x74
>>> >> rtfree(f059e120,daa41080,da6f5d14,c023e36c,daa44680) at rtfree+0x2e
>>> >> in_rtchange(daa44680,0) at in_rtchange+0x14
>>> >> in_pcbnotifyall(c0421f58,91fa5e77,0,c023d65a) at in_pcbnotifyall+0x46
>>> >> udp_ctlinput(f,d73c9bf8,0,0,da6f5d68) at udp_ctlinput+0x10f
>>> >> netmsg_pru_ctlinput(d73c9b1c,0,c0421f24,da6f5d84,c0233f7a) at
>>> >> netmsg_pru_ctlinput+0x13
>>> >> netmsg_service(d73c9b1c,1,0,c03f3934,ff800000) at netmsg_service+0x44
>>> >> netmsg_service_loop(c0421f24,0,0,0,0) at netmsg_service_loop+0x18
>>> >> lwkt_exit() at lwkt_exit
>>> >
>>> > Ok, I see.  We need to locate the msgport based on 'cmd' paraeter
>>> > passed to pr_ctlport too.  And forwarding in_pcbnotifyall() to all
>>> > related msgports (i.e. all udp msgports or all tcp msgports).  I will
>>> > give you a patch to try when I get home.
>>> >
>>>
>>> Please test/review following patch:
>>> http://leaf.dragonflybsd.org/~sephe/tcp_udp_ctlinput.diff
>>>
>>
>> Panic:
>>
>> panic: netmsg_put_port: self-referential deadlock on netport
>> mp_lock = 00000000; cpuid = 0
>> Trace beginning at frame 0xda6f5ca0
>> panic(da6f5cc4,da6f5cf4,2b063bc9,f,da6f5cc8) at panic+0x14d
>> panic(c037181c,da6f5cf4,da6f5cdc,c01bdc9a,c041dfd0) at panic+0x14d
>> netmsg_put_port(c041dfd0,da6f5cf4) at netmsg_put_port+0x3e
>> lwkt_domsg(c041dfd0,da6f5cf4,0,0) at lwkt_domsg+0x4f
>> udp_ctlinput(f,d73c9bf8,0,0,da6f5d68) at udp_ctlinput+0x151
>> netmsg_pru_ctlinput(d73c9b1c,0,c0422664,da6f5d84,c0234082) at
>> netmsg_pru_ctlinput+0x13
>> netmsg_service(d73c9b1c,1,0,c03f4034,ff800000) at netmsg_service+0x44
>> netmsg_service_loop(c0422664,0,0,0,0) at netmsg_service_loop+0x18
>> lwkt_exit() at lwkt_exit
>> boot() called on cpu#0
>> Uptime: 15m44s
>>
>> (kgdb) bt
>> #0  dumpsys () at ./machine/thread.h:83
>> #1  0xc01b3c1d in boot (howto=260) at
>> /usr/src/sys/kern/kern_shutdown.c:375
>> #2  0xc01b3ee0 in panic (fmt=0xc037181c "netmsg_put_port:
>> self-referential deadlock on netport") at
>> /usr/src/sys/kern/kern_shutdown.c:800
>> #3  0xc0234009 in netmsg_put_port (port=0xc041dfd0, lmsg=0xda6f5cf4) at
>> /usr/src/sys/net/netisr.c:137
>> #4  0xc01bdc9a in lwkt_domsg (port=0x0, msg=0xda6f5cf4, flags=0) at
>> /usr/src/sys/sys/msgport2.h:52
>> #5  0xc025321b in udp_ctlinput (cmd=15, sa=0xd73c9bf8, vip=0x0) at
>> /usr/src/sys/netinet/udp_usrreq.c:700
>> #6  0xc01e5bad in netmsg_pru_ctlinput (msg=0xd73c9b1c) at
>> /usr/src/sys/kern/uipc_msg.c:648
>> #7  0xc0233ca8 in netmsg_service (msg=0xd73c9b1c, mpsafe_mode=1,
>> mplocked=0) at /usr/src/sys/net/netisr.c:286
>> #8  0xc0234082 in netmsg_service_loop (arg=0xc0422664) at
>> /usr/src/sys/net/netisr.c:333
>> #9  0xc01bafee in lwkt_deschedule_self (td=Cannot access memory at
>> address 0x8
>> ) at /usr/src/sys/kern/lwkt_thread.c:228
>> Backtrace stopped: previous frame inner to this frame (corrupt stack?)
>>
>> kernel and vmcore are being uploaded to leaf. (*.22)
>
> Please test this one:
> http://leaf.dragonflybsd.org/~sephe/tcp_udp_ctlinput.diff1

If no problem pops up, I plan to commit the above patch on Nov.11

Best Regards,
sephe

-- 
Live Free or Die