Re: rt remote free backtrace (#2)

[Peter Avalos <pavalos@xxxxxxxxxxxx>]

On Thu, Oct 30, 2008 at 10:04:10PM +0800, Sepherosa Ziehau wrote:
> On Thu, Oct 30, 2008 at 10:06 AM, Sepherosa Ziehau <sepherosa@gmail.com> wrote:
> > On Thu, Oct 30, 2008 at 12:12 AM, Peter Avalos <pavalos@theshell.com> wrote:
> >> On Thu, Sep 11, 2008 at 07:36:02PM +0800, Sepherosa Ziehau wrote:
> >>> Hi all,
> >>>
> >>> I have committed a patch to HEAD, which will print backtrace if
> >>> rtfree_remote() is called.  If you see this kind of messages on your
> >>> console, please post them to bugs@.  If you think your box could
> >>> afford panicking (mainly to get coredumps), you could set
> >>> net.route.remote_free_panic to 1, and post the URL to the coredump so
> >>> we could fetch and debug it.
> >>>
> >>
> >> Here's a different one I got twice yesterday:
> >>
> >> rt remote free rt_cpuid 1, mycpuid 0
> >> Trace beginning at frame 0xda6f5c90
> >> rtfree_remote(c0371380,1,0,da6f5cf4,c023dc8d) at rtfree_remote+0x74
> >> rtfree_remote(f059e120,1) at rtfree_remote+0x74
> >> rtfree(f059e120,daa41080,da6f5d14,c023e36c,daa44680) at rtfree+0x2e
> >> in_rtchange(daa44680,0) at in_rtchange+0x14
> >> in_pcbnotifyall(c0421f58,91fa5e77,0,c023d65a) at in_pcbnotifyall+0x46
> >> udp_ctlinput(f,d73c9bf8,0,0,da6f5d68) at udp_ctlinput+0x10f
> >> netmsg_pru_ctlinput(d73c9b1c,0,c0421f24,da6f5d84,c0233f7a) at
> >> netmsg_pru_ctlinput+0x13
> >> netmsg_service(d73c9b1c,1,0,c03f3934,ff800000) at netmsg_service+0x44
> >> netmsg_service_loop(c0421f24,0,0,0,0) at netmsg_service_loop+0x18
> >> lwkt_exit() at lwkt_exit
> >
> > Ok, I see.  We need to locate the msgport based on 'cmd' paraeter
> > passed to pr_ctlport too.  And forwarding in_pcbnotifyall() to all
> > related msgports (i.e. all udp msgports or all tcp msgports).  I will
> > give you a patch to try when I get home.
> >
> 
> Please test/review following patch:
> http://leaf.dragonflybsd.org/~sephe/tcp_udp_ctlinput.diff
> 

Panic:

panic: netmsg_put_port: self-referential deadlock on netport
mp_lock = 00000000; cpuid = 0
Trace beginning at frame 0xda6f5ca0
panic(da6f5cc4,da6f5cf4,2b063bc9,f,da6f5cc8) at panic+0x14d
panic(c037181c,da6f5cf4,da6f5cdc,c01bdc9a,c041dfd0) at panic+0x14d
netmsg_put_port(c041dfd0,da6f5cf4) at netmsg_put_port+0x3e
lwkt_domsg(c041dfd0,da6f5cf4,0,0) at lwkt_domsg+0x4f
udp_ctlinput(f,d73c9bf8,0,0,da6f5d68) at udp_ctlinput+0x151
netmsg_pru_ctlinput(d73c9b1c,0,c0422664,da6f5d84,c0234082) at
netmsg_pru_ctlinput+0x13
netmsg_service(d73c9b1c,1,0,c03f4034,ff800000) at netmsg_service+0x44
netmsg_service_loop(c0422664,0,0,0,0) at netmsg_service_loop+0x18
lwkt_exit() at lwkt_exit
boot() called on cpu#0
Uptime: 15m44s

(kgdb) bt
#0  dumpsys () at ./machine/thread.h:83
#1  0xc01b3c1d in boot (howto=260) at
/usr/src/sys/kern/kern_shutdown.c:375
#2  0xc01b3ee0 in panic (fmt=0xc037181c "netmsg_put_port:
self-referential deadlock on netport") at
/usr/src/sys/kern/kern_shutdown.c:800
#3  0xc0234009 in netmsg_put_port (port=0xc041dfd0, lmsg=0xda6f5cf4) at
/usr/src/sys/net/netisr.c:137
#4  0xc01bdc9a in lwkt_domsg (port=0x0, msg=0xda6f5cf4, flags=0) at
/usr/src/sys/sys/msgport2.h:52
#5  0xc025321b in udp_ctlinput (cmd=15, sa=0xd73c9bf8, vip=0x0) at
/usr/src/sys/netinet/udp_usrreq.c:700
#6  0xc01e5bad in netmsg_pru_ctlinput (msg=0xd73c9b1c) at
/usr/src/sys/kern/uipc_msg.c:648
#7  0xc0233ca8 in netmsg_service (msg=0xd73c9b1c, mpsafe_mode=1,
mplocked=0) at /usr/src/sys/net/netisr.c:286
#8  0xc0234082 in netmsg_service_loop (arg=0xc0422664) at
/usr/src/sys/net/netisr.c:333
#9  0xc01bafee in lwkt_deschedule_self (td=Cannot access memory at
address 0x8
) at /usr/src/sys/kern/lwkt_thread.c:228
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

kernel and vmcore are being uploaded to leaf. (*.22)

--Peter

Attachment: pgp00000.pgp
Description: PGP signature