Re: Interrupt recursion smashes kernel memory

["Simon 'corecode' Schubert" <corecode@xxxxxxxxxxxx>]

Matthew Dillon wrote:
>     The kernel stack is rather small.  I think it's only 8K or 12K.  It is
>     possible that the nvidia driver is exhausting it just with its normal
>     operation.

The stack is full of interrupt frames, so I am sure that the interrupts 
are being serviced before the old ones can iret:

> :Checking the return addresses, most frames have return addresses of:
> :
> :0xc028fc90 <doreti+0>:  pop    %eax
> :0xc028fc91 <doreti+1>:  mov    $0x0,%eax
> :0xc028fc9d <doreti+13>: cli    
> :
> :or
> :
> :0xc029774f <Xicu_slowintr11+143>:       jmp    0xc028fc90 <doreti>

here is the dump of the overwritten memory area:
(kgdb) x/128x entry->prev
0xd6e25dc0:     0xc029774f      0x00000008      0x00203286      0x00000000
0xd6e25dd0:     0x00000010      0x00000018      0x00000010      0x00000010
0xd6e25de0:     0x0000001c      0xd6814d00      0xd6e26244      0xd6e25e00
0xd6e25df0:     0xd6814d00      0xd6814d00      0x00000000      0x00000000
0xd6e25e00:     0x00000000      0x00000000      0x00000000      0xc028fc9d
0xd6e25e10:     0x00000008      0x00203286      0x00000010      0x00000018
0xd6e25e20:     0x00000010      0x00000010      0x0000001c      0xd6814d00
0xd6e25e30:     0xd6e26244      0xd6e25e48      0xd6814d00      0xd6814d00
0xd6e25e40:     0x00000000      0x00000000      0x00000000      0x00000000
0xd6e25e50:     0x00000000      0xc028fc90      0x00000008      0x00203296
0xd6e25e60:     0x00000000      0x00000010      0x00000018      0x00000010
0xd6e25e70:     0x00000010      0x0000001c      0xd6814d00      0xd6e26244
0xd6e25e80:     0xd6e25e94      0xd6814d00      0xd6814d00      0x00000000
0xd6e25e90:     0x00000000      0x00000000      0x00000000      0x00000000
0xd6e25ea0:     0xc029774f      0x00000008      0x00203286      0x00000000
0xd6e25eb0:     0x00000010      0x00000018      0x00000010      0x00000010
0xd6e25ec0:     0x0000001c      0xd6814d00      0xd6e26244      0xd6e25ee0
0xd6e25ed0:     0xd6814d00      0xd6814d00      0x00000000      0x00000000
0xd6e25ee0:     0x00000000      0x00000000      0x00000000      0xc028fc90
0xd6e25ef0:     0x00000008      0x00203282      0x00000000      0x00000010
0xd6e25f00:     0x00000018      0x00000010      0x00000010      0x0000001c
0xd6e25f10:     0xd6814d00      0xd6e26244      0xd6e25f2c      0xd6814d00
0xd6e25f20:     0xd6814d00      0x00000000      0x00000000      0x00000000
0xd6e25f30:     0x00000000      0x00000000      0xc028fc90      0x00000008
0xd6e25f40:     0x00203286      0x00000000      0x00000010      0x00000018
0xd6e25f50:     0x00000010      0x00000010      0x0000001c      0xd6814d00
0xd6e25f60:     0xd6e26244      0xd6e25f78      0xd6814d00      0xd6814d00
0xd6e25f70:     0x00000000      0x00000000      0x00000000      0x00000000
0xd6e25f80:     0x00000000      0xc028fc9d      0x00000008      0x00203282
0xd6e25f90:     0x00000010      0x00000018      0x00000010      0x00000010
0xd6e25fa0:     0x0000001c      0xd6814d00      0xd6e26244      0xd6e25fc0
0xd6e25fb0:     0xd6814d00      0xd6814d00      0x00000000      0x00000000
(kgdb)
0xd6e25fc0:     0x00000000      0x00000000      0x00000000      0xc029774f
0xd6e25fd0:     0x00000008      0x00203286      0x00000000      0x00000010
0xd6e25fe0:     0x00000018      0x00000010      0x00000010      0x0000001c
0xd6e25ff0:     0xd6814d00      0xd6e26244      0xd6e2600c      0xd6814d00
0xd6e26000:     0xd6814d00      0x00000000      0x00000000      0x00000000
0xd6e26010:     0x00000000      0x00000000      0xc028fc90      0x00000008
0xd6e26020:     0x00203286      0x00000000      0x00000010      0x00000018
0xd6e26030:     0x00000010      0x00000010      0x0000001c      0xd6814d00
0xd6e26040:     0xd6e26244      0xd6e26058      0xd6814d00      0xd6814d00
0xd6e26050:     0x00000000      0x00000000      0x00000000      0x00000000
0xd6e26060:     0x00000000      0xc028fc9d      0x00000008      0x00203286
0xd6e26070:     0x00000010      0x00000018      0x00000010      0x00000010
0xd6e26080:     0x0000001c      0xd6814d00      0xd6e26244      0xd6e260a0
0xd6e26090:     0xd6814d00      0xd6814d00      0x00000000      0x00000000
0xd6e260a0:     0x00000000      0x00000000      0x00000000      0xc028fc90
0xd6e260b0:     0x00000008      0x00203286      0x00000000      0x00000010
0xd6e260c0:     0x00000018      0x00000010      0x00000010      0x0000001c
0xd6e260d0:     0xd6814d00      0xd6e26244      0xd6e260ec      0xd6814d00
0xd6e260e0:     0xd6814d00      0x00000000      0x00000000      0x00000000
0xd6e260f0:     0x00000000      0x00000000      0xc028fc90      0x00000008
0xd6e26100:     0x00203282      0x00000000      0x00000010      0x00000018
0xd6e26110:     0x00000010      0x00000010      0x0000001c      0xd6814d00
0xd6e26120:     0xd6e26244      0xd6e26138      0xd6814d00      0xd6814d00
0xd6e26130:     0x00000000      0x00000000      0x00000000      0x00000000
0xd6e26140:     0x00000000      0xc029774f      0x00000008      0x00203296
0xd6e26150:     0x00000000      0x00000010      0x00000018      0x00000010
0xd6e26160:     0x00000010      0x0000001c      0xd6814d00      0xd6e26244
0xd6e26170:     0xd6e26184      0xd6814d00      0xd6814d00      0x00000000
0xd6e26180:     0x00000000      0x00000000      0x00000000      0x00000000
0xd6e26190:     0xc028fc90      0x00000008      0x00203286      0x00000000
0xd6e261a0:     0x00000010      0x00000018      0x00000010      0x00000010
0xd6e261b0:     0x0000001c      0xd6814d00      0xd6e26244      0xd6e261d0

I'll work from the upper addresses downwards:

frame (eflags)	eip		function
0xd6e26198	0xc028fc90 <doreti>:    pop    %eax
0xd6e2614c	0xc029774f <Xicu_slowintr11+143>: jmp 0xc028fc90 <doreti>
0xd6e26100	0xc028fc90 <doreti>:    pop    %eax
0xd6e260b4	0xc028fc90 <doreti>:    pop    %eax
0xd6e2606c	0xc028fc9d <doreti+13>: cli
0xd6e26020	0xc028fc90 <doreti>:    pop    %eax
0xd6e25fd4	0xc029774f <Xicu_slowintr11+143>: jmp 0xc028fc90 <doreti>
0xd6e25f8c	0xc028fc9d <doreti+13>: cli
0xd6e25f40	0xc028fc90 <doreti>:    pop    %eax
0xd6e25ef4	0xc028fc90 <doreti>:    pop    %eax
0xd6e25ea8	0xc029774f <Xicu_slowintr11+143>: jmp 0xc028fc90 <doreti>
0xd6e25e5c	0xc028fc90 <doreti>:    pop    %eax
0xd6e25e14	0xc028fc9d <doreti+13>: cli
0xd6e25dc8	0xc029774f <Xicu_slowintr11+143>: jmp 0xc028fc90 <doreti>

I've also found stacks going up to

0xc028fe40 <splz>:      pushf
via
0xc018b7cf <lwkt_yield_quick+42>:       cmpl   $0x0,0xc031eac8
0xc018bc5a <lwkt_schedule+315>: add    $0xc,%esp

All these locations are within the ISR.  There *is* interrupt recursion 
going on.

>     Reentrancy is protected.  The interrupt is masked when taken and only
>     unmasked after the interrupt procedure has completed operation.  In
>     the case of scheduled interrupts the interrupt is masked when the
>     interrupt is taken and unmasked by the interrupt thread after it
>     finishes processing it.

I see.  Still, something is wrong.  Maybe my ICU is broken and sometimes 
passes interrupts despite them being disabled?

>     Is IRQ11 the video interrupt during your tests?  It kinda sounds like
>     normal calls to the nvidia driver are causing the problem.

Yes, intr 11 is used by the video card.  I really can't see how this could 
be normal calls, because after all, all of these stack frames are in the 
interrupt path.

cheers
  simon

--
Serve - BSD     +++  RENT this banner advert  +++    ASCII Ribbon   /"\
Work - Mac      +++  space for low €€€ NOW!1  +++      Campaign     \ /
Party Enjoy Relax   |   http://dragonflybsd.org      Against  HTML   \
Dude 2c 2 the max   !   http://golden-apple.biz       Mail + News   / \