3Ware 9500 - incorrect attachment of twa driver to card causing constant rebuilds of system (mirrored) volume?

[Tomaž Borštnar <tomaz.borstnar@xxxxxxxx>]

Hello!

	I have Tyan motherboard with 3Ware 9500-8 controller which is recognized just fine. Installer properly installs Dfly on it and 
everything. Now that system is running we are trying to set it up according to our needs - few jails and such. After rebooting the 
machine via reboot 3ware card ALWAYS shows that it needs to rebuild the first array which hosts OS. Then it takes some 23 minutes 
to do it (a pair of 73G Raptors) and lights show actual activity. Why does dmesg below show 2 twa devices with second one having 
warnings?

Machine is:

Tyan motherboard with dual Opteron 242
2GB RAM
3Ware 9500-8 with 2 RAID-1 volumes

DragonFly dkdev_master 1.1-CURRENT DragonFly 1.1-CURRENT #0: Tue Sep  7 00:15:03 GMT 2004 
root@xxxxxxxxxxxxxxxxxxxxxx:/usr/obj/usr/src/sys/GENERIC  i386

(I am in the middle of updating to latest sources...)

here is dmesg:

Copyright (c) 2003, 2004 The DragonFly Project.
Copyright (c) 1992-2003 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
DragonFly 1.1-CURRENT #0: Tue Sep  7 00:15:03 GMT 2004
    root@xxxxxxxxxxxxxxxxxxxxxx:/usr/obj/usr/src/sys/GENERIC
TSC clock: 1594084800 Hz, i8254 clock: 1193037 Hz
CPU: AMD Opteron(tm) Processor 242 (1594.28-MHz 686-class CPU)
  Origin = "AuthenticAMD"  Id = 0xf5a  Stepping = 10
  Features=0x78bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CM
OV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2>
  AMD Features=0xe0500000<<b20>,AMIE,<b29>,DSP,3DNow!>
real memory  = 2147418112 (2097088K bytes)
avail memory = 2078502912 (2029788K bytes)
Preloaded elf kernel "/kernel" at 0xc065a000.
Preloaded elf module "/modules/acpi.ko" at 0xc065a1e8.
Pentium Pro MTRR support enabled
md0: Malloc disk
pcibios: BIOS version 2.10
Using $PIR table, 15 entries at 0xc00f48c0
npx0: <math processor> on motherboard
npx0: INT 16 interface
Using XMM optimized bcopy/copyin/copyout
acpi0: <A M I OEMXSDT> on motherboard
installed MI handler for int 9
acpi0: Power Button (fixed)
cpu0: <ACPI CPU> on acpi0
pcib1: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pcib1: couldn't read bus number from cfg space
pcib1: trying bus number 1
pci1: <ACPI PCI bus> on pcib1
3ware device driver for 9000 series storage controllers, version: 2.40.00.000
twa0: <3ware 9000 series Storage Controller> port 0x9800-0x98ff mem 0xfb800000-0
xfbffffff,0xfc7ffc00-0xfc7ffcff irq 9 at device 1.0 on pci1
twa0: WARNING: (0x04: 0x0008): Unclean shutdown detected: unit=0
twa0: 8 ports, Firmware FE9X 2.02.00.011, BIOS BE9X 2.02.01.037
acpi_button0: <Power Button> on acpi0
pcib0: <Host to PCI bridge> at pcibus 0 on motherboard
pci0: <PCI bus> on pcib0
pcib2: <PCI to PCI bridge (vendor=1022 device=7460)> at device 6.0 on pci0
pci3: <PCI bus> on pcib2
ohci0: <OHCI (generic) USB controller> mem 0xfeafc000-0xfeafcfff irq 10 at devic
e 0.0 on pci3
installed MI handler for int 10
usb0: OHCI version 1.0, legacy support
usb0: <OHCI (generic) USB controller> on ohci0
usb0: USB revision 1.0
uhub0: AMD OHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub0: 3 ports with 3 removable, self powered
ohci1: <OHCI (generic) USB controller> mem 0xfeafd000-0xfeafdfff irq 10 at devic
e 0.1 on pci3
usb1: OHCI version 1.0, legacy support
usb1: <OHCI (generic) USB controller> on ohci1
usb1: USB revision 1.0
uhub1: AMD OHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub1: 3 ports with 3 removable, self powered
atapci0: <SiI 3114 SATA controller> port 0xa800-0xa80f,0xac00-0xac03,0xb000-0xb0
07,0xb400-0xb403,0xbc00-0xbc07 mem 0xfeafec00-0xfeafefff irq 10 at device 5.0 on
 pci3
ata2: at 0xbc00 on atapci0
ata3: at 0xb000 on atapci0
pci3: <ATI Mach64-GR graphics accelerator> at 6.0 irq 11
fxp0: <Intel 82551 Pro/100 Ethernet> port 0xa400-0xa43f mem 0xfeaa0000-0xfeabfff
f,0xfeafb000-0xfeafbfff irq 11 at device 8.0 on pci3
installed MI handler for int 11
miibus0: <MII bus> on fxp0
inphy0: <i82555 10/100 media interface> on miibus0
inphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
fxp0: MAC address: 00:e0:81:2b:1c:54
isab0: <PCI to ISA bridge (vendor=1022 device=7468)> at device 7.0 on pci0
isa0: <ISA bus> on isab0
atapci1: <AMD 8111 UltraATA/133 controller> port 0xffa0-0xffaf at device 7.1 on
pci0
ata0: at 0x1f0 irq 14 on atapci1
installed MI handler for int 14
ata1: at 0x170 irq 15 on atapci1
installed MI handler for int 15
pci0: <unknown card> (vendor=0x1022, dev=0x746a) at 7.2 irq 10
pci0: <PCI to Other bridge (vendor=1022 device=746b)> at 7.3
pcib3: <PCI to PCI bridge (vendor=1022 device=7450)> at device 10.0 on pci0
pci2: <PCI bus> on pcib3
bge0: <Broadcom BCM5704C Dual Gigabit Ethernet, ASIC rev. 0x2003> mem 0xfc8b0000
-0xfc8bffff,0xfc8c0000-0xfc8cffff irq 5 at device 9.0 on pci2
installed MI handler for int 5
miibus1: <MII bus> on bge0
brgphy0: <BCM5704 10/100/1000baseTX PHY> on miibus1
brgphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseTX, 1000baseTX
-FDX, auto
bge0: MAC address: 00:e0:81:2b:1c:74
bge1: <Broadcom BCM5704C Dual Gigabit Ethernet, ASIC rev. 0x2003> mem 0xfc8e0000
-0xfc8effff,0xfc8f0000-0xfc8fffff irq 9 at device 9.1 on pci2
miibus2: <MII bus> on bge1
brgphy1: <BCM5704 10/100/1000baseTX PHY> on miibus2
brgphy1:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseTX, 1000baseTX
-FDX, auto
bge1: MAC address: 00:e0:81:2b:1c:75
pci0: <unknown card> (vendor=0x1022, dev=0x7451) at 10.1
pcib4: <PCI to PCI bridge (vendor=1022 device=7450)> at device 11.0 on pci0
pci4: <PCI bus> on pcib4
twa1: <3ware 9000 series Storage Controller> port 0x9800-0x98ff mem 0xfb800000-0
xfbffffff,0xfc7ffc00-0xfc7ffcff irq 9 at device 1.0 on pci4
twa1: can't allocate register window.
twa(187): WARNING: cdevsw removed multiple times!
twa: Warning: cdevsw_remove() called while 2 device refs still exist! (major 187
)
device_probe_and_attach: twa1 attach returned 6
pci0: <unknown card> (vendor=0x1022, dev=0x7451) at 11.1
orm0: <Option ROMs> at iomem 0xc0000-0xc7fff,0xc8000-0xcc7ff,0xcc800-0xcdfff on
isa0
pmtimer0 on isa0
fdc0: <NEC 72065B or clone> at port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on isa0
installed MI handler for int 6
fdc0: FIFO enabled, 8 bytes threshold
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> flags 0x1 irq 1 on atkbdc0
kbd0 at atkbd0
installed MI handler for int 1
psm0: failed to get data.
psm0: <PS/2 Mouse> irq 12 on atkbdc0
installed MI handler for int 12
psm0: model IntelliMouse, device ID 3
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
sio0 at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0
sio0: type 16550A
installed MI handler for int 4
sio1 at port 0x2f8-0x2ff irq 3 on isa0
sio1: type 16550A
installed MI handler for int 3
ppc0: <Parallel port> at port 0x378-0x37f irq 7 on isa0
ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode
ppbus0: <Parallel port bus> on ppc0
plip0: <PLIP network interface> on ppbus0
lpt0: <Printer> on ppbus0
lpt0: Interrupt-driven port
ppi0: <Parallel I/O> on ppbus0
installed MI handler for int 7
installed MI handler for int 0
acpi_cpu: throttling enabled, 8 steps (100% to 12.5%), currently 100.0%
acd0: CDROM <CD-224E> at ata1-master PIO4
Mounting root from ufs:/dev/da0s1a
da0 at twa0 bus 0 target 0 lun 0
da0: <3ware Logical Disk 00 1.00> Fixed Direct Access SCSI-0 device
da0: 100.000MB/s transfers
da0: 66747MB (136697856 512 byte sectors: 255H 63S/T 8509C)
da1 at twa0 bus 0 target 1 lun 0
da1: <3ware Logical Disk 01 1.00> Fixed Direct Access SCSI-0 device
da1: 100.000MB/s transfers
da1: 238408MB (488259584 512 byte sectors: 255H 63S/T 30392C)
cd0 at ata1 bus 0 target 0 lun 0
cd0: <TEAC CD-224E 1.9A> Removable CD-ROM SCSI-0 device
cd0: 16.000MB/s transfers
cd0: Attempt to query device size failed: NOT READY, Medium not present

Excerpt from /var/log/messages:
Sep 15 19:23:26 dkdev_master kernel: twa0: INFO: (0x04: 0x000c): Background initialize started: unit=0
Sep 15 19:46:30 dkdev_master kernel: twa0: INFO: (0x04: 0x0007): Background initialize done: unit=0

pciconf -l -v:
twa0@pci1:1:0:  class=0x010400 card=0x100213c1 chip=0x100213c1 rev=0x00 hdr=0x00
    vendor   = '3ware Inc.'
    class    = mass storage
    subclass = RAID
pcib2@pci0:6:0: class=0x060400 card=0x000000c0 chip=0x74601022 rev=0x07 hdr=0x01
    vendor   = 'Advanced Micro Devices (AMD)'
    device   = 'AMD-8111 PCI Bridge'
    class    = bridge
    subclass = PCI-PCI
isab0@pci0:7:0: class=0x060100 card=0x74681022 chip=0x74681022 rev=0x05 hdr=0x00
    vendor   = 'Advanced Micro Devices (AMD)'
    device   = 'AMD-8111 LPC Bridge'
    class    = bridge
    subclass = PCI-ISA
atapci1@pci0:7:1:       class=0x01018a card=0x74691022 chip=0x74691022 rev=0x03
hdr=0x00
    vendor   = 'Advanced Micro Devices (AMD)'
    device   = 'AMD-8111 UltraATA/133 Controller'
    class    = mass storage
    subclass = ATA
none0@pci0:7:2: class=0x0c0500 card=0x746a1022 chip=0x746a1022 rev=0x02 hdr=0x00
    vendor   = 'Advanced Micro Devices (AMD)'
    device   = 'AMD-8111 SMBus 2.0 Controller'
    class    = serial bus
    subclass = SMBus
none1@pci0:7:3: class=0x068000 card=0x746b1022 chip=0x746b1022 rev=0x05 hdr=0x00
    vendor   = 'Advanced Micro Devices (AMD)'
    device   = 'AMD-8111 ACPI Controller'
    class    = bridge
    subclass = PCI-unknown
pcib3@pci0:10:0:        class=0x060400 card=0x000000a0 chip=0x74501022 rev=0x12
hdr=0x01
    vendor   = 'Advanced Micro Devices (AMD)'
    class    = bridge
    subclass = PCI-PCI
none2@pci0:10:1:        class=0x080010 card=0x36c01022 chip=0x74511022 rev=0x01
hdr=0x00
    vendor   = 'Advanced Micro Devices (AMD)'
    class    = base peripheral
    subclass = interrupt controller
pcib4@pci0:11:0:        class=0x060400 card=0x000000a0 chip=0x74501022 rev=0x12
hdr=0x01
    vendor   = 'Advanced Micro Devices (AMD)'
    class    = bridge
    subclass = PCI-PCI
none3@pci0:11:1:        class=0x080010 card=0x36c01022 chip=0x74511022 rev=0x01
hdr=0x00
    vendor   = 'Advanced Micro Devices (AMD)'
    class    = base peripheral
    subclass = interrupt controller
hostb0@pci0:24:0:       class=0x060000 card=0x00000000 chip=0x11001022 rev=0x00
hdr=0x00
    vendor   = 'Advanced Micro Devices (AMD)'
    class    = bridge
    subclass = HOST-PCI
hostb1@pci0:24:1:       class=0x060000 card=0x00000000 chip=0x11011022 rev=0x00
hdr=0x00
    vendor   = 'Advanced Micro Devices (AMD)'
    class    = bridge
    subclass = HOST-PCI
hostb2@pci0:24:2:       class=0x060000 card=0x00000000 chip=0x11021022 rev=0x00
hdr=0x00
    vendor   = 'Advanced Micro Devices (AMD)'
    class    = bridge
    subclass = HOST-PCI
hostb3@pci0:24:3:       class=0x060000 card=0x00000000 chip=0x11031022 rev=0x00
hdr=0x00
    vendor   = 'Advanced Micro Devices (AMD)'
    class    = bridge
    subclass = HOST-PCI
hostb4@pci0:25:0:       class=0x060000 card=0x00000000 chip=0x11001022 rev=0x00
hdr=0x00
    vendor   = 'Advanced Micro Devices (AMD)'
    class    = bridge
    subclass = HOST-PCI
hostb5@pci0:25:1:       class=0x060000 card=0x00000000 chip=0x11011022 rev=0x00
hdr=0x00
    vendor   = 'Advanced Micro Devices (AMD)'
    class    = bridge
    subclass = HOST-PCI
hostb6@pci0:25:2:       class=0x060000 card=0x00000000 chip=0x11021022 rev=0x00
hdr=0x00
    vendor   = 'Advanced Micro Devices (AMD)'
    class    = bridge
    subclass = HOST-PCI
hostb7@pci0:25:3:       class=0x060000 card=0x00000000 chip=0x11031022 rev=0x00
hdr=0x00
    vendor   = 'Advanced Micro Devices (AMD)'
    class    = bridge
    subclass = HOST-PCI
ohci0@pci3:0:0: class=0x0c0310 card=0x74641022 chip=0x74641022 rev=0x0b hdr=0x00
    vendor   = 'Advanced Micro Devices (AMD)'
    device   = 'OpenHCI USB Host Controller'
    class    = serial bus
    subclass = USB
ohci1@pci3:0:1: class=0x0c0310 card=0x74641022 chip=0x74641022 rev=0x0b hdr=0x00
    vendor   = 'Advanced Micro Devices (AMD)'
    device   = 'OpenHCI USB Host Controller'
    class    = serial bus
    subclass = USB
atapci0@pci3:5:0:       class=0x018000 card=0x31141095 chip=0x31141095 rev=0x02
hdr=0x00
    vendor   = 'Silicon Image Inc (Was: CMD Technology Inc)'
    class    = mass storage
none4@pci3:6:0: class=0x030000 card=0x80081002 chip=0x47521002 rev=0x27 hdr=0x00
    vendor   = 'ATI Technologies'
    device   = 'Rage XL PCI'
    class    = display
    subclass = VGA
fxp0@pci3:8:0:  class=0x020000 card=0x10408086 chip=0x12298086 rev=0x10 hdr=0x00
    vendor   = 'Intel Corporation'
    device   = '82557/8/9 EtherExpress PRO/100(B) Ethernet Adapter'
    class    = network
    subclass = ethernet
bge0@pci2:9:0:  class=0x020000 card=0x164414e4 chip=0x164814e4 rev=0x03 hdr=0x00
    vendor   = 'Broadcom Corporation'
    class    = network
    subclass = ethernet
bge1@pci2:9:1:  class=0x020000 card=0x164414e4 chip=0x164814e4 rev=0x03 hdr=0x00
    vendor   = 'Broadcom Corporation'
    class    = network
    subclass = ethernet
none5@pci1:1:0: class=0x010400 card=0x100213c1 chip=0x100213c1 rev=0x00 hdr=0x00
    vendor   = '3ware Inc.'
    class    = mass storage
    subclass = RAID