CRC FAILED: LAYER2

[YONETANI Tomokazu <qhwt+dfly@xxxxxxxxxx>]

Hi.
I've just caught this panic this morning.  It's an Athlon 64 X2 running
an SMP kernel built from source as of July 5th (a few commits after the
fixes to fdfree()).  It's a 1Tbyte HAMMER-only filesystem and I accidentally
run `hammer version-upgrade' command on it a several days ago, in case
it matters.  The machine is turned on at 7:00 am every day, runs cvsync
to retrieve a few CVS repositories, and convert them to git repos.
According to today's log something went wrong during the second invocation
of cvsync.  The script then proceeded to converting the first CVS repository,
when it caught this panic.  The kernel and vmcore are uploaded at my leaf
account as ~y0netan1/crash/{kernel,vmcore}.4 .

Thanks in advance.

(kgdb) bt
#0  dumpsys () at ./machine/thread.h:83
#1  0xc019ece5 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:375
#2  0xc019efaa in panic (fmt=0xc030227d "CRC FAILED: LAYER2")
    at /usr/src/sys/kern/kern_shutdown.c:801
#3  0xc026318f in hammer_blockmap_free (trans=0xd87b211c,
    zone_offset=10376293665613545856, bytes=128)
    at /usr/src/sys/vfs/hammer/hammer_blockmap.c:804
#4  0xc0273837 in hammer_delete_at_cursor (cursor=0xd87bec9c,
    delete_flags=<value optimized out>, delete_tid=370686325120,
    delete_ts=1246918444, track=0, stat_bytes=0x0)
    at /usr/src/sys/vfs/hammer/hammer_object.c:2319
#5  0xc0273a5c in hammer_ip_delete_record (cursor=0xd87bec9c, ip=0xddf75c50,
    tid=370686325120) at /usr/src/sys/vfs/hammer/hammer_object.c:2179
#6  0xc026af88 in hammer_update_inode (cursor=0xd87bec9c, ip=0xddf75c50)
    at /usr/src/sys/vfs/hammer/hammer_inode.c:1161
#7  0xc026b809 in hammer_sync_inode (trans=0xd87b211c, ip=0xddf75c50)
    at /usr/src/sys/vfs/hammer/hammer_inode.c:2889
#8  0xc026958a in hammer_flusher_slave_thread (arg=0xd060a2f0)
    at /usr/src/sys/vfs/hammer/hammer_flusher.c:491
#9  0xc01a71a1 in lwkt_deschedule_self (td=Cannot access memory at address 0x8
)
    at /usr/src/sys/kern/lwkt_thread.c:263
Backtrace stopped: previous frame inner to this frame (corrupt stack?)
(kgdb) fr 3
#3  0xc026318f in hammer_blockmap_free (trans=0xd87b211c,
    zone_offset=10376293665613545856, bytes=128)
    at /usr/src/sys/vfs/hammer/hammer_blockmap.c:804
804                             panic("CRC FAILED: LAYER2");
(kgdb) list
799             if (error)
800                     goto failed;
801             if (layer2->entry_crc != crc32(layer2, HAMMER_LAYER2_CRCSIZE)) {
802                     hammer_lock_ex(&hmp->blkmap_lock);
803                     if (layer2->entry_crc != crc32(layer2, HAMMER_LAYER2_CRCSIZE))
804                             panic("CRC FAILED: LAYER2");
805                     hammer_unlock(&hmp->blkmap_lock);
806             }
807
808             hammer_lock_ex(&hmp->blkmap_lock);