Re: HAMMER problems with ciss

[György Vilmos <vilmos.gyorgy@xxxxxxxxx>]

On 08/08/2009, György Vilmos <vilmos.gyorgy@gmail.com> wrote:
> Hi Matt,
>
> On 07/08/2009, Matthew Dillon <dillon@apollo.backplane.com> wrote:
>>
>> :Hi All,
>> :
>> :I'm trying to do some testing with dfly 2.2 (both release and the last
>> :release snapshot) and HAMMER.
>> :For the first run I issued a simple dd if=/dev/zero of=testfile bs=1M
>> :command on the filesystem, which wrote some stuff out and immediately
>> :switched the file system into read only mode.
>> :
>> :UFS works fine.
>> :
>> :The kernel logs this:
>> :(da1:ciss1:0:1:0): SYNCHRONIZE CACHE(10). CDB: 35 0 0 0 0 0 0 0 0 0
>> :(da1:ciss1:0:1:0): CAM Status: SCSI Status Error
>> :(da1:ciss1:0:1:0): SCSI Status: Check Condition
>> :(da1:ciss1:0:1:0): ILLEGAL REQUEST asc:20,0
>> :(da1:ciss1:0:1:0): Invalid command operation code
>> :(da1:ciss1:0:1:0): Unretryable error
>> :HAMMER(test): Critical error inode=-1 while flushing meta-data
>> :HAMMER(test): Forcing read-only mode
>> :HAMMER(test): Critical error inode=-1 while flushing meta-data
>> :HAMMER(test): Critical write error during flush, refusing to sync UNDO
>> FIFO
>> :
>> :This is with a HP P400 controller with battery backed write cache,
>> :which -as it seems- doesn't support the SYNCHRONIZE CACHE command.
>> :FreeBSD's ciss driver apparently has workaround for this.
>> :
>> :Any chance of merging that change into dfly?
>> :
>> :Regards,
>>
>>     Hmm.  I looked at the driver code and we seem to have the same
>>     workaround.  The workaround is disabled in both the FreeBSD driver
>>     and our driver.  Please try changing line 899 in dev/raid/ciss.c,
>>     change the #if 0 to an #if 1.
>>
>> #if 0
>>     /* XXX later revisions may not need this */
>>     sc->ciss_flags |= CISS_FLAG_FAKE_SYNCH;
>> #endif
>>
>>     And see if that fixes the problem.  Also post all the ciss lines
>>     from /var/run/dmesg.boot, maybe I can do a more specific check of
>>     adapter version to set the flag for.
> ciss0: <HP Smart Array P800> port 0x5000-0x50ff mem
> 0xfdef0000-0xfdef0fff,0xfdf00000-0xfdffffff irq 7 at device 0.0 on
> pci14
> ciss1: <HP Smart Array P400> port 0x4000-0x40ff mem
> 0xfdbf0000-0xfdbf0fff,0xfdc00000-0xfdcfffff irq 10 at device 0.0 on
> pci6
> da0 at ciss1 bus 0 target 0 lun 0
> da1 at ciss1 bus 0 target 1 lun 0
> da2 at ciss1 bus 0 target 2 lun 0
> but I think it would be the same for all SA family members...
>
> Works fine with this change, thanks.
Albeit a little bit behind UFS:

hammer:
seqwr
#bklsize[kB]    MBps    iops    min     avg     max     95%     - idok ms-ban
1               .07     71.81   0.0000  0.0000  0.0007  0.0000
2               .14     75.19   0.0000  0.0000  0.0004  0.0000
4               .28     71.86   0.0000  0.0000  0.0004  0.0001
8               .60     77.37   0.0000  0.0000  0.0004  0.0001
16              1.15    73.92   0.0000  0.0000  0.0163  0.0001
32              2.43    77.88   0.0000  0.0001  0.0161  0.0001
64              4.70    75.27   0.0001  0.0001  0.0011  0.0001
128             8.14    65.14   0.0001  0.0001  0.0170  0.0001

UFS:
#bklsize[kB]	MBps	iops	min	avg	max	95%	- idok ms-ban
1		.44	460.00 	0.0000	0.0000	0.0003	0.0000
2		.84	434.38 	0.0000	0.0000	0.0145	0.0000
4		1.52	390.20 	0.0000	0.0000	0.0543	0.0000
8		2.53	325.05 	0.0000	0.0000	0.0158	0.0000
16		3.80	243.54 	0.0000	0.0000	0.0093	0.0000
32		7.54	241.59 	0.0000	0.0001	0.1237	0.0001
64		15.08	241.38 	0.0001	0.0001	0.0142	0.0001
128		25.17	201.37 	0.0001	0.0002	0.1309	0.0002
256		41.02	164.08 	0.0002	0.0004	0.1311	0.0004
512		60.64	121.29 	0.0006	0.0008	0.0861	0.0008

BTW, the UFS numbers were made with 2.2R UP kernel, while the hammer
numbers are with 2.3 (latest) SMP. Can this cause such a big
difference, or is it the different inner workings instead?

-- 
http://suckit.blog.hu/