PERC3/DC, PERC4/DC Write Performance

We purchased four dual-2.8Ghz 2650s, with PERC3/DC cards, along with 
two PowerVault 220S arrays (Ultra320 version), to create two storage 
clusters as recommended by Dell Power Solutions and Red Hat Enterprise.

Cluster testing went relatively smoothly. Failover worked under both 
RHEL AS 2.1 and 3.0. However, we noticed writes to the shared storage 
were slower than expected. So, we went back to a basic configuration 
with factory defaults to diagnose the problem:

With a single 36gb-10K-U320 disk in the PV220S, connected to one 2650, 
on RedHat 9 installed by the Open Manage CD (v7.5), no clustered SCSI, 
a sequential write could be done at 20 MB/s.

Changing to "WRITETHRU" mode on a single drive, required by clustering, 
sequential writes dropped to 10 MB/s. On a 5-disk RAID5, writes fell to 
8 MB/s.

In comparison, we can write to a PE2650 single internal disk, with the 
on-board Adaptec AIC-7899, at 35 MB/s:

sync; time dd ibs=1048576 obs=1048576 count=1024 if=/dev/zero 
of=/var/gigfile;
1024+0 records in 1024+0 records out
real    0m25.274s	user    0m2.690s	sys     0m11.730s
time sync
real    0m3.645s	user    0m0.000s	sys     0m0.040s
(1024 MB/28.9 seconds = 35.4 MB/s)

We tried the following to improve PERC3/DC writes, with little effect:

* Check logs and settings:
	No failures or events in the BIOS, "messages", or megamon/raid monitor.
	Disk, channel and "megaraid" all report U160 speed, proper termination.

* Change the defaults on the PERC3/DC (to):
	Write Policy = WRBACK (WTHRU dropped peformance by 40%)
	Read Policy = ADAPTIVE (tried NONE, READ AHEAD)
	Cache Policy = DIRECTIO (tried CACHEDIO, which gave a small gain)
	BIOS: Enabled (DISABLED, required by clustering)
	Stripe size: 64k (others had no effect)
	
* Update all BIOS and Firmware:
	PV220S revision E.14, Ultra320 Version
	PE2650 version A15 through A17
	PERC3/DC Dell firmware 1.92 (BIOS 3.31)
	(also tried LSI Elite 1600 Firmware C170 through 111Q)

* Update software Megaraid driver
	Tried versions 1.18h, 1.18j and the latest 2.00.9

* Change OS:
	Tested with RedHat AS 2.1, RedHat 9.0, RedHat AS 3.0.

* Change disk models:
	Atlas U320 IV 10K 36Gb are included. Also tried IBM Ultrastar U320 10K 
36Gb.

* Change channels and cables:
	Configured PV220S on Channel 1 instead of 0. New LVD cables tried.
	
* Change slots:
	Moved the PERC3/DC from slot 1 (66mhz shared) to slot 3 (133mhz 
dedicated).
	
* Change enclosures and bus modes:
	Tested with a second PV220S in shared, split-bus and clustered modes.
	Also tried with the internal backplane, without a PV220S. Same results.

* Change PowerEdges:
	Two seperate PowerEdges tested individually.

After speaking with several Dell technicians, and escalating the issue, 
we exchanged the PERC3/DC's for PERC4/DC's.

With a PERC4/DC, factory-defaulted sequential writes to a single disk 
(without writethrough), went from 20 MB/s to nearly 40 MB/s. However, 
just by switching to "writethrough", they dropped to 10 MB/s (8 MB/s on 
RAID5).

We then spoke with an Advanced Software Support technician, who was 
able to forward our results to the Dell Hardware Escalation team. They 
concluded the following:

"The server and power vault are working as designed and we do not 
expect a solution to the performance difference on this configuration."

We hope that Dell chooses to more truthfully advertise the performance 
tradeoff of the cluster configuration they recommend. If more accurate 
information were to be provided prior to sale, customers would no 
longer be suprised by this issue.

_______________________________________________
Linux-PowerEdge mailing list
Linux-PowerEdge <at> dell.com
http://lists.us.dell.com/mailman/listinfo/linux-poweredge
Please read the FAQ at http://lists.us.dell.com/faq or search the list archives at http://lists.us.dell.com/htdig/