Horrible Write Performance with Smartarray P400 in DL320, DL320s and DL1xx

We are a small hosting company in austria and are using P400 controllers in about 20 HP Servers (DL 320(s), 160, 180)
Now, since more servers have mucher to do, we recognize very strange performanceproblems on writing to a logical drive
on the P400 Controllers even if the server is idle.

some tests we made, shows the performance problems:

OS: Debian Linux, we also tried 2.6.25 kernel and Ubuntu, CentOS and FreeBSD

Controller: P400 Rev 1 or 3, with BBWC and without, all with 256 MB Ram, latest FW or older (see detailed info attached from hpacucli controller details and hpaducli output)

Server: DL320 (FW Upgraded to latest Version with Firmware Maintenance 8.10)

if we make write stuff to the disk you can see the problem:

time sh -c "dd if=/dev/zero of=/tmp/test bs=1024k count=4000; sync"
4000+0 records in
4000+0 records out
4194304000 bytes (4.2 GB) copied, 325.159 seconds, 12.9 MB/s
real 7m49.905s
user 0m0.004s
sys 0m12.121s
(you see, syncing needs also 2 minutes after dd has finished, after this the serverload is about 5, before 0.01)

we tried almost everything to find the problem, her is a list what we tried:

- updated p400 fw to latest version
- upgraded servers fw to latest version with fw-maintenance 8.10
- tried other linux distri
- tried other linux kernel (latest 2.6.25)
- other HP Server: we tried DL320s, DL320, DL160 and DL180
- tried other p400

Now we dont know what to do now, we have no more ideas how we could solve the problem :(

It would be very very very nice if anybody could help us with this issue!

Greetings from austria (the country without kangoroos),

Florian Schicker