[openib-general] BandWidth doubt

[john t]

Hi,

I got following readings in one of my experiments:

Single 64-bit xeon machine (2 dual-core 3.2 GHz Intel CPUs, linux FC4, OFED
1.0) with two Mellanox DDR (4x) HCAs (each having two ports and each
connected to a PCI x8 interface) is connected to a switch (all the 4 DDR
(4x) ports are connected to the switch).

If I send data from mthca0-1 to mthca0-1 meaning from same port to the same
port i.e. same port doing send/recv (also same cable doing send/recv) I get
a BW of around 10 Gb/sec.

Similarly, from mthca1-1 to mthca1-1 I get same i.e. around 10 Gb/sec.

So, individual port-to-port gives 10 Gb/sec.

But when I use them together i.e when I send the data from mthca0-1 to
mthca0-1 AND from mthca1-1 to mthca1-1 at the same time (simultaneously) I
get a BW of 6.7 Gb/sec on each port. This is less than 10 Gb/sec that is
expected. Note that mthca0 and mthca1 are connected to two different PCI-x8
interfaces, so there is no question of bandwidth splitting. What could be
causing such a behaviour ??

Just to add if the same thing is done between two different hosts i.e. If I
send data from mthca0-0 and mthca1-1 of one host to mthca0-0 and mthca1-1 of
other host, I get expected BW i.e. 10 Gb/sec on each port/link.

Regards,
John
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20061110/6ad6185b/attachment.html>