[ofa-general] Infiniband performance

[Jan Ruffing]

Chuck Hartley wrote:
> What kind of machines are the cards installed in?
- Intel Xeon Quadcore; CPU E5420 @ 2.50GHz
- 4 GB RAM
- Super X7DCL Mainboard

(The 10 GBit/s TCP bandwidth has also been meassured on superior 64 bit
machines running Ubuntu 8.04, also using Mellanox Infinihost III Lx DDR
HCA cards in a 8x PCIe slots, when using a basic Infiniband installation.
When trying to set up Infiniband by manually installing the OFED 1.3.
packages on one of the machines, the maximum bandwidth meassured locally
was also 12 GBits/s.
The trial-and-error installation ran into some problems when trying to
set up the more advanced Infiniband features, though. So the decision
was made to test Infiniband on a system that supported the OFED install
script to get a cleaner installation; the results have been summarized
in the previous mail.)

>   Do you have the cards in 8x PCIe slots?
The cards are in 8x PCIe slots.

BIOS settings for the slots are as follows:

8x PCIe, Slot 6:
Option ROM Scan   [Enabled]
Enable Master     [Enabled]
Latency Timer     [Default]

No other PCI(e)-Slots have any cards installed.

(According to the manual, "Each PCI Express port on the MCH provides 4
GB/s bidirectional bandwidth if configured as a x8 port[...]". This
would equal 32 Gbit/s, and even if unidirectional speed was half of that
(I'm no expert on the PCIe bus), it would still be above the meassured
Infiniband bandwidth by about 4 Gbit/s unidirectional. So I'm somewhat
hesitant to assume PCI Express as the bottleneck for an Infiniband
bandwidth of 12 GBit/s.)


-- 
Jan Ruffing
Software Developer

Motama GmbH
Lortzingstraße 10 · 66111 Saarbrücken · Germany
tel +49 681 940 85 50 · fax +49 681 940 85 49
ruffing at motama.com · www.motama.com

Companies register · district council Saarbrücken · HRB 15249
CEOs · Dr.-Ing. Marco Lohse, Michael Repplinger

This e-mail may contain confidential and/or privileged information. 
If you are not the intended recipient (or have received this e-mail 
in error) please notify the sender immediately and destroy this 
e-mail. Any unauthorized copying, disclosure or distribution of the 
material in this e-mail is strictly forbidden.