[openib-general] Multi-port HCA

[Bernard King-Smith]

"john t" <johnt1johnt2 at gmail.com> wrote on 10/05/2006 08:18:31 AM:

> Hi Bernard,
> 
> I had a configuration issue. I fixed it and now I get same BW (i.e. 
> around 10 Gb/sec) on each port provided I use ports on different HCA
> cards. If I use two ports of the same HCA card then BW gets divided 
> between these two ports. I am using Mellanox HCA cards and doing 
> simple send/recv using uverbs. 
> 
> Do you think it could be an issue with Mallanox driver or could it 
> be due to system/PCI-E limitation.

I haven't looked closely at the Mellanox driver, but I suspect it is 
there. When we were looking at the EHCA driver we found that if you have 
two links using the same event queue ( one per adapter ), and if the 
completion queue handling is serialized off the one interrupt per adapter, 
then one and two links get the same aggregate bandwidth. The problem is if 
you have two completion queues for the two links, they are handled 
serially. You need separate threads off the interrupt, each processing one 
of the completion queues in parallel on different CPU's to get the scaling 
you expect for 2 links. 

I don't think it is the PCI-e bus because it can handle much more than 20 
Gb/s. However, I also don't know how the Mellanox chip works for 2 ports.

> 
> Regards,
> John T.
> 
Bernie King-Smith 
IBM Corporation
Server Group
Cluster System Performance 
wombat2 at us.ibm.com    (845)433-8483
Tie. 293-8483 or wombat2 on NOTES 

"We are not responsible for the world we are born into, only for the world 
we leave when we die.
So we have to accept what has gone before us and work to change the only 
thing we can,
-- The Future." William Shatner
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20061005/7e4b6543/attachment.html>