[ofa-general] RE: msi-x seems to cause large performance variation for connectx / datagram mode

[Sagi Rotem]

Or,
Did you take a look at the cpu utilization while running ? Did u see
higher cpu utilization while getting the 850MB/sec ?
How many cpu's were actually working ?
I will try and reproduce your results.
Sagi 

-----Original Message-----
From: Or Gerlitz [mailto:ogerlitz at voltaire.com] 
Sent: Wednesday, June 25, 2008 6:34 PM
To: Sagi Rotem
Cc: general at lists.openfabrics.org
Subject: Re: msi-x seems to cause large performance variation for
connectx / datagram mode

> digging a little more, I see now that if I disable msi-x, the 
> performance gets to a fixed value of about 620MB/s with connectx which

> is the lower range that msi-x gives me...

setting the irq daemon off, I see that if I set the cpu affinity of the
netperf process to be as the one of the mlx4 irq I get fixed performance
of 650 MB/s

If I set it to different CPU, then eventually I get 480MB/s constantly
and 620MB/s constantly if is a different core on the same CPU...

HCA			MB/s	msi_x	irqdaemon	netperf affinity
=======================================================================
ConnectX-ConnectX	620     0 	on		none

ConnectX-ConnectX	620-850 1 	on		none
ConnectX-ConnectX	650     1 	off		same as irq
ConnectX-ConnectX	620     1 	off		different core
ConnectX-ConnectX	480     1 	off		different cpu


Or.