[ofa-general] MLX HCA: CQ request notification for multiple completions not implemented?

[Yicheng Jia]

>> My concern is the performance. The overhead of software poll_cq loop 
>> is quite significant if there are multiple pieces of small amount of 
>> data to be transferred on both sender/receiver sides. For instance, on 
>> the sender, the data I have are 1k, 1k, 2k, 1k..., on the receiver 
>> side, the data size and blocks are the same, 1k, 1k, 2k, 1k.... Do you 
>> have a good solution for such kind of problem?

>How many QPs do you use?
>(and how outstanding WR from every QP?)

Only one QP. Is it better to alloc multiple QPs and evenly distribute WRs 
among those QPs?

Best,
Yicheng




Dotan Barak <dotanba at gmail.com> 
05/28/2008 12:24 PM

To
Yicheng Jia <YJia at tmriusa.com>
cc
general at lists.openfabrics.org
Subject
Re: [ofa-general] MLX HCA: CQ request notification for multiple 
completions not implemented?






Yicheng Jia wrote:
>
> Thanks for your reply. I'm using one CQ for all the WRs. Do you know 
> why there's no ARM-N support in MLX drivers?
I don't know if i can speak in the name of Mellanox/MLX driver 
maintainers, but i think that the
reason is lack of demand for this feature (but i can't be sure).

> My concern is the performance. The overhead of software poll_cq loop 
> is quite significant if there are multiple pieces of small amount of 
> data to be transferred on both sender/receiver sides. For instance, on 
> the sender, the data I have are 1k, 1k, 2k, 1k..., on the receiver 
> side, the data size and blocks are the same, 1k, 1k, 2k, 1k.... Do you 
> have a good solution for such kind of problem?
How many QPs do you use?
(and how outstanding WR from every QP?)

Dotan
> Best,
> Yicheng
>
>
>
> *Dotan Barak <dotanba at gmail.com>*
>
> 05/23/2008 01:27 PM
>
> 
> To
>                Yicheng Jia <YJia at tmriusa.com>
> cc
>                general at lists.openfabrics.org
> Subject
>                Re: [ofa-general] MLX HCA: CQ request notification for 
multiple 
> completions not implemented?
>
>
>
> 
>
>
>
>
>
> Hi.
>
> Yicheng Jia wrote:
> >
> > Hi Folks,
> >
> > I'm trying to use CQ Event notification for multiple completions
> > (ARM_N) according to Mellanox Lx III user manual for scatter/gathering
> > RDMA. However I couldn't find it in current MLX driver. It seems to me
> > that only ARM_NEXT and ARM_SOLICIT are implemented. So if there are
> > multiple work requests, I have to use "poll_cq" to synchronously wait
> > until all the requests are done, is it correct? Is there a way to do
> > asynchronous multiple send by subscribing for a ARM_N event?
> You are right: the low level drivers of Mellanox devices doesn't support
> ARM-N
> (This feature is supported by the devices, but it wasn't implemented in
> the low level drivers).
>
> You are right, in order to read all of the completions you need to use
> poll_cq.
>
> By the way: Do you have you have to create a completion for any WR?
> (if you are using one QP, this will maybe solve your problem).
>
> Dotan
>
> 
_____________________________________________________________________________
> Scanned by IBM Email Security Management Services powered by 
> MessageLabs. For more information please visit http://www.ers.ibm.com
> 
_____________________________________________________________________________
> <http://www.ers.ibm.com/>
>
> 
_____________________________________________________________________________
> Scanned by IBM Email Security Management Services powered by 
> MessageLabs. For more information please visit http://www.ers.ibm.com
> 
_____________________________________________________________________________


_____________________________________________________________________________
Scanned by IBM Email Security Management Services powered by MessageLabs. 
For more information please visit http://www.ers.ibm.com
_____________________________________________________________________________



_____________________________________________________________________________
Scanned by IBM Email Security Management Services powered by MessageLabs. For more information please visit http://www.ers.ibm.com
_____________________________________________________________________________
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20080528/5bb756fb/attachment.html>