[ofa-general] infiniband multicast (libibverbs)

[Kelly Burkhart]

> -----Original Message-----
> From: Dotan Barak [mailto:dotanba at gmail.com] 
> 
> Kelly Burkhart wrote:
> > If I specify more than about 6500 messages/sec, I skip some 
> sequences
> > and receive others multiple times.  I always receive the 
> same number of
> > messages the client sent.  It appears as though all of the 
> messages come
> > through, but I'm missing some and reading others twice.
> >   
> Do you use the "volatile" when you access the pointed memory buffer?

I do not.  I noticed this with the post_buf and poll_buf variables
in pingpong_context, but they're not used in send_lat.  I assumed
they only replied to RDMA.

Do I need to be using volatile anywhere with UD send?


> > I suspect that there is some trick to more reliable 
> multicast messaging
> > that I don't know about.  Does anyone have hints for 
> multicasting high
> > message rates with a small percentage of drops or misses?
> >   
> Do you have worst results than the ib_send_bw.c?
> Can you try to send unicast messages (with minimum changes) to see if 
> the issue is related to multicast send?
> 
> Anyway, you should remember that multicast messages are being 
> sent over 
> UD QPs and messages can be dropped.

I solved (or hid) my problem by recv-ing into multiple buffers.  I do
realize that multicast messages can be dropped, but I want to know what
level of one-way reliability and message rate I can achieve.

Since I was receiving all messages before, I don't think my results 
were different than ib_send_bw.  My problem was not realizing that my 
buffer could be clobbered prior to me polling the cq for the work
completion.

Thanks,

-K