[ofa-general] infiniband multicast (libibverbs)

[Dotan Barak]

Kelly Burkhart wrote:
> I'm experimenting with multicast and am having an interesting issue.
> The setup is ripped mostly from ib_send_lat.c.  I have a client which
> sends and a server which reads.  All sends/recieves use a 2048 byte
> message.
>
> The client can send any number of messages at any message rate.  The
> client spins in a tight loop while sending to reduce bursts of messages
> (1000 messgages/sec are spread out over the sec).  The client embeds a
> sequence number in the message.
>
> After setup, the server does this:
>
> post 2048 recvs
> for(;;) {
>     ibv_poll_cq in loop, waiting for completion
>     post recv
>     check sequence number
> }
>
> If I specify more than about 6500 messages/sec, I skip some sequences
> and receive others multiple times.  I always receive the same number of
> messages the client sent.  It appears as though all of the messages come
> through, but I'm missing some and reading others twice.
>   
Do you use the "volatile" when you access the pointed memory buffer?
> I suspect that there is some trick to more reliable multicast messaging
> that I don't know about.  Does anyone have hints for multicasting high
> message rates with a small percentage of drops or misses?
>   
Do you have worst results than the ib_send_bw.c?
Can you try to send unicast messages (with minimum changes) to see if 
the issue is related to multicast send?

Anyway, you should remember that multicast messages are being sent over 
UD QPs and messages can be dropped.

Dotan