[ofa-general] infiniband multicast (libibverbs)

[Kelly Burkhart]

I'm experimenting with multicast and am having an interesting issue.
The setup is ripped mostly from ib_send_lat.c.  I have a client which
sends and a server which reads.  All sends/recieves use a 2048 byte
message.

The client can send any number of messages at any message rate.  The
client spins in a tight loop while sending to reduce bursts of messages
(1000 messgages/sec are spread out over the sec).  The client embeds a
sequence number in the message.

After setup, the server does this:

post 2048 recvs
for(;;) {
    ibv_poll_cq in loop, waiting for completion
    post recv
    check sequence number
}

If I specify more than about 6500 messages/sec, I skip some sequences
and receive others multiple times.  I always receive the same number of
messages the client sent.  It appears as though all of the messages come
through, but I'm missing some and reading others twice.

I suspect that there is some trick to more reliable multicast messaging
that I don't know about.  Does anyone have hints for multicasting high
message rates with a small percentage of drops or misses?

Thanks,

-K