[libfabric-users] Max IOV length with verbs provider

[Hefty, Sean]

> using the verbs provider (libfabric v1.12.1), I use the fi_sendmsg to send messages.
> The flag FI_INJECT_COMPLETE is used and message I/O vector has a variable length
> (iov_count).
> 
> Independently from the total size of the message, if iov_count > 20 the messages are
> not sent and EAGAIN is returned.
> This means that I can send, for example, 24 kB messages with 3 IOV entries, but not
> with 24 entries.
> 
> To be precise, EAGAIN is returned by ibv_post_send [1] and iov_count is the only
> parameter that changes in the test.
> 
> I understand that multiple causes can underlie an EAGAIN, but given the role of
> iov_count is there something I can check to start with?

Check fi_tx_attr/fi_rx_attr::iov_limit.  That's the max iov size that the provider can support.  It's possible if you run a debug version of the code, you might hit an assertion.

- Sean