<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
Dear User community, Sean,
<div class=""><br class="">
</div>
<div class="">I would like to close this thread reporting my solution for dealing with long IOV and the verbs provider.</div>
<div class=""><br class="">
</div>
<div class="">Libfabric sets IOV length limit of the verbs provider to 4 by default.</div>
<div class="">This limit be changed via the environment variables</div>
<div class=""><br class="">
</div>
<div class=""><span class="Apple-tab-span" style="white-space:pre"></span>FI_VERBS_TX_IOV_LIMIT</div>
<div class=""><span class="Apple-tab-span" style="white-space:pre"></span>FI_VERBS_RX_IOV_LIMIT</div>
<div class=""><br class="">
</div>
<div class="">These values are passed to the verbs provider that compares them to the maximum values allowed by the device.</div>
<div class="">In fact, the actual limitation comes from the hardware and, in the case of Mellanox Connect-X NICs it is can be read with</div>
<div class=""><tt class="">ibv_devinfo -v <font face="Helvetica" class="">under the
</font><font face="Menlo" class="">max_sg</font><font face="Helvetica" class=""> field. For the Connect-X5 </font><span style="font-family: Menlo;" class="">max_sg </span><font face="Helvetica" class="">is 30, for </font></tt>Connect-X4 it is 32.</div>
<div class=""><br class="">
</div>
<div class="">In short, setting the env variables and respecting these limits in my application I can reliably transfer IOVs with up to 30 items.</div>
<div class=""><br class="">
</div>
<div class="">Kind regards,</div>
<div class="">Carlo</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">PS: long IOVs seem not to be a good idea [1], but in my specific use case there's little choice.</div>
<div class="">[1] <a href="https://www.rdmamojo.com/2013/06/08/tips-and-tricks-to-optimize-your-rdma-code/" class="">https://www.rdmamojo.com/2013/06/08/tips-and-tricks-to-optimize-your-rdma-code/</a></div>
<div class="">
<div class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<span style="font-family: Times;" class="">_____________________</span></div>
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<span style="font-family: Times;" class="">Carlo A. Gottardo</span></div>
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<span style="font-family: Times;" class="">Postdoc at Nikhef</span></div>
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<span style="font-family: Times;" class="">Skype: carlogottardo</span></div>
</div>
</div>
</div>
</div>
</div>
<br class="Apple-interchange-newline">
</div>
<br class="Apple-interchange-newline">
<br class="Apple-interchange-newline">
</div>
<div><br class="">
<blockquote type="cite" class="">
<div class="">On 21 Jun 2021, at 18:14, Hefty, Sean <<a href="mailto:sean.hefty@intel.com" class="">sean.hefty@intel.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="">
<blockquote type="cite" class="">using the verbs provider (libfabric v1.12.1), I use the fi_sendmsg to send messages.<br class="">
The flag FI_INJECT_COMPLETE is used and message I/O vector has a variable length<br class="">
(iov_count).<br class="">
<br class="">
Independently from the total size of the message, if iov_count > 20 the messages are<br class="">
not sent and EAGAIN is returned.<br class="">
This means that I can send, for example, 24 kB messages with 3 IOV entries, but not<br class="">
with 24 entries.<br class="">
<br class="">
To be precise, EAGAIN is returned by ibv_post_send [1] and iov_count is the only<br class="">
parameter that changes in the test.<br class="">
<br class="">
I understand that multiple causes can underlie an EAGAIN, but given the role of<br class="">
iov_count is there something I can check to start with?<br class="">
</blockquote>
<br class="">
Check fi_tx_attr/fi_rx_attr::iov_limit. That's the max iov size that the provider can support. It's possible if you run a debug version of the code, you might hit an assertion.<br class="">
<br class="">
- Sean<br class="">
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</body>
</html>