<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><br class="">
<div><br class=""><blockquote type="cite" class=""><div class="">On Jan 16, 2020, at 11:03 AM, Philip Davis <<a href="mailto:philip.e.davis@rutgers.edu" class="">philip.e.davis@rutgers.edu</a>> wrote:</div><br class="Apple-interchange-newline"><div class="">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" class="">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
Hi Steve,
<div class=""><br class="">
</div>
<div class="">Thanks for the quick response.</div>
<div class=""><br class="">
</div>
<div class="">I am expecting to use the the rxm provider for verbs, but in fi_info I do not see an FI_EP_MSG-type verbs provider.</div></div></div></blockquote><div><br class=""></div>Could you provide the output for “ibv_devinfo -v” and “lsmod | grep ib”?</div><div><br class=""></div><div>Steve</div><div><br class=""><blockquote type="cite" class=""><div class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div class=""><br class="">
</div>
<div class="">
<div class="">provider: tcp;ofi_rxm</div>
<div class=""> fabric: TCP-IP</div>
<div class=""> domain: tcp</div>
<div class=""> version: 1.0</div>
<div class=""> type: FI_EP_RDM</div>
<div class=""> protocol: FI_PROTO_RXM</div>
<div class="">provider: tcp;ofi_rxm</div>
<div class=""> fabric: TCP-IP</div>
<div class=""> domain: tcp</div>
<div class=""> version: 1.0</div>
<div class=""> type: FI_EP_RDM</div>
<div class=""> protocol: FI_PROTO_RXM</div>
<div class="">provider: tcp;ofi_rxm</div>
<div class=""> fabric: TCP-IP</div>
<div class=""> domain: tcp</div>
<div class=""> version: 1.0</div>
<div class=""> type: FI_EP_RDM</div>
<div class=""> protocol: FI_PROTO_RXM</div>
<div class="">provider: tcp;ofi_rxm</div>
<div class=""> fabric: TCP-IP</div>
<div class=""> domain: tcp</div>
<div class=""> version: 1.0</div>
<div class=""> type: FI_EP_RDM</div>
<div class=""> protocol: FI_PROTO_RXM</div>
<div class="">provider: tcp;ofi_rxm</div>
<div class=""> fabric: TCP-IP</div>
<div class=""> domain: tcp</div>
<div class=""> version: 1.0</div>
<div class=""> type: FI_EP_RDM</div>
<div class=""> protocol: FI_PROTO_RXM</div>
<div class="">provider: tcp;ofi_rxm</div>
<div class=""> fabric: TCP-IP</div>
<div class=""> domain: tcp</div>
<div class=""> version: 1.0</div>
<div class=""> type: FI_EP_RDM</div>
<div class=""> protocol: FI_PROTO_RXM</div>
<div class="">provider: verbs;ofi_rxd</div>
<div class=""> fabric: IB-0xfe80000000000000</div>
<div class=""> domain: mlx4_0-dgram</div>
<div class=""> version: 1.0</div>
<div class=""> type: FI_EP_RDM</div>
<div class=""> protocol: FI_PROTO_RXD</div>
<div class="">provider: UDP;ofi_rxd</div>
<div class=""> fabric: UDP-IP</div>
<div class=""> domain: udp</div>
<div class=""> version: 1.0</div>
<div class=""> type: FI_EP_RDM</div>
<div class=""> protocol: FI_PROTO_RXD</div>
<div class="">provider: UDP;ofi_rxd</div>
<div class=""> fabric: UDP-IP</div>
<div class=""> domain: udp</div>
<div class=""> version: 1.0</div>
<div class=""> type: FI_EP_RDM</div>
<div class=""> protocol: FI_PROTO_RXD</div>
<div class="">provider: UDP;ofi_rxd</div>
<div class=""> fabric: UDP-IP</div>
<div class=""> domain: udp</div>
<div class=""> version: 1.0</div>
<div class=""> type: FI_EP_RDM</div>
<div class=""> protocol: FI_PROTO_RXD</div>
<div class="">provider: UDP;ofi_rxd</div>
<div class=""> fabric: UDP-IP</div>
<div class=""> domain: udp</div>
<div class=""> version: 1.0</div>
<div class=""> type: FI_EP_RDM</div>
<div class=""> protocol: FI_PROTO_RXD</div>
<div class="">provider: UDP;ofi_rxd</div>
<div class=""> fabric: UDP-IP</div>
<div class=""> domain: udp</div>
<div class=""> version: 1.0</div>
<div class=""> type: FI_EP_RDM</div>
<div class=""> protocol: FI_PROTO_RXD</div>
<div class="">provider: UDP;ofi_rxd</div>
<div class=""> fabric: UDP-IP</div>
<div class=""> domain: udp</div>
<div class=""> version: 1.0</div>
<div class=""> type: FI_EP_RDM</div>
<div class=""> protocol: FI_PROTO_RXD</div>
<div class="">provider: verbs</div>
<div class=""> fabric: IB-0xfe80000000000000</div>
<div class=""> domain: mlx4_0-dgram</div>
<div class=""> version: 1.0</div>
<div class=""> type: FI_EP_DGRAM</div>
<div class=""> protocol: FI_PROTO_IB_UD</div>
<div class="">provider: UDP</div>
<div class=""> fabric: UDP-IP</div>
<div class=""> domain: udp</div>
<div class=""> version: 1.1</div>
<div class=""> type: FI_EP_DGRAM</div>
<div class=""> protocol: FI_PROTO_UDP</div>
<div class="">provider: UDP</div>
<div class=""> fabric: UDP-IP</div>
<div class=""> domain: udp</div>
<div class=""> version: 1.1</div>
<div class=""> type: FI_EP_DGRAM</div>
<div class=""> protocol: FI_PROTO_UDP</div>
<div class="">provider: UDP</div>
<div class=""> fabric: UDP-IP</div>
<div class=""> domain: udp</div>
<div class=""> version: 1.1</div>
<div class=""> type: FI_EP_DGRAM</div>
<div class=""> protocol: FI_PROTO_UDP</div>
<div class="">provider: UDP</div>
<div class=""> fabric: UDP-IP</div>
<div class=""> domain: udp</div>
<div class=""> version: 1.1</div>
<div class=""> type: FI_EP_DGRAM</div>
<div class=""> protocol: FI_PROTO_UDP</div>
<div class="">provider: UDP</div>
<div class=""> fabric: UDP-IP</div>
<div class=""> domain: udp</div>
<div class=""> version: 1.1</div>
<div class=""> type: FI_EP_DGRAM</div>
<div class=""> protocol: FI_PROTO_UDP</div>
<div class="">provider: UDP</div>
<div class=""> fabric: UDP-IP</div>
<div class=""> domain: udp</div>
<div class=""> version: 1.1</div>
<div class=""> type: FI_EP_DGRAM</div>
<div class=""> protocol: FI_PROTO_UDP</div>
<div class="">provider: sockets</div>
<div class=""> fabric: 10.1.0.0/16</div>
<div class=""> domain: em1</div>
<div class=""> version: 2.0</div>
<div class=""> type: FI_EP_MSG</div>
<div class=""> protocol: FI_PROTO_SOCK_TCP</div>
<div class="">provider: sockets</div>
<div class=""> fabric: 10.1.0.0/16</div>
<div class=""> domain: em1</div>
<div class=""> version: 2.0</div>
<div class=""> type: FI_EP_DGRAM</div>
<div class=""> protocol: FI_PROTO_SOCK_TCP</div>
<div class="">provider: sockets</div>
<div class=""> fabric: 10.1.0.0/16</div>
<div class=""> domain: em1</div>
<div class=""> version: 2.0</div>
<div class=""> type: FI_EP_RDM</div>
<div class=""> protocol: FI_PROTO_SOCK_TCP</div>
<div class="">provider: sockets</div>
<div class=""> fabric: 10.157.14.0/24</div>
<div class=""> domain: em2</div>
<div class=""> version: 2.0</div>
<div class=""> type: FI_EP_MSG</div>
<div class=""> protocol: FI_PROTO_SOCK_TCP</div>
<div class="">provider: sockets</div>
<div class=""> fabric: 10.157.14.0/24</div>
<div class=""> domain: em2</div>
<div class=""> version: 2.0</div>
<div class=""> type: FI_EP_DGRAM</div>
<div class=""> protocol: FI_PROTO_SOCK_TCP</div>
<div class="">provider: sockets</div>
<div class=""> fabric: 10.157.14.0/24</div>
<div class=""> domain: em2</div>
<div class=""> version: 2.0</div>
<div class=""> type: FI_EP_RDM</div>
<div class=""> protocol: FI_PROTO_SOCK_TCP</div>
<div class="">provider: sockets</div>
<div class=""> fabric: fe80::/64</div>
<div class=""> domain: em1</div>
<div class=""> version: 2.0</div>
<div class=""> type: FI_EP_MSG</div>
<div class=""> protocol: FI_PROTO_SOCK_TCP</div>
<div class="">provider: sockets</div>
<div class=""> fabric: fe80::/64</div>
<div class=""> domain: em1</div>
<div class=""> version: 2.0</div>
<div class=""> type: FI_EP_DGRAM</div>
<div class=""> protocol: FI_PROTO_SOCK_TCP</div>
<div class="">provider: sockets</div>
<div class=""> fabric: fe80::/64</div>
<div class=""> domain: em1</div>
<div class=""> version: 2.0</div>
<div class=""> type: FI_EP_RDM</div>
<div class=""> protocol: FI_PROTO_SOCK_TCP</div>
<div class="">provider: sockets</div>
<div class=""> fabric: fe80::/64</div>
<div class=""> domain: em2</div>
<div class=""> version: 2.0</div>
<div class=""> type: FI_EP_MSG</div>
<div class=""> protocol: FI_PROTO_SOCK_TCP</div>
<div class="">provider: sockets</div>
<div class=""> fabric: fe80::/64</div>
<div class=""> domain: em2</div>
<div class=""> version: 2.0</div>
<div class=""> type: FI_EP_DGRAM</div>
<div class=""> protocol: FI_PROTO_SOCK_TCP</div>
<div class="">provider: sockets</div>
<div class=""> fabric: fe80::/64</div>
<div class=""> domain: em2</div>
<div class=""> version: 2.0</div>
<div class=""> type: FI_EP_RDM</div>
<div class=""> protocol: FI_PROTO_SOCK_TCP</div>
<div class="">provider: sockets</div>
<div class=""> fabric: 127.0.0.0/8</div>
<div class=""> domain: lo</div>
<div class=""> version: 2.0</div>
<div class=""> type: FI_EP_MSG</div>
<div class=""> protocol: FI_PROTO_SOCK_TCP</div>
<div class="">provider: sockets</div>
<div class=""> fabric: 127.0.0.0/8</div>
<div class=""> domain: lo</div>
<div class=""> version: 2.0</div>
<div class=""> type: FI_EP_DGRAM</div>
<div class=""> protocol: FI_PROTO_SOCK_TCP</div>
<div class="">provider: sockets</div>
<div class=""> fabric: 127.0.0.0/8</div>
<div class=""> domain: lo</div>
<div class=""> version: 2.0</div>
<div class=""> type: FI_EP_RDM</div>
<div class=""> protocol: FI_PROTO_SOCK_TCP</div>
<div class="">provider: sockets</div>
<div class=""> fabric: ::1/128</div>
<div class=""> domain: lo</div>
<div class=""> version: 2.0</div>
<div class=""> type: FI_EP_MSG</div>
<div class=""> protocol: FI_PROTO_SOCK_TCP</div>
<div class="">provider: sockets</div>
<div class=""> fabric: ::1/128</div>
<div class=""> domain: lo</div>
<div class=""> version: 2.0</div>
<div class=""> type: FI_EP_DGRAM</div>
<div class=""> protocol: FI_PROTO_SOCK_TCP</div>
<div class="">provider: sockets</div>
<div class=""> fabric: ::1/128</div>
<div class=""> domain: lo</div>
<div class=""> version: 2.0</div>
<div class=""> type: FI_EP_RDM</div>
<div class=""> protocol: FI_PROTO_SOCK_TCP</div>
<div class="">provider: tcp</div>
<div class=""> fabric: TCP-IP</div>
<div class=""> domain: tcp</div>
<div class=""> version: 0.1</div>
<div class=""> type: FI_EP_MSG</div>
<div class=""> protocol: FI_PROTO_SOCK_TCP</div>
<div class="">provider: tcp</div>
<div class=""> fabric: TCP-IP</div>
<div class=""> domain: tcp</div>
<div class=""> version: 0.1</div>
<div class=""> type: FI_EP_MSG</div>
<div class=""> protocol: FI_PROTO_SOCK_TCP</div>
<div class="">provider: tcp</div>
<div class=""> fabric: TCP-IP</div>
<div class=""> domain: tcp</div>
<div class=""> version: 0.1</div>
<div class=""> type: FI_EP_MSG</div>
<div class=""> protocol: FI_PROTO_SOCK_TCP</div>
<div class="">provider: tcp</div>
<div class=""> fabric: TCP-IP</div>
<div class=""> domain: tcp</div>
<div class=""> version: 0.1</div>
<div class=""> type: FI_EP_MSG</div>
<div class=""> protocol: FI_PROTO_SOCK_TCP</div>
<div class="">provider: tcp</div>
<div class=""> fabric: TCP-IP</div>
<div class=""> domain: tcp</div>
<div class=""> version: 0.1</div>
<div class=""> type: FI_EP_MSG</div>
<div class=""> protocol: FI_PROTO_SOCK_TCP</div>
<div class="">provider: tcp</div>
<div class=""> fabric: TCP-IP</div>
<div class=""> domain: tcp</div>
<div class=""> version: 0.1</div>
<div class=""> type: FI_EP_MSG</div>
<div class=""> protocol: FI_PROTO_SOCK_TCP</div>
<div class="">provider: shm</div>
<div class=""> fabric: shm</div>
<div class=""> domain: shm</div>
<div class=""> version: 1.0</div>
<div class=""> type: FI_EP_RDM</div>
<div class=""> protocol: FI_PROTO_SHM</div>
</div>
<div class=""><br class="">
</div>
<div class="">Thanks,</div>
<div class="">Philip</div>
<div class="">
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On Jan 16, 2020, at 11:03 AM, Steve Welch <<a href="mailto:swelch@systemfabricworks.com" class="">swelch@systemfabricworks.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">
<div class="" style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
<div class="">Hi Phillip,</div>
<div class=""><br class="">
</div>
<div class="">Since you are specifying an FI_EP_RDM in your hints I assume you want to utilize the RXM provider on top of the Verbs core provider (i.e. ofi_rxm;verbs). The Verbs provider does not offer native FI_RDM_EP support. To use either XRC (or FI_EP_RDM
endpoint) you will have to use RXM, but I am unaware of any IB provider that supported XRC that did not support RC.</div>
<div class=""><br class="">
</div>
<div class="">If you issue a 'fi_info -p verbs -v’ it will list all the verbs domains supported and the underlying protocol and you could verify if RC should be supported (via RXM for FI_EP_RDM). If you issue 'fi_info -p “ofi_rxm;verbs”', you should see multiple
domains for the “ofi_rxm;verbs” provider combination. XRC domains have the “-xrc” suffix.</div>
<div class=""><br class="">
</div>
<div class="">If you must use XRC and the RXM/Verbs combination then you will need to set the environment variable FI_OFI_RXM_USE_SRX=1 and RXM will handle the shared RX details.</div>
<div class=""><br class="">
</div>
<div class="">Steve</div>
<div class=""><br class="">
</div>
</div>
</div>
<div style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<br class="">
<blockquote type="cite" class="">
<div class="">On Jan 16, 2020, at 8:56 AM, Philip Davis <<a href="mailto:philip.e.davis@rutgers.edu" class="">philip.e.davis@rutgers.edu</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="">Hello,<br class="">
<br class="">
I am working with a user that is running on an older Infiniband cluster. Using libfaric with the following hints:<br class="">
<br class="">
hints->caps = FI_MSG | FI_SEND | FI_RECV | FI_REMOTE_READ |<br class="">
FI_REMOTE_WRITE | FI_RMA | FI_READ | FI_WRITE;<br class="">
hints->mode = FI_CONTEXT | FI_LOCAL_MR | FI_CONTEXT2 | FI_MSG_PREFIX |<br class="">
FI_ASYNC_IOV | FI_RX_CQ_DATA;<br class="">
hints->domain_attr->mr_mode = FI_MR_BASIC;<br class="">
hints->domain_attr->control_progress = FI_PROGRESS_AUTO;<br class="">
hints->domain_attr->data_progress = FI_PROGRESS_AUTO;<br class="">
hints->ep_attr->type = FI_EP_RDM;<br class="">
<br class="">
<br class="">
No verbs providers are found. Looking through the debug output, I suspect this is the crucial line:<br class="">
<br class="">
libfabric:verbs:fabric:fi_ibv_get_matching_info():1213<info> hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints<br class="">
<br class="">
I take it that the underlying hardware is only compatible with FI_PROTO_RDMA_CM_IB_XRC protocol for MSG endpoints, and it looks like I need to have FI_SHARED_CONTEXT enabled for these endpoints to be supported. I’m having some trouble understanding the implications
of using FI_SHARED_CONTEXT. If I only ever use one endpoint, is there any functional or performance impact to setting this? I’d rather not change to using shared contexts unconditionally, so is there a good way for me to detect this situation other than to
do a maximally permissive fi_getinfo and iterate through the verbs results?<br class="">
<br class="">
Thanks,<br class="">
Philip<br class="">
_______________________________________________<br class="">
Libfabric-users mailing list<br class="">
<a href="mailto:Libfabric-users@lists.openfabrics.org" class="">Libfabric-users@lists.openfabrics.org</a><br class="">
<a href="https://lists.openfabrics.org/mailman/listinfo/libfabric-users" class="">https://lists.openfabrics.org/mailman/listinfo/libfabric-users</a></div>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</div></blockquote></div><br class=""></body></html>