<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class=""><div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class="">Hi Phillip,</div><div class=""><br class=""></div><div class="">Since you are specifying an FI_EP_RDM in your hints I assume you want to utilize the RXM provider on top of the Verbs core provider (i.e. ofi_rxm;verbs). The Verbs provider does not offer native FI_RDM_EP support. To use either XRC (or FI_EP_RDM endpoint) you will have to use RXM, but I am unaware of any IB provider that supported XRC that did not support RC.</div><div class=""><br class=""></div><div class="">If you issue a 'fi_info -p verbs -v’ it will list all the verbs domains supported and the underlying protocol and you could verify if RC should be supported (via RXM for FI_EP_RDM). If you issue 'fi_info -p “ofi_rxm;verbs”', you should see multiple domains for the “ofi_rxm;verbs” provider combination. XRC domains have the “-xrc” suffix.</div><div class=""><br class=""></div><div class="">If you must use XRC and the RXM/Verbs combination then you will need to set the environment variable FI_OFI_RXM_USE_SRX=1 and RXM will handle the shared RX details.</div><div class=""><br class=""></div><div class="">Steve</div><div class=""><br class=""></div></div>
</div>
<div><br class=""><blockquote type="cite" class=""><div class="">On Jan 16, 2020, at 8:56 AM, Philip Davis <<a href="mailto:philip.e.davis@rutgers.edu" class="">philip.e.davis@rutgers.edu</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class="">Hello,<br class=""><br class="">I am working with a user that is running on an older Infiniband cluster. Using libfaric with the following hints:<br class=""><br class="">hints->caps = FI_MSG | FI_SEND | FI_RECV | FI_REMOTE_READ |<br class=""> FI_REMOTE_WRITE | FI_RMA | FI_READ | FI_WRITE;<br class=""> hints->mode = FI_CONTEXT | FI_LOCAL_MR | FI_CONTEXT2 | FI_MSG_PREFIX |<br class=""> FI_ASYNC_IOV | FI_RX_CQ_DATA;<br class=""> hints->domain_attr->mr_mode = FI_MR_BASIC;<br class=""> hints->domain_attr->control_progress = FI_PROGRESS_AUTO;<br class=""> hints->domain_attr->data_progress = FI_PROGRESS_AUTO;<br class=""> hints->ep_attr->type = FI_EP_RDM;<br class=""><br class=""><br class="">No verbs providers are found. Looking through the debug output, I suspect this is the crucial line:<br class=""><br class="">libfabric:verbs:fabric:fi_ibv_get_matching_info():1213<info> hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints<br class=""><br class="">I take it that the underlying hardware is only compatible with FI_PROTO_RDMA_CM_IB_XRC protocol for MSG endpoints, and it looks like I need to have FI_SHARED_CONTEXT enabled for these endpoints to be supported. I’m having some trouble understanding the implications of using FI_SHARED_CONTEXT. If I only ever use one endpoint, is there any functional or performance impact to setting this? I’d rather not change to using shared contexts unconditionally, so is there a good way for me to detect this situation other than to do a maximally permissive fi_getinfo and iterate through the verbs results?<br class=""><br class="">Thanks,<br class="">Philip<br class="">_______________________________________________<br class="">Libfabric-users mailing list<br class=""><a href="mailto:Libfabric-users@lists.openfabrics.org" class="">Libfabric-users@lists.openfabrics.org</a><br class="">https://lists.openfabrics.org/mailman/listinfo/libfabric-users<br class=""></div></div></blockquote></div><br class=""></body></html>