<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body>
<div class="BodyFragment"><font size="2"><span style="font-size:11pt;">
<div class="PlainText">Hi Sean,<br>
<br>
> On Jan 16, 2020, at 11:11 AM, Hefty, Sean <sean.hefty@intel.com> wrote:<br>
> <br>
>> I am working with a user that is running on an older Infiniband cluster. Using libfaric<br>
>> with the following hints:<br>
>> <br>
>> hints->caps = FI_MSG | FI_SEND | FI_RECV | FI_REMOTE_READ |<br>
>> FI_REMOTE_WRITE | FI_RMA | FI_READ | FI_WRITE;<br>
>> hints->mode = FI_CONTEXT | FI_LOCAL_MR | FI_CONTEXT2 | FI_MSG_PREFIX |<br>
>> FI_ASYNC_IOV | FI_RX_CQ_DATA;<br>
>> hints->domain_attr->mr_mode = FI_MR_BASIC;<br>
> <br>
> You may want to consider updating to the newer mr_mode bits. This field was changed starting in the 1.5 release.<br>
> <br>
>> hints->domain_attr->control_progress = FI_PROGRESS_AUTO;<br>
>> hints->domain_attr->data_progress = FI_PROGRESS_AUTO;<br>
>> hints->ep_attr->type = FI_EP_RDM;<br>
> <br>
> This is requesting RDM endpoints, not MSG. Is this the intent for your app, and the issue is that it can't find the verbs support underneath?<br>
<br>
That’s right, I am expecting to use RXM for libfabric >= 1.7, and the issue is there is no verbs provider that supports FI_EP_MSG<br>
> <br>
>> No verbs providers are found. Looking through the debug output, I suspect this is the<br>
>> crucial line:<br>
> <br>
> If you run fi_info, do you see the verbs provider there?<br>
<br>
I see verbs providers that support FI_EP_DGRAM, but not FI_EP_MSG.<br>
<br>
> <br>
>> libfabric:verbs:fabric:fi_ibv_get_matching_info():1213<info> hints->ep_attr->rx_ctx_cnt<br>
>> != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints<br>
> <br>
> This is checking for XRC QP support. Skipping these if the hardware doesn't support it should be fine.<br>
<br>
Okay, thanks. I was thinking that’s why the FI_EP_MSG support wasn’t there.<br>
<br>
> <br>
>> I take it that the underlying hardware is only compatible with FI_PROTO_RDMA_CM_IB_XRC<br>
>> protocol for MSG endpoints, and it looks like I need to have FI_SHARED_CONTEXT enabled<br>
>> for these endpoints to be supported. I’m having some trouble understanding the<br>
>> implications of using FI_SHARED_CONTEXT. If I only ever use one endpoint, is there any<br>
>> functional or performance impact to setting this? I’d rather not change to using shared<br>
>> contexts unconditionally, so is there a good way for me to detect this situation other<br>
>> than to do a maximally permissive fi_getinfo and iterate through the verbs results?<br>
> <br>
> You don't need to use shared contexts or XRC. When you mention only using one endpoint, do you mean one MSG endpoint or one RDM endpoint?<br>
<br>
One RDM endpoint.<br>
<br>
> <br>
> What version of libfabric are you using? Attaching the full debug output from the startup checks might help isolate the problem.<br>
<br>
This output is from 1.7.2. The user reports the same problem with 1.6.x and 1.9.x, but I unfortuantely don’t have access to debug logs from those versions. I have attached the debug log from the startup phase. I’m afraid it’s a bit of a mess because it’s multiple
unlabeled ranks.<br>
> <br>
> - Sean<br>
</div>
</span></font></div>
<div class="BodyFragment"><font size="2"><span style="font-size:11pt;">
<div class="PlainText"><br>
Thank you for your help.<br>
<br>
Philip<br>
<br>
</div>
</span></font></div>
</body>
</html>