[libfabric-users] [External] gni provider and FI_REMOTE_CQ_DATA

Wed Sep 9 11:18:49 PDT 2020

Logged (with potential fix) as https://github.com/ofiwg/libfabric/issues/6237.

On Sep 9, 2020, at 9:12 AM, D'Alessandro, Luke K <ldalessa at iu.edu<mailto:ldalessa at iu.edu>> wrote:

Yet another reeky to myself (sorry). Hope this helps someone out at some point in the future.

On Sep 4, 2020, at 4:22 PM, D'Alessandro, Luke K <ldalessa at iu.edu<mailto:ldalessa at iu.edu>> wrote:

So I just discovered in the log:

libfabric:31689:gni:ep_data:gnix_ops_allowed():887<debug> [31689:2] flags:0x2220204, FI_REMOTE_CQ_DATA, FI_FENCE, FI_INJECT
libfabric:31689:gni:ep_data:gnix_ops_allowed():889<debug> [31689:2] peer_caps:0x118000000312004, FI_MULTI_RECV, FI_TRIGGER, FI_FENCE
libfabric:31689:gni:ep_data:gnix_ops_allowed():891<debug> [31689:2] caps:0x118000000312004, FI_RMA, FI_REMOTE_WRITE, FI_MULTI_RECV, FI_TRIGGER, FI_FENCE, FI_LOCAL_COMM, FI_REMOTE_COMM, FI_RMA_EVENT
libfabric:31689:gni:cq:_gnix_cq_add_error():325<info> [31689:2] creating error event entry

And some hunting in a debug build shows me that I’m failing at https://github.com/ofiwg/libfabric/blob/master/prov/gni/src/gnix_rma.c#L1224.

I guess that I haven’t set up the endpoint/cq appropriately, so I’ll keep poking at that to see where I have gone wrong.

It turns out that we need to request both FI_WRITE and FI_REMOTE_WRITE in order to support remote CQ write events with fi_gni, even though we never try and create local write completion or do local write operations. With this in place I am getting the immediate data through to the remote end.

***Unfortunately***

This does not transmit the address information, even after adding FI_SOURCE to our requested capabilities. I’ve stepped through a debug libfabric build and it appears that, even though FI_SOURCE is advertised by GNI, it is unsupported in the rma code path. fi_gni uses a two-put model to send immediate data, the first is a GNI_PostFma operation to perform the op, and the second is a GNI_SmsgSendWTag operation to send the immediate data.

The GNI_SmsgSendWTag operation only sends the immediate data value, and not the target address. The relevant message structure and send operation are found here:

https://github.com/ofiwg/libfabric/blob/master/prov/gni/include/gnix_nic.h#L303
https://github.com/ofiwg/libfabric/blob/master/prov/gni/src/gnix_rma.c#L364

And the relevant remote CQ handler is found here:

https://github.com/ofiwg/libfabric/blob/master/prov/gni/src/gnix_rma.c#L262
https://github.com/ofiwg/libfabric/blob/master/prov/gni/src/gnix_rma.c#L272

You can clearly see that the address is uniformly set to NULL.

I’m not sure why the message structure doesn’t contain the address as well, it feels like if the provider is agreeing to FI_SOURCE that it should be actually providing the source. It could be that there is some pressure to reduce the size of the message structure here? Or perhaps it is an oversight… I’ll probably go ahead and submit it as an issue for the GNI provider.

Thanks,
Luke

Thanks,
Luke

On Sep 3, 2020, at 4:07 PM, D'Alessandro, Luke K <ldalessa at iu.edu<mailto:ldalessa at iu.edu>> wrote:

This message was sent from a non-IU address. Please exercise caution when clicking links or opening attachments from external sources.

Hi All,

I have some test code that depends on FI_REMOTE_CQ_DATA which I’ve debugged using the  UDP;ofi_rxd provider.

I am trying to run that code using the gni provider on an XC40 but I’m not ever seeing the remote CQ events there. Is there something special I need to set up to get remote CQ events with gni?

I request:

fi_info *hints = fi_allocinfo();
hints->caps                   = FI_RMA | FI_REMOTE_WRITE | FI_RMA_EVENT;
hints->mode                   = FI_CONTEXT | FI_CONTEXT2;
hints->domain_attr->mr_mode   = FI_MR_BASIC;
hints->ep_attr->type          = FI_EP_RDM;
hints->tx_attr->msg_order     = FI_ORDER_WAW | FI_ORDER_RMA_WAW;
hints->rx_attr->msg_order     = FI_ORDER_WAW | FI_ORDER_RMA_WAW;
hints->rx_attr->caps          = FI_RMA | FI_REMOTE_WRITE | FI_RMA_EVENT;

And I successfully receive:

0: # Provider                           Fabric               Domain Version     EP_TYPE    Protocol
0: # gni                                   gni /sys/class/gni/kgni0     1.1   FI_EP_RDM   FI_EP_RDM
0: # gni;ofi_rxd                           gni /sys/class/gni/kgni0   111.0   FI_EP_RDM   FI_EP_RDM
...
1: # Provider                           Fabric               Domain Version     EP_TYPE    Protocol
1: # gni                                   gni /sys/class/gni/kgni0     1.1   FI_EP_RDM   FI_EP_RDM
1: # gni;ofi_rxd                           gni /sys/class/gni/kgni0   111.0   FI_EP_RDM   FI_EP_RDM

I move through sequence of initialization calls that seem to be standard from what I can tell, resulting in an endpoint that is enabled successfully.

static fi_context ep_ctx[2];
check(fi_endpoint, domain, info, &ep, ep_ctx);
check(fi_ep_bind, ep, &tx->fid, FI_TRANSMIT | FI_SELECTIVE_COMPLETION);
check(fi_ep_bind, ep, &rx->fid, FI_RECV);
check(fi_ep_bind, ep, &av->fid, 0);
check(fi_enable, ep);

Messages are sent with fi_writemsg and FI_REMOTE_CQ_DATA, and neither fail nor signal FI_EAGAIN (this is a little alarming as I have tx/rx size of 500 and I send more than that through the endpoint, I guess they just vanish into the ether).

int e = fi_writemsg(ep, &msg, FI_REMOTE_CQ_DATA);

if (likely(!e)) {
  return true;
}

if (likely(e == -FI_EAGAIN)) {
  return false;
}

fmt::print(stderr, "[{}] has unhandled tx error {}: {}\n", mpi::rank(), -e, fi_strerror(-e));

Unfortunately I never see any completions on the target rank (unlike UDP;ofi_rxd where things are fine).

Is there some magic that I need with gni to make FI_REMOTE_CQ_DATA work?

Thanks,
Luke
_______________________________________________
Libfabric-users mailing list
Libfabric-users at lists.openfabrics.org<mailto:Libfabric-users at lists.openfabrics.org>
https://lists.openfabrics.org/mailman/listinfo/libfabric-users

_______________________________________________
Libfabric-users mailing list
Libfabric-users at lists.openfabrics.org<mailto:Libfabric-users at lists.openfabrics.org>
https://lists.openfabrics.org/mailman/listinfo/libfabric-users

_______________________________________________
Libfabric-users mailing list
Libfabric-users at lists.openfabrics.org<mailto:Libfabric-users at lists.openfabrics.org>
https://lists.openfabrics.org/mailman/listinfo/libfabric-users

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/libfabric-users/attachments/20200909/d7b39802/attachment-0001.htm>