[libfabric-users] fi_read verbs ENODATA

[Hefty, Sean]

> hints->caps = FI_RMA|FI_MSG;
> 
> hints->ep_attr->type = FI_EP_MSG;
> 
> hints->addr_format = FI_SOCKADDR_IN;
> 
> hints->fabric_attr->prov_name = g_strdup("verbs");
> 
> hints->mode = FI_LOCAL_MR;

For verbs and any recent release of libfabric, you will want to remove setting this.  Instead, set:

hints->domain_attr->mr_mode = FI_MR_LOCAL | FI_MR_VIRT_ADDR |
					FI_MR_ALLOCATED | FI_MR_PROV_KEY;

>      mode: [ FI_RX_CQ_DATA ]

You will want to add this to hints->mode if you plan on using CQ data (verbs refers to this as immediate data).  You can leave this off if not.
 
> Output with FI_DEBUG_LEVEL="info" returns much info about how certain
> variables are not set (like rx_size), which shouldnt be a problem, right?

Correct - that just means that the provider will pick a default.

> So where is the problem with my call? Isnt the call I make in the
> application the same as the one in the command line?

The problem is that verbs requires additional support by the application, mostly to handle memory registration restrictions.  Those are the mr_mode bits called out above.

> The only difference I can spot is the FI_SOURCE-flag for the
> get_info-call and I was told here before that verbs-provider supports
> said flag.
> 
> 
> Or does the hints->domain_attr->mr_mode need to be set to FI_MR_BASIC
> despite it being non-defined for libfabric version 1.5 and later?

I'll try not to confuse things, but...

You can set mr_mode = FI_MR_BASIC *instead of* using the individual bits mentioned above.  However, if you do, you will also need to set hints->mode = FI_LOCAL_MR.

I recommend setting mr_mode as mentioned above, as it will be the only way to access more general hardware.

- Sean