[libfabric-users] fi_cq_open fails on the verbs provider

[Jörn Schumacher]

Hi,

I managed to setup a reliable connection using libfabric and the sockets 
provider. I use FI_WAIT_FD with epoll to manage completions. Now I am 
trying to run the code on the verbs provider, but it fails on the client 
side while creating a CQ:

>   struct fi_cq_attr cq_attr;
>   cq_attr.size = 0;      /* # entries for CQ */
>   cq_attr.flags = 0;     /* operation flags */
>   cq_attr.format = FI_CQ_FORMAT_CONTEXT;    /* completion format */
>   cq_attr.wait_obj= FI_WAIT_FD;  /* requested wait object */
>   cq_attr.signaling_vector = 0; /* interrupt affinity */
>   cq_attr.wait_cond = FI_WAIT_NONE; /* wait condition format */
>   cq_attr.wait_set = NULL;  /* optional wait set */
>
>   if(ret = fi_cq_open(socket->domain, &cq_attr, &socket->cq, NULL))
>   {
>     ERROR("fi_cq_open failed: %d '%s'", ret, fi_strerror(-ret));
>   }

fi_cq_open returns -22 ('Invalid argument'). The manpage of fi_verbs 
says FI_WAIT_FD is supported. FI_CQ_FORMAT_UNSPEC also did not help. 
What could be the source of this error?

Here is the full code:
https://gist.github.com/joerns/7833f418edb28e348d98#file-client-c-L83

Thanks for any help,
Jörn