[libfabric-users] IB: Resource temporarily unavailable

[François Tessier]

Hi Jithin,

I ran again these experiments while setting the env parameter for debug
logs. On the client side, when I use "-f verbs <IP server IB interface>,
here are the logs:

libfabric:core:core:fi_param_define_():226<info> registered var provider
libfabric:core:core:fi_param_get_():275<info> variable provider=<not set>
libfabric:core:core:fi_param_define_():226<info> registered var
provider_path
libfabric:core:core:fi_param_get_():275<info> variable
provider_path=<not set>
libfabric:core:core:fi_register_provider():123<info> registering
provider: verbs (1.0)
libfabric:sockets:core:fi_param_define_():226<info> registered var
pe_waittime
libfabric:sockets:core:fi_param_define_():226<info> registered var
max_conn_retry
libfabric:sockets:core:fi_param_define_():226<info> registered var
def_conn_map_sz
libfabric:sockets:core:fi_param_define_():226<info> registered var def_av_sz
libfabric:sockets:core:fi_param_define_():226<info> registered var def_cq_sz
libfabric:sockets:core:fi_param_define_():226<info> registered var def_eq_sz
libfabric:sockets:core:fi_param_define_():226<info> registered var
pe_affinity
libfabric:sockets:core:fi_param_define_():226<info> registered var
dgram_drop_rate
libfabric:core:core:fi_register_provider():123<info> registering
provider: sockets (1.0)
libfabric:verbs:core:fi_ibv_check_hints():414<info> Unsupported capabilities
libfabric:verbs:core:fi_ibv_msg_ep_connect():148<info> src_addr:
172.23.100.114:47212
libfabric:verbs:core:fi_ibv_msg_ep_connect():155<info> dst_addr:
172.23.100.221:12345
transmit(): common/shared.c:805, ret=-11 (Resource temporarily unavailable)
libfabric:sockets:core:fi_param_undefine():163<debug> Removing param:
pe_waittime
libfabric:sockets:core:fi_param_undefine():163<debug> Removing param:
max_conn_retry
libfabric:sockets:core:fi_param_undefine():163<debug> Removing param:
def_conn_map_sz
libfabric:sockets:core:fi_param_undefine():163<debug> Removing param:
def_av_sz
libfabric:sockets:core:fi_param_undefine():163<debug> Removing param:
def_cq_sz
libfabric:sockets:core:fi_param_undefine():163<debug> Removing param:
def_eq_sz
libfabric:sockets:core:fi_param_undefine():163<debug> Removing param:
pe_affinity
libfabric:sockets:core:fi_param_undefine():163<debug> Removing param:
dgram_drop_rate

If I use "-f sockets" on both sides with the IP address of the IB
interface, it works and the performances are quite good (which is
curious considering the use of IP over IB...).

Regards,

François

On 16/02/2016 11:16, Jose, Jithin wrote:
> Hi Francios,
>
> The usage seems fine. Do you have any logs from the run? Additional logs may help to identify the issue here.
>
> Logs can be enabled by setting the env parameter FI_LOG_LEVEL=<warn|debug>. Debug logs are enabled only in debug build (--enable-debug).
>
> For the second run, are you using the sockets provider or the verbs provider (-f option)? Sockets provider should be able to run on both interfaces.
>
> - Jithin
>
>
>
>
>
> -----Original Message-----
> From: <libfabric-users-bounces at lists.openfabrics.org> on behalf of François Tessier <ftessier at anl.gov>
> Date: Monday, February 15, 2016 at 12:37 PM
> To: "libfabric-users at lists.openfabrics.org" <libfabric-users at lists.openfabrics.org>
> Subject: [libfabric-users] IB: Resource temporarily unavailable
>
>> Hello,
>>
>> I am trying to use libfabric and particularly through the given examples
>> in the fabtest repository.
>>
>> On our local platform, I tried to run fi_msg_pingpong :
>>
>>    - Server side: ./fi_msg_pingpong -b 12345 -f sockets -I 1000 -s
>> 10.233.100.99
>>    - Client side: ./fi_msg_pingpong -p 12345 -f sockets 10.233.100.99
>>
>> It works perfectly. However, when I try to use verbs instead of sockets
>> (while modifying the address with the one of the IB interface), the
>> client gives as output "transmit(): common/shared.c:805, ret=-11
>> (Resource temporarily unavailable)". Is there something wrong with my
>> use? I compiled libfabric with options to enable both sockets and verbs.
>> Our platform has a FDR Infiniband interconnect.
>>
>> Thank you for your help,
>>
>> Regards,
>>
>> François
>>
>> -- 
>> Dr. François TESSIER
>> Argonne National Laboratory
>> LCF Division - Bldg 240, 4E 12
>> Tel : +1 (630)-252-5068
>> http://www.francoistessier.info
>> PGP 0x8096B5FA
>>
>> _______________________________________________
>> Libfabric-users mailing list
>> Libfabric-users at lists.openfabrics.org
>> http://lists.openfabrics.org/mailman/listinfo/libfabric-users