[libfabric-users] IB: Resource temporarily unavailable

[Ilango, Arun]

Hi Francois,

I'm able to reproduce the issue on my setup using verbs provider and I'm looking into it currently. Will get back to you soon.

Thanks,
Arun.

-----Original Message-----
From: libfabric-users-bounces at lists.openfabrics.org [mailto:libfabric-users-bounces at lists.openfabrics.org] On Behalf Of François Tessier
Sent: Tuesday, February 16, 2016 10:07 AM
To: Jose, Jithin <jithin.jose at intel.com>; libfabric-users at lists.openfabrics.org
Subject: Re: [libfabric-users] IB: Resource temporarily unavailable

Hi Jithin,

I ran again these experiments while setting the env parameter for debug logs. On the client side, when I use "-f verbs <IP server IB interface>, here are the logs:

libfabric:core:core:fi_param_define_():226<info> registered var provider libfabric:core:core:fi_param_get_():275<info> variable provider=<not set> libfabric:core:core:fi_param_define_():226<info> registered var provider_path libfabric:core:core:fi_param_get_():275<info> variable provider_path=<not set> libfabric:core:core:fi_register_provider():123<info> registering
provider: verbs (1.0)
libfabric:sockets:core:fi_param_define_():226<info> registered var pe_waittime libfabric:sockets:core:fi_param_define_():226<info> registered var max_conn_retry libfabric:sockets:core:fi_param_define_():226<info> registered var def_conn_map_sz libfabric:sockets:core:fi_param_define_():226<info> registered var def_av_sz libfabric:sockets:core:fi_param_define_():226<info> registered var def_cq_sz libfabric:sockets:core:fi_param_define_():226<info> registered var def_eq_sz libfabric:sockets:core:fi_param_define_():226<info> registered var pe_affinity libfabric:sockets:core:fi_param_define_():226<info> registered var dgram_drop_rate libfabric:core:core:fi_register_provider():123<info> registering
provider: sockets (1.0)
libfabric:verbs:core:fi_ibv_check_hints():414<info> Unsupported capabilities libfabric:verbs:core:fi_ibv_msg_ep_connect():148<info> src_addr:
172.23.100.114:47212
libfabric:verbs:core:fi_ibv_msg_ep_connect():155<info> dst_addr:
172.23.100.221:12345
transmit(): common/shared.c:805, ret=-11 (Resource temporarily unavailable) libfabric:sockets:core:fi_param_undefine():163<debug> Removing param:
pe_waittime
libfabric:sockets:core:fi_param_undefine():163<debug> Removing param:
max_conn_retry
libfabric:sockets:core:fi_param_undefine():163<debug> Removing param:
def_conn_map_sz
libfabric:sockets:core:fi_param_undefine():163<debug> Removing param:
def_av_sz
libfabric:sockets:core:fi_param_undefine():163<debug> Removing param:
def_cq_sz
libfabric:sockets:core:fi_param_undefine():163<debug> Removing param:
def_eq_sz
libfabric:sockets:core:fi_param_undefine():163<debug> Removing param:
pe_affinity
libfabric:sockets:core:fi_param_undefine():163<debug> Removing param:
dgram_drop_rate

If I use "-f sockets" on both sides with the IP address of the IB interface, it works and the performances are quite good (which is curious considering the use of IP over IB...).

Regards,

François

On 16/02/2016 11:16, Jose, Jithin wrote:
> Hi Francios,
>
> The usage seems fine. Do you have any logs from the run? Additional logs may help to identify the issue here.
>
> Logs can be enabled by setting the env parameter FI_LOG_LEVEL=<warn|debug>. Debug logs are enabled only in debug build (--enable-debug).
>
> For the second run, are you using the sockets provider or the verbs provider (-f option)? Sockets provider should be able to run on both interfaces.
>
> - Jithin
>
>
>
>
>
> -----Original Message-----
> From: <libfabric-users-bounces at lists.openfabrics.org> on behalf of 
> François Tessier <ftessier at anl.gov>
> Date: Monday, February 15, 2016 at 12:37 PM
> To: "libfabric-users at lists.openfabrics.org" 
> <libfabric-users at lists.openfabrics.org>
> Subject: [libfabric-users] IB: Resource temporarily unavailable
>
>> Hello,
>>
>> I am trying to use libfabric and particularly through the given 
>> examples in the fabtest repository.
>>
>> On our local platform, I tried to run fi_msg_pingpong :
>>
>>    - Server side: ./fi_msg_pingpong -b 12345 -f sockets -I 1000 -s
>> 10.233.100.99
>>    - Client side: ./fi_msg_pingpong -p 12345 -f sockets 10.233.100.99
>>
>> It works perfectly. However, when I try to use verbs instead of 
>> sockets (while modifying the address with the one of the IB 
>> interface), the client gives as output "transmit(): 
>> common/shared.c:805, ret=-11 (Resource temporarily unavailable)". Is 
>> there something wrong with my use? I compiled libfabric with options to enable both sockets and verbs.
>> Our platform has a FDR Infiniband interconnect.
>>
>> Thank you for your help,
>>
>> Regards,
>>
>> François
>>
>> --
>> Dr. François TESSIER
>> Argonne National Laboratory
>> LCF Division - Bldg 240, 4E 12
>> Tel : +1 (630)-252-5068
>> http://www.francoistessier.info
>> PGP 0x8096B5FA
>>
>> _______________________________________________
>> Libfabric-users mailing list
>> Libfabric-users at lists.openfabrics.org
>> http://lists.openfabrics.org/mailman/listinfo/libfabric-users

_______________________________________________
Libfabric-users mailing list
Libfabric-users at lists.openfabrics.org
http://lists.openfabrics.org/mailman/listinfo/libfabric-users