<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
</head>
<body dir="ltr">
<style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
<div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif;" dir="ltr">
<p>Not sure if it's connected but if I ssh into compute nodes and run fi_pingpong on two nodes, the client dies with</p>
<p><br>
</p>
<p></p>
<div>/apps/daint/UES/biddisco/gcc/8.3.0/libfabric/bin/fi_pingpong 148.187.32.221<br>
[error] /apps/daint/UES/biddisco/src/libfabric-cray/util/pingpong.c:521 : ctrl/read: no data or remote connection closed</div>
<p></p>
<p><br>
</p>
<p>and the sever dies with <br>
</p>
<p></p>
<div>nid00220:~$ /apps/daint/UES/biddisco/gcc/8.3.0/libfabric/bin/fi_pingpong <br>
libfabric:55223:core:core:fi_param_define_():231<debug> registered var perf_cntr<br>
libfabric:55223:core:core:fi_param_get_():280<info> variable perf_cntr=<not set><br>
libfabric:55223:core:core:fi_param_define_():231<debug> registered var hook<br>
libfabric:55223:core:core:fi_param_get_():280<info> variable hook=<not set><br>
libfabric:55223:core:core:fi_param_define_():231<debug> registered var mr_cache_max_size<br>
libfabric:55223:core:core:fi_param_define_():231<debug> registered var mr_cache_max_count<br>
libfabric:55223:core:core:fi_param_define_():231<debug> registered var mr_cache_monitor<br>
libfabric:55223:core:core:fi_param_get_():280<info> variable mr_cache_max_size=<not set><br>
libfabric:55223:core:core:fi_param_get_():280<info> variable mr_cache_max_count=<not set><br>
libfabric:55223:core:core:fi_param_get_():280<info> variable mr_cache_monitor=<not set><br>
libfabric:55223:core:mr:ofi_default_cache_size():56<info> default cache size=468306659<br>
libfabric:55223:core:core:fi_param_define_():231<debug> registered var provider<br>
libfabric:55223:core:core:fi_param_define_():231<debug> registered var fork_unsafe<br>
libfabric:55223:core:core:fi_param_define_():231<debug> registered var universe_size<br>
libfabric:55223:core:core:fi_param_get_():280<info> variable provider=<not set><br>
libfabric:55223:core:core:fi_param_define_():231<debug> registered var provider_path<br>
libfabric:55223:core:core:fi_param_get_():280<info> variable provider_path=<not set><br>
libfabric:55223:core:core:ofi_register_provider():396<debug> no provider structure or name<br>
libfabric:55223:core:core:ofi_register_provider():396<debug> no provider structure or name<br>
libfabric:55223:core:core:ofi_register_provider():396<debug> no provider structure or name<br>
libfabric:55223:gni:fabric:__gnix_ccm_init():171<debug> [55223:1] Reading job info file /tmp/ccm_alps_info<br>
libfabric:55223:gni:fabric:__gnix_alps_init():284<warn> [55223:1] lli get response failed, alps_status=4(No such file or directory)<br>
libfabric:55223:gni:fabric:_gnix_nics_per_rank():672<warn> [55223:1] __gnix_app_init() failed, ret=-5(No such file or directory)<br>
libfabric:55223:gni:fabric:_gnix_nic_init():1414<warn> [55223:1] _gnix_nics_per_rank failed: -5<br>
libfabric:55223:core:core:ofi_register_provider():402<info> registering provider: gni (1.1)<br>
libfabric:55223:core:core:ofi_register_provider():396<debug> no provider structure or name<br>
libfabric:55223:core:core:ofi_register_provider():396<debug> no provider structure or name<br>
libfabric:55223:core:core:ofi_register_provider():396<debug> no provider structure or name<br>
libfabric:55223:core:core:ofi_register_provider():396<debug> no provider structure or name<br>
libfabric:55223:core:core:ofi_register_provider():396<debug> no provider structure or name<br>
libfabric:55223:core:core:ofi_register_provider():396<debug> no provider structure or name<br>
libfabric:55223:core:core:ofi_register_provider():396<debug> no provider structure or name<br>
libfabric:55223:core:core:ofi_register_provider():396<debug> no provider structure or name<br>
libfabric:55223:core:core:ofi_register_provider():396<debug> no provider structure or name<br>
libfabric:55223:core:core:ofi_register_provider():396<debug> no provider structure or name<br>
libfabric:55223:core:core:ofi_register_provider():396<debug> no provider structure or name<br>
libfabric:55223:core:core:ofi_register_provider():396<debug> no provider structure or name<br>
libfabric:55223:core:core:ofi_register_provider():402<info> registering provider: ofi_hook_debug (110.10)<br>
libfabric:55223:core:core:ofi_register_provider():402<info> registering provider: ofi_hook_noop (110.10)<br>
libfabric:55223:gni:fabric:_gnix_ep_getinfo():457<trace> [55223:1] <br>
libfabric:55223:gni:fabric:_gnix_ep_getinfo():457<trace> [55223:1] <br>
libfabric:55223:gni:fabric:_gnix_ep_getinfo():507<debug> [55223:1] Passed EP attributes check<br>
libfabric:55223:gni:fabric:_gnix_ep_getinfo():522<debug> [55223:1] Passed mode check<br>
libfabric:55223:gni:fabric:_gnix_ep_getinfo():532<debug> [55223:1] Passed caps check gnix_info->caps = 0x0f1c000000313f1e<br>
libfabric:55223:gni:fabric:_gnix_ep_getinfo():547<debug> [55223:1] Passed TX attributes check<br>
libfabric:55223:gni:fabric:_gnix_ep_getinfo():565<debug> [55223:1] Passed fabric name check<br>
libfabric:55223:gni:fabric:__gnix_getinfo_resolve_node():417<info> [55223:1] node: (null) service: (null)<br>
libfabric:55223:gni:fabric:__gnix_getinfo_resolve_node():422<info> [55223:1] src_pe: 0xdc src_port: 0x0<br>
libfabric:55223:gni:fabric:_gnix_ep_getinfo():658<debug> [55223:1] Passed the domain attributes check<br>
libfabric:55223:gni:fabric:_gnix_ep_getinfo():677<debug> [55223:1] Returning EP type: FI_EP_DGRAM<br>
libfabric:55223:gni:fabric:_gnix_ep_getinfo():457<trace> [55223:1] <br>
libfabric:55223:core:core:fi_getinfo_():967<debug> fi_getinfo: provider gni returned success<br>
libfabric:55223:gni:core:_gnix_ref_init():254<debug> [55223:1] 0x616e08 refs 1<br>
libfabric:55223:core:core:fi_fabric_():1154<info> Opened fabric: gni<br>
libfabric:55223:gni:eq:gnix_eq_open():380<trace> [55223:1] <br>
libfabric:55223:gni:eq:gnix_verify_eq_attr():103<trace> [55223:1] <br>
libfabric:55223:gni:core:_gnix_ref_init():254<debug> [55223:1] 0x616eb8 refs 1<br>
libfabric:55223:gni:core:gnix_eq_open():398<debug> [55223:1] 0x616e08 refs 2<br>
libfabric:55223:gni:eq:gnix_eq_set_wait():76<trace> [55223:1] <br>
libfabric:55223:gni:eq:gnix_wait_open():536<trace> [55223:1] <br>
libfabric:55223:gni:eq:gnix_verify_wait_attr():367<trace> [55223:1] <br>
libfabric:55223:gni:eq:gnix_init_wait_obj():387<trace> [55223:1] <br>
libfabric:55223:gni:core:gnix_wait_open():564<debug> [55223:1] 0x616e08 refs 3<br>
libfabric:55223:gni:ep_ctrl:__gnix_wait_start_progress():175<trace> [55223:1] <br>
libfabric:55223:gni:ep_ctrl:__gnix_wait_start_progress():179<trace> [55223:1] <br>
libfabric:55223:gni:fabric:gnix_write_proc_job():528<warn> [55223:1] write(disable_affinity_apply) failed, errno=Invalid argument<br>
libfabric:55223:gni:eq:__gnix_wait_start_progress():185<warn> [55223:1] _gnix_job_disable call returned -22<br>
libfabric:55223:gni:domain:gnix_domain_open():579<trace> [55223:1] <br>
libfabric:55223:gni:fabric:gnix_domain_open():591<info> [55223:1] failed to find authorization key, creating new authorization key<br>
libfabric:55223:gni:ep_ctrl:__gnix_wait_nic_prog_thread_fn():72<trace> [55223:2] <br>
libfabric:55223:gni:fabric:__gnix_ccm_init():171<debug> [55223:1] Reading job info file /tmp/ccm_alps_info<br>
libfabric:55223:gni:fabric:__gnix_alps_init():284<warn> [55223:1] lli get response failed, alps_status=4(No such file or directory)<br>
libfabric:55223:gni:fabric:gnixu_get_rdma_credentials():437<warn> [55223:1] __gnix_app_init() failed, ret=-5(No such file or directory)<br>
libfabric:55223:gni:domain:_gnix_auth_key_enable():347<info> [55223:1] pkey=00002aaa ptag=171 key_partition_size=0 key_offset=0 enabled<br>
libfabric:55223:gni:domain:gnix_domain_open():597<info> [55223:1] authorization key=0x61a1a0 ptag 171 cookie 0x2aaa<br>
libfabric:55223:gni:core:gnix_domain_open():652<debug> [55223:1] 0x616e08 refs 4<br>
libfabric:55223:gni:core:_gnix_ref_init():254<debug> [55223:1] 0x61a2e0 refs 1<br>
libfabric:55223:gni:mr:_gnix_auth_key_enable():354<debug> [55223:1] authorization key already enabled, auth_key=0x61a1a0<br>
libfabric:55223:gni:mr:_gnix_mr_reg():222<trace> [55223:1] <br>
libfabric:55223:gni:mr:_gnix_mr_reg():224<info> [55223:1] reg: buf=0x2aaaadfe7000 len=12587008<br>
libfabric:55223:gni:mr:_gnix_mr_cache_init():998<trace> [55223:1] <br>
libfabric:55223:gni:mr:_gnix_mr_cache_init():998<trace> [55223:1] <br>
libfabric:55223:gni:mr:_gnix_mr_cache_register():1541<trace> [55223:1] <br>
libfabric:55223:gni:mr:_gnix_notifier_get_event():270<debug> [55223:1] nothing to read from kdreg :(<br>
libfabric:55223:gni:fabric:__gnix_smrn_read_events():139<debug> [55223:1] no more events to be read<br>
libfabric:55223:gni:mr:__mr_cache_search_inuse():1205<debug> [55223:1] could not find key in inuse, key=2aaaadfe7000:c01000<br>
libfabric:55223:gni:mr:_gnix_notifier_get_event():270<debug> [55223:1] nothing to read from kdreg :(<br>
libfabric:55223:gni:fabric:__gnix_smrn_read_events():139<debug> [55223:1] no more events to be read<br>
libfabric:55223:gni:mr:_gnix_notifier_get_event():270<debug> [55223:1] nothing to read from kdreg :(<br>
libfabric:55223:gni:fabric:__gnix_smrn_read_events():139<debug> [55223:1] no more events to be read<br>
libfabric:55223:gni:mr:__mr_cache_search_stale():1335<debug> [55223:1] searching for stale entry, key=2aaaadfe7000:c01000<br>
libfabric:55223:gni:mr:__gnix_register_region():692<debug> [55223:1] addr 0x2aaaadfe7000 len 12587008 flags 0x0<br>
libfabric:55223:gni:ep_ctrl:gnix_nic_alloc():954<trace> [55223:1] <br>
libfabric:55223:gni:ep_ctrl:gnix_nic_alloc():1059<warn> [55223:1] GNI_CdmAttach returned GNI_RC_INVALID_PARAM<br>
libfabric:55223:gni:fabric:_gnix_dump_gni_res():729<warn> [55223:1] Device Resources:<br>
dev res: MDD, avail: 4089 res: 409 held: 0 total: 4095<br>
dev res: CQ, avail: 2042 res: 10 held: 0 total: 2047<br>
dev res: FMA, avail: 126 res: 4 held: 0 total: 127<br>
dev res: CE, avail: 4 res: 0 held: 0 total: 4<br>
dev res: DLA, avail: 16384 res: 1024 held: 0 total: 16384<br>
dev res: TCR, avail: 65292 res: 0 held: 0 total: 16<br>
dev res: DVA, avail: 4398046511104 res: 1099511627776 held: 0 total: 4398046511104<br>
dev res: VMDH, avail: 4 res: 0 held: 0 total: 4<br>
libfabric:55223:gni:fabric:_gnix_dump_gni_res():745<warn> [55223:1] Job Resources:<br>
libfabric:55223:gni:mr:__gnix_generic_register():609<info> [55223:1] could not allocate nic to do mr_reg, ret=-22<br>
libfabric:55223:gni:mr:__mr_cache_create_registration():1465<info> [55223:1] failed to register memory with callback<br>
fi_mr_reg(): /apps/daint/UES/biddisco/src/libfabric-cray/util/pingpong.c:1329, ret=-12 (Cannot allocate memory)<br>
libfabric:55223:gni:eq:gnix_eq_close():452<trace> [55223:1] <br>
libfabric:55223:gni:core:gnix_eq_close():459<debug> [55223:1] 0x616eb8 refs 0<br>
libfabric:55223:gni:core:__eq_destruct():243<debug> [55223:1] 0x616e08 refs 3<br>
libfabric:55223:gni:eq:gnix_wait_close():505<trace> [55223:1] <br>
libfabric:55223:gni:core:gnix_wait_close():520<debug> [55223:1] 0x616e08 refs 2<br>
libfabric:55223:gni:ep_ctrl:__gnix_wait_stop_progress():201<trace> [55223:1] <br>
libfabric:55223:gni:domain:gnix_domain_close():218<trace> [55223:1] <br>
libfabric:55223:gni:mr:__mr_cache_flush():1109<trace> [55223:1] <br>
libfabric:55223:gni:mr:__mr_cache_flush():1111<debug> [55223:1] starting flush on memory registration cache<br>
libfabric:55223:gni:mr:__mr_cache_flush():1155<debug> [55223:1] flushed 0 of 0 entries from memory registration cache<br>
libfabric:55223:gni:mr:__mr_cache_flush():1109<trace> [55223:1] <br>
libfabric:55223:gni:mr:__mr_cache_flush():1111<debug> [55223:1] starting flush on memory registration cache<br>
libfabric:55223:gni:mr:__mr_cache_flush():1155<debug> [55223:1] flushed 0 of 0 entries from memory registration cache<br>
libfabric:55223:gni:core:gnix_domain_close():265<debug> [55223:1] 0x61a2e0 refs 0<br>
libfabric:55223:gni:domain:__domain_destruct():77<trace> [55223:1] <br>
libfabric:55223:gni:mr:_gnix_mr_cache_destroy():1071<trace> [55223:1] <br>
libfabric:55223:gni:mr:__mr_cache_flush():1109<trace> [55223:1] <br>
libfabric:55223:gni:mr:__mr_cache_flush():1111<debug> [55223:1] starting flush on memory registration cache<br>
libfabric:55223:gni:mr:__mr_cache_flush():1155<debug> [55223:1] flushed 0 of 0 entries from memory registration cache<br>
libfabric:55223:gni:mr:_gnix_mr_cache_destroy():1071<trace> [55223:1] <br>
libfabric:55223:gni:mr:__mr_cache_flush():1109<trace> [55223:1] <br>
libfabric:55223:gni:mr:__mr_cache_flush():1111<debug> [55223:1] starting flush on memory registration cache<br>
libfabric:55223:gni:mr:__mr_cache_flush():1155<debug> [55223:1] flushed 0 of 0 entries from memory registration cache<br>
libfabric:55223:gni:core:__domain_destruct():103<debug> [55223:1] 0x616e08 refs 1<br>
libfabric:55223:gni:domain:gnix_domain_close():274<info> [55223:1] gnix_domain_close invoked returning 0<br>
libfabric:55223:gni:core:gnix_fabric_close():194<debug> [55223:1] 0x616e08 refs 0</div>
<div><br>
</div>
<div>JB<br>
</div>
<br>
<p></p>
</div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Libfabric-users <libfabric-users-bounces@lists.openfabrics.org> on behalf of Biddiscombe, John A. <biddisco@cscs.ch><br>
<b>Sent:</b> 16 June 2020 18:33:53<br>
<b>To:</b> libfabric-users@lists.openfabrics.org<br>
<b>Subject:</b> [libfabric-users] Error allocating domain</font>
<div> </div>
</div>
<div>
<div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif;" dir="ltr">
<p></p>
<div>I've got this log when I dump out my own messages, and also enable debugging in libfabric - can anyone tell what's wrong from the message. Code that used to work seems to have stopped. I upgraded to libfabric 1.10.1 tag and rebuilt, but it didn't change.</div>
<div><br>
</div>
<div>The only thing that springs to mind is that the application is also using MPI on the cray at the same time, so when this code is called, mpi_init would have already been called, and perhaps somehow the nic is inaccessible - hence the error. I'm sure it
used to work - and if I use ranks = 1 - it runs - so perhaps mpi detects just one rank and does no initialization, but when I use N>1 ranks, it dies. Any suggestions welcome. Thanks</div>
<div><br>
</div>
<div>JB</div>
<div><br>
</div>
<div><br>
</div>
<div><DEB> 0000056511 0x2aaaaab2dec0 cpu 000 nid00219(0) CONTROL Allocating domain <br>
libfabric:69061:gni:core:_gnix_ref_init():254<debug> [69061:1] 0x8579d8 refs 1<br>
libfabric:69061:core:core:fi_fabric_():1154<info> Opened fabric: gni<br>
libfabric:69061:gni:domain:gnix_domain_open():579<trace> [69061:1] <br>
libfabric:69061:gni:fabric:gnix_domain_open():591<info> [69061:1] failed to find authorization key, creating new authorization key<br>
libfabric:69061:gni:domain:_gnix_auth_key_enable():347<info> [69061:1] pkey=dd920000 ptag=14 key_partition_size=0 key_offset=0 enabled<br>
libfabric:69061:gni:domain:gnix_domain_open():597<info> [69061:1] authorization key=0x857a10 ptag 14 cookie 0xdd920000<br>
libfabric:69061:gni:mr:_gnix_notifier_open():88<warn> [69061:1] kdreg device open failed: Device or resource busy<br>
<ERR> 0000056576 0x2aaaaab2dec0 cpu 000 nid00219(0) ERROR__ fi_domain : Device or resource busy</div>
<br>
<p></p>
</div>
</div>
</body>
</html>