<div dir="ltr">HI John,<div><br></div><div>I'm seeing this same problem at NERSC/edison.  I'll use that system to debug this problem.</div><div><br></div><div>Howard</div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">2017-02-15 13:40 GMT-07:00 Biddiscombe, John A. <span dir="ltr"><<a href="mailto:biddisco@cscs.ch" target="_blank">biddisco@cscs.ch</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Sung<br>
<br>
just fyi : I checked out the v2.2.0 branch of criterion and recompiled it and libfabric  and got broadly the same results, slightly different number of fails, but the same pattern.<br>
<br>
daint103:/scratch/snx3000/<wbr>biddisco/src/libfabric-cray (master *=)$ ~/apps/libfabric/bin/run_<wbr>gnitest<br>
[----] Warning! The test `api_cq::msg_send_only` crashed during its setup or teardown.<br>
Unidentified node: Error detected by libibgni.so.  Subsequent operation may be unreliable.  IAA did not recognize this as an MPI process<br>
Unidentified node: Error detected by libibgni.so.  Subsequent operation may be unreliable.  IAA did not recognize this as an MPI process<br>
[----] prov/gni/test/cm.c:203: Assertion failed: fi_endpoint<br>
[FAIL] cm_basic::srv_setup: (0.44s)<br>
<div><div class="h5">[----] prov/gni/test/rdm_dgram_stx.c:<wbr>165: Assertion failed: fi_endpoint<br>
[----] Warning! The test `dgram_rma_stx::inject` crashed during its setup or teardown.<br>
[----] prov/gni/test/rdm_dgram_stx.c:<wbr>165: Assertion failed: fi_endpoint<br>
[----] Warning! The test `dgram_rma_stx::inject_write` crashed during its setup or teardown.<br>
[----] prov/gni/test/rdm_dgram_stx.c:<wbr>165: Assertion failed: fi_endpoint<br>
[----] Warning! The test `dgram_rma_stx::inject_write_<wbr>retrans` crashed during its setup or teardown.<br>
[----] prov/gni/test/rdm_dgram_stx.c:<wbr>165: Assertion failed: fi_endpoint<br>
[----] Warning! The test `dgram_rma_stx::inject_<wbr>writedata` crashed during its setup or teardown.<br>
[----] prov/gni/test/rdm_dgram_stx.c:<wbr>165: Assertion failed: fi_endpoint<br>
[----] Warning! The test `dgram_rma_stx::inject_<wbr>writedata_retrans` crashed during its setup or teardown.<br>
[----] prov/gni/test/rdm_dgram_stx.c:<wbr>165: Assertion failed: fi_endpoint<br>
[----] Warning! The test `dgram_rma_stx::read` crashed during its setup or teardown.<br>
[----] prov/gni/test/rdm_dgram_stx.c:<wbr>165: Assertion failed: fi_endpoint<br>
[----] Warning! The test `dgram_rma_stx::read_<wbr>alignment` crashed during its setup or teardown.<br>
[----] prov/gni/test/rdm_dgram_stx.c:<wbr>165: Assertion failed: fi_endpoint<br>
[----] Warning! The test `dgram_rma_stx::readmsg` crashed during its setup or teardown.<br>
[----] prov/gni/test/rdm_dgram_stx.c:<wbr>165: Assertion failed: fi_endpoint<br>
[----] Warning! The test `dgram_rma_stx::readv` crashed during its setup or teardown.<br>
[----] prov/gni/test/rdm_dgram_stx.c:<wbr>165: Assertion failed: fi_endpoint<br>
[----] Warning! The test `dgram_rma_stx::write` crashed during its setup or teardown.<br>
[----] prov/gni/test/rdm_dgram_stx.c:<wbr>165: Assertion failed: fi_endpoint<br>
[----] Warning! The test `dgram_rma_stx::write_<wbr>alignment` crashed during its setup or teardown.<br>
[----] prov/gni/test/rdm_dgram_stx.c:<wbr>165: Assertion failed: fi_endpoint<br>
[----] Warning! The test `dgram_rma_stx::write_<wbr>alignment_retrans` crashed during its setup or teardown.<br>
[----] prov/gni/test/rdm_dgram_stx.c:<wbr>165: Assertion failed: fi_endpoint<br>
[----] Warning! The test `dgram_rma_stx::write_autoreg` crashed during its setup or teardown.<br>
[----] prov/gni/test/rdm_dgram_stx.c:<wbr>165: Assertion failed: fi_endpoint<br>
[----] Warning! The test `dgram_rma_stx::write_autoreg_<wbr>uncached` crashed during its setup or teardown.<br>
[----] prov/gni/test/rdm_dgram_stx.c:<wbr>165: Assertion failed: fi_endpoint<br>
[----] Warning! The test `dgram_rma_stx::write_error` crashed during its setup or teardown.<br>
[----] prov/gni/test/rdm_dgram_stx.c:<wbr>165: Assertion failed: fi_endpoint<br>
[----] Warning! The test `dgram_rma_stx::write_fence` crashed during its setup or teardown.<br>
[----] prov/gni/test/rdm_dgram_stx.c:<wbr>165: Assertion failed: fi_endpoint<br>
[----] Warning! The test `dgram_rma_stx::write_fence_<wbr>retrans` crashed during its setup or teardown.<br>
[----] prov/gni/test/rdm_dgram_stx.c:<wbr>165: Assertion failed: fi_endpoint<br>
[----] Warning! The test `dgram_rma_stx::write_retrans` crashed during its setup or teardown.<br>
[----] prov/gni/test/rdm_dgram_stx.c:<wbr>165: Assertion failed: fi_endpoint<br>
[----] Warning! The test `dgram_rma_stx::writedata` crashed during its setup or teardown.<br>
[----] prov/gni/test/rdm_dgram_stx.c:<wbr>165: Assertion failed: fi_endpoint<br>
[----] Warning! The test `dgram_rma_stx::writedata_<wbr>retrans` crashed during its setup or teardown.<br>
[----] prov/gni/test/rdm_dgram_stx.c:<wbr>165: Assertion failed: fi_endpoint<br>
[----] Warning! The test `dgram_rma_stx::writemsg` crashed during its setup or teardown.<br>
[----] prov/gni/test/rdm_dgram_stx.c:<wbr>165: Assertion failed: fi_endpoint<br>
[----] Warning! The test `dgram_rma_stx::writemsg_<wbr>retrans` crashed during its setup or teardown.<br>
[----] prov/gni/test/rdm_dgram_stx.c:<wbr>165: Assertion failed: fi_endpoint<br>
[----] Warning! The test `dgram_rma_stx::writev` crashed during its setup or teardown.<br>
[----] prov/gni/test/rdm_dgram_stx.c:<wbr>165: Assertion failed: fi_endpoint<br>
[----] Warning! The test `dgram_rma_stx::writev_<wbr>retrans` crashed during its setup or teardown.<br>
[----] prov/gni/test/rdm_dgram_stx.c:<wbr>165: Assertion failed: fi_endpoint<br>
</div></div>[----] Warning! The test `rdm_rma_stx::inject` crashed during its setup or teardown.<br>
<span class="">[----] prov/gni/test/rdm_dgram_stx.c:<wbr>165: Assertion failed: fi_endpoint<br>
</span>[----] Warning! The test `rdm_rma_stx::inject_write` crashed during its setup or teardown.<br>
<span class="">[----] prov/gni/test/rdm_dgram_stx.c:<wbr>165: Assertion failed: fi_endpoint<br>
</span>[----] Warning! The test `rdm_rma_stx::inject_write_<wbr>retrans` crashed during its setup or teardown.<br>
<span class="">[----] prov/gni/test/rdm_dgram_stx.c:<wbr>165: Assertion failed: fi_endpoint<br>
</span>[----] Warning! The test `rdm_rma_stx::inject_<wbr>writedata` crashed during its setup or teardown.<br>
<span class="">[----] prov/gni/test/rdm_dgram_stx.c:<wbr>165: Assertion failed: fi_endpoint<br>
</span>[----] Warning! The test `rdm_rma_stx::inject_<wbr>writedata_retrans` crashed during its setup or teardown.<br>
<span class="">[----] prov/gni/test/rdm_dgram_stx.c:<wbr>165: Assertion failed: fi_endpoint<br>
</span>[----] Warning! The test `rdm_rma_stx::read` crashed during its setup or teardown.<br>
<span class="">[----] prov/gni/test/rdm_dgram_stx.c:<wbr>165: Assertion failed: fi_endpoint<br>
</span>[----] Warning! The test `rdm_rma_stx::read_alignment` crashed during its setup or teardown.<br>
<span class="">[----] prov/gni/test/rdm_dgram_stx.c:<wbr>165: Assertion failed: fi_endpoint<br>
</span>[----] Warning! The test `rdm_rma_stx::read_alignment_<wbr>retrans` crashed during its setup or teardown.<br>
<span class="">[----] prov/gni/test/rdm_dgram_stx.c:<wbr>165: Assertion failed: fi_endpoint<br>
</span>[----] Warning! The test `rdm_rma_stx::read_error` crashed during its setup or teardown.<br>
<span class="">[----] prov/gni/test/rdm_dgram_stx.c:<wbr>165: Assertion failed: fi_endpoint<br>
</span>[----] Warning! The test `rdm_rma_stx::read_retrans` crashed during its setup or teardown.<br>
<span class="">[----] prov/gni/test/rdm_dgram_stx.c:<wbr>165: Assertion failed: fi_endpoint<br>
</span>[----] Warning! The test `rdm_rma_stx::readmsg` crashed during its setup or teardown.<br>
<span class="">[----] prov/gni/test/rdm_dgram_stx.c:<wbr>165: Assertion failed: fi_endpoint<br>
</span>[----] Warning! The test `rdm_rma_stx::readmsg_retrans` crashed during its setup or teardown.<br>
<span class="">[----] prov/gni/test/rdm_dgram_stx.c:<wbr>165: Assertion failed: fi_endpoint<br>
</span>[----] Warning! The test `rdm_rma_stx::readv` crashed during its setup or teardown.<br>
<span class="">[----] prov/gni/test/rdm_dgram_stx.c:<wbr>165: Assertion failed: fi_endpoint<br>
</span>[----] Warning! The test `rdm_rma_stx::readv_retrans` crashed during its setup or teardown.<br>
<span class="">[----] prov/gni/test/rdm_dgram_stx.c:<wbr>165: Assertion failed: fi_endpoint<br>
</span>[----] Warning! The test `rdm_rma_stx::trigger` crashed during its setup or teardown.<br>
<span class="">[----] prov/gni/test/rdm_dgram_stx.c:<wbr>165: Assertion failed: fi_endpoint<br>
</span>[----] Warning! The test `rdm_rma_stx::write` crashed during its setup or teardown.<br>
<span class="">[----] prov/gni/test/rdm_dgram_stx.c:<wbr>165: Assertion failed: fi_endpoint<br>
</span>[----] Warning! The test `rdm_rma_stx::write_alignment` crashed during its setup or teardown.<br>
<span class="">[----] prov/gni/test/rdm_dgram_stx.c:<wbr>165: Assertion failed: fi_endpoint<br>
</span>[----] Warning! The test `rdm_rma_stx::write_alignment_<wbr>retrans` crashed during its setup or teardown.<br>
<span class="">[----] prov/gni/test/rdm_dgram_stx.c:<wbr>165: Assertion failed: fi_endpoint<br>
</span>[----] Warning! The test `rdm_rma_stx::write_autoreg` crashed during its setup or teardown.<br>
<span class="">[----] prov/gni/test/rdm_dgram_stx.c:<wbr>165: Assertion failed: fi_endpoint<br>
</span>[----] Warning! The test `rdm_rma_stx::write_autoreg_<wbr>uncached` crashed during its setup or teardown.<br>
<span class="">[----] prov/gni/test/rdm_dgram_stx.c:<wbr>165: Assertion failed: fi_endpoint<br>
</span>[----] Warning! The test `rdm_rma_stx::write_error` crashed during its setup or teardown.<br>
<span class="">[----] prov/gni/test/rdm_dgram_stx.c:<wbr>165: Assertion failed: fi_endpoint<br>
</span>[----] Warning! The test `rdm_rma_stx::write_fence` crashed during its setup or teardown.<br>
<span class="">[----] prov/gni/test/rdm_dgram_stx.c:<wbr>165: Assertion failed: fi_endpoint<br>
</span>[----] Warning! The test `rdm_rma_stx::write_fence_<wbr>retrans` crashed during its setup or teardown.<br>
<span class="">[----] prov/gni/test/rdm_dgram_stx.c:<wbr>165: Assertion failed: fi_endpoint<br>
</span>[----] Warning! The test `rdm_rma_stx::write_retrans` crashed during its setup or teardown.<br>
<span class="">[----] prov/gni/test/rdm_dgram_stx.c:<wbr>165: Assertion failed: fi_endpoint<br>
</span>[----] Warning! The test `rdm_rma_stx::writedata` crashed during its setup or teardown.<br>
<span class="">[----] prov/gni/test/rdm_dgram_stx.c:<wbr>165: Assertion failed: fi_endpoint<br>
</span>[----] Warning! The test `rdm_rma_stx::writedata_<wbr>retrans` crashed during its setup or teardown.<br>
<span class="">[----] prov/gni/test/rdm_dgram_stx.c:<wbr>165: Assertion failed: fi_endpoint<br>
</span>[----] Warning! The test `rdm_rma_stx::writemsg` crashed during its setup or teardown.<br>
<span class="">[----] prov/gni/test/rdm_dgram_stx.c:<wbr>165: Assertion failed: fi_endpoint<br>
</span>[----] Warning! The test `rdm_rma_stx::writemsg_<wbr>retrans` crashed during its setup or teardown.<br>
<span class="">[----] prov/gni/test/rdm_dgram_stx.c:<wbr>165: Assertion failed: fi_endpoint<br>
</span>[----] Warning! The test `rdm_rma_stx::writev` crashed during its setup or teardown.<br>
<span class="">[----] prov/gni/test/rdm_dgram_stx.c:<wbr>165: Assertion failed: fi_endpoint<br>
</span>[----] Warning! The test `rdm_rma_stx::writev_retrans` crashed during its setup or teardown.<br>
[----] prov/gni/test/sep.c:2343: Assertion failed: fi_scalable_ep<br>
[FAIL] scalable::av_insert: (0.46s)<br>
[----] prov/gni/test/sep.c:177: Assertion failed: fi_scalable_ep<br>
[----] Warning! The test `scalablem::all` crashed during its setup or teardown.<br>
[----] prov/gni/test/sep.c:177: Assertion failed: fi_scalable_ep<br>
[----] Warning! The test `scalablem::misc` crashed during its setup or teardown.<br>
[----] prov/gni/test/sep.c:177: Assertion failed: fi_scalable_ep<br>
[----] Warning! The test `scalablet::all` crashed during its setup or teardown.<br>
Unidentified node: Error detected by libibgni.so.  Subsequent operation may be unreliable.  IAA did not recognize this as an MPI process<br>
[----] prov/gni/test/vc.c:271: Assertion failed: fi_endpoint<br>
[----] Warning! The test `vc_conn_ping_auto::ep_<wbr>connect_inter_cm` crashed during its setup or teardown.<br>
Unidentified node: Error detected by libibgni.so.  Subsequent operation may be unreliable.  IAA did not recognize this as an MPI process<br>
[----] prov/gni/test/vc.c:271: Assertion failed: fi_endpoint<br>
[----] Warning! The test `vc_conn_ping_auto::ep_<wbr>connect_inter_cm_pp` crashed during its setup or teardown.<br>
Unidentified node: Error detected by libibgni.so.  Subsequent operation may be unreliable.  IAA did not recognize this as an MPI process<br>
[----] prov/gni/test/vc.c:271: Assertion failed: fi_endpoint<br>
[----] Warning! The test `vc_conn_ping_auto::ep_<wbr>connect_intra_cm` crashed during its setup or teardown.<br>
Unidentified node: Error detected by libibgni.so.  Subsequent operation may be unreliable.  IAA did not recognize this as an MPI process<br>
[----] prov/gni/test/vc.c:271: Assertion failed: fi_endpoint<br>
[----] Warning! The test `vc_conn_ping_auto::ep_<wbr>connect_intra_cm_pp` crashed during its setup or teardown.<br>
Unidentified node: Error detected by libibgni.so.  Subsequent operation may be unreliable.  IAA did not recognize this as an MPI process<br>
[----] prov/gni/test/vc.c:271: Assertion failed: fi_endpoint<br>
[----] Warning! The test `vc_conn_ping_auto::ep_<wbr>connect_self` crashed during its setup or teardown.<br>
Unidentified node: Error detected by libibgni.so.  Subsequent operation may be unreliable.  IAA did not recognize this as an MPI process<br>
[----] prov/gni/test/vc.c:271: Assertion failed: fi_endpoint<br>
[----] Warning! The test `vc_conn_ping_auto::ep_<wbr>connect_self_pp` crashed during its setup or teardown.<br>
Unidentified node: Error detected by libibgni.so.  Subsequent operation may be unreliable.  IAA did not recognize this as an MPI process<br>
[----] prov/gni/test/vc.c:271: Assertion failed: fi_endpoint<br>
[----] Warning! The test `vc_conn_ping_manual::ep_<wbr>connect_inter_cm_pp` crashed during its setup or teardown.<br>
Unidentified node: Error detected by libibgni.so.  Subsequent operation may be unreliable.  IAA did not recognize this as an MPI process<br>
[----] prov/gni/test/vc.c:271: Assertion failed: fi_endpoint<br>
[----] Warning! The test `vc_conn_ping_manual::ep_<wbr>connect_intra_cm` crashed during its setup or teardown.<br>
Unidentified node: Error detected by libibgni.so.  Subsequent operation may be unreliable.  IAA did not recognize this as an MPI process<br>
[----] prov/gni/test/vc.c:271: Assertion failed: fi_endpoint<br>
[----] Warning! The test `vc_conn_ping_manual::ep_<wbr>connect_intra_cm_pp` crashed during its setup or teardown.<br>
Unidentified node: Error detected by libibgni.so.  Subsequent operation may be unreliable.  IAA did not recognize this as an MPI process<br>
[----] prov/gni/test/vc.c:271: Assertion failed: fi_endpoint<br>
[----] Warning! The test `vc_conn_ping_manual::ep_<wbr>connect_self` crashed during its setup or teardown.<br>
Unidentified node: Error detected by libibgni.so.  Subsequent operation may be unreliable.  IAA did not recognize this as an MPI process<br>
[----] prov/gni/test/vc.c:271: Assertion failed: fi_endpoint<br>
[----] Warning! The test `vc_conn_ping_manual::ep_<wbr>connect_self_pp` crashed during its setup or teardown.<br>
[====] Synthesis: Tested: 631 | Passing: 561 | Failing: 70 | Crashing: 68<br>
<br>
</blockquote></div><br></div>