[libfabric-users] FI_EP_MSG on cray

Biddiscombe, John A. biddisco at cscs.ch
Wed Feb 15 12:40:45 PST 2017


Sung

just fyi : I checked out the v2.2.0 branch of criterion and recompiled it and libfabric  and got broadly the same results, slightly different number of fails, but the same pattern.

daint103:/scratch/snx3000/biddisco/src/libfabric-cray (master *=)$ ~/apps/libfabric/bin/run_gnitest
[----] Warning! The test `api_cq::msg_send_only` crashed during its setup or teardown.
Unidentified node: Error detected by libibgni.so.  Subsequent operation may be unreliable.  IAA did not recognize this as an MPI process
Unidentified node: Error detected by libibgni.so.  Subsequent operation may be unreliable.  IAA did not recognize this as an MPI process
[----] prov/gni/test/cm.c:203: Assertion failed: fi_endpoint
[FAIL] cm_basic::srv_setup: (0.44s)
[----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
[----] Warning! The test `dgram_rma_stx::inject` crashed during its setup or teardown.
[----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
[----] Warning! The test `dgram_rma_stx::inject_write` crashed during its setup or teardown.
[----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
[----] Warning! The test `dgram_rma_stx::inject_write_retrans` crashed during its setup or teardown.
[----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
[----] Warning! The test `dgram_rma_stx::inject_writedata` crashed during its setup or teardown.
[----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
[----] Warning! The test `dgram_rma_stx::inject_writedata_retrans` crashed during its setup or teardown.
[----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
[----] Warning! The test `dgram_rma_stx::read` crashed during its setup or teardown.
[----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
[----] Warning! The test `dgram_rma_stx::read_alignment` crashed during its setup or teardown.
[----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
[----] Warning! The test `dgram_rma_stx::readmsg` crashed during its setup or teardown.
[----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
[----] Warning! The test `dgram_rma_stx::readv` crashed during its setup or teardown.
[----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
[----] Warning! The test `dgram_rma_stx::write` crashed during its setup or teardown.
[----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
[----] Warning! The test `dgram_rma_stx::write_alignment` crashed during its setup or teardown.
[----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
[----] Warning! The test `dgram_rma_stx::write_alignment_retrans` crashed during its setup or teardown.
[----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
[----] Warning! The test `dgram_rma_stx::write_autoreg` crashed during its setup or teardown.
[----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
[----] Warning! The test `dgram_rma_stx::write_autoreg_uncached` crashed during its setup or teardown.
[----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
[----] Warning! The test `dgram_rma_stx::write_error` crashed during its setup or teardown.
[----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
[----] Warning! The test `dgram_rma_stx::write_fence` crashed during its setup or teardown.
[----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
[----] Warning! The test `dgram_rma_stx::write_fence_retrans` crashed during its setup or teardown.
[----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
[----] Warning! The test `dgram_rma_stx::write_retrans` crashed during its setup or teardown.
[----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
[----] Warning! The test `dgram_rma_stx::writedata` crashed during its setup or teardown.
[----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
[----] Warning! The test `dgram_rma_stx::writedata_retrans` crashed during its setup or teardown.
[----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
[----] Warning! The test `dgram_rma_stx::writemsg` crashed during its setup or teardown.
[----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
[----] Warning! The test `dgram_rma_stx::writemsg_retrans` crashed during its setup or teardown.
[----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
[----] Warning! The test `dgram_rma_stx::writev` crashed during its setup or teardown.
[----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
[----] Warning! The test `dgram_rma_stx::writev_retrans` crashed during its setup or teardown.
[----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
[----] Warning! The test `rdm_rma_stx::inject` crashed during its setup or teardown.
[----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
[----] Warning! The test `rdm_rma_stx::inject_write` crashed during its setup or teardown.
[----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
[----] Warning! The test `rdm_rma_stx::inject_write_retrans` crashed during its setup or teardown.
[----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
[----] Warning! The test `rdm_rma_stx::inject_writedata` crashed during its setup or teardown.
[----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
[----] Warning! The test `rdm_rma_stx::inject_writedata_retrans` crashed during its setup or teardown.
[----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
[----] Warning! The test `rdm_rma_stx::read` crashed during its setup or teardown.
[----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
[----] Warning! The test `rdm_rma_stx::read_alignment` crashed during its setup or teardown.
[----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
[----] Warning! The test `rdm_rma_stx::read_alignment_retrans` crashed during its setup or teardown.
[----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
[----] Warning! The test `rdm_rma_stx::read_error` crashed during its setup or teardown.
[----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
[----] Warning! The test `rdm_rma_stx::read_retrans` crashed during its setup or teardown.
[----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
[----] Warning! The test `rdm_rma_stx::readmsg` crashed during its setup or teardown.
[----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
[----] Warning! The test `rdm_rma_stx::readmsg_retrans` crashed during its setup or teardown.
[----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
[----] Warning! The test `rdm_rma_stx::readv` crashed during its setup or teardown.
[----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
[----] Warning! The test `rdm_rma_stx::readv_retrans` crashed during its setup or teardown.
[----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
[----] Warning! The test `rdm_rma_stx::trigger` crashed during its setup or teardown.
[----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
[----] Warning! The test `rdm_rma_stx::write` crashed during its setup or teardown.
[----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
[----] Warning! The test `rdm_rma_stx::write_alignment` crashed during its setup or teardown.
[----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
[----] Warning! The test `rdm_rma_stx::write_alignment_retrans` crashed during its setup or teardown.
[----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
[----] Warning! The test `rdm_rma_stx::write_autoreg` crashed during its setup or teardown.
[----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
[----] Warning! The test `rdm_rma_stx::write_autoreg_uncached` crashed during its setup or teardown.
[----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
[----] Warning! The test `rdm_rma_stx::write_error` crashed during its setup or teardown.
[----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
[----] Warning! The test `rdm_rma_stx::write_fence` crashed during its setup or teardown.
[----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
[----] Warning! The test `rdm_rma_stx::write_fence_retrans` crashed during its setup or teardown.
[----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
[----] Warning! The test `rdm_rma_stx::write_retrans` crashed during its setup or teardown.
[----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
[----] Warning! The test `rdm_rma_stx::writedata` crashed during its setup or teardown.
[----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
[----] Warning! The test `rdm_rma_stx::writedata_retrans` crashed during its setup or teardown.
[----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
[----] Warning! The test `rdm_rma_stx::writemsg` crashed during its setup or teardown.
[----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
[----] Warning! The test `rdm_rma_stx::writemsg_retrans` crashed during its setup or teardown.
[----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
[----] Warning! The test `rdm_rma_stx::writev` crashed during its setup or teardown.
[----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
[----] Warning! The test `rdm_rma_stx::writev_retrans` crashed during its setup or teardown.
[----] prov/gni/test/sep.c:2343: Assertion failed: fi_scalable_ep
[FAIL] scalable::av_insert: (0.46s)
[----] prov/gni/test/sep.c:177: Assertion failed: fi_scalable_ep
[----] Warning! The test `scalablem::all` crashed during its setup or teardown.
[----] prov/gni/test/sep.c:177: Assertion failed: fi_scalable_ep
[----] Warning! The test `scalablem::misc` crashed during its setup or teardown.
[----] prov/gni/test/sep.c:177: Assertion failed: fi_scalable_ep
[----] Warning! The test `scalablet::all` crashed during its setup or teardown.
Unidentified node: Error detected by libibgni.so.  Subsequent operation may be unreliable.  IAA did not recognize this as an MPI process
[----] prov/gni/test/vc.c:271: Assertion failed: fi_endpoint
[----] Warning! The test `vc_conn_ping_auto::ep_connect_inter_cm` crashed during its setup or teardown.
Unidentified node: Error detected by libibgni.so.  Subsequent operation may be unreliable.  IAA did not recognize this as an MPI process
[----] prov/gni/test/vc.c:271: Assertion failed: fi_endpoint
[----] Warning! The test `vc_conn_ping_auto::ep_connect_inter_cm_pp` crashed during its setup or teardown.
Unidentified node: Error detected by libibgni.so.  Subsequent operation may be unreliable.  IAA did not recognize this as an MPI process
[----] prov/gni/test/vc.c:271: Assertion failed: fi_endpoint
[----] Warning! The test `vc_conn_ping_auto::ep_connect_intra_cm` crashed during its setup or teardown.
Unidentified node: Error detected by libibgni.so.  Subsequent operation may be unreliable.  IAA did not recognize this as an MPI process
[----] prov/gni/test/vc.c:271: Assertion failed: fi_endpoint
[----] Warning! The test `vc_conn_ping_auto::ep_connect_intra_cm_pp` crashed during its setup or teardown.
Unidentified node: Error detected by libibgni.so.  Subsequent operation may be unreliable.  IAA did not recognize this as an MPI process
[----] prov/gni/test/vc.c:271: Assertion failed: fi_endpoint
[----] Warning! The test `vc_conn_ping_auto::ep_connect_self` crashed during its setup or teardown.
Unidentified node: Error detected by libibgni.so.  Subsequent operation may be unreliable.  IAA did not recognize this as an MPI process
[----] prov/gni/test/vc.c:271: Assertion failed: fi_endpoint
[----] Warning! The test `vc_conn_ping_auto::ep_connect_self_pp` crashed during its setup or teardown.
Unidentified node: Error detected by libibgni.so.  Subsequent operation may be unreliable.  IAA did not recognize this as an MPI process
[----] prov/gni/test/vc.c:271: Assertion failed: fi_endpoint
[----] Warning! The test `vc_conn_ping_manual::ep_connect_inter_cm_pp` crashed during its setup or teardown.
Unidentified node: Error detected by libibgni.so.  Subsequent operation may be unreliable.  IAA did not recognize this as an MPI process
[----] prov/gni/test/vc.c:271: Assertion failed: fi_endpoint
[----] Warning! The test `vc_conn_ping_manual::ep_connect_intra_cm` crashed during its setup or teardown.
Unidentified node: Error detected by libibgni.so.  Subsequent operation may be unreliable.  IAA did not recognize this as an MPI process
[----] prov/gni/test/vc.c:271: Assertion failed: fi_endpoint
[----] Warning! The test `vc_conn_ping_manual::ep_connect_intra_cm_pp` crashed during its setup or teardown.
Unidentified node: Error detected by libibgni.so.  Subsequent operation may be unreliable.  IAA did not recognize this as an MPI process
[----] prov/gni/test/vc.c:271: Assertion failed: fi_endpoint
[----] Warning! The test `vc_conn_ping_manual::ep_connect_self` crashed during its setup or teardown.
Unidentified node: Error detected by libibgni.so.  Subsequent operation may be unreliable.  IAA did not recognize this as an MPI process
[----] prov/gni/test/vc.c:271: Assertion failed: fi_endpoint
[----] Warning! The test `vc_conn_ping_manual::ep_connect_self_pp` crashed during its setup or teardown.
[====] Synthesis: Tested: 631 | Passing: 561 | Failing: 70 | Crashing: 68



More information about the Libfabric-users mailing list