[libfabric-users] FI_EP_MSG on cray

[Howard Pritchard]

HI John,

I'm seeing this same problem at NERSC/edison.  I'll use that system to
debug this problem.

Howard


2017-02-15 13:40 GMT-07:00 Biddiscombe, John A. <biddisco at cscs.ch>:

> Sung
>
> just fyi : I checked out the v2.2.0 branch of criterion and recompiled it
> and libfabric  and got broadly the same results, slightly different number
> of fails, but the same pattern.
>
> daint103:/scratch/snx3000/biddisco/src/libfabric-cray (master *=)$
> ~/apps/libfabric/bin/run_gnitest
> [----] Warning! The test `api_cq::msg_send_only` crashed during its setup
> or teardown.
> Unidentified node: Error detected by libibgni.so.  Subsequent operation
> may be unreliable.  IAA did not recognize this as an MPI process
> Unidentified node: Error detected by libibgni.so.  Subsequent operation
> may be unreliable.  IAA did not recognize this as an MPI process
> [----] prov/gni/test/cm.c:203: Assertion failed: fi_endpoint
> [FAIL] cm_basic::srv_setup: (0.44s)
> [----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
> [----] Warning! The test `dgram_rma_stx::inject` crashed during its setup
> or teardown.
> [----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
> [----] Warning! The test `dgram_rma_stx::inject_write` crashed during its
> setup or teardown.
> [----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
> [----] Warning! The test `dgram_rma_stx::inject_write_retrans` crashed
> during its setup or teardown.
> [----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
> [----] Warning! The test `dgram_rma_stx::inject_writedata` crashed during
> its setup or teardown.
> [----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
> [----] Warning! The test `dgram_rma_stx::inject_writedata_retrans`
> crashed during its setup or teardown.
> [----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
> [----] Warning! The test `dgram_rma_stx::read` crashed during its setup or
> teardown.
> [----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
> [----] Warning! The test `dgram_rma_stx::read_alignment` crashed during
> its setup or teardown.
> [----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
> [----] Warning! The test `dgram_rma_stx::readmsg` crashed during its setup
> or teardown.
> [----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
> [----] Warning! The test `dgram_rma_stx::readv` crashed during its setup
> or teardown.
> [----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
> [----] Warning! The test `dgram_rma_stx::write` crashed during its setup
> or teardown.
> [----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
> [----] Warning! The test `dgram_rma_stx::write_alignment` crashed during
> its setup or teardown.
> [----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
> [----] Warning! The test `dgram_rma_stx::write_alignment_retrans` crashed
> during its setup or teardown.
> [----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
> [----] Warning! The test `dgram_rma_stx::write_autoreg` crashed during its
> setup or teardown.
> [----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
> [----] Warning! The test `dgram_rma_stx::write_autoreg_uncached` crashed
> during its setup or teardown.
> [----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
> [----] Warning! The test `dgram_rma_stx::write_error` crashed during its
> setup or teardown.
> [----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
> [----] Warning! The test `dgram_rma_stx::write_fence` crashed during its
> setup or teardown.
> [----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
> [----] Warning! The test `dgram_rma_stx::write_fence_retrans` crashed
> during its setup or teardown.
> [----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
> [----] Warning! The test `dgram_rma_stx::write_retrans` crashed during its
> setup or teardown.
> [----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
> [----] Warning! The test `dgram_rma_stx::writedata` crashed during its
> setup or teardown.
> [----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
> [----] Warning! The test `dgram_rma_stx::writedata_retrans` crashed
> during its setup or teardown.
> [----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
> [----] Warning! The test `dgram_rma_stx::writemsg` crashed during its
> setup or teardown.
> [----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
> [----] Warning! The test `dgram_rma_stx::writemsg_retrans` crashed during
> its setup or teardown.
> [----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
> [----] Warning! The test `dgram_rma_stx::writev` crashed during its setup
> or teardown.
> [----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
> [----] Warning! The test `dgram_rma_stx::writev_retrans` crashed during
> its setup or teardown.
> [----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
> [----] Warning! The test `rdm_rma_stx::inject` crashed during its setup or
> teardown.
> [----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
> [----] Warning! The test `rdm_rma_stx::inject_write` crashed during its
> setup or teardown.
> [----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
> [----] Warning! The test `rdm_rma_stx::inject_write_retrans` crashed
> during its setup or teardown.
> [----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
> [----] Warning! The test `rdm_rma_stx::inject_writedata` crashed during
> its setup or teardown.
> [----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
> [----] Warning! The test `rdm_rma_stx::inject_writedata_retrans` crashed
> during its setup or teardown.
> [----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
> [----] Warning! The test `rdm_rma_stx::read` crashed during its setup or
> teardown.
> [----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
> [----] Warning! The test `rdm_rma_stx::read_alignment` crashed during its
> setup or teardown.
> [----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
> [----] Warning! The test `rdm_rma_stx::read_alignment_retrans` crashed
> during its setup or teardown.
> [----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
> [----] Warning! The test `rdm_rma_stx::read_error` crashed during its
> setup or teardown.
> [----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
> [----] Warning! The test `rdm_rma_stx::read_retrans` crashed during its
> setup or teardown.
> [----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
> [----] Warning! The test `rdm_rma_stx::readmsg` crashed during its setup
> or teardown.
> [----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
> [----] Warning! The test `rdm_rma_stx::readmsg_retrans` crashed during its
> setup or teardown.
> [----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
> [----] Warning! The test `rdm_rma_stx::readv` crashed during its setup or
> teardown.
> [----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
> [----] Warning! The test `rdm_rma_stx::readv_retrans` crashed during its
> setup or teardown.
> [----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
> [----] Warning! The test `rdm_rma_stx::trigger` crashed during its setup
> or teardown.
> [----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
> [----] Warning! The test `rdm_rma_stx::write` crashed during its setup or
> teardown.
> [----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
> [----] Warning! The test `rdm_rma_stx::write_alignment` crashed during its
> setup or teardown.
> [----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
> [----] Warning! The test `rdm_rma_stx::write_alignment_retrans` crashed
> during its setup or teardown.
> [----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
> [----] Warning! The test `rdm_rma_stx::write_autoreg` crashed during its
> setup or teardown.
> [----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
> [----] Warning! The test `rdm_rma_stx::write_autoreg_uncached` crashed
> during its setup or teardown.
> [----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
> [----] Warning! The test `rdm_rma_stx::write_error` crashed during its
> setup or teardown.
> [----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
> [----] Warning! The test `rdm_rma_stx::write_fence` crashed during its
> setup or teardown.
> [----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
> [----] Warning! The test `rdm_rma_stx::write_fence_retrans` crashed
> during its setup or teardown.
> [----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
> [----] Warning! The test `rdm_rma_stx::write_retrans` crashed during its
> setup or teardown.
> [----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
> [----] Warning! The test `rdm_rma_stx::writedata` crashed during its setup
> or teardown.
> [----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
> [----] Warning! The test `rdm_rma_stx::writedata_retrans` crashed during
> its setup or teardown.
> [----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
> [----] Warning! The test `rdm_rma_stx::writemsg` crashed during its setup
> or teardown.
> [----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
> [----] Warning! The test `rdm_rma_stx::writemsg_retrans` crashed during
> its setup or teardown.
> [----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
> [----] Warning! The test `rdm_rma_stx::writev` crashed during its setup or
> teardown.
> [----] prov/gni/test/rdm_dgram_stx.c:165: Assertion failed: fi_endpoint
> [----] Warning! The test `rdm_rma_stx::writev_retrans` crashed during its
> setup or teardown.
> [----] prov/gni/test/sep.c:2343: Assertion failed: fi_scalable_ep
> [FAIL] scalable::av_insert: (0.46s)
> [----] prov/gni/test/sep.c:177: Assertion failed: fi_scalable_ep
> [----] Warning! The test `scalablem::all` crashed during its setup or
> teardown.
> [----] prov/gni/test/sep.c:177: Assertion failed: fi_scalable_ep
> [----] Warning! The test `scalablem::misc` crashed during its setup or
> teardown.
> [----] prov/gni/test/sep.c:177: Assertion failed: fi_scalable_ep
> [----] Warning! The test `scalablet::all` crashed during its setup or
> teardown.
> Unidentified node: Error detected by libibgni.so.  Subsequent operation
> may be unreliable.  IAA did not recognize this as an MPI process
> [----] prov/gni/test/vc.c:271: Assertion failed: fi_endpoint
> [----] Warning! The test `vc_conn_ping_auto::ep_connect_inter_cm` crashed
> during its setup or teardown.
> Unidentified node: Error detected by libibgni.so.  Subsequent operation
> may be unreliable.  IAA did not recognize this as an MPI process
> [----] prov/gni/test/vc.c:271: Assertion failed: fi_endpoint
> [----] Warning! The test `vc_conn_ping_auto::ep_connect_inter_cm_pp`
> crashed during its setup or teardown.
> Unidentified node: Error detected by libibgni.so.  Subsequent operation
> may be unreliable.  IAA did not recognize this as an MPI process
> [----] prov/gni/test/vc.c:271: Assertion failed: fi_endpoint
> [----] Warning! The test `vc_conn_ping_auto::ep_connect_intra_cm` crashed
> during its setup or teardown.
> Unidentified node: Error detected by libibgni.so.  Subsequent operation
> may be unreliable.  IAA did not recognize this as an MPI process
> [----] prov/gni/test/vc.c:271: Assertion failed: fi_endpoint
> [----] Warning! The test `vc_conn_ping_auto::ep_connect_intra_cm_pp`
> crashed during its setup or teardown.
> Unidentified node: Error detected by libibgni.so.  Subsequent operation
> may be unreliable.  IAA did not recognize this as an MPI process
> [----] prov/gni/test/vc.c:271: Assertion failed: fi_endpoint
> [----] Warning! The test `vc_conn_ping_auto::ep_connect_self` crashed
> during its setup or teardown.
> Unidentified node: Error detected by libibgni.so.  Subsequent operation
> may be unreliable.  IAA did not recognize this as an MPI process
> [----] prov/gni/test/vc.c:271: Assertion failed: fi_endpoint
> [----] Warning! The test `vc_conn_ping_auto::ep_connect_self_pp` crashed
> during its setup or teardown.
> Unidentified node: Error detected by libibgni.so.  Subsequent operation
> may be unreliable.  IAA did not recognize this as an MPI process
> [----] prov/gni/test/vc.c:271: Assertion failed: fi_endpoint
> [----] Warning! The test `vc_conn_ping_manual::ep_connect_inter_cm_pp`
> crashed during its setup or teardown.
> Unidentified node: Error detected by libibgni.so.  Subsequent operation
> may be unreliable.  IAA did not recognize this as an MPI process
> [----] prov/gni/test/vc.c:271: Assertion failed: fi_endpoint
> [----] Warning! The test `vc_conn_ping_manual::ep_connect_intra_cm`
> crashed during its setup or teardown.
> Unidentified node: Error detected by libibgni.so.  Subsequent operation
> may be unreliable.  IAA did not recognize this as an MPI process
> [----] prov/gni/test/vc.c:271: Assertion failed: fi_endpoint
> [----] Warning! The test `vc_conn_ping_manual::ep_connect_intra_cm_pp`
> crashed during its setup or teardown.
> Unidentified node: Error detected by libibgni.so.  Subsequent operation
> may be unreliable.  IAA did not recognize this as an MPI process
> [----] prov/gni/test/vc.c:271: Assertion failed: fi_endpoint
> [----] Warning! The test `vc_conn_ping_manual::ep_connect_self` crashed
> during its setup or teardown.
> Unidentified node: Error detected by libibgni.so.  Subsequent operation
> may be unreliable.  IAA did not recognize this as an MPI process
> [----] prov/gni/test/vc.c:271: Assertion failed: fi_endpoint
> [----] Warning! The test `vc_conn_ping_manual::ep_connect_self_pp`
> crashed during its setup or teardown.
> [====] Synthesis: Tested: 631 | Passing: 561 | Failing: 70 | Crashing: 68
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/libfabric-users/attachments/20170216/40d030af/attachment.html>