<div dir="ltr">Hi John,<div><br></div><div>Could you try the <span style="font-size:12.8px">run_</span><span style="font-size:12.8px">gnitest script with this UGNI debug level set? I'd like to understand why that's failing for you.</span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">I cannot get fi_pingpong to work with FI_EP_MSG for GNI provider. It should work though. I filed an issue on the GNI downstream provider repo.</span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">Howard</span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px"><br></span></div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">2017-02-13 13:21 GMT-07:00 Biddiscombe, John A. <span dir="ltr"><<a href="mailto:biddisco@cscs.ch" target="_blank">biddisco@cscs.ch</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="white" lang="EN-GB" link="#0563C1" vlink="#954F72">
<div class="m_1393725984742040119WordSection1">
<p class="MsoNormal">Howard, here’s some output …<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">The machine is the cray piz daint at CSCS,<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Allocation as follows<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">salloc -N 2 -C mc --time=02:00:00 –exclusive<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">daint102:/scratch/snx3000/<wbr>biddisco/build$ export UGNI_DEBUG=10<u></u><u></u></p>
<p class="MsoNormal">daint102:/scratch/snx3000/<wbr>biddisco/build$ ./frun.sh ~/apps/fabtests/bin/fi_msg<u></u><u></u></p>
<p class="MsoNormal">running /users/biddisco/apps/fabtests/<wbr>bin/fi_msg on nid00[722,724]<u></u><u></u></p>
<p class="MsoNormal">nid00722 is 148.187.34.215<u></u><u></u></p>
<p class="MsoNormal">Generated command is srun -n 2 --ntasks-per-node=1 -l --multi-prog ./scalable.conf<u></u><u></u></p>
<p class="MsoNormal">0 /users/biddisco/apps/fabtests/<wbr>bin/fi_msg -p gni<u></u><u></u></p>
<p class="MsoNormal">1 /users/biddisco/apps/fabtests/<wbr>bin/fi_msg -p gni 148.187.34.215<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">0: [ 44] GNII_DebugInit: GNII_debug_level: 10 GNII_subsys_debug: 0 GNII_debug_mask: 0x0 GNII_debug_inst_id: 44<u></u><u></u></p>
<p class="MsoNormal">0: [ 44] JOB: GNII_GetKernelVersion: kgni version major = 0x0 minor 0x45 code 0xb9 built with major = 0x0 minor = 0x45 code 0x4e24<u></u><u></u></p>
<p class="MsoNormal">0: [ 44] JOB: GNI_GetJobResInfo: job resource: FMA (6) used: 0 limit: 123<u></u><u></u></p>
<p class="MsoNormal">0: [ 44] JOB: GNII_GetKernelVersion: kgni version major = 0x0 minor 0x45 code 0xb9 built with major = 0x0 minor = 0x45 code 0x4e24<u></u><u></u></p>
<p class="MsoNormal">0: [ 44] JOB: GNI_GetJobResInfo: job resource: CQ (5) used: 0 limit: 509<u></u><u></u></p>
<p class="MsoNormal">0: fi_getinfo(): common/shared.c:454, ret=-61 (No data available)<u></u><u></u></p>
<p class="MsoNormal">1: [ 36] GNII_DebugInit: GNII_debug_level: 10 GNII_subsys_debug: 0 GNII_debug_mask: 0x0 GNII_debug_inst_id: 36<u></u><u></u></p>
<p class="MsoNormal">1: [ 36] JOB: GNII_GetKernelVersion: kgni version major = 0x0 minor 0x45 code 0xb9 built with major = 0x0 minor = 0x45 code 0x4e24<u></u><u></u></p>
<p class="MsoNormal">1: [ 36] JOB: GNI_GetJobResInfo: job resource: FMA (6) used: 0 limit: 123<u></u><u></u></p>
<p class="MsoNormal">1: [ 36] JOB: GNII_GetKernelVersion: kgni version major = 0x0 minor 0x45 code 0xb9 built with major = 0x0 minor = 0x45 code 0x4e24<u></u><u></u></p>
<p class="MsoNormal">1: [ 36] JOB: GNI_GetJobResInfo: job resource: CQ (5) used: 0 limit: 509<u></u><u></u></p>
<p class="MsoNormal">1: [ 36] JOB: GNII_GetKernelVersion: kgni version major = 0x0 minor 0x45 code 0xb9 built with major = 0x0 minor = 0x45 code 0x4e24<u></u><u></u></p>
<p class="MsoNormal">1: [ 36] FMA: GNI_CdmAttach: FMA window size: 32768<u></u><u></u></p>
<p class="MsoNormal">1: [ 36] FMA: GNI_CdmAttach: NOPRIV_ERR masked<u></u><u></u></p>
<p class="MsoNormal">1: [ 36] JOB: GNI_CdmAttach: ptag = 36 inst_id = 13864961 fma_window = 0x0000000000000000 fma_ctrl = 0x0000000000000000<u></u><u></u></p>
<p class="MsoNormal">1: [ 36] CQ: GNI_CqCreate: entry_count: 1361 reqs: 1361 adjusted entries: 1395 alloc_count: 1396<u></u><u></u></p>
<p class="MsoNormal">1: [ 36] CQ: cq_create: ioctl(GNI_IOC_CQ_CREATE)<u></u><u></u></p>
<p class="MsoNormal">1: [ 36] CQ: cq_create: #1 cq created, kern_cq_descr = 1, mode = 2, rd_index_ptr = 0x2aaaaaad7ba0, queue = 0x2aaaaaad5000, intr_mask = (nil)<u></u><u></u></p>
<p class="MsoNormal">1: [ 36] CQ: GNI_CqCreate: entry_count: 1361 reqs: 1361 adjusted entries: 1395 alloc_count: 1396<u></u><u></u></p>
<p class="MsoNormal">1: [ 36] CQ: cq_create: ioctl(GNI_IOC_CQ_CREATE)<u></u><u></u></p>
<p class="MsoNormal">1: [ 36] CQ: cq_create: #1 cq created, kern_cq_descr = 394, mode = 20, rd_index_ptr = 0x2aaaaaadc000, queue = 0x2aaaaaad8000, intr_mask = (nil)<u></u><u></u></p>
<p class="MsoNormal">1: [ 36] FLBTE: GNII_FlbteInit: FLBTE: tx_counter 0x2aaaaaace008, chan 2, max_len -1, total 511<u></u><u></u></p>
<p class="MsoNormal">1: [ 36] CQ: GNI_CqCreate: entry_count: 2048 reqs: 2048 adjusted entries: 2559 alloc_count: 2560<u></u><u></u></p>
<p class="MsoNormal">1: [ 36] CQ: cq_create: ioctl(GNI_IOC_CQ_CREATE)<u></u><u></u></p>
<p class="MsoNormal">1: [ 36] CQ: cq_create: #1 cq created, kern_cq_descr = 395, mode = 4, rd_index_ptr = 0x2aaaaaae6000, queue = 0x2aaaaaade000, intr_mask = (nil)<u></u><u></u></p>
<p class="MsoNormal">1: [ 36] CQ: GNI_CqCreate: entry_count: 2048 reqs: 2048 adjusted entries: 2559 alloc_count: 2560<u></u><u></u></p>
<p class="MsoNormal">1: [ 36] CQ: cq_create: ioctl(GNI_IOC_CQ_CREATE)<u></u><u></u></p>
<p class="MsoNormal">1: [ 36] CQ: cq_create: #1 cq created, kern_cq_descr = 396, mode = 5, rd_index_ptr = 0x2aaaaaaef000, queue = 0x2aaaaaae7000, intr_mask = 0x2aaaaaacf000<u></u><u></u></p>
<p class="MsoNormal">1: [ 36] CQ: GNI_CqCreate: entry_count: 16384 reqs: 16384 adjusted entries: 16895 alloc_count: 16896<u></u><u></u></p>
<p class="MsoNormal">1: [ 36] CQ: cq_create: ioctl(GNI_IOC_CQ_CREATE) returned error - Invalid argument<u></u><u></u></p>
<p class="MsoNormal">1: [ 36] CQ: GNI_CqCreate: GNI_IOC_CQ_CREATE with PHYS_MEM failed trying without PHYS_MEM<u></u><u></u></p>
<p class="MsoNormal">1: [ 36] MR: GNI_MemRegister: Mem reg of 135168 length at addr 0x2aaaaaaf0000<u></u><u></u></p>
<p class="MsoNormal">1: [ 36] CQ: cq_create: ioctl(GNI_IOC_CQ_CREATE)<u></u><u></u></p>
<p class="MsoNormal">1: [ 36] CQ: cq_create: #1 cq created, kern_cq_descr = 397, mode = 0, rd_index_ptr = 0x2aaaaab11000, queue = 0x2aaaaaaf0000, intr_mask = (nil)<u></u><u></u></p>
<p class="MsoNormal">1: [ 36] CQ: GNI_CqCreate: entry_count: 16384 reqs: 16384 adjusted entries: 16895 alloc_count: 16896<u></u><u></u></p>
<p class="MsoNormal">1: [ 36] CQ: cq_create: ioctl(GNI_IOC_CQ_CREATE) returned error - Invalid argument<u></u><u></u></p>
<p class="MsoNormal">1: [ 36] CQ: GNI_CqCreate: GNI_IOC_CQ_CREATE with PHYS_MEM failed trying without PHYS_MEM<u></u><u></u></p>
<p class="MsoNormal">1: [ 36] MR: GNI_MemRegister: Mem reg of 135168 length at addr 0x2aaaaab23000<u></u><u></u></p>
<p class="MsoNormal">1: [ 36] CQ: cq_create: ioctl(GNI_IOC_CQ_CREATE)<u></u><u></u></p>
<p class="MsoNormal">1: [ 36] CQ: cq_create: #1 cq created, kern_cq_descr = 398, mode = 1, rd_index_ptr = 0x2aaaaab12000, queue = 0x2aaaaab23000, intr_mask = 0x2aaaaaacf004<u></u><u></u></p>
<p class="MsoNormal">1: [ 36] MR: GNI_MemRegister: Mem reg of 136314880 length at addr 0x2aaaae400000<u></u><u></u></p>
<p class="MsoNormal">srun: error: nid00722: task 0: Exited with exit code 61<u></u><u></u></p>
<p class="MsoNormal">srun: Terminating job step 789872.11<u></u><u></u></p>
<p class="MsoNormal">srun: Job step aborted: Waiting up to 32 seconds for job step to finish.<u></u><u></u></p>
<p class="MsoNormal">srun: error: nid00724: task 1: Killed<u></u><u></u></p>
<p class="MsoNormal">daint102:/scratch/snx3000/<wbr>biddisco/build$<u></u><u></u></p>
</div>
</div>
<br>______________________________<wbr>_________________<br>
Libfabric-users mailing list<br>
<a href="mailto:Libfabric-users@lists.openfabrics.org">Libfabric-users@lists.<wbr>openfabrics.org</a><br>
<a href="http://lists.openfabrics.org/mailman/listinfo/libfabric-users" rel="noreferrer" target="_blank">http://lists.openfabrics.org/<wbr>mailman/listinfo/libfabric-<wbr>users</a><br>
<br></blockquote></div><br></div>