<div dir="ltr">HI John,<div><br></div><div>These messages look like the type you get if you don't have exclusive access to the node. Does your system use</div><div>ALPS or SLURM? There's another factor as well, do these nodes have GPUs? This may impact your jobs Aries hw resource limits. We don't typically test libfabric on Cray XC nodes with GPUs.</div><div><br></div><div>Howard</div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">2017-02-14 7:26 GMT-07:00 Biddiscombe, John A. <span dir="ltr"><<a href="mailto:biddisco@cscs.ch" target="_blank">biddisco@cscs.ch</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-GB" link="blue" vlink="purple">
<div class="m_-6187743838770924939WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Sorry Howard, this went into my spam folder and I missed it.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">I have run the gni test and it creates rather a lot of output when debug is enabled.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">I’ve put the output (800MB) here
<a href="ftp://ftp.cscs.ch/out/biddisco/cray/gnitestout.txt" target="_blank">ftp://ftp.cscs.ch/out/<wbr>biddisco/cray/gnitestout.txt</a><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">The synopsis is
<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">[====] Synthesis: Tested: 631 | Passing: 573 | Failing: 58 | Crashing: 57<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">with the majority of errors being of the form<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">[ 240] CQ: cq_create: ioctl(GNI_IOC_CQ_CREATE) returned error - Invalid argument<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">with occasional<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">[ 240] JOB: GNI_CdmAttach: ioctl(GNI_IOC_NIC_SETATTR) NIC[0] returned error - No space left on device<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">(but from what I read the gnitest only runs on one node, so it may not be much use).<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Thanks for taking the time to investigate.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">PS. I forgot to ask - if the FI_EP_MSG or gni is due in 1.5.0 then what sort of timescale would one expect that to be in?<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">JB<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Howard Pritchard [mailto:<a href="mailto:hppritcha@gmail.com" target="_blank">hppritcha@gmail.com</a>]
<br>
<b>Sent:</b> 14 February 2017 00:08<br>
<b>To:</b> Biddiscombe, John A.<br>
<b>Cc:</b> <a href="mailto:libfabric-users@lists.openfabrics.org" target="_blank">libfabric-users@lists.<wbr>openfabrics.org</a><br>
<b>Subject:</b> Re: [libfabric-users] FI_EP_MSG on cray<u></u><u></u></span></p><div><div class="h5">
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">Hi John,<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Could you try the <span style="font-size:9.5pt">run_gnitest script with this UGNI debug level set? I'd like to understand why that's failing for you.</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt">I cannot get fi_pingpong to work with FI_EP_MSG for GNI provider. It should work though. I filed an issue on the GNI downstream provider repo.</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt">Howard</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">2017-02-13 13:21 GMT-07:00 Biddiscombe, John A. <<a href="mailto:biddisco@cscs.ch" target="_blank">biddisco@cscs.ch</a>>:<u></u><u></u></p>
<div>
<div>
<p class="MsoNormal">Howard, here’s some output …<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">The machine is the cray piz daint at CSCS,<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">Allocation as follows<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">salloc -N 2 -C mc --time=02:00:00 –exclusive<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">daint102:/scratch/snx3000/<wbr>biddisco/build$ export UGNI_DEBUG=10<u></u><u></u></p>
<p class="MsoNormal">daint102:/scratch/snx3000/<wbr>biddisco/build$ ./frun.sh ~/apps/fabtests/bin/fi_msg<u></u><u></u></p>
<p class="MsoNormal">running /users/biddisco/apps/fabtests/<wbr>bin/fi_msg on nid00[722,724]<u></u><u></u></p>
<p class="MsoNormal">nid00722 is 148.187.34.215<u></u><u></u></p>
<p class="MsoNormal">Generated command is srun -n 2 --ntasks-per-node=1 -l --multi-prog ./scalable.conf<u></u><u></u></p>
<p class="MsoNormal">0 /users/biddisco/apps/fabtests/<wbr>bin/fi_msg -p gni<u></u><u></u></p>
<p class="MsoNormal">1 /users/biddisco/apps/fabtests/<wbr>bin/fi_msg -p gni 148.187.34.215<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">0: [ 44] GNII_DebugInit: GNII_debug_level: 10 GNII_subsys_debug: 0 GNII_debug_mask: 0x0 GNII_debug_inst_id: 44<u></u><u></u></p>
<p class="MsoNormal">0: [ 44] JOB: GNII_GetKernelVersion: kgni version major = 0x0 minor 0x45 code 0xb9 built with major = 0x0 minor = 0x45 code 0x4e24<u></u><u></u></p>
<p class="MsoNormal">0: [ 44] JOB: GNI_GetJobResInfo: job resource: FMA (6) used: 0 limit: 123<u></u><u></u></p>
<p class="MsoNormal">0: [ 44] JOB: GNII_GetKernelVersion: kgni version major = 0x0 minor 0x45 code 0xb9 built with major = 0x0 minor = 0x45 code 0x4e24<u></u><u></u></p>
<p class="MsoNormal">0: [ 44] JOB: GNI_GetJobResInfo: job resource: CQ (5) used: 0 limit: 509<u></u><u></u></p>
<p class="MsoNormal">0: fi_getinfo(): common/shared.c:454, ret=-61 (No data available)<u></u><u></u></p>
<p class="MsoNormal">1: [ 36] GNII_DebugInit: GNII_debug_level: 10 GNII_subsys_debug: 0 GNII_debug_mask: 0x0 GNII_debug_inst_id: 36<u></u><u></u></p>
<p class="MsoNormal">1: [ 36] JOB: GNII_GetKernelVersion: kgni version major = 0x0 minor 0x45 code 0xb9 built with major = 0x0 minor = 0x45 code 0x4e24<u></u><u></u></p>
<p class="MsoNormal">1: [ 36] JOB: GNI_GetJobResInfo: job resource: FMA (6) used: 0 limit: 123<u></u><u></u></p>
<p class="MsoNormal">1: [ 36] JOB: GNII_GetKernelVersion: kgni version major = 0x0 minor 0x45 code 0xb9 built with major = 0x0 minor = 0x45 code 0x4e24<u></u><u></u></p>
<p class="MsoNormal">1: [ 36] JOB: GNI_GetJobResInfo: job resource: CQ (5) used: 0 limit: 509<u></u><u></u></p>
<p class="MsoNormal">1: [ 36] JOB: GNII_GetKernelVersion: kgni version major = 0x0 minor 0x45 code 0xb9 built with major = 0x0 minor = 0x45 code 0x4e24<u></u><u></u></p>
<p class="MsoNormal">1: [ 36] FMA: GNI_CdmAttach: FMA window size: 32768<u></u><u></u></p>
<p class="MsoNormal">1: [ 36] FMA: GNI_CdmAttach: NOPRIV_ERR masked<u></u><u></u></p>
<p class="MsoNormal">1: [ 36] JOB: GNI_CdmAttach: ptag = 36 inst_id = 13864961 fma_window = 0x0000000000000000 fma_ctrl = 0x0000000000000000<u></u><u></u></p>
<p class="MsoNormal">1: [ 36] CQ: GNI_CqCreate: entry_count: 1361 reqs: 1361 adjusted entries: 1395 alloc_count: 1396<u></u><u></u></p>
<p class="MsoNormal">1: [ 36] CQ: cq_create: ioctl(GNI_IOC_CQ_CREATE)<u></u><u></u></p>
<p class="MsoNormal">1: [ 36] CQ: cq_create: #1 cq created, kern_cq_descr = 1, mode = 2, rd_index_ptr = 0x2aaaaaad7ba0, queue = 0x2aaaaaad5000, intr_mask = (nil)<u></u><u></u></p>
<p class="MsoNormal">1: [ 36] CQ: GNI_CqCreate: entry_count: 1361 reqs: 1361 adjusted entries: 1395 alloc_count: 1396<u></u><u></u></p>
<p class="MsoNormal">1: [ 36] CQ: cq_create: ioctl(GNI_IOC_CQ_CREATE)<u></u><u></u></p>
<p class="MsoNormal">1: [ 36] CQ: cq_create: #1 cq created, kern_cq_descr = 394, mode = 20, rd_index_ptr = 0x2aaaaaadc000, queue = 0x2aaaaaad8000, intr_mask = (nil)<u></u><u></u></p>
<p class="MsoNormal">1: [ 36] FLBTE: GNII_FlbteInit: FLBTE: tx_counter 0x2aaaaaace008, chan 2, max_len -1, total 511<u></u><u></u></p>
<p class="MsoNormal">1: [ 36] CQ: GNI_CqCreate: entry_count: 2048 reqs: 2048 adjusted entries: 2559 alloc_count: 2560<u></u><u></u></p>
<p class="MsoNormal">1: [ 36] CQ: cq_create: ioctl(GNI_IOC_CQ_CREATE)<u></u><u></u></p>
<p class="MsoNormal">1: [ 36] CQ: cq_create: #1 cq created, kern_cq_descr = 395, mode = 4, rd_index_ptr = 0x2aaaaaae6000, queue = 0x2aaaaaade000, intr_mask = (nil)<u></u><u></u></p>
<p class="MsoNormal">1: [ 36] CQ: GNI_CqCreate: entry_count: 2048 reqs: 2048 adjusted entries: 2559 alloc_count: 2560<u></u><u></u></p>
<p class="MsoNormal">1: [ 36] CQ: cq_create: ioctl(GNI_IOC_CQ_CREATE)<u></u><u></u></p>
<p class="MsoNormal">1: [ 36] CQ: cq_create: #1 cq created, kern_cq_descr = 396, mode = 5, rd_index_ptr = 0x2aaaaaaef000, queue = 0x2aaaaaae7000, intr_mask = 0x2aaaaaacf000<u></u><u></u></p>
<p class="MsoNormal">1: [ 36] CQ: GNI_CqCreate: entry_count: 16384 reqs: 16384 adjusted entries: 16895 alloc_count: 16896<u></u><u></u></p>
<p class="MsoNormal">1: [ 36] CQ: cq_create: ioctl(GNI_IOC_CQ_CREATE) returned error - Invalid argument<u></u><u></u></p>
<p class="MsoNormal">1: [ 36] CQ: GNI_CqCreate: GNI_IOC_CQ_CREATE with PHYS_MEM failed trying without PHYS_MEM<u></u><u></u></p>
<p class="MsoNormal">1: [ 36] MR: GNI_MemRegister: Mem reg of 135168 length at addr 0x2aaaaaaf0000<u></u><u></u></p>
<p class="MsoNormal">1: [ 36] CQ: cq_create: ioctl(GNI_IOC_CQ_CREATE)<u></u><u></u></p>
<p class="MsoNormal">1: [ 36] CQ: cq_create: #1 cq created, kern_cq_descr = 397, mode = 0, rd_index_ptr = 0x2aaaaab11000, queue = 0x2aaaaaaf0000, intr_mask = (nil)<u></u><u></u></p>
<p class="MsoNormal">1: [ 36] CQ: GNI_CqCreate: entry_count: 16384 reqs: 16384 adjusted entries: 16895 alloc_count: 16896<u></u><u></u></p>
<p class="MsoNormal">1: [ 36] CQ: cq_create: ioctl(GNI_IOC_CQ_CREATE) returned error - Invalid argument<u></u><u></u></p>
<p class="MsoNormal">1: [ 36] CQ: GNI_CqCreate: GNI_IOC_CQ_CREATE with PHYS_MEM failed trying without PHYS_MEM<u></u><u></u></p>
<p class="MsoNormal">1: [ 36] MR: GNI_MemRegister: Mem reg of 135168 length at addr 0x2aaaaab23000<u></u><u></u></p>
<p class="MsoNormal">1: [ 36] CQ: cq_create: ioctl(GNI_IOC_CQ_CREATE)<u></u><u></u></p>
<p class="MsoNormal">1: [ 36] CQ: cq_create: #1 cq created, kern_cq_descr = 398, mode = 1, rd_index_ptr = 0x2aaaaab12000, queue = 0x2aaaaab23000, intr_mask = 0x2aaaaaacf004<u></u><u></u></p>
<p class="MsoNormal">1: [ 36] MR: GNI_MemRegister: Mem reg of 136314880 length at addr 0x2aaaae400000<u></u><u></u></p>
<p class="MsoNormal">srun: error: nid00722: task 0: Exited with exit code 61<u></u><u></u></p>
<p class="MsoNormal">srun: Terminating job step 789872.11<u></u><u></u></p>
<p class="MsoNormal">srun: Job step aborted: Waiting up to 32 seconds for job step to finish.<u></u><u></u></p>
<p class="MsoNormal">srun: error: nid00724: task 1: Killed<u></u><u></u></p>
<p class="MsoNormal">daint102:/scratch/snx3000/<wbr>biddisco/build$<u></u><u></u></p>
</div>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
______________________________<wbr>_________________<br>
Libfabric-users mailing list<br>
<a href="mailto:Libfabric-users@lists.openfabrics.org" target="_blank">Libfabric-users@lists.<wbr>openfabrics.org</a><br>
<a href="http://lists.openfabrics.org/mailman/listinfo/libfabric-users" target="_blank">http://lists.openfabrics.org/<wbr>mailman/listinfo/libfabric-<wbr>users</a><u></u><u></u></p>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div></div></div>
</div>
</blockquote></div><br></div>