<div dir="ltr">Hi John,<div><br></div><div>This is odd. Could you check what resource limits you're getting when you salloc a node:</div><div><br></div><div>salloc -N 1 --exclusive</div><div>
<p class="gmail-p1"><span class="gmail-s1">srun -n 1 cat /sys/class/gni/kgni0/resources</span></p><p class="gmail-p1"><span class="gmail-s1"><br></span></p><p class="gmail-p1"><span class="gmail-s1">and post the output? You should be seeing something like:</span></p><p class="gmail-p1"><span class="gmail-s1">--- PTag: 104 PKey: 0x1734 JobId: 0x900000d62 RefCount: 1 Suspend: Idle ---</span></p><p class="gmail-p1"><span class="gmail-s1">Name Used Limit HWM </span></p><p class="gmail-p1"><span class="gmail-s1">MDD 0 1806 0 <br></span>CQ 0 495 <br>FMA 0 123 <br>SFMA 0 123 <br>RDMA 0 -1 <br>DIRECT 0 -1 <br>IOMMU 0 134217728 <br>PCI-IOMMU 0 -1 <br>CE 0 1 <br>DLA 0 15360 <br>non-VMDH 0 -1 <br>SMDD Hold 0 -1 </p><p class="gmail-p1"><span class="gmail-s1">--- PTag: 105 PKey: 0x1735 JobId: 0x900000d62 RefCount: 1 Suspend: Idle ---</span></p><p class="gmail-p1"><span class="gmail-s1">Name Used Limit HWM </span></p><p class="gmail-p1"><span class="gmail-s1">MDD 0 1806 0 <br></span>CQ 0 495 <br>FMA 0 123 <br>SFMA 0 123 <br>RDMA 0 -1 <br>DIRECT 0 -1 <br>IOMMU 0 134217728 <br>PCI-IOMMU 0 -1 <br>CE 0 1 <br>DLA 0 15360 <br>non-VMDH 0 -1 <br>SMDD Hold 0 -1 </p><p class="gmail-p1"><span class="gmail-s1">Howard</span></p></div></div><div class="gmail_extra"><br><div class="gmail_quote">2017-02-14 14:36 GMT-07:00 Biddiscombe, John A. <span dir="ltr"><<a href="mailto:biddisco@cscs.ch" target="_blank">biddisco@cscs.ch</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="white" lang="EN-GB" link="blue" vlink="purple">
<div class="m_2485307529250579315WordSection1">
<p class="MsoNormal"><span style="font-size:14.0pt;font-family:Calibri">Yes, the version I’m using is just as is found here
<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:14.0pt;font-family:Calibri"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:14.0pt;font-family:Calibri"><a href="https://github.com/ofi-cray/libfabric-cray/blob/master/prov/gni/test/run_gnitest#L42" target="_blank">https://github.com/ofi-cray/<wbr>libfabric-cray/blob/master/<wbr>prov/gni/test/run_gnitest#L42</a><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:14.0pt;font-family:Calibri"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:14.0pt;font-family:Calibri">so it should be set<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:14.0pt;font-family:Calibri"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:14.0pt;font-family:Calibri">JB<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:14.0pt;font-family:Calibri"><u></u> <u></u></span></p>
<div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span style="font-family:Calibri;color:black">From: </span>
</b><span style="font-family:Calibri;color:black">Howard Pritchard <<a href="mailto:hppritcha@gmail.com" target="_blank">hppritcha@gmail.com</a>><br>
<b>Date: </b>Tuesday, 14 February 2017 at 21:05</span></p><div><div class="h5"><br>
<b>To: </b>John Biddiscombe <<a href="mailto:biddisco@cscs.ch" target="_blank">biddisco@cscs.ch</a>><br>
<b>Cc: </b>"<a href="mailto:libfabric-users@lists.openfabrics.org" target="_blank">libfabric-users@lists.<wbr>openfabrics.org</a>" <<a href="mailto:libfabric-users@lists.openfabrics.org" target="_blank">libfabric-users@lists.<wbr>openfabrics.org</a>><br>
<b>Subject: </b>Re: [libfabric-users] FI_EP_MSG on cray<u></u><u></u></div></div><p></p>
</div><div><div class="h5">
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Hi John, <u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">On the Cray with KNL+omnipath you'll end up using the PSM2 provider.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Could you double check that your copy of run_gnitest has<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="m_2485307529250579315gmail-p1"><span class="m_2485307529250579315gmail-s1">export</span><span class="m_2485307529250579315gmail-s2"> UGNI_FMA_SHARED=</span><span class="m_2485307529250579315gmail-s3">0</span><u></u><u></u></p>
<p class="m_2485307529250579315gmail-p1">is being set?<u></u><u></u></p>
<p class="m_2485307529250579315gmail-p1">Howard<u></u><u></u></p>
<p class="m_2485307529250579315gmail-p1"><u></u> <u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">2017-02-14 12:10 GMT-07:00 Biddiscombe, John A. <<a href="mailto:biddisco@cscs.ch" target="_blank">biddisco@cscs.ch</a>>:<u></u><u></u></p>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<p class="MsoNormal">The nodes I requested (in exclusive mode) are, non gpu nodes.<br>
<br>
XC40 Compute Nodes<br>
Intel® Xeon® E5-2695 v4 @ 2.10GHz (2 x 18 cores, 64/128 GB RAM) (daint-mc)<br>
<br>
my slurm allocation uses<br>
<br>
salloc -N 2 -C mc --time=02:00:00 --exclusive<br>
<br>
I can try on GPU nodes to see if anything is different.<br>
<br>
I have another cray with KNL + omnipath I’ll test on just out of curiosity.<br>
<br>
JB<br>
<br>
From: Howard Pritchard <<a href="mailto:hppritcha@gmail.com" target="_blank">hppritcha@gmail.com</a>><br>
Date: Tuesday, 14 February 2017 at 19:47<br>
To: John Biddiscombe <<a href="mailto:biddisco@cscs.ch" target="_blank">biddisco@cscs.ch</a>><br>
Cc: "<a href="mailto:libfabric-users@lists.openfabrics.org" target="_blank">libfabric-users@lists.<wbr>openfabrics.org</a>" <<a href="mailto:libfabric-users@lists.openfabrics.org" target="_blank">libfabric-users@lists.<wbr>openfabrics.org</a>><u></u><u></u></p>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">Subject: Re: [libfabric-users] FI_EP_MSG on cray<br>
<br>
HI John,<br>
<br>
These messages look like the type you get if you don't have exclusive access to the node. Does your system use<br>
ALPS or SLURM? There's another factor as well, do these nodes have GPUs? This may impact your jobs Aries hw resource limits. We don't typically test libfabric on Cray XC nodes with GPUs.<br>
<br>
Howard<br>
<br>
<br>
2017-02-14 7:26 GMT-07:00 Biddiscombe, John A. <<a href="mailto:biddisco@cscs.ch" target="_blank">biddisco@cscs.ch</a>>:<br>
Sorry Howard, this went into my spam folder and I missed it.<br>
<br>
I have run the gni test and it creates rather a lot of output when debug is enabled.<br>
I’ve put the output (800MB) here <a href="ftp://ftp.cscs.ch/out/biddisco/cray/gnitestout.txt" target="_blank">
ftp://ftp.cscs.ch/out/<wbr>biddisco/cray/gnitestout.txt</a><br>
<br>
The synopsis is<br>
[====] Synthesis: Tested: 631 | Passing: 573 | Failing: 58 | Crashing: 57<br>
<br>
with the majority of errors being of the form<br>
[ 240] CQ: cq_create: ioctl(GNI_IOC_CQ_CREATE) returned error - Invalid argument<br>
with occasional<br>
[ 240] JOB: GNI_CdmAttach: ioctl(GNI_IOC_NIC_SETATTR) NIC[0] returned error - No space left on device<br>
<br>
(but from what I read the gnitest only runs on one node, so it may not be much use).<br>
<br>
Thanks for taking the time to investigate.<br>
<br>
PS. I forgot to ask - if the FI_EP_MSG or gni is due in 1.5.0 then what sort of timescale would one expect that to be in?<br>
<br>
JB<br>
<br>
From: Howard Pritchard [mailto:<a href="mailto:hppritcha@gmail.com" target="_blank">hppritcha@gmail.com</a>]<br>
Sent: 14 February 2017 00:08<br>
To: Biddiscombe, John A.<br>
Cc: <a href="mailto:libfabric-users@lists.openfabrics.org" target="_blank">libfabric-users@lists.<wbr>openfabrics.org</a><br>
Subject: Re: [libfabric-users] FI_EP_MSG on cray<br>
<br>
Hi John,<br>
<br>
Could you try the run_gnitest script with this UGNI debug level set? I'd like to understand why that's failing for you.<br>
<br>
I cannot get fi_pingpong to work with FI_EP_MSG for GNI provider. It should work though. I filed an issue on the GNI downstream provider repo.<br>
<br>
Howard<br>
<br>
<br>
<br>
<br>
<br>
2017-02-13 13:21 GMT-07:00 Biddiscombe, John A. <<a href="mailto:biddisco@cscs.ch" target="_blank">biddisco@cscs.ch</a>>:<br>
Howard, here’s some output …<br>
<br>
The machine is the cray piz daint at CSCS,<br>
<br>
Allocation as follows<br>
<br>
salloc -N 2 -C mc --time=02:00:00 –exclusive<br>
<br>
daint102:/scratch/snx3000/<wbr>biddisco/build$ export UGNI_DEBUG=10<br>
daint102:/scratch/snx3000/<wbr>biddisco/build$ ./frun.sh ~/apps/fabtests/bin/fi_msg<br>
running /users/biddisco/apps/fabtests/<wbr>bin/fi_msg on nid00[722,724]<br>
nid00722 is 148.187.34.215<br>
Generated command is srun -n 2 --ntasks-per-node=1 -l --multi-prog ./scalable.conf<br>
0 /users/biddisco/apps/fabtests/<wbr>bin/fi_msg -p gni<br>
1 /users/biddisco/apps/fabtests/<wbr>bin/fi_msg -p gni 148.187.34.215<br>
<br>
0: [ 44] GNII_DebugInit: GNII_debug_level: 10 GNII_subsys_debug: 0 GNII_debug_mask: 0x0 GNII_debug_inst_id: 44<br>
0: [ 44] JOB: GNII_GetKernelVersion: kgni version major = 0x0 minor 0x45 code 0xb9 built with major = 0x0 minor = 0x45 code 0x4e24<br>
0: [ 44] JOB: GNI_GetJobResInfo: job resource: FMA (6) used: 0 limit: 123<br>
0: [ 44] JOB: GNII_GetKernelVersion: kgni version major = 0x0 minor 0x45 code 0xb9 built with major = 0x0 minor = 0x45 code 0x4e24<br>
0: [ 44] JOB: GNI_GetJobResInfo: job resource: CQ (5) used: 0 limit: 509<br>
0: fi_getinfo(): common/shared.c:454, ret=-61 (No data available)<br>
1: [ 36] GNII_DebugInit: GNII_debug_level: 10 GNII_subsys_debug: 0 GNII_debug_mask: 0x0 GNII_debug_inst_id: 36<br>
1: [ 36] JOB: GNII_GetKernelVersion: kgni version major = 0x0 minor 0x45 code 0xb9 built with major = 0x0 minor = 0x45 code 0x4e24<br>
1: [ 36] JOB: GNI_GetJobResInfo: job resource: FMA (6) used: 0 limit: 123<br>
1: [ 36] JOB: GNII_GetKernelVersion: kgni version major = 0x0 minor 0x45 code 0xb9 built with major = 0x0 minor = 0x45 code 0x4e24<br>
1: [ 36] JOB: GNI_GetJobResInfo: job resource: CQ (5) used: 0 limit: 509<br>
1: [ 36] JOB: GNII_GetKernelVersion: kgni version major = 0x0 minor 0x45 code 0xb9 built with major = 0x0 minor = 0x45 code 0x4e24<br>
1: [ 36] FMA: GNI_CdmAttach: FMA window size: 32768<br>
1: [ 36] FMA: GNI_CdmAttach: NOPRIV_ERR masked<br>
1: [ 36] JOB: GNI_CdmAttach: ptag = 36 inst_id = 13864961 fma_window = 0x0000000000000000 fma_ctrl = 0x0000000000000000<br>
1: [ 36] CQ: GNI_CqCreate: entry_count: 1361 reqs: 1361 adjusted entries: 1395 alloc_count: 1396<br>
1: [ 36] CQ: cq_create: ioctl(GNI_IOC_CQ_CREATE)<br>
1: [ 36] CQ: cq_create: #1 cq created, kern_cq_descr = 1, mode = 2, rd_index_ptr = 0x2aaaaaad7ba0, queue = 0x2aaaaaad5000, intr_mask = (nil)<br>
1: [ 36] CQ: GNI_CqCreate: entry_count: 1361 reqs: 1361 adjusted entries: 1395 alloc_count: 1396<br>
1: [ 36] CQ: cq_create: ioctl(GNI_IOC_CQ_CREATE)<br>
1: [ 36] CQ: cq_create: #1 cq created, kern_cq_descr = 394, mode = 20, rd_index_ptr = 0x2aaaaaadc000, queue = 0x2aaaaaad8000, intr_mask = (nil)<br>
1: [ 36] FLBTE: GNII_FlbteInit: FLBTE: tx_counter 0x2aaaaaace008, chan 2, max_len -1, total 511<br>
1: [ 36] CQ: GNI_CqCreate: entry_count: 2048 reqs: 2048 adjusted entries: 2559 alloc_count: 2560<br>
1: [ 36] CQ: cq_create: ioctl(GNI_IOC_CQ_CREATE)<br>
1: [ 36] CQ: cq_create: #1 cq created, kern_cq_descr = 395, mode = 4, rd_index_ptr = 0x2aaaaaae6000, queue = 0x2aaaaaade000, intr_mask = (nil)<br>
1: [ 36] CQ: GNI_CqCreate: entry_count: 2048 reqs: 2048 adjusted entries: 2559 alloc_count: 2560<br>
1: [ 36] CQ: cq_create: ioctl(GNI_IOC_CQ_CREATE)<br>
1: [ 36] CQ: cq_create: #1 cq created, kern_cq_descr = 396, mode = 5, rd_index_ptr = 0x2aaaaaaef000, queue = 0x2aaaaaae7000, intr_mask = 0x2aaaaaacf000<br>
1: [ 36] CQ: GNI_CqCreate: entry_count: 16384 reqs: 16384 adjusted entries: 16895 alloc_count: 16896<br>
1: [ 36] CQ: cq_create: ioctl(GNI_IOC_CQ_CREATE) returned error - Invalid argument<br>
1: [ 36] CQ: GNI_CqCreate: GNI_IOC_CQ_CREATE with PHYS_MEM failed trying without PHYS_MEM<br>
1: [ 36] MR: GNI_MemRegister: Mem reg of 135168 length at addr 0x2aaaaaaf0000<br>
1: [ 36] CQ: cq_create: ioctl(GNI_IOC_CQ_CREATE)<br>
1: [ 36] CQ: cq_create: #1 cq created, kern_cq_descr = 397, mode = 0, rd_index_ptr = 0x2aaaaab11000, queue = 0x2aaaaaaf0000, intr_mask = (nil)<br>
1: [ 36] CQ: GNI_CqCreate: entry_count: 16384 reqs: 16384 adjusted entries: 16895 alloc_count: 16896<br>
1: [ 36] CQ: cq_create: ioctl(GNI_IOC_CQ_CREATE) returned error - Invalid argument<br>
1: [ 36] CQ: GNI_CqCreate: GNI_IOC_CQ_CREATE with PHYS_MEM failed trying without PHYS_MEM<br>
1: [ 36] MR: GNI_MemRegister: Mem reg of 135168 length at addr 0x2aaaaab23000<br>
1: [ 36] CQ: cq_create: ioctl(GNI_IOC_CQ_CREATE)<br>
1: [ 36] CQ: cq_create: #1 cq created, kern_cq_descr = 398, mode = 1, rd_index_ptr = 0x2aaaaab12000, queue = 0x2aaaaab23000, intr_mask = 0x2aaaaaacf004<br>
1: [ 36] MR: GNI_MemRegister: Mem reg of 136314880 length at addr 0x2aaaae400000<br>
srun: error: nid00722: task 0: Exited with exit code 61<br>
srun: Terminating job step 789872.11<br>
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.<br>
srun: error: nid00724: task 1: Killed<br>
daint102:/scratch/snx3000/<wbr>biddisco/build$<br>
<br>
______________________________<wbr>_________________<br>
Libfabric-users mailing list<br>
<a href="mailto:Libfabric-users@lists.openfabrics.org" target="_blank">Libfabric-users@lists.<wbr>openfabrics.org</a><br>
<a href="http://lists.openfabrics.org/mailman/listinfo/libfabric-users" target="_blank">http://lists.openfabrics.org/<wbr>mailman/listinfo/libfabric-<wbr>users</a><br>
<br>
<br>
<br>
<u></u><u></u></p>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div></div></div>
</div>
</blockquote></div><br></div>