<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Aptos;
panose-1:2 11 0 4 2 2 2 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Aptos",sans-serif;}
span.EmailStyle200
{mso-style-type:personal-reply;
font-family:"Aptos",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:11.0pt;
mso-ligatures:none;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="#467886" vlink="#96607D" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">Update:<o:p></o:p></p>
<p class="MsoNormal">Issue resolved. Problem was found to be slow NIC provider code in rdma-core.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div id="mail-editor-reference-message-container">
<div>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-bottom:12.0pt"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">Libfabric-users <libfabric-users-bounces@lists.openfabrics.org> on behalf of Niyaz Murshed <Niyaz.Murshed@arm.com><br>
<b>Date: </b>Monday, September 23, 2024 at 9:24</span><span style="font-size:12.0pt;font-family:"Arial",sans-serif;color:black"> </span><span style="font-size:12.0pt;color:black">AM<br>
<b>To: </b>libfabric-users@lists.openfabrics.org <libfabric-users@lists.openfabrics.org><br>
<b>Cc: </b>nd <nd@arm.com><br>
<b>Subject: </b>Re: [libfabric-users] fi_rma_bw error<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal">Further debugging this, I see that the server (which accepts the WRITE) sends RNR NAK.<o:p></o:p></p>
<p class="MsoNormal">This shows that the CLIENT is sending WRITE request faster than SERVER can accept WRITE requests.<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<div id="mail-editor-reference-message-container">
<div>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-bottom:12.0pt"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">Libfabric-users <libfabric-users-bounces@lists.openfabrics.org> on behalf of Niyaz Murshed <Niyaz.Murshed@arm.com><br>
<b>Date: </b>Thursday, September 19, 2024 at 10:33</span><span style="font-size:12.0pt;font-family:"Arial",sans-serif;color:black"> </span><span style="font-size:12.0pt;color:black">AM<br>
<b>To: </b>libfabric-users@lists.openfabrics.org <libfabric-users@lists.openfabrics.org><br>
<b>Subject: </b>[libfabric-users] fi_rma_bw error</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Hello, <o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">I am seeing some issues with size more than 38000b when running fi_rma_bw test. Has something changed recently.<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">root@nvidia-grace-2-1:/# fi_rma_bw -s 192.168.100.200 192.168.100.100 -e msg -o write -d roceP2p1s0 -S 32000 -p verbs<o:p></o:p></p>
<p class="MsoNormal">bytes iters total time MB/sec usec/xfer Mxfers/sec<o:p></o:p></p>
<p class="MsoNormal">31k 20k 610m 0.44s 1454.91 21.99 0.05<o:p></o:p></p>
<p class="MsoNormal">root@nvidia-grace-2-1:/# fi_rma_bw -s 192.168.100.200 192.168.100.100 -e msg -o write -d roceP2p1s0 -S 36000 -p verbs<o:p></o:p></p>
<p class="MsoNormal">bytes iters total time MB/sec usec/xfer Mxfers/sec<o:p></o:p></p>
<p class="MsoNormal">35k 20k 686m 0.52s 1379.17 26.10 0.04<o:p></o:p></p>
<p class="MsoNormal">root@nvidia-grace-2-1:/# fi_rma_bw -s 192.168.100.200 192.168.100.100 -e msg -o write -d roceP2p1s0 -S 38000 -p verbs<o:p></o:p></p>
<p class="MsoNormal">bytes iters total time MB/sec usec/xfer Mxfers/sec<o:p></o:p></p>
<p class="MsoNormal">37k 20k 724m 0.56s 1366.78 27.80 0.04<o:p></o:p></p>
<p class="MsoNormal">root@nvidia-grace-2-1:/# fi_rma_bw -s 192.168.100.200 192.168.100.100 -e msg -o write -d roceP2p1s0 -S 40000 -p verbs<o:p></o:p></p>
<p class="MsoNormal">[error] fabtests:common/shared.c:2995: cq_readerr 5 (Input/output error), provider errno: 2 (local QP operation error)<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">root@nvidia-grace-2-1:/# fi_rma_bw -s 192.168.100.200 192.168.100.100 -e msg -o write -d roceP2p1s0 -p verbs<o:p></o:p></p>
<p class="MsoNormal">bytes iters total time MB/sec usec/xfer Mxfers/sec<o:p></o:p></p>
<p class="MsoNormal">64 20k 1.2m 0.02s 66.06 0.97 1.03<o:p></o:p></p>
<p class="MsoNormal">256 20k 4.8m 0.02s 327.37 0.78 1.28<o:p></o:p></p>
<p class="MsoNormal">1k 20k 19m 0.02s 1306.79 0.78 1.28<o:p></o:p></p>
<p class="MsoNormal">4k 20k 78m 0.05s 1525.88 2.68 0.37<o:p></o:p></p>
<p class="MsoNormal">[error] fabtests:common/shared.c:2995: cq_readerr 5 (Input/output error), provider errno: 2 (local QP operation error)<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">Any suggestion where to look for error?<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">Regards,<o:p></o:p></p>
<p class="MsoNormal">Niyaz<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<p class="MsoNormal"><span style="font-size:12.0pt">IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose
the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
</span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>