<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
So, I have a question related to Hugo’s question… other than <a href="https://ofiwg.github.io/libfabric/" class="">https://ofiwg.github.io/libfabric/</a> and fi_provider, are there documents I can read on how to be a better maintainer of the PSM2 provider?
<div class=""><br class="">
<div class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div style="font-size: 16px;">-----</div>
<div style="font-size: 16px;">Michael Heinz</div>
<div><a href="mailto:michael.william.heinz@cornelisnetworks.com" class="">michael.william.heinz@cornelisnetworks.com</a></div>
</div>
</div>
<div><br class="">
<blockquote type="cite" class="">
<div class="">On Apr 14, 2021, at 3:00 PM, <a href="mailto:ofiwg-request@lists.openfabrics.org" class="">
ofiwg-request@lists.openfabrics.org</a> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="">Send ofiwg mailing list submissions to<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span><a href="mailto:ofiwg@lists.openfabrics.org" class="">ofiwg@lists.openfabrics.org</a><br class="">
<br class="">
To subscribe or unsubscribe via the World Wide Web, visit<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span>https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.openfabrics.org%2Fmailman%2Flistinfo%2Fofiwg&data=04%7C01%7Cmichael.william.heinz%40cornelisnetworks.com%7Cac3c4925ee2344d3ff1f08d8ff7761e4%7C4dbdb7da74ee4b458747ef5ce5ebe68a%7C0%7C0%7C637540235521136674%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=rgTe5E7EnhJPa6P4cVZvGlMm6%2BNRAdHRsve79KQTOSM%3D&reserved=0<br class="">
or, via email, send a message with subject or body 'help' to<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span>ofiwg-request@lists.openfabrics.org<br class="">
<br class="">
You can reach the person managing the list at<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span>ofiwg-owner@lists.openfabrics.org<br class="">
<br class="">
When replying, please edit your Subject line so it is more specific<br class="">
than "Re: Contents of ofiwg digest..."<br class="">
<br class="">
<br class="">
Today's Topics:<br class="">
<br class="">
1. Resource Management RX CQ and address vectors (BOLLORE, HUGO)<br class="">
2. Re: Resource Management RX CQ and address vectors (Hefty, Sean)<br class="">
<br class="">
<br class="">
----------------------------------------------------------------------<br class="">
<br class="">
Message: 1<br class="">
Date: Wed, 14 Apr 2021 16:33:54 +0000<br class="">
From: "BOLLORE, HUGO" <hugo.bollore@atos.net><br class="">
To: "ofiwg@lists.openfabrics.org" <ofiwg@lists.openfabrics.org><br class="">
Subject: [ofiwg] Resource Management RX CQ and address vectors<br class="">
Message-ID: <475d88f7-1b8e-abbb-e054-b9c5b8e5da96@atos.net><br class="">
Content-Type: text/plain; charset="utf-8"<br class="">
<br class="">
Hello,<br class="">
<br class="">
I'm working with Mehdi Bendahhou (Atos) on a new provider.<br class="">
We have a few questions about the API specification.<br class="">
<br class="">
First, we were wondering how to deal with the MULTI_RECV option when <br class="">
checking overruns of the RX CQ.<br class="">
When fi_recv is called with this option we cannot know how many messages <br class="">
will be received and thus how many completions may result of this operation.<br class="">
If resource management is enabled, documentation states that the <br class="">
provider must return -FI_EAGAIN if an operation could result in CQ overruns.<br class="">
Is there a specific case to apply with the MULTI_RECV option ? If not, <br class="">
what should we do in this situation ?<br class="">
<br class="">
Second, we are unsure about the state of an address vector during the <br class="">
execution of an application.<br class="">
Are address vectors static as soon as the endpoint is enabled or can <br class="">
they change dynamically ?<br class="">
<br class="">
Last, in a connectionless endpoint, is it required for a receiving <br class="">
application to have the sender address in its address vector if the <br class="">
fi_recv is posted with a src_addr to FI_ADDR_UNSPEC and/or <br class="">
FI_DIRECTED_RECV is disabled ?<br class="">
<br class="">
If this helps to answer the last two questions, we are trying to <br class="">
determine if at any given point an endpoint is able to retrieve a list <br class="">
of all address/endpoints it is communicating with (both in emission and <br class="">
reception).<br class="">
<br class="">
Thanks in advance to those who can enlighten us !<br class="">
Hugo Bolloré - Atos<br class="">
<br class="">
------------------------------<br class="">
<br class="">
Message: 2<br class="">
Date: Wed, 14 Apr 2021 17:22:05 +0000<br class="">
From: "Hefty, Sean" <sean.hefty@intel.com><br class="">
To: "BOLLORE, HUGO" <hugo.bollore@atos.net>,<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span>"ofiwg@lists.openfabrics.org" <ofiwg@lists.openfabrics.org><br class="">
Subject: Re: [ofiwg] Resource Management RX CQ and address vectors<br class="">
Message-ID:<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span><DM6PR11MB4609A51ED8C1A6694E0E55559E4E9@DM6PR11MB4609.namprd11.prod.outlook.com><br class="">
<span class="Apple-tab-span" style="white-space:pre"></span><br class="">
Content-Type: text/plain; charset="utf-8"<br class="">
<br class="">
<blockquote type="cite" class="">I'm working with Mehdi Bendahhou (Atos) on a new provider.<br class="">
We have a few questions about the API specification.<br class="">
</blockquote>
<br class="">
Welcome!<br class="">
<br class="">
<blockquote type="cite" class="">First, we were wondering how to deal with the MULTI_RECV option when<br class="">
checking overruns of the RX CQ.<br class="">
When fi_recv is called with this option we cannot know how many messages<br class="">
will be received and thus how many completions may result of this operation.<br class="">
If resource management is enabled, documentation states that the<br class="">
provider must return -FI_EAGAIN if an operation could result in CQ overruns.<br class="">
Is there a specific case to apply with the MULTI_RECV option ? If not,<br class="">
what should we do in this situation ?<br class="">
</blockquote>
<br class="">
This probably needs to be handled as part of flow control, to back off the sender if handling the receive would result in CQ overflow. In the current implementations that I'm aware of that handle MULTI_RECV, CQ overflow is handled. The overflow entries are
queued in separate location. For example, the util_cq code will handle this.<br class="">
<br class="">
We can discuss if the man pages need to be updated exclude multi-recv from resource management requirements.<br class="">
<br class="">
<blockquote type="cite" class="">Second, we are unsure about the state of an address vector during the<br class="">
execution of an application.<br class="">
Are address vectors static as soon as the endpoint is enabled or can<br class="">
they change dynamically ?<br class="">
</blockquote>
<br class="">
They can change dynamically, and some applications do this (fabtest for one). But ultimately, this comes down to what your hardware can do and its target applications. If needed, you can document this restriction. I know there are environments where the
implementation ends up using a static address vector, possibly pre-loaded before the app starts and shared between processes.<br class="">
<br class="">
<blockquote type="cite" class="">Last, in a connectionless endpoint, is it required for a receiving<br class="">
application to have the sender address in its address vector if the<br class="">
fi_recv is posted with a src_addr to FI_ADDR_UNSPEC and/or<br class="">
FI_DIRECTED_RECV is disabled ?<br class="">
</blockquote>
<br class="">
No, the receiver does not have to have the sender's address in its AV. Unidirectional transfers are supported. The use of an out of band, separate communication library, should not be required.<br class="">
<br class="">
<blockquote type="cite" class="">If this helps to answer the last two questions, we are trying to<br class="">
determine if at any given point an endpoint is able to retrieve a list<br class="">
of all address/endpoints it is communicating with (both in emission and<br class="">
reception).<br class="">
</blockquote>
<br class="">
It's not required from an API perspective.<br class="">
<br class="">
FWIW, the util providers (rxm, rxd) handle this by maintaining a map between addresses in the local AV, versus those peers that its communicating with. It's possible for the AV to contains addresses where remote communication has not (yet) be setup. Likewise,
the endpoint may communicate with peers that are not in the local AV.<br class="">
<br class="">
- Sean<br class="">
<br class="">
------------------------------<br class="">
<br class="">
Subject: Digest Footer<br class="">
<br class="">
_______________________________________________<br class="">
ofiwg mailing list<br class="">
ofiwg@lists.openfabrics.org<br class="">
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.openfabrics.org%2Fmailman%2Flistinfo%2Fofiwg&data=04%7C01%7Cmichael.william.heinz%40cornelisnetworks.com%7Cac3c4925ee2344d3ff1f08d8ff7761e4%7C4dbdb7da74ee4b458747ef5ce5ebe68a%7C0%7C0%7C637540235521136674%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=rgTe5E7EnhJPa6P4cVZvGlMm6%2BNRAdHRsve79KQTOSM%3D&reserved=0<br class="">
<br class="">
<br class="">
------------------------------<br class="">
<br class="">
End of ofiwg Digest, Vol 93, Issue 4<br class="">
************************************<br class="">
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</body>
</html>