<div dir="ltr">Hi Sean,<div><br></div><div><div class="gmail_extra">Some comments inlined, but first about the idea of the msgq being instantiated via a method in the<br>fabric class. I was thinking of the most general case, but I understand the case for<br>
having it be instantiated via a domain class method. If a vendor supported a protocol that</div><div class="gmail_extra">internal to the domain could handle multiple rails, that would solve the problem of tag/rx buffer</div>
<div class="gmail_extra">matching across multiple rails. </div><div class="gmail_extra"><br></div><div class="gmail_extra">Perhaps that's how mxm and psm work already?</div><div class="gmail_extra"><br></div><div class="gmail_extra">
If that's the case, then I'd agree having a msgq object being instantiated via a domain method would</div><div class="gmail_extra">be more appropriate.</div><div class="gmail_extra"><br></div><div class="gmail_extra">
<br></div><div class="gmail_extra"><br><div class="gmail_quote">2014-05-22 16:01 GMT-06:00 Hefty, Sean <span dir="ltr"><<a href="mailto:sean.hefty@intel.com" target="_blank">sean.hefty@intel.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
If I understand your issue, it's not with the send calls being associated with an endpoint, only the relationship of a buffer at the target side. I would think this applies to all receive buffers and any target buffers of an RMA or atomic operation. I didn't follow your asymmetric comment.<br>
<br>
RMA target buffers are currently associated with a domain (using libfabric terms). Moving that to the fabric level has an implication that a buffer may be associated with different access keys. Trying to move receive buffers above the domain would have a similar issue.<br>
<br>
Attempting to share receive buffers above the domain would likely result in synchronization issues between devices and providers in such a way that support for zero copy would be compromised. Even SRQ is restricted to a single domain.<br>
</blockquote><div><br></div><div>Okay, SRQ example is good reason to have the msgq being "child" of domain.</div><div> <br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
>From an architectural viewpoint, I guess the question is, does it make sense that receive buffers never be directly associated with endpoints? This seems to be the general decision point that this discussion is leading to.<br>
</blockquote><div><br></div><div>I think this needs more discussion. I'd think users of SRQ feature in ibverbs may have some ideas</div><div>about pluses and minuses of associating buffers with endpoints vs the way RX buffers are posted to </div>
<div>SRQs now with ibverbs.</div><div><br></div><div>It might also be interesting to have some PSM and/or MXM gurus provide insight here. Both of these API's</div><div>have a MSGQ like concept. Although it seems just from looking at available header files, and, for example,</div>
<div>OpenMPI usage of these APIs, that the MSGQ also provides some kind of communication envelope as well.</div><div><br></div><div>Actually for the PSM provider in libfabric, its pretty clear that the MSGQ is an important concept for that</div>
<div>API.</div><div> </div><div>Howard</div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
- Sean<br>
<br>
<br>
> No I would not create an RMA class. It seems to me like RMA is a transport<br>
> method, and as<br>
> such should be a method of the ep class. What I didn't like about the<br>
> receive buffer method<br>
> (with or without tags actually since the IB SRQ capability also falls under<br>
> this umbrella) was<br>
> that it seemed asymmetric. For FID_MSG type EP, with the current proposal,<br>
> there is<br>
> no ability to allow receive buffers to be posted that could "match"<br>
> incoming sends from<br>
> a set of EP's. Here I mean match in the general sense. Only if a vendor<br>
> supported<br>
> FID_RDM/FID_DGRAM could this be done as the proposal currently stands.<br>
><br>
> I think I'm restating what Rich said.<br>
><br>
> Howard<br>
><br>
><br>
> On Thu, May 22, 2014 at 1:29 PM, Hefty, Sean <<a href="mailto:sean.hefty@intel.com">sean.hefty@intel.com</a>> wrote:<br>
><br>
><br>
> Thanks, Howard, this is helpful.<br>
><br>
> Regarding the 'tag match class' that you mention, would you create an<br>
> 'rma class' as a peer, with the RMA operations defined in a similar<br>
> fashion? If not, why not? Would this also extend to all other data<br>
> transfer operations? I.e. message queue (send/receive) and atomics, plus<br>
> any others defined in the future?<br>
><br>
><br>
> > -----Original Message-----<br>
> > From: Howard Pritchard [mailto:<a href="mailto:hppritcha@gmail.com">hppritcha@gmail.com</a>]<br>
> > Sent: Thursday, May 22, 2014 11:53 AM<br>
> > To: Richard Graham<br>
> > Cc: Hefty, Sean; <a href="mailto:ofiwg@lists.openfabrics.org">ofiwg@lists.openfabrics.org</a>; ofiwg-<br>
> > <a href="mailto:mpi@lists.openfabrics.org">mpi@lists.openfabrics.org</a><br>
> > Subject: Re: [ofiwg] Call today<br>
> ><br>
> > Hi Folks,<br>
> ><br>
> > here is a diagram of a concept that was discussed in a side<br>
> conversation at<br>
> > the last OFA workshop. I'd thought that a msgq (aka tag matcher<br>
> class)<br>
> > object<br>
> > should be instantiated via a method of the fabric class.<br>
> ><br>
> > red lines in the diagram indicated the pointee can be associated<br>
> with the<br>
> > class<br>
> > being pointed to by the arrow, using the bind method of the class<br>
> being<br>
> > pointed<br>
> > to.<br>
> ><br>
> > the search_by_addr method of the msgq is for use with FID_RDM<br>
> endpoints,<br>
> > while search_by_ep method is when the msgq is associated with<br>
> multipled<br>
> > FID_MSG type endpoints.<br>
> ><br>
> > Note the slide is a little old since the EC class has been divided<br>
> now into<br>
> > a EQ and counter type completion notification mechanisms.<br>
> ><br>
> > Hoping this will maybe help a little here.<br>
> ><br>
> > Howard<br>
> ><br>
> ><br>
> ><br>
> > On Thu, May 22, 2014 at 11:59 AM, Richard Graham<br>
> <<a href="mailto:richardg@mellanox.com">richardg@mellanox.com</a>><br>
> > wrote:<br>
> ><br>
> ><br>
> > Please see inline<br>
> ><br>
> > -----Original Message-----<br>
> > From: Hefty, Sean [mailto:<a href="mailto:sean.hefty@intel.com">sean.hefty@intel.com</a>]<br>
> > Sent: Thursday, May 22, 2014 12:43 PM<br>
> > To: Richard Graham; <a href="mailto:ofiwg@lists.openfabrics.org">ofiwg@lists.openfabrics.org</a>; ofiwg-<br>
> > <a href="mailto:mpi@lists.openfabrics.org">mpi@lists.openfabrics.org</a><br>
> > Cc: Paul Grun (<a href="mailto:grun@cray.com">grun@cray.com</a>); Liran Liss<br>
> > Subject: RE: Call today<br>
> ><br>
> > With permission, copying mailing list on side thread that<br>
> popped up.<br>
> ><br>
> > I understand MPI has wild card receives. But tagged<br>
> semantics are<br>
> > useful even when associated with a generic endpoint concept, or a<br>
> specific<br>
> > address. Note the proposed endpoint concept is not necessarily<br>
> bound to a<br>
> > specific piece of hardware, though it may be based on the provider<br>
> > implementation. The tagged operations themselves may be<br>
> implemented by<br>
> > hardware and are not restricted to being purely a software<br>
> construct.<br>
> > [rich] If the attempt here is to provide a building block<br>
> that will<br>
> > map to different use-case scenarios, then need to have an<br>
> architecture that<br>
> > will map well onto the areas of interest. MPI is just one such<br>
> upper level<br>
> > service, one that has been called out specifically in the context<br>
> of the<br>
> > proposal you have been presenting. So, following on this (the<br>
> precise<br>
> > definition of end point is still rather fuzzy at this stage) in<br>
> general,<br>
> > there is no such one-to-one mapping of and endpoint to an MPI<br>
> matching<br>
> > context, but there can be an association of a matching context with<br>
> one or<br>
> > more endpoints. What I am suggesting here is that we keep data<br>
> notions<br>
> > around data transfer orthogonal to what is done with the data (tag<br>
> > matching, in this case). How the functionality is implemented<br>
> (hardware<br>
> > or not) is separate from how the stack in architected<br>
> ><br>
> > Tagged interfaces, as well as other interfaces such as<br>
> message<br>
> > queues, may still exist above the endpoint. But that layering of<br>
> > interfaces seems better suited above the fabric interfaces (e.g.<br>
> MPI),<br>
> > rather than included with it. This seems more debatable to me<br>
> though, and<br>
> > we could examine whether a domain or fabric object should have<br>
> send/receive<br>
> > capabilities.<br>
> > [rich] Need to keep separate how data is transferred (perhaps<br>
> with<br>
> > functions that we may call send/recv) from the ULP's use of this<br>
> data<br>
> > (perhaps also using the a similar naming scheme of send/recv).<br>
> ><br>
> > - Sean<br>
> ><br>
> > > -----Original Message-----<br>
> > > From: Richard Graham [mailto:<a href="mailto:richardg@mellanox.com">richardg@mellanox.com</a>]<br>
> > > Sent: Wednesday, May 21, 2014 11:09 AM<br>
> > > To: Hefty, Sean<br>
> > > Cc: Paul Grun (<a href="mailto:grun@cray.com">grun@cray.com</a>); Liran Liss<br>
> > > Subject: RE: Call today<br>
> > ><br>
> > > Tag matching as it comes to MPI semantics is not local to a<br>
> given<br>
> > pair<br>
> > > of processes, e.g. MPI has a wild card receive that can<br>
> take data<br>
> > from<br>
> > > any source, and therefore the matching context is broader<br>
> than just<br>
> > a<br>
> > > single pair of source and destination.<br>
> > ><br>
> > > Rich<br>
> > ><br>
> > > -----Original Message-----<br>
> > > From: Hefty, Sean [mailto:<a href="mailto:sean.hefty@intel.com">sean.hefty@intel.com</a>]<br>
> > > Sent: Wednesday, May 21, 2014 1:13 PM<br>
> > > To: Richard Graham<br>
> > > Cc: Paul Grun (<a href="mailto:grun@cray.com">grun@cray.com</a>); Liran Liss<br>
> > > Subject: RE: Call today<br>
> > ><br>
> > > Tag matching, RMA, atomics, and message operations are<br>
> currently<br>
> > > associated with an endpoint, but the functions are<br>
> independent of<br>
> > the<br>
> > > communication protocol in use. Conceptually, it seems<br>
> reasonable<br>
> > to<br>
> > > think of tag matching as a merging of message and RMA write<br>
> > operations.<br>
> > ><br>
> > > I agree that an endpoint is associated with the data<br>
> source/sink.<br>
> > > There is no implied mapping between a process and an<br>
> endpoint.<br>
> > ><br>
> > ><br>
> > > > -----Original Message-----<br>
> > > > From: Richard Graham [mailto:<a href="mailto:richardg@mellanox.com">richardg@mellanox.com</a>]<br>
> > > > Sent: Tuesday, May 20, 2014 9:22 PM<br>
> > > > To: Hefty, Sean<br>
> > > > Cc: Paul Grun (<a href="mailto:grun@cray.com">grun@cray.com</a>); Liran Liss<br>
> > > > Subject: RE: Call today<br>
> > > ><br>
> > > > I suppose that you could consider tag-matching as part of<br>
> > transport.<br>
> > > > However, I would argue that such protocols should be<br>
> independent<br>
> > of<br>
> > > > whether or not a reliable or unreliable communication<br>
> protocol is<br>
> > > > used<br>
> > > (at least<br>
> > > > when it comes to the tag support needed for MPI).<br>
> Also, I<br>
> > associate an<br>
> > > > end-point with either the source and/or the sync of data.<br>
> In MPI<br>
> > > > tag matching is associated with mpi-level<br>
> (process,communicator)<br>
> > > > pair, and therefore the tag-matching context may be<br>
> associated<br>
> > with<br>
> > > > many end-<br>
> > > points.<br>
> > > > I would therefore keep tag-matching as a separate<br>
> concept.<br>
> > > ><br>
> > > > Rich<br>
> > > ><br>
> > > > -----Original Message-----<br>
> > > > From: Hefty, Sean [mailto:<a href="mailto:sean.hefty@intel.com">sean.hefty@intel.com</a>]<br>
> > > > Sent: Tuesday, May 20, 2014 1:26 PM<br>
> > > > To: Richard Graham<br>
> > > > Cc: Paul Grun (<a href="mailto:grun@cray.com">grun@cray.com</a>); Liran Liss<br>
> > > > Subject: RE: Call today<br>
> > > ><br>
> > > > Tag-matching is a transport object (protocol), so I do<br>
> think it<br>
> > > > makes sense being associated with a transport level<br>
> object (i.e.<br>
> > endpoint).<br>
> > > ><br>
> > > > I thought you were referring to the SRQ, which may or may<br>
> not be<br>
> > a<br>
> > > > transport level object. If the sharing of data buffer(s)<br>
> among<br>
> > > > multiple connections is not considered a transport<br>
> object, then I<br>
> > > > agree, it may make sense to have it be a separate object<br>
> with its<br>
> > > > own<br>
> > > interfaces.<br>
> > > > Alternatively, it could also be a property of endpoints<br>
> to share<br>
> > > > receive buffers.<br>
> > > ><br>
> > > > When the SRQ appears in the transport object (protocol),<br>
> it may<br>
> > get<br>
> > > > more complex.<br>
> > > ><br>
> > > > For initial thoughts, sharing receive buffers could be<br>
> handled<br>
> > by:<br>
> > > ><br>
> > > > 1. Creating an explicit SRQ object as a 'peer' to an<br>
> endpoint.<br>
> > SRQ<br>
> > > > would have the ability to associate receive buffers with<br>
> it.<br>
> > > > Endpoints would need to be associated with an SRQ to make<br>
> use of<br>
> > it.<br>
> > > > 2. Create an SRQ 'endpoint' object. A send-receive<br>
> endpoint<br>
> > could<br>
> > > > be created from and inherent the SRQ interfaces.<br>
> > > > 3. Add an endpoint property to allow sharing data<br>
> buffers.<br>
> > Shared<br>
> > > > buffers could be posted to a domain object, or,<br>
> alternatively,<br>
> > any<br>
> > > endpoint.<br>
> > > ><br>
> > > > Ultimately, the question becomes a matter of where the<br>
> 'post<br>
> > receive<br>
> > > > buffer' operation resides, and the behavior of any 'post<br>
> receive<br>
> > buffer'<br>
> > > > call which may reside elsewhere. E.g. SRQ::PostRecv()<br>
> versus<br>
> > > > EP::PostRecv(), what is the behavior of EP::PostRecv() if<br>
> buffer<br>
> > > > sharing is enabled?<br>
> > > ><br>
> > > > These assume SRQ as a non-transport object, or at least<br>
> one that<br>
> > is<br>
> > > > not visible to the application.<br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > > > Liran mentioned that you wanted me to repeat what I<br>
> said - my<br>
> > only<br>
> > > > > comment was that we not couple transport (connection<br>
> based<br>
> > > > > transport) with tag- matching (or any other object<br>
> supported by<br>
> > > > > the<br>
> > > library).<br>
> > > > > These are two different concepts, and should be kept<br>
> separate.<br>
> > > > ><br>
> > > > ><br>
> > > > ><br>
> > > > > Rich<br>
> ><br>
> > _______________________________________________<br>
> > ofiwg mailing list<br>
> > <a href="mailto:ofiwg@lists.openfabrics.org">ofiwg@lists.openfabrics.org</a><br>
> > <a href="http://lists.openfabrics.org/mailman/listinfo/ofiwg" target="_blank">http://lists.openfabrics.org/mailman/listinfo/ofiwg</a><br>
> ><br>
> ><br>
><br>
><br>
><br>
<br>
</blockquote></div><br></div></div></div>