<div dir="ltr">Hi Sean,<div><br></div><div>No I would not create an RMA class. It seems to me like RMA is a transport method, and as</div><div>such should be a method of the ep class. What I didn't like about the receive buffer method</div>
<div>(with or without tags actually since the IB SRQ capability also falls under this umbrella) was</div><div>that it seemed asymmetric. For FID_MSG type EP, with the current proposal, there is</div><div>no ability to allow receive buffers to be posted that could "match" incoming sends from</div>
<div>a set of EP's. Here I mean match in the general sense. Only if a vendor supported </div><div>FID_RDM/FID_DGRAM could this be done as the proposal currently stands.</div><div><br></div><div>I think I'm restating what Rich said.</div>
<div><br></div><div>Howard</div><div><br></div><div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, May 22, 2014 at 1:29 PM, Hefty, Sean <span dir="ltr"><<a href="mailto:sean.hefty@intel.com" target="_blank">sean.hefty@intel.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Thanks, Howard, this is helpful.<br>
<br>
Regarding the 'tag match class' that you mention, would you create an 'rma class' as a peer, with the RMA operations defined in a similar fashion? If not, why not? Would this also extend to all other data transfer operations? I.e. message queue (send/receive) and atomics, plus any others defined in the future?<br>
<br>
<br>
> -----Original Message-----<br>
> From: Howard Pritchard [mailto:<a href="mailto:hppritcha@gmail.com">hppritcha@gmail.com</a>]<br>
> Sent: Thursday, May 22, 2014 11:53 AM<br>
> To: Richard Graham<br>
> Cc: Hefty, Sean; <a href="mailto:ofiwg@lists.openfabrics.org">ofiwg@lists.openfabrics.org</a>; ofiwg-<br>
> <a href="mailto:mpi@lists.openfabrics.org">mpi@lists.openfabrics.org</a><br>
> Subject: Re: [ofiwg] Call today<br>
><br>
> Hi Folks,<br>
><br>
> here is a diagram of a concept that was discussed in a side conversation at<br>
> the last OFA workshop. I'd thought that a msgq (aka tag matcher class)<br>
> object<br>
> should be instantiated via a method of the fabric class.<br>
><br>
> red lines in the diagram indicated the pointee can be associated with the<br>
> class<br>
> being pointed to by the arrow, using the bind method of the class being<br>
> pointed<br>
> to.<br>
><br>
> the search_by_addr method of the msgq is for use with FID_RDM endpoints,<br>
> while search_by_ep method is when the msgq is associated with multipled<br>
> FID_MSG type endpoints.<br>
><br>
> Note the slide is a little old since the EC class has been divided now into<br>
> a EQ and counter type completion notification mechanisms.<br>
><br>
> Hoping this will maybe help a little here.<br>
><br>
> Howard<br>
><br>
><br>
><br>
> On Thu, May 22, 2014 at 11:59 AM, Richard Graham <<a href="mailto:richardg@mellanox.com">richardg@mellanox.com</a>><br>
> wrote:<br>
><br>
><br>
> Please see inline<br>
><br>
> -----Original Message-----<br>
> From: Hefty, Sean [mailto:<a href="mailto:sean.hefty@intel.com">sean.hefty@intel.com</a>]<br>
> Sent: Thursday, May 22, 2014 12:43 PM<br>
> To: Richard Graham; <a href="mailto:ofiwg@lists.openfabrics.org">ofiwg@lists.openfabrics.org</a>; ofiwg-<br>
> <a href="mailto:mpi@lists.openfabrics.org">mpi@lists.openfabrics.org</a><br>
> Cc: Paul Grun (<a href="mailto:grun@cray.com">grun@cray.com</a>); Liran Liss<br>
> Subject: RE: Call today<br>
><br>
> With permission, copying mailing list on side thread that popped up.<br>
><br>
> I understand MPI has wild card receives. But tagged semantics are<br>
> useful even when associated with a generic endpoint concept, or a specific<br>
> address. Note the proposed endpoint concept is not necessarily bound to a<br>
> specific piece of hardware, though it may be based on the provider<br>
> implementation. The tagged operations themselves may be implemented by<br>
> hardware and are not restricted to being purely a software construct.<br>
> [rich] If the attempt here is to provide a building block that will<br>
> map to different use-case scenarios, then need to have an architecture that<br>
> will map well onto the areas of interest. MPI is just one such upper level<br>
> service, one that has been called out specifically in the context of the<br>
> proposal you have been presenting. So, following on this (the precise<br>
> definition of end point is still rather fuzzy at this stage) in general,<br>
> there is no such one-to-one mapping of and endpoint to an MPI matching<br>
> context, but there can be an association of a matching context with one or<br>
> more endpoints. What I am suggesting here is that we keep data notions<br>
> around data transfer orthogonal to what is done with the data (tag<br>
> matching, in this case). How the functionality is implemented (hardware<br>
> or not) is separate from how the stack in architected<br>
><br>
> Tagged interfaces, as well as other interfaces such as message<br>
> queues, may still exist above the endpoint. But that layering of<br>
> interfaces seems better suited above the fabric interfaces (e.g. MPI),<br>
> rather than included with it. This seems more debatable to me though, and<br>
> we could examine whether a domain or fabric object should have send/receive<br>
> capabilities.<br>
> [rich] Need to keep separate how data is transferred (perhaps with<br>
> functions that we may call send/recv) from the ULP's use of this data<br>
> (perhaps also using the a similar naming scheme of send/recv).<br>
><br>
> - Sean<br>
><br>
> > -----Original Message-----<br>
> > From: Richard Graham [mailto:<a href="mailto:richardg@mellanox.com">richardg@mellanox.com</a>]<br>
> > Sent: Wednesday, May 21, 2014 11:09 AM<br>
> > To: Hefty, Sean<br>
> > Cc: Paul Grun (<a href="mailto:grun@cray.com">grun@cray.com</a>); Liran Liss<br>
> > Subject: RE: Call today<br>
> ><br>
> > Tag matching as it comes to MPI semantics is not local to a given<br>
> pair<br>
> > of processes, e.g. MPI has a wild card receive that can take data<br>
> from<br>
> > any source, and therefore the matching context is broader than just<br>
> a<br>
> > single pair of source and destination.<br>
> ><br>
> > Rich<br>
> ><br>
> > -----Original Message-----<br>
> > From: Hefty, Sean [mailto:<a href="mailto:sean.hefty@intel.com">sean.hefty@intel.com</a>]<br>
> > Sent: Wednesday, May 21, 2014 1:13 PM<br>
> > To: Richard Graham<br>
> > Cc: Paul Grun (<a href="mailto:grun@cray.com">grun@cray.com</a>); Liran Liss<br>
> > Subject: RE: Call today<br>
> ><br>
> > Tag matching, RMA, atomics, and message operations are currently<br>
> > associated with an endpoint, but the functions are independent of<br>
> the<br>
> > communication protocol in use. Conceptually, it seems reasonable<br>
> to<br>
> > think of tag matching as a merging of message and RMA write<br>
> operations.<br>
> ><br>
> > I agree that an endpoint is associated with the data source/sink.<br>
> > There is no implied mapping between a process and an endpoint.<br>
> ><br>
> ><br>
> > > -----Original Message-----<br>
> > > From: Richard Graham [mailto:<a href="mailto:richardg@mellanox.com">richardg@mellanox.com</a>]<br>
> > > Sent: Tuesday, May 20, 2014 9:22 PM<br>
> > > To: Hefty, Sean<br>
> > > Cc: Paul Grun (<a href="mailto:grun@cray.com">grun@cray.com</a>); Liran Liss<br>
> > > Subject: RE: Call today<br>
> > ><br>
> > > I suppose that you could consider tag-matching as part of<br>
> transport.<br>
> > > However, I would argue that such protocols should be independent<br>
> of<br>
> > > whether or not a reliable or unreliable communication protocol is<br>
> > > used<br>
> > (at least<br>
> > > when it comes to the tag support needed for MPI). Also, I<br>
> associate an<br>
> > > end-point with either the source and/or the sync of data. In MPI<br>
> > > tag matching is associated with mpi-level (process,communicator)<br>
> > > pair, and therefore the tag-matching context may be associated<br>
> with<br>
> > > many end-<br>
> > points.<br>
> > > I would therefore keep tag-matching as a separate concept.<br>
> > ><br>
> > > Rich<br>
> > ><br>
> > > -----Original Message-----<br>
> > > From: Hefty, Sean [mailto:<a href="mailto:sean.hefty@intel.com">sean.hefty@intel.com</a>]<br>
> > > Sent: Tuesday, May 20, 2014 1:26 PM<br>
> > > To: Richard Graham<br>
> > > Cc: Paul Grun (<a href="mailto:grun@cray.com">grun@cray.com</a>); Liran Liss<br>
> > > Subject: RE: Call today<br>
> > ><br>
> > > Tag-matching is a transport object (protocol), so I do think it<br>
> > > makes sense being associated with a transport level object (i.e.<br>
> endpoint).<br>
> > ><br>
> > > I thought you were referring to the SRQ, which may or may not be<br>
> a<br>
> > > transport level object. If the sharing of data buffer(s) among<br>
> > > multiple connections is not considered a transport object, then I<br>
> > > agree, it may make sense to have it be a separate object with its<br>
> > > own<br>
> > interfaces.<br>
> > > Alternatively, it could also be a property of endpoints to share<br>
> > > receive buffers.<br>
> > ><br>
> > > When the SRQ appears in the transport object (protocol), it may<br>
> get<br>
> > > more complex.<br>
> > ><br>
> > > For initial thoughts, sharing receive buffers could be handled<br>
> by:<br>
> > ><br>
> > > 1. Creating an explicit SRQ object as a 'peer' to an endpoint.<br>
> SRQ<br>
> > > would have the ability to associate receive buffers with it.<br>
> > > Endpoints would need to be associated with an SRQ to make use of<br>
> it.<br>
> > > 2. Create an SRQ 'endpoint' object. A send-receive endpoint<br>
> could<br>
> > > be created from and inherent the SRQ interfaces.<br>
> > > 3. Add an endpoint property to allow sharing data buffers.<br>
> Shared<br>
> > > buffers could be posted to a domain object, or, alternatively,<br>
> any<br>
> > endpoint.<br>
> > ><br>
> > > Ultimately, the question becomes a matter of where the 'post<br>
> receive<br>
> > > buffer' operation resides, and the behavior of any 'post receive<br>
> buffer'<br>
> > > call which may reside elsewhere. E.g. SRQ::PostRecv() versus<br>
> > > EP::PostRecv(), what is the behavior of EP::PostRecv() if buffer<br>
> > > sharing is enabled?<br>
> > ><br>
> > > These assume SRQ as a non-transport object, or at least one that<br>
> is<br>
> > > not visible to the application.<br>
> > ><br>
> > ><br>
> > ><br>
> > > > Liran mentioned that you wanted me to repeat what I said - my<br>
> only<br>
> > > > comment was that we not couple transport (connection based<br>
> > > > transport) with tag- matching (or any other object supported by<br>
> > > > the<br>
> > library).<br>
> > > > These are two different concepts, and should be kept separate.<br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > > Rich<br>
><br>
> _______________________________________________<br>
> ofiwg mailing list<br>
> <a href="mailto:ofiwg@lists.openfabrics.org">ofiwg@lists.openfabrics.org</a><br>
> <a href="http://lists.openfabrics.org/mailman/listinfo/ofiwg" target="_blank">http://lists.openfabrics.org/mailman/listinfo/ofiwg</a><br>
><br>
><br>
<br>
</blockquote></div><br></div></div></div>