<div>Hi Robert</div><div><br></div><div>For the current OFI MTL inside Open MPI a provider needs to support FI_EP_RDM. </div><div><br></div><div>Howard</div><div><br><div class="gmail_quote"><div>Jeff Squyres (jsquyres) <<a href="mailto:jsquyres@cisco.com">jsquyres@cisco.com</a>> schrieb am Mi. 8. Feb. 2017 um 06:41:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">+1 -- each MPI is different.<br class="gmail_msg">
<br class="gmail_msg">
Open MPI nominally supports the same Libfabric tag matching interfaces as MPICH/ch4, but the exact requirements may be slightly different (I don't know offhand).<br class="gmail_msg">
<br class="gmail_msg">
Additionally, some vendors have chosen to support different Libfabric interfaces in Open MPI. For example, Cisco supports its usNIC hardware with the EP_DGRAM Libfabric interfaces.<br class="gmail_msg">
<br class="gmail_msg">
<br class="gmail_msg">
> On Feb 7, 2017, at 6:22 PM, Blocksome, Michael <<a href="mailto:michael.blocksome@intel.com" class="gmail_msg" target="_blank">michael.blocksome@intel.com</a>> wrote:<br class="gmail_msg">
><br class="gmail_msg">
> For compiling MPICH/CH4/OFI the pertinent ofi features required of the provider are specified via “capability sets” (basically a collection of ofi features).<br class="gmail_msg">
><br class="gmail_msg">
> <a href="https://github.com/pmodels/mpich/blob/master/src/mpid/ch4/netmod/ofi/ofi_capability_sets.h" rel="noreferrer" class="gmail_msg" target="_blank">https://github.com/pmodels/mpich/blob/master/src/mpid/ch4/netmod/ofi/ofi_capability_sets.h</a><br class="gmail_msg">
><br class="gmail_msg">
> /*<br class="gmail_msg">
> * The definitions map to these capability sets:<br class="gmail_msg">
> *<br class="gmail_msg">
> * MPIDI_OFI_ENABLE_DATA fi_tsenddata (and other functions with immediate data)<br class="gmail_msg">
> * Uses FI_REMOTE_CQ_DATA, FI_DIRECTED_RECV<br class="gmail_msg">
> * MPIDI_OFI_ENABLE_AV_TABLE Use FI_AV_TABLE instead of FI_AV_MAP<br class="gmail_msg">
> * MPIDI_OFI_ENABLE_SCALABLE_ENDPOINTS fi_scalable_ep instead of fi_ep<br class="gmail_msg">
> * domain_attr.max_ep_tx_ctx > 1<br class="gmail_msg">
> * MPIDI_OFI_ENABLE_STX_RMA Use shared transmit contexts for RMA<br class="gmail_msg">
> * Uses FI_SHARED_CONTEXT<br class="gmail_msg">
> * MPIDI_OFI_ENABLE_MR_SCALABLE Use FI_MR_SCALABLE instead of FI_MR_BASIC<br class="gmail_msg">
> * If using runtime mode, this will be set to FI_MR_UNSPEC<br class="gmail_msg">
> * MPIDI_OFI_ENABLE_TAGGED Use FI_TAGGED interface instead of FI_MSG<br class="gmail_msg">
> * MPIDI_OFI_ENABLE_AM Use FI_MSG and FI_MULTI_RECV for active messages<br class="gmail_msg">
> * MPIDI_OFI_ENABLE_RMA Use FI_ATOMICS and FI_RMA interfaces<br class="gmail_msg">
> * MPIDI_OFI_FETCH_ATOMIC_IOVECS The maximum number of iovecs that can be used for fetch_atomic operations<br class="gmail_msg">
> */<br class="gmail_msg">
><br class="gmail_msg">
> most things you can turn off, but some stuff is required. You need to support one of the two address vector types, a memory region type (if rma and/or atomics), etc. In general, you need “FI_MSG and FI_MULTI_RECV” to get MPICH bootstrapped, the rest is sugar.<br class="gmail_msg">
><br class="gmail_msg">
> Of course all MPIs are different … this is just for MPICH.<br class="gmail_msg">
><br class="gmail_msg">
> mike.<br class="gmail_msg">
><br class="gmail_msg">
> -----Original Message-----<br class="gmail_msg">
> From: Libfabric-users [mailto:<a href="mailto:libfabric-users-bounces@lists.openfabrics.org" class="gmail_msg" target="_blank">libfabric-users-bounces@lists.openfabrics.org</a>] On Behalf Of Hefty, Sean<br class="gmail_msg">
> Sent: Tuesday, February 7, 2017 5:18 PM<br class="gmail_msg">
> To: Robert Cauble <<a href="mailto:rcauble@google.com" class="gmail_msg" target="_blank">rcauble@google.com</a>><br class="gmail_msg">
> Cc: <a href="mailto:ofiwg@lists.openfabrics.org" class="gmail_msg" target="_blank">ofiwg@lists.openfabrics.org</a>; <a href="mailto:libfabric-users@lists.openfabrics.org" class="gmail_msg" target="_blank">libfabric-users@lists.openfabrics.org</a><br class="gmail_msg">
> Subject: Re: [libfabric-users] provider developer documentation<br class="gmail_msg">
><br class="gmail_msg">
>> You mention "minimal amount of the API" -- I would like to do that for<br class="gmail_msg">
>> starters. Are there guidelines WRT which operations/features are<br class="gmail_msg">
>> required by various MPI implementations and implications for not<br class="gmail_msg">
>> supporting them (loss of functionality vs loss of performance?)<br class="gmail_msg">
><br class="gmail_msg">
> I think this greatly depends on the MPI. :) The target minimal amount of work depends on your hardware. The expectation is that the utility layers will provide the rest. Note that the utility layers are designed for performance, just may not be optimal for the underlying hardware. (We are actively working on the utility code, so fully drop in support isn't there yet.)<br class="gmail_msg">
><br class="gmail_msg">
> If your target HW just sends/receives packets, you can aim for the functionality supported by the UDP provider -- DGRAM EPs with simple send/receive support. If your target HW works best with connection-oriented communication, I would mimic the verbs provider -- MSG EPs with RMA and shared receive contexts. For reliable-unconnected hardware, I would implement support for the tagged interfaces first.<br class="gmail_msg">
><br class="gmail_msg">
> I'm not familiar enough with the MPI implementations to know what features are optional versus required.<br class="gmail_msg">
> _______________________________________________<br class="gmail_msg">
> Libfabric-users mailing list<br class="gmail_msg">
> <a href="mailto:Libfabric-users@lists.openfabrics.org" class="gmail_msg" target="_blank">Libfabric-users@lists.openfabrics.org</a><br class="gmail_msg">
> <a href="http://lists.openfabrics.org/mailman/listinfo/libfabric-users" rel="noreferrer" class="gmail_msg" target="_blank">http://lists.openfabrics.org/mailman/listinfo/libfabric-users</a><br class="gmail_msg">
> _______________________________________________<br class="gmail_msg">
> ofiwg mailing list<br class="gmail_msg">
> <a href="mailto:ofiwg@lists.openfabrics.org" class="gmail_msg" target="_blank">ofiwg@lists.openfabrics.org</a><br class="gmail_msg">
> <a href="http://lists.openfabrics.org/mailman/listinfo/ofiwg" rel="noreferrer" class="gmail_msg" target="_blank">http://lists.openfabrics.org/mailman/listinfo/ofiwg</a><br class="gmail_msg">
<br class="gmail_msg">
<br class="gmail_msg">
--<br class="gmail_msg">
Jeff Squyres<br class="gmail_msg">
<a href="mailto:jsquyres@cisco.com" class="gmail_msg" target="_blank">jsquyres@cisco.com</a><br class="gmail_msg">
<br class="gmail_msg">
_______________________________________________<br class="gmail_msg">
Libfabric-users mailing list<br class="gmail_msg">
<a href="mailto:Libfabric-users@lists.openfabrics.org" class="gmail_msg" target="_blank">Libfabric-users@lists.openfabrics.org</a><br class="gmail_msg">
<a href="http://lists.openfabrics.org/mailman/listinfo/libfabric-users" rel="noreferrer" class="gmail_msg" target="_blank">http://lists.openfabrics.org/mailman/listinfo/libfabric-users</a><br class="gmail_msg">
</blockquote></div></div>