[Users] APM, LMC, and iSER; oh my!

[Robert LeBlanc]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

I was hoping someone here would have had experience with some of the
technologies listed here. Is there a different mailing list that this
question may be more appropriate?

Thanks,
-----BEGIN PGP SIGNATURE-----
Version: Mailvelope v1.3.6
Comment: https://www.mailvelope.com

wsFcBAEBCAAQBQJXA/u9CRDmVDuy+mK58QAAwtIP/Rb2daxyGWu8ve0ro2qX
GK4Qog+XMYbn+7GIXQUFATkDzYDdWLL69vO12WbF1I4q5I1b04FXgVktUh/1
unbBKwhByf4XVvmLn+6EdYaZJpn1iyLjxzp8RLJe9wA9NvBb9Fhx/qrWxYOg
14KB6+ht28US0Gaxcl3epjp/SIGDHQRehbN1WIjffqeE6dPqc9iP5NepCvom
G7sXXoYPo2pSR4IYYBqyaKSi9VX5zKxt898XHPNyf4RS5PG6JsAVkxF4iHIy
c1YFgZ1qSXJ7p/6OFcxz+NBUnFj/uQh/9XbTgpv/RslgEh+S+WxcUl1RukVa
b9yBAWxm0YODvkScgJglVk28lP0p6GULR48llx7GJIRyWC1Sz3Pz3f5p/GuR
XP1oP+hAD7ULEB0Qo5Xr0dLSLR0CFQ2LdNUtGCEdCplavMimmgUajzea2QPo
hOx4DJR2LarOuBTXSur6T8WlxLa7JCWql8eXQAWMTd+uDeIFmm2Tj8+CLXfE
92Cs7f1UkunhXsswrbzpD07fyH2Tzng8l4I2AOgPYuN6JGT7wVNnXkHOKgPg
ak6AOlSXmPwCmkkKP39h2PwKkLzm3/fywsWStTGiK39i583BGj6jjsDxkmC7
5vpIXfmIJBhYLf0h6FVloBUCfU0HZx27dfKWkA0kFcsxUmX1q1BOaLuH3lCl
/cHi
=1gyN
-----END PGP SIGNATURE-----
----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1


On Wed, Mar 30, 2016 at 1:58 PM, Robert LeBlanc <robert at leblancnet.us> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> I've been trying to understand Automatic Path Migration (APM) in order
> to provide extreme stability in our Infiniband fabric[0][1]. My
> research has also lead me to LID Mask Count (LMC) as a possibility for
> providing hot-spot avoidance[2]. In fact it seems like the hot-spot
> avoidance could also provide detection of failed paths and remove them
> from the stripe set, providing benefits of both approaches.
>
> From the articles, it seems that additional code had to be written to
> leverage either of the technologies and was targeted for MPI
> applications. In the APM case three modules: Alternate Path
> Specification module, Path Loading Request module, and Path Migration
> module which handles different aspects of configuring APM and
> controlling the failover/failback of the paths. In the case of [2] a
> shim module was created that performed the striping and consolidating
> of the data across the links.
>
> We'd like to leverage one or both of these features in our
> environment. Enabling LMC in the subnet manager seems pretty straight
> forward, then the application has to leverage the multiple LIDs. I
> can't find any good documentation regarding configuring the
> environment for APM. Based on what I've read, you have to send the
> alternate path to the verbs command to modify the QP, but I'm not sure
> how to get the alternate path to begin with. Could this be another LID
> pair from LMC, does it require a separate PKEY?
>
> In the "APM support for IPoIB" [3] thread, it sounds like APM and
> potentially [2] can't be performed over discrete adapters which would
> be really helpful for us. It sounds like there may have been some
> progress in this area over the last 3.5 years, but I can't seem to
> find anything.
>
> Our primary use is iSER at the moment, and even with three links, we
> have situations where the majority of the paths go through a single
> switch. We would like to have paths forced through different switches
> and balance the loads as much as possible. I'd be grateful for any
> links to documentation or any discussions that will help me get past
> this roadblock in my understanding. We are currently exporting the
> iSER target multiple times and using multipath on the client, but it
> seems Infiniband could do this a lot better.
>
> [0] https://www.researchgate.net/publication/220952412_Automatic_Path_Migration_over_InfiniBand_Early_Experiences
> [1] http://hpc.pnl.gov/people/vishnu/public/vishnu_cluster09.pdf
> [2] http://hpc.pnl.gov/people/vishnu/public/vishnu_ccgrid07.pdf
> [3] http://comments.gmane.org/gmane.linux.drivers.rdma/13529
>
> Thanks,
> - ----------------
> Robert LeBlanc
> PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1
> -----BEGIN PGP SIGNATURE-----
> Version: Mailvelope v1.3.6
> Comment: https://www.mailvelope.com
>
> wsFcBAEBCAAQBQJW/C+iCRDmVDuy+mK58QAAx20QALbKVtHKj7klgPJVMz9w
> GXe+crZRFHqf/xtSfdh9xY9wau8U/zoPu1LDLxhZbtt3QD6evB2tZ9YN/lbD
> /I9oznFHA2+0fWuPSwdptSfxPXIW0L9Ajl5SW59yO06GohAIVR7npqCL+xiL
> fDP5qcYgOLiGGdAZ/QsZHlPW1NrpHIOnxtbNZiFUehhQtXvmWD1P15UPStw8
> U1+PC26DDe0Svk8CPWF79ckfia1FuYqRNFGPJub64WkZpuA8pLjPi2mTnsJo
> 43cibeggx0uE6/EL0XDvqtjOZgFiBvh7wlXqMmHB+cbyepDUaCyt2CSz+nnr
> UXc/Pbx5JB/NfqBmC+fh8z91kLoIh7Q+CTFZGXs3Ho7lush9SOUkNFo2W8sr
> TR5fSoHw1804AqEOX3+9LHfb3m5p4imWxl6uTpdVBW/vWGn5BdAeOFlgndeU
> cu4NwK0JNZDNDw3KKSsr+iQM2+H8vwhV3Ayw4dAL9Uu0ZPku6vp8aFFrpOyY
> MdOVQgUsoXrqmKDgyheILxzpt3kH/E/GK6g5/w6oO5Ohj56x/cRF81ylVCDx
> VnVZeDAFUr9Sr/4fNaHc+HGXC4KjOs5GliEJmXDHZlgAtmAjddhIxDkEWEAX
> 8axiEVpOHwwboZAtYC66AKZEPBZjq+TZjUiuceUxw7dVDoB+tA+ACmLZL39s
> 01nz
> =/M34
> -----END PGP SIGNATURE-----