[Users] APM, LMC, and iSER; oh my!

Thu Apr 7 00:48:22 PDT 2016

IMO, LMC is not a precondition for APM, you can also refer to:
http://downloads.openfabrics.org/Media/Monterey_2012/2012_Workshop_Mon_RDMACM_AMP_Support_Liran_Liss.pdf

This is about the RDMA_CM APM support in kernel side. The code is in
the link I pointed to you before.

2016-04-06 17:47 GMT+02:00 Robert LeBlanc <robert at leblancnet.us>:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Thanks, I'll look into it. Am I correct in assuming the requirement of
> LMC in order to take advantage of APM?
>
> Thanks,
> -----BEGIN PGP SIGNATURE-----
> Version: Mailvelope v1.3.6
> Comment: https://www.mailvelope.com
>
> wsFcBAEBCAAQBQJXBS+FCRDmVDuy+mK58QAAIP4P/R1owJXHyjcwEqVOLtGL
> c9MptjXjO+xb+kDnBm3N3ZVTUQqzfgzitfWvZ2FphGK55S6kqngWglovMM/U
> Kz/e9xnl/IfCZXeipazmmWjP5ZHwkSWNgVolFYoMRu+SBr252d8pw86DA6cl
> 3FZ0JQtUNrtC+UiWzuzmwXXxYv3gMeeKXSH/QggiHzhr+1cnGhZAMtbrHd5O
> 48RLH97bpFiOFKwTQ/syAFAsd0grlNqSvYrkX/fVtdxQMikJj5m9PmFYHFfQ
> TPKCncqT8NgPWDik4138RrvRrhHVK4n5VqNffyNBRTqTsMx/hPqOQTzsJg6v
> OdjXtSV+FD6vwXsxnPFGlTz7uKvu3QpBEOupx9Ux7ij5vLRoHYR7gvm84X4N
> +WdLEyqw5QXS3ysiNZmVn5s9jZz5CqSQ9VMf44i0l9NwWU7TR7pELicnsxes
> b/xOyHwpJldNjKwyrIvFuJ+RcBuRSRGY/ASm2yb/IcRUjfaA4xyU5jnxK5Ul
> LQ2Fk0/Uv3t6ZY71jOMHj3NmiPzahIY56fgkJhSgz7+8tU4h4eg7mIYfgBTk
> kDcqZ2KFPoQ4GuwDdrdW0vfQ27yhGPoOXggLYTqSW8HzJbE97UbrSNCPds6Q
> G0tiGoJhUd3+sdtAQ8j5r1AJGuzrd8HxkouZiA3mpAU8j08yttVe1ZpmdztN
> kI+K
> =OhV6
> -----END PGP SIGNATURE-----
> ----------------
> Robert LeBlanc
> PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1
>
>
> On Wed, Apr 6, 2016 at 2:59 AM, Jack Wang <xjtuwjp at gmail.com> wrote:
>> About APM, Oracle has patch for that, FYI:
>>
>> https://oss.oracle.com/git/?p=linux-uek3-3.8.git;a=commitdiff;h=c8ead45c3e5abed09586a4d51826e429c225cba1
>>
>> 2016-04-05 19:54 GMT+02:00 Robert LeBlanc <robert at leblancnet.us>:
>>> -----BEGIN PGP SIGNED MESSAGE-----
>>> Hash: SHA256
>>>
>>> I was hoping someone here would have had experience with some of the
>>> technologies listed here. Is there a different mailing list that this
>>> question may be more appropriate?
>>>
>>> Thanks,
>>> -----BEGIN PGP SIGNATURE-----
>>> Version: Mailvelope v1.3.6
>>> Comment: https://www.mailvelope.com
>>>
>>> wsFcBAEBCAAQBQJXA/u9CRDmVDuy+mK58QAAwtIP/Rb2daxyGWu8ve0ro2qX
>>> GK4Qog+XMYbn+7GIXQUFATkDzYDdWLL69vO12WbF1I4q5I1b04FXgVktUh/1
>>> unbBKwhByf4XVvmLn+6EdYaZJpn1iyLjxzp8RLJe9wA9NvBb9Fhx/qrWxYOg
>>> 14KB6+ht28US0Gaxcl3epjp/SIGDHQRehbN1WIjffqeE6dPqc9iP5NepCvom
>>> G7sXXoYPo2pSR4IYYBqyaKSi9VX5zKxt898XHPNyf4RS5PG6JsAVkxF4iHIy
>>> c1YFgZ1qSXJ7p/6OFcxz+NBUnFj/uQh/9XbTgpv/RslgEh+S+WxcUl1RukVa
>>> b9yBAWxm0YODvkScgJglVk28lP0p6GULR48llx7GJIRyWC1Sz3Pz3f5p/GuR
>>> XP1oP+hAD7ULEB0Qo5Xr0dLSLR0CFQ2LdNUtGCEdCplavMimmgUajzea2QPo
>>> hOx4DJR2LarOuBTXSur6T8WlxLa7JCWql8eXQAWMTd+uDeIFmm2Tj8+CLXfE
>>> 92Cs7f1UkunhXsswrbzpD07fyH2Tzng8l4I2AOgPYuN6JGT7wVNnXkHOKgPg
>>> ak6AOlSXmPwCmkkKP39h2PwKkLzm3/fywsWStTGiK39i583BGj6jjsDxkmC7
>>> 5vpIXfmIJBhYLf0h6FVloBUCfU0HZx27dfKWkA0kFcsxUmX1q1BOaLuH3lCl
>>> /cHi
>>> =1gyN
>>> -----END PGP SIGNATURE-----
>>> ----------------
>>> Robert LeBlanc
>>> PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1
>>>
>>>
>>> On Wed, Mar 30, 2016 at 1:58 PM, Robert LeBlanc <robert at leblancnet.us> wrote:
>>>> -----BEGIN PGP SIGNED MESSAGE-----
>>>> Hash: SHA256
>>>>
>>>> I've been trying to understand Automatic Path Migration (APM) in order
>>>> to provide extreme stability in our Infiniband fabric[0][1]. My
>>>> research has also lead me to LID Mask Count (LMC) as a possibility for
>>>> providing hot-spot avoidance[2]. In fact it seems like the hot-spot
>>>> avoidance could also provide detection of failed paths and remove them
>>>> from the stripe set, providing benefits of both approaches.
>>>>
>>>> From the articles, it seems that additional code had to be written to
>>>> leverage either of the technologies and was targeted for MPI
>>>> applications. In the APM case three modules: Alternate Path
>>>> Specification module, Path Loading Request module, and Path Migration
>>>> module which handles different aspects of configuring APM and
>>>> controlling the failover/failback of the paths. In the case of [2] a
>>>> shim module was created that performed the striping and consolidating
>>>> of the data across the links.
>>>>
>>>> We'd like to leverage one or both of these features in our
>>>> environment. Enabling LMC in the subnet manager seems pretty straight
>>>> forward, then the application has to leverage the multiple LIDs. I
>>>> can't find any good documentation regarding configuring the
>>>> environment for APM. Based on what I've read, you have to send the
>>>> alternate path to the verbs command to modify the QP, but I'm not sure
>>>> how to get the alternate path to begin with. Could this be another LID
>>>> pair from LMC, does it require a separate PKEY?
>>>>
>>>> In the "APM support for IPoIB" [3] thread, it sounds like APM and
>>>> potentially [2] can't be performed over discrete adapters which would
>>>> be really helpful for us. It sounds like there may have been some
>>>> progress in this area over the last 3.5 years, but I can't seem to
>>>> find anything.
>>>>
>>>> Our primary use is iSER at the moment, and even with three links, we
>>>> have situations where the majority of the paths go through a single
>>>> switch. We would like to have paths forced through different switches
>>>> and balance the loads as much as possible. I'd be grateful for any
>>>> links to documentation or any discussions that will help me get past
>>>> this roadblock in my understanding. We are currently exporting the
>>>> iSER target multiple times and using multipath on the client, but it
>>>> seems Infiniband could do this a lot better.
>>>>
>>>> [0] https://www.researchgate.net/publication/220952412_Automatic_Path_Migration_over_InfiniBand_Early_Experiences
>>>> [1] http://hpc.pnl.gov/people/vishnu/public/vishnu_cluster09.pdf
>>>> [2] http://hpc.pnl.gov/people/vishnu/public/vishnu_ccgrid07.pdf
>>>> [3] http://comments.gmane.org/gmane.linux.drivers.rdma/13529
>>>>
>>>> Thanks,
>>>> - ----------------
>>>> Robert LeBlanc
>>>> PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1
>>>> -----BEGIN PGP SIGNATURE-----
>>>> Version: Mailvelope v1.3.6
>>>> Comment: https://www.mailvelope.com
>>>>
>>>> wsFcBAEBCAAQBQJW/C+iCRDmVDuy+mK58QAAx20QALbKVtHKj7klgPJVMz9w
>>>> GXe+crZRFHqf/xtSfdh9xY9wau8U/zoPu1LDLxhZbtt3QD6evB2tZ9YN/lbD
>>>> /I9oznFHA2+0fWuPSwdptSfxPXIW0L9Ajl5SW59yO06GohAIVR7npqCL+xiL
>>>> fDP5qcYgOLiGGdAZ/QsZHlPW1NrpHIOnxtbNZiFUehhQtXvmWD1P15UPStw8
>>>> U1+PC26DDe0Svk8CPWF79ckfia1FuYqRNFGPJub64WkZpuA8pLjPi2mTnsJo
>>>> 43cibeggx0uE6/EL0XDvqtjOZgFiBvh7wlXqMmHB+cbyepDUaCyt2CSz+nnr
>>>> UXc/Pbx5JB/NfqBmC+fh8z91kLoIh7Q+CTFZGXs3Ho7lush9SOUkNFo2W8sr
>>>> TR5fSoHw1804AqEOX3+9LHfb3m5p4imWxl6uTpdVBW/vWGn5BdAeOFlgndeU
>>>> cu4NwK0JNZDNDw3KKSsr+iQM2+H8vwhV3Ayw4dAL9Uu0ZPku6vp8aFFrpOyY
>>>> MdOVQgUsoXrqmKDgyheILxzpt3kH/E/GK6g5/w6oO5Ohj56x/cRF81ylVCDx
>>>> VnVZeDAFUr9Sr/4fNaHc+HGXC4KjOs5GliEJmXDHZlgAtmAjddhIxDkEWEAX
>>>> 8axiEVpOHwwboZAtYC66AKZEPBZjq+TZjUiuceUxw7dVDoB+tA+ACmLZL39s
>>>> 01nz
>>>> =/M34
>>>> -----END PGP SIGNATURE-----
>>> _______________________________________________
>>> Users mailing list
>>> Users at lists.openfabrics.org
>>> http://lists.openfabrics.org/mailman/listinfo/users