[libfabric-users] Two issues while using libfabric

[jin.hao1 at zte.com.cn]

Hi,Sir


Our team have met several problems while developing network system on Mellanox ConnectX using libfabric-1.7.0-1.el7.


We would appreciate your advices on these issues.





A, B, and C three nodes build links with each other, each node works as both a fabric client and a fabric server.For example, Node A had 2 RDMA links : A-->B and  C-->A.  And  we had met two problems while using  libfabric:


1. Multiple nodes build links in pairs, how to register memory to fi_write to different nodes?


we found that these links had different fi_info and fi_domain, so the shared data must be registered onto both links, and each link had it's own key/desc for the same memory, which maked app much more complicated. This is unacceptable.  We believed that Links  on the same RNIC should share the same domain.  


https://github.com/ofiwg/libfabric/issues/6259 )








2. All the  nodes have same same hardware and same Centos os,  some nodes fi_mr_reg failed with error -12, while others worked OK.


We found it failed when libfabric using the fi_ibv_mr_cache_ops mode and it worked ok when using fi_ibv_mr_ops.  We don't know how to resolve it.


Is there any documents or manuals  about these modes ?  How can we confirm it's working  mode ?






Thank you for your assistance.










金浩 jinhao


 IT Development
Engineer



Data Intelligence Platform Development Dept.II/Central R＆D Institute/System Product






南京市雨花台区软件大道50号南京研发中心二区3号楼4楼 
4/F, Building No. 3, ZTE Corporation, Yuhuatai District,


Nanjing, P.R.China, 210012

T: +86 25-88014501

M: +86 13611591690
E: jin.hao1 at zte.com.cn 
www.zte.com.cn
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/libfabric-users/attachments/20201118/4ddf6dc1/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: zMail_snapScreen_tmp.jpg
Type: image/jpeg
Size: 68237 bytes
Desc: not available
URL: <http://lists.openfabrics.org/pipermail/libfabric-users/attachments/20201118/4ddf6dc1/attachment-0003.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 24242e5637af428891c4db731e7765ad.jpg
Type: image/jpeg
Size: 2064 bytes
Desc: not available
URL: <http://lists.openfabrics.org/pipermail/libfabric-users/attachments/20201118/4ddf6dc1/attachment-0004.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 9ae3e214c17d49ed935d87c674ba3ee2.jpg
Type: image/jpeg
Size: 6015 bytes
Desc: not available
URL: <http://lists.openfabrics.org/pipermail/libfabric-users/attachments/20201118/4ddf6dc1/attachment-0005.jpg>