[libfabric-users] Two issues while using  libfabric

jin.hao1 at zte.com.cn jin.hao1 at zte.com.cn
Tue Nov 17 18:12:27 PST 2020


Our team have met several problems while developing network system on Mellanox ConnectX using libfabric-1.7.0-1.el7.

We would appreciate your advices on these issues.

A, B, and C three nodes build links with each other, each node works as both a fabric client and a fabric server.For example, Node A had 2 RDMA links : A-->B and  C-->A.  And  we had met two problems while using  libfabric:

1. Multiple nodes build links in pairs, how to register memory to fi_write to different nodes?

we found that these links had different fi_info and fi_domain, so the shared data must be registered onto both links, and each link had it's own key/desc for the same memory, which maked app much more complicated. This is unacceptable.  We believed that Links  on the same RNIC should share the same domain.  

https://github.com/ofiwg/libfabric/issues/6259 )

2. All the  nodes have same same hardware and same Centos os,  some nodes fi_mr_reg failed with error -12, while others worked OK.

We found it failed when libfabric using the fi_ibv_mr_cache_ops mode and it worked ok when using fi_ibv_mr_ops.  We don't know how to resolve it.

Is there any documents or manuals  about these modes ?  How can we confirm it's working  mode ?

Thank you for your assistance.

金浩 jinhao

 IT Development

Data Intelligence Platform Development Dept.II/Central R&D Institute/System Product

4/F, Building No. 3, ZTE Corporation, Yuhuatai District,

Nanjing, P.R.China, 210012

T: +86 25-88014501

M: +86 13611591690
E: jin.hao1 at zte.com.cn 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/libfabric-users/attachments/20201118/4ddf6dc1/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: zMail_snapScreen_tmp.jpg
Type: image/jpeg
Size: 68237 bytes
Desc: not available
URL: <http://lists.openfabrics.org/pipermail/libfabric-users/attachments/20201118/4ddf6dc1/attachment-0003.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 24242e5637af428891c4db731e7765ad.jpg
Type: image/jpeg
Size: 2064 bytes
Desc: not available
URL: <http://lists.openfabrics.org/pipermail/libfabric-users/attachments/20201118/4ddf6dc1/attachment-0004.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 9ae3e214c17d49ed935d87c674ba3ee2.jpg
Type: image/jpeg
Size: 6015 bytes
Desc: not available
URL: <http://lists.openfabrics.org/pipermail/libfabric-users/attachments/20201118/4ddf6dc1/attachment-0005.jpg>

More information about the Libfabric-users mailing list