[ofa-general] Re: [PATCH 0/8 v3] RDMAoE support

[Or Gerlitz]

Liran Liss wrote:
> I think that not having a user-space SA API is a big hole - we should not accept this situation and work to fill in the gap. Once such an API exists, all non-rdmacm apps will probably use it - implementing path queries and multicast joins is a tedious effort, as
> the convergence in the kernel has shown. 

In short: no (big hole) and no (all apps will run to use it). In more 
details, if its a big hole, how come almost no one asked for it? next, 
many (most, maybe all) of the MPI people claim that their biggest/sole 
problem with the rdma-cm is the non scalability of the SA being their 
bottleneck (personally, I am not convinced the SA is the bottleneck) and 
as such they are not going in that direction. So we are still remain 
with SRP, does Mellanox see SRP deployments running over rdmaoe?

> this is still TBD. We will need to define reasonable policies to integrate with DCB features, such as PFC. Any ideas on this are more than welcome!
In your previous post "the rmdaoa "sa" code does more than just path 
resolution - it provides SL, MTU, rate, packet lifetime, and other 
params." you were using present so I'm not sure how I was supposed to 
guess that this is TBD. On the other hand, thinking about the SA thing a 
bit further, this is something to take care of.

Basically, I understand that the idea is to replace LRH/GRH/BTH/ETH with 
something like MAC/VLAN/GRH/BTH/ETH - so SL is PCP, PKEY is VLAN (here 
we are hit directly by the IB bug of putting the pkey in the L4 header). 
Note that such mechanics (e.g PCP, VLAN, and DCBX related things) can be 
implemented more naturally in the rdma-cm layer

> We understood that most people want to see this separated until the standard is finalized. In the meantime, we will try to reduce code duplication as much as possible.

its open source, either send me pointer to such postings, or tell them 
to comment on the list


> not at all. Its just a matter of what comes first - implementation is ongoing

so this path-set doesn't play with e.g RDS using IPv4 addresses? I 
understood it does, so please clarify.

> the only API changes are the sa-like path-queries and multicast joins; verbs are not changed in any way. In addition, we implemented the 'get_mac' method (for translating gids to macs) in the generic user-kernel ABI so any RDMAoE implementation can use it. I don't understand what concerns you; please explain.
Any change to ib_verbs.h is an API change, and indeed there were few 
comments from Woody and others. you were commenting back, etc. What I'm 
saying is that  I'd like to see the group who works on the software 
rdmaoe driver commenting too.

Or.