[openib-general] [PATCH][iWARP] Added provider CM verbsandquery provider methods

[Caitlin Bestler]

> > Linux already have the infrastructure for zero-copy send, with some 
> > hardware help it is possible to implement zero-copy receive too.
> > Moving data in memory is out of the question.
> > 
> > Anyway I think this questions should be answered before moving this 
> > discussion to netdev.
> 
> I don't know how to do it and I've thought quite a bit about it. 
> 
> If you want the semantics specified for RDDP over a reliable 
> transport, then you need to do TCP in hardware. 
> Consider RDMA_WRITE. The target buffer is specified in the 
> RDDP header which is in turn carried as part of the TCP payload. 
> Therefore in order for the hardware to get to the RDDP 
> header, it must crack the TCP header. The TCP processing 
> can't be stateless because the hardware must discard things 
> such as duplicates or risk overwriting an application buffer 
> that has already been written, delivered, and reused for some 
> other purpose. There are many other examples, but the point 
> is TCP state information is needed by the hardware.
> 

I fully agree with this analysis. 

To go further, if the receive buffer is specified by *either*
RDMA or  TCP semantics then you cannot place directly to that
buffer from the hardware unless the hardware owns the TCP
connection.

Doing zero-copy at the Ethernet or IP layer does not accomplish
anything -- the application is thinking TCP or RDMA.

It is also important to remember that there are multiple aspects
to  how RDMA and/or a QP/CQ interface improves efficiency. Placing
directly from the raw receive buffer to the user's buffer without
requiring a context switch is only part of it. The elimination of
excess notifications to the user is equally vital. Any stateless
design will not only target the wrong buffers, it will have no idea
as to when the application really needs to be woken up.

This is not really the right forum to discuss whether these hardware
enhancements are valuable. If they were not valuable companies would
not be building them. Building products presupposes end customers
that are willing to pay for those products. If there are no customers
then these features will go away no matter what is decided here. If
the customers want them they will find a way to deploy the hardware.
So the question is how to integrate these features in a way that
preserve the authority of the main stack in defending against attacks,
regulating traffic, etc. The alternative is not for hardware offload
to go away, but for it not to be integrated. I hope we can all agree
that the latter is not a desirable outcome.