[ofa-general] Re: [PATCH 0/3] ib/ipoib: Enable IPoIB-UD 4K MTU support

[Roland Dreier]

 > The current IPoIB-UD implementation is limited IPoIB payload size to
 > 2048 through hard coding IPOIB_PACKET_SIZE. The implementation is
 > designed for kernel PAGE_SIZE equals or greater than 4K. If the kernel
 > PAGE_SIZE is equals to 2K, memory buffer allocation will be failure when
 > lack of large buffer of memory. However most of the Distros does support
 > PAGE_SIZE >= 4K. So this implementation has no problem for 2048 payload.
 > This implementation is simple but it prevents HCA device who does
 > support 4096 payload from performing, like IBM eHCA2.

Not sure I understand this.  Is there any possible configuration of
any architecture where Linux runs where PAGE_SIZE < 4096?

 > This patch allows IPoIB-UD MTU up to 4092 (4K - IPOIB_ENCAP_LEN) when
 > HCA can support 4K MTU. In this patch, APIs for S/G buffer allocation in
 > IPoIB-CM mode has been made generic so IPoIB-UD and IPoIB-CM can share
 > the S/G code.

This approach seems overly complex to me, since it ends up going
through all the CM buffer fragment bookkeeping for the simple UD path.

However, I now realize that my earlier idea of allocating a scratch
buffer for the GRH and just allocating a 4096 byte skb doesn't work,
because the skb_shinfo ends up being allocated along with the buffer,
so trying to allocate a 4096-byte skb will bloat the data past a
single page, which is what we're trying to avoid.

So how about the following?  When using a UD MTU of 4096 with a page
size of 4096, allocate an skb of size 44 for the GRH and ethertype,
and then allocate a single page for the fragment list.  This means
that the IP packet will start nicely 16-byte aligned for free, and all
the bookkeeping is very simple.

 - R.