[ofa-general] Re: [PATCH 0/3] ib/ipoib: Enable IPoIB-UD 4K MTU support
Roland Dreier
rdreier at cisco.com
Fri Feb 1 15:35:23 PST 2008
> The current IPoIB-UD implementation is limited IPoIB payload size to
> 2048 through hard coding IPOIB_PACKET_SIZE. The implementation is
> designed for kernel PAGE_SIZE equals or greater than 4K. If the kernel
> PAGE_SIZE is equals to 2K, memory buffer allocation will be failure when
> lack of large buffer of memory. However most of the Distros does support
> PAGE_SIZE >= 4K. So this implementation has no problem for 2048 payload.
> This implementation is simple but it prevents HCA device who does
> support 4096 payload from performing, like IBM eHCA2.
Not sure I understand this. Is there any possible configuration of
any architecture where Linux runs where PAGE_SIZE < 4096?
> This patch allows IPoIB-UD MTU up to 4092 (4K - IPOIB_ENCAP_LEN) when
> HCA can support 4K MTU. In this patch, APIs for S/G buffer allocation in
> IPoIB-CM mode has been made generic so IPoIB-UD and IPoIB-CM can share
> the S/G code.
This approach seems overly complex to me, since it ends up going
through all the CM buffer fragment bookkeeping for the simple UD path.
However, I now realize that my earlier idea of allocating a scratch
buffer for the GRH and just allocating a 4096 byte skb doesn't work,
because the skb_shinfo ends up being allocated along with the buffer,
so trying to allocate a 4096-byte skb will bloat the data past a
single page, which is what we're trying to avoid.
So how about the following? When using a UD MTU of 4096 with a page
size of 4096, allocate an skb of size 44 for the GRH and ethertype,
and then allocate a single page for the fragment list. This means
that the IP packet will start nicely 16-byte aligned for free, and all
the bookkeeping is very simple.
- R.
More information about the general
mailing list