[openib-general] Re: [PATCH] [SRP] support for it_iu lengthnegotiation

[Kenneth L Jeffries]

> My objections are the following (as I said in my previous mail):
>  - I don't like allocating a 1 KB IU for every send IU, since most of
>    that memory will probably never be used.
>  - I'm not convinced that it's _ever_ a win to have the target do
>    another RDMA to fetch the indirect buffer list.  You need to
>    convince me that it's not better to simply tell the upper layers
>    what the limit on s/g list length is to fit in the current IU size.

I also don't want to allocate 1KB IU's. If IU's were fixed size, I'd want
(probably, depending on performance testing) a fixed size of 350 bytes
(from Fab Tiller's 64KB i/o, 4KB pages, Windows) or possibly even
the mininum DDBD (as Fab Tiller also says).  1KB IU's with thousands
of RC's causes me a lot of wasted space heartburn.

[as an aside, it sure would be nice if we could do an SRP-3 (since SRP-2
is dead) where multiple direct descriptors would be allowed. The only
way to get multiple descriptors now is with indirect descriptors.]

I am pretty sure that someone doing a video server might want to do, say,
1MB i/o's. 1MB with 4KB pages means 256 descriptors and an iu of
something over 4096 bytes. I definitely don't want to be told by the srp
initiator that I need to use 4KB iu's. (So we agree there.)

Your second point has a couple of parts. On the srp target side, if rmda
reads of additional indirect buffer descriptors is done only 1% of the time
and the trade off is much better memory utilization (ie. smaller iu's) then
from the target's point of view there probably is a big win in doing the
extra descriptor fetches.  

The other side is the number of trips from the application thru the the
scsi and srp layers per i/o. But again, if extra trips are made only 1% of
the time, then my guess is that smaller iu's would be better. 

I do find some appeal in having the internal initiator iu size be able to
be larger (easily) than the on-the-wire iu size. If it were hard to do then
the appeal would not outweigh the cost. 

As long as the target is able to set the iu size and the target can set the iu size
to be fairly small, then I'm ok with just passing that size on to the scsi upper
layer. I'm also ok with a per-connection internal initiator iu size if someone 
wants to code that.

Ken Jeffries