[ofa-general] OFED SRP Client / StorageGear Target / Performance with Modified Write Protocol

[Ken Jeffries]

Our implicit assumption has been that since T10 abandoned SRP 2 that the
T10/SRP community had little interest in SRP enhancements. If there is
IB/SRP community interest we would certainly persue a T10 project of
some sort. If StorageGear is the only interested party, then not so much.

Our general target is small clusters that benefit from SSDs. An SSD takes
advantage of IB much more fully than an IB RAID box does. Since we support
up to 4 hca's, we envision up to 8 system clusters that use our system
without needing an IB switch. Enabling low cost switchless clusters doesn't
sell many switches directly but enabling low cost IB does help the IB
market in general. 

We think these clusters will want to do random "small" i/o's and that "small"
will almost always be larger than 512 bytes. 

Yes we use giant IUs to be able to hold at least one block. "giant" is 
relative though. Using srp_sg_tablesize=255 results in an IU of 4148
bytes which is plenty to hold one 4096 byte block. Since our motherboard
supports up to 64GB, the overhead of the large IU's is a non-issue for
us. Of course the client only transmits the used portion of the IU so
non-iu-data-writes remain small on the wire. The client side code 
simply uses an additional s/g entry passed to the IB layer so no client
side copy is done.

Ken

-----Original Message-----
From: Roland Dreier [mailto:rdreier at cisco.com] 
Sent: Monday, July 30, 2007 4:38 PM
To: Ken Jeffries
Cc: general at lists.openfabrics.org
Subject: Re: [ofa-general] OFED SRP Client / StorageGear Target / Performance with Modified Write Protocol


 > We have been doing a fair amount of performance testing on our SRP target.
 > One thing we found early on was that client writes were considerably slower
 > than client reads. We addressed this by patching the SRP client code so
 > that it could include the client write data in the SRP CMD IU if it would
 > fit. This notion is in iSER but is not in standard SRP. Architecturally,
 > the capability is signaled using an additional data buffer format bit.
 > We find that client write performance is considerably improved by using
 > this capability. We are calling SRP spec compliant writes "standard
 > writes" and our modified writes "iu data writes".

I think this may make sense but you probably want to involve T10 to
get it standardized somehow.  Also, although I know having a big IOP
number is important for various non-technical reasons, are there any
realistic storage workloads that do lots of single-block writes?

Also I guess you need to use giant IUs to be able to hold at least one
block in the IU?

 - R.