[Scst-devel] [ofa-general] RE: SRP aggregate bandwidth decreasing as threads increase

[Sam Haxor]

If we pin the thread to 'a' CPU(say CPU-X) then can we pass on a hint to the BE driver to process the response completion on the same 'CPU-X' ? This way the CPU-cache will be utilized efficiently. I don't know how the IB driver etc works. So can't contribute in terms of code right now. But this is how any sample implementation could/should look like - 

BE driver creates the following - 

QUEUES[NR_CPUS];
QUEUES {
CMD QUEUE[SOME_QUEUE_DEPTH];
RSP   QUEUE[SOME_QUEUE_DEPTH];
};

1) scsi-mid down-calls BE driver, and also passes a hint aka 'thread-CPU-X'.
2) BE transmits cmd on QUEUES[thread-CPU-X]->CMD QUEUE[slot_index];
3) The adapter(HBA/HCA) will interrupt the BE driver on the 'thread-CPU-X'.
    3.1) Now it is the BE drivers responsibility to affinitize the response-draining with the corresponding CPU @ driver load
           time.

Ciao



----- Original Message ----
> From: Bart Van Assche <bart.vanassche at gmail.com>
> To: "Sufficool, Stanley" <ssufficool at rov.sbcounty.gov>
> Cc: Chris Worley <worleys at gmail.com>; scst-devel <scst-devel at lists.sourceforge.net>; OpenIB <general at lists.openfabrics.org>
> Sent: Monday, May 18, 2009 1:22:43 PM
> Subject: Re: [Scst-devel] [ofa-general] RE: SRP aggregate bandwidth decreasing as threads increase
> 
> On Mon, May 18, 2009 at 6:52 PM, Sufficool, Stanley
> wrote:
> > IIRC, The SRP Target code has many context switches that throttle
> > performance at higher thread counts.
> 
> Depends on which version of ib_srpt you are using. The ib_srpt kernel
> module has a parameter called "thread" which allows to control whether
> disk I/O is handled in another thread than the one that communicates
> over InfiniBand (thread=1) or in the same thread (thread=0). For older
> versions of the ib_srpt kernel module the default was thread=1, which
> caused indeed a lot of context switches. On December 3, 2008 (SCST
> Subversion revision 594) the default has been changed from thread=1 to
> thread=0 because the latter results in better performance.
> 
> Bart.
> 
> ------------------------------------------------------------------------------
> Crystal Reports - New Free Runtime and 30 Day Trial
> Check out the new simplified licensing option that enables 
> unlimited royalty-free distribution of the report engine 
> for externally facing server and web deployment. 
> http://p.sf.net/sfu/businessobjects
> _______________________________________________
> Scst-devel mailing list
> Scst-devel at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/scst-devel