[openib-general] Re: IBM utilizing and testing openIB release 1 RCx

[Kevin Reilly]

Thanks Mike,
   Certainly increasing the default send/recv queues sizes is better then
what is in
the current release-1 openIB stream. I think our admins are setting up
different sizes
on the file-servers then the clients. We could just make the defaults be
the worst case
but that doesn't seem right to use more memory then needed on a server.

   Some servers can be big powerful file servers with lots of TCP
connections, some server
might be 1 or 2 way blades and some might have there storage not going over
the IB network
and using IB for only MPI traffic.

   What do you think about a simple module load parameter to let the admin
set the send and
recv queue values to what they need on that server? Each HCA vendor can say
what the max
they support is.


Kevin J. Reilly
STSM, HPC Architecture
-Federation/HPS  Chief Engineer
-HPC interconnect architect
(office) 845-433-7976  (tieline) 8-293-7976



                                                                           
             "Michael S.                                                   
             Tsirkin"                                                      
             <mst at mellanox.co.                                          To 
             il>                       Kevin Reilly/Poughkeepsie/IBM at IBMUS 
                                                                        cc 
             03/25/2006 12:04          openib-general at openib.org           
             PM                                                    Subject 
                                       Re: IBM utilizing and testing       
                                       openIB release 1 RCx                
             Please respond to                                             
                "Michael S.                                                
                 Tsirkin"                                                  
             <mst at mellanox.co.                                             
                    il>                                                    
                                                                           
                                                                           




Quoting r. Kevin Reilly <kjreilly at us.ibm.com>:
>   The biggest problem we have at the moment is being able to tune the
> IPoverIB drivers  sendq_size and recvq_size.  We simply
> can get IP to work without tuning these parameters in some situations.
For
> example, running our cluster filesystem file server over
> IPoverIB.  The server that is a filesystem server needs to have many
socket
> connections open and we use a big 8 or 16 way server
> for this purpose. We very quickly overrun the IPoverIB drivers send and
> receive queue. We are submitting a patch so we can tune
> these because we couldn't get things to work with the defaults and not
way
> to tune.

Which queue sizes do you use?
Maybe we can just have everyone use the bigger values and be done with it.

--
Michael S. Tsirkin
Staff Engineer, Mellanox Technologies