[ewg] Need help for Infiniband optimisation for our cluster (MTU...)

[Mike Heinz]

Richard - that's odd, I don't see an "ibv_portstat" command on my boxes - do you know what package provides it?

-----Original Message-----
From: ewg-bounces at lists.openfabrics.org [mailto:ewg-bounces at lists.openfabrics.org] On Behalf Of Richard Croucher
Sent: Tuesday, December 07, 2010 11:17 AM
To: 'giggzounet'; ewg at lists.openfabrics.org
Subject: Re: [ewg] Need help for Infiniband optimisation for our cluster (MTU...)

The InfiniBand standard allows a MTU of 4096 bytes but the HCA you are using
limits this to 2048. This can be set and queried using your SM management.

On the server side, the ibv_portstat command will show the current MTU size.

All the information you need is in the docs. 
There are also multiple parties, including myself, who offer training
workshops for this stuff.

Richard


-----Original Message-----
From: ewg-bounces at lists.openfabrics.org
[mailto:ewg-bounces at lists.openfabrics.org] On Behalf Of giggzounet
Sent: 07 December 2010 15:32
To: ewg at lists.openfabrics.org
Subject: Re: [ewg] Need help for Infiniband optimisation for our cluster
(MTU...)

Hi,

Thx for your answer!

Particularly the explication between connected and datagram mode (I see
that with the IMB1 benchmarks of mpi)!

The hardware we are using in details:
- on the master: Mellanox MHGH18-XTC ConnectX with VPI adapter, single
port 20Gb/s, PCIe2.0 x8 2.5GT/s
- on the nodes: Integrated Mellanox DDR Infiniband 20Gbs ConnectX with
QSFP Connector.

How can I know the limit of the MTU size ?


On the Infiniband we are just using mpi with different CFD programs. But
always with mpi (intel mpi or openmpi). Sould I use QoS ?

Thx for your help!


Le 07/12/2010 16:09, Richard Croucher a écrit :
> Connected mode will provide more throughput but datagram mode will provide
> lower latency.  
> You don't say what HCA's you are using.  Some of the optimizations for
> Connected mode are only available for the newer ConnectX QDR HCA's.
> 
> Your HCA will probably limit the MTU size.  Leave this as large as
possible.
> 
> If you are only running a single application on the InfiniBand you need
not
> bother with QoS.   If you are running multiple, then you do need to set
> this.  This is quite complex since you need to define V'L's, their
> arbitration policies and assign SL's to them.  This is described in the
> OpenSM docs.  This is relevant even if you are using the embedded SM in
the
> switch.
> 
> AS a newbie, take a look in the ../OFED/docs  
> There is probably all you need there. Mellanox also have some useful docs
on
> their website.
> 
> -----Original Message-----
> From: ewg-bounces at lists.openfabrics.org
> [mailto:ewg-bounces at lists.openfabrics.org] On Behalf Of giggzounet
> Sent: 07 December 2010 14:01
> To: ewg at lists.openfabrics.org
> Subject: [ewg] Need help for Infiniband optimisation for our cluster
> (MTU...)
> 
> Hi,
> 
> I'm new on this list. We have in our laboratory a little cluster:
> - master 8 cores
> - 8 nodes with 12 cores
> - DDR infiniband switch Mellanox MTS3600R
> 
> On these machines we have an oscar cluster with CentOS 5.5. We have
> installed the ofed packages 1.5.1. The default config for the infiniband
> is used. So infiniband is running in connected mode.
> 
> Our cluster is used to solve CFD (Computational Fluid Dynamics)
> problems. And I'm trying to optimize the infiniband network and so I
> have several questions:
> 
> - Is it the right mailing list to ask ? (if not...where should I post ?)
> 
> - Is there a how-to on infiniband optimisation ?
> 
> - CFD computations need a lot of bandwidth. There are a lot of data
> exchange through MPI (we are using intel mpi). Has the infiniband mode
> (connected or datagram) influence in this case ? What is the "best" MTU
> for those computation ?
> 
> 
> Best regards,
> Guillaume
> 
> _______________________________________________
> ewg mailing list
> ewg at lists.openfabrics.org
> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg


_______________________________________________
ewg mailing list
ewg at lists.openfabrics.org
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg

_______________________________________________
ewg mailing list
ewg at lists.openfabrics.org
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg