[ewg] Fwd: OpenSM for dual Rail

[Hal Rosenstock]

On 7/26/2014 7:57 AM, Hal Rosenstock wrote:
> Hi Atul,
> 
> On 7/26/2014 5:45 AM, Atul Yadav wrote:
>> Dear Admin,
>>
>> In our exiting setup we are having 18 server with dual IB cards.
>> And 2 IB switch.
>>
>> We are planning to use MPI traffic on IB Port1 and Lustre Traffic on IB
>> port 2.
>>
>> IB Port 1 		Hostname 		IB Port 2
>> IB Switch  1 	IP Address 	Hostname 	IP Address 	IB Switch  2
>> Port 1 	192.168.2.1 	Compute-1 	192.168.3.1 	Port 1
>> Port 2 	192.168.2.2 	Compute-2 	192.168.3.2 	Port 2
>> Port 3 	192.168.2.3 	Compute-3 	192.168.3.3 	Port 3
>> Port 4 	192.168.2.4 	Compute-4 	192.168.3.4 	Port 4
>> Port 5 	192.168.2.5 	Compute-5 	192.168.3.5 	Port 5
>> Port 6 	192.168.2.6 	Compute-6 	192.168.3.6 	Port 6
>> Port 7 	192.168.2.7 	Compute-7 	192.168.3.7 	Port 7
>> Port 8 	192.168.2.8 	Compute-8 	192.168.3.8 	Port 8
>> Port 9 	192.168.2.9 	Compute-9 	192.168.3.9 	Port 9
>> Port 10 	192.168.2.10 	Compute-10 	192.168.3.10 	Port 10
>> Port 11 	192.168.2.11 	Compute-11 	192.168.3.11 	Port 11
>> Port 12 	192.168.2.12 	Compute-12 	192.168.3.12 	Port 12
>> Port 13 	192.168.2.13 	Compute-13 	192.168.3.13 	Port 13
>> Port 14 	192.168.2.14 	Compute-14 	192.168.3.14 	Port 14
>> Port 15 	192.168.2.15 	Compute-15 	192.168.3.15 	Port 15
>> Port 16 	192.168.2.16 	Compute-16 	192.168.3.16 	Port 16
>> Port 17 	192.168.2.17 	Compute-17 	192.168.3.17 	Port 17
>> Port 18 	192.168.2.18 	Compute-18 	192.168.3.18 	Port 18
>>
>>
>>
>> need your help in opensm configuration and setup ....
> 
> You need an SM per subnet. They should be setup to have different subnet
> prefixes.
> 
> Do you plan on running OpenSM for each subnet on the same or different
> machines ?
> 
> It's a little simpler if it's different machines.
> 
> If it's on same machine, basically you need 2 options files (one for
> each subnet/OpenSM) and make the output (/var/log and /var/cache/opensm
> folders) go to separate files/directories. You can do that with the
> various config options.
> 
> Also, see:
> https://www.mail-archive.com/linux-rdma@vger.kernel.org/msg03559.html
> 
> I don't see OSM_LOG_DIR in recent OpenSMs but there is log_prefix option.

Here's the general process I'd follow for running 2 OpenSM instances (on
2 different ports) on same machine:

Generate opensm conf file for your OpenSM via opensm -c
Make two copies opensm-rail1.conf opensm-rail2.conf
edit these as follows:
change port to port GUID for this SM
change at least 1 subnet_prefix (must be different)
change log_file and dump_files_dir to be unique

set OSM_CACHE_DIR environment variable for one of the OpenSM instances

-- Hal

> -- Hal
> 
>> Thank You
>> Atul Yadav
>>
>>
>>
>> _______________________________________________
>> ewg mailing list
>> ewg at lists.openfabrics.org
>> http://lists.openfabrics.org/mailman/listinfo/ewg
>