[Users] can someone clarify and help with partitioning?

[Weiny, Ira]

> -----Original Message-----
> From: users-bounces at lists.openfabrics.org [mailto:users-
> bounces at lists.openfabrics.org] On Behalf Of Johannes Mäulen
> Sent: Thursday, August 08, 2013 3:48 AM
> To: users at lists.openfabrics.org
> Subject: [Users] can someone clarify and help with partitioning?
> 
> Hi there!
> I've got very little experience with IB so far, and I would like a to ask a few
> questions.
> Perhaps I can get some help here?

I'm a bit unsure of the traffic pattern here but I'll try an answer some questions.

> I'd like to use IB for fast Interconnects between four machines, all with IPoIB.
> My setup looks like the following:
> I've got a small IB Switch (Flextronics 8port SDR, it has no subnet manager)
> and 4 dual port Cards. (2 Voltaire, 2 HP)
> Box1 and box2(windows) are connected directly(192.168.1.0/30)via ib0, box3
> and box4(debian linux) are connected directly(192.168.3.0/30) via ib0.
> Box1,box2,box3,box4 are connected to the Flextronics switch via ib1,
> 192.168.2.0/24.
> All Interfaces are running in Connected Mode with MTU 65520.
> 
> I've tried to illustrate it:
> 
> box1 ib0----192.168.1.0/30------- ib0   box2
> ib1					ib1
> |________  			________|
> 	      |  			|
> 		Ib-switch
> 		192.168.2.0/24
> ________|			|_______
> |				      	|
> ib1					ib1
> box3 ib0----192.168.3.0/30------- ib0	box4
> 
> 
> How many Subnet-Managers do i need to install?

>From what you have shown above you have 3 subnets.

	1) the ib switch subnet which connects port 2 of all the HCA's.
	2) the point to point link between the Linux boxes.
	3) the point to point link between the windows boxes.

Every IB subnet requires an SM.  Therefore, from your description, you will need 3 SM's.  Since you have 2 SM's running now are some of the ports not "Active"?

> (currently I have one running on windows and one running on linux) Do I
> need to build three partitions?

IB partitions are for partitioning an IB subnet.  So if you don't want "ib1" to be able to talk to other "ib1's" then you will need partitions.  If the idea is to keep the ib0 and ib1 traffic separate then you will not need any partitions as you have 3 subnets which are separate already.

> 
> I read ipoib_release_notes.txt:
> [...]
> ==========================================================
> ==================
> ===
> 3. Known Issues
> ==========================================================
> ==================
> ===
> 1. If a host has multiple interfaces and (a) each interface belongs to a
>    different IP subnet, (b) they all use the same InfiniBand Partition, and
> (c)
>    they are connected to the same IB Switch, then the host violates the IP rule
>    requiring different broadcast domains. Consequently, the host may build an
>    incorrect ARP table.
> 
>    The correct setting of a multi-homed IPoIB host is achieved by using a
>    different PKEY for each IP subnet.
> [...]
> 
> But I'm unsure if this applies to my lab.
> Yes, I have multiple hosts with multiple interfaces
> (a) CHECK
> Yes, right now they use the same IB Partition. As all subnet managers are
> running with default config... 		(b) CHECK
> YESNO,Parts of it are connected to the same IB Switch.
> (c) UNSURE

Again I'm not sure of the traffic pattern and what you are trying to accomplish with partitions.

Hope this helps,
Ira

> 
> Many thanks in advance for your help!
> 
> Johannes