[ofa-general] Installing openIB on Linux FC5

[Hal Rosenstock]

On Wed, 2007-06-06 at 10:45, Hossein Pourreza wrote:
> On Wed, Jun 06, 2007 at 10:21:31AM -0400, Hal Rosenstock wrote:
> > On Wed, 2007-06-06 at 10:08, Hossein Pourreza wrote:
> > > Hi,
> > > 
> > > Many thanks for your reply. I really appreciate that.
> > > 
> > > Our cluster uses Mellanox Technologies MT23108 InfiniHost (rev a1) and Sun 9P switch. 
> > > Out switch has its own SubnetManager and whenever I try to run opensm, I get an error 
> > > saying that there is another sm running with a mismatch key.
> > > 
> > > The result of running ibstat is like this:
> > > 
> > > 		CA type: MT23108
> > >         Number of ports: 2
> > >         Firmware version: 3.3.2
> > >         Hardware version: a1
> > >         Node GUID: 0x0003ba0001001788
> > >         System image GUID: 0x0003ba000100178b
> > >         Port 1:
> > >                 State: Active
> > >                 Physical state: LinkUp
> > >                 Rate: 10
> > >                 Base lid: 2
> > >                 LMC: 0
> > >                 SM lid: 1
> > >                 Capability mask: 0x00510a68
> > >                 Port GUID: 0x0003ba0001001789
> > > 		Port 2:
> > >                 State: Down
> > >                 Physical state: Polling
> > >                 Rate: 2
> > >                 Base lid: 0
> > >                 LMC: 0
> > >                 SM lid: 0
> > >                 Capability mask: 0x00510a68
> > >                 Port GUID: 0x0003ba000100178a
> > > 
> > > Is there anything wrong with this output?
> > 
> > Nothing wrong with the output :-) but is your port connected ? It
> > appears there is some connectivity problem as Physical state is not
> > LinkUp (and hence State is  Down) so SM cannot configure it.
> 
> I only use port 1 of each HCA and I just connected those to the switch. Should I
> connect both ports? There are only 9 ports available on our switch and we have 5
> nodes (10 ports in total).

My bad :-( I just looked at port 2. Port 1 looks fine (active and has
base and SM LIDs).

-- Hal

> Thanks again for all you help
> Hossein
> 
> 
> > 
> > -- Hal
> > 
> > > Many thanks for your kind help
> > > Hossein
> > > On Wed, Jun 06, 2007 at 09:09:53AM +0300, Tziporet Koren wrote:
> > > > Hossein Pourreza wrote:
> > > > >Hi all,
> > > > >
> > > > >I am new to infiniband stuff and am trying to configure an infiniband-based
> > > > >cluster using Linux FC 5. I downloaded the OFED-1.0 and tried to install 
> > > > >it on
> > > > >cluster nodes. Now I can load the kernel modules without any error but I 
> > > > >cannot
> > > > >run a simple test like ibv_ud_pingpong to check the connectivity of nodes 
> > > > >in
> > > > >user-level.
> > > > >
> > > > >
> > > > >  
> > > > Have you run opensm?
> > > > You can run ibstat on each node to see ports are active
> > > > 
> > > > Tziporet