[openib-general] IB initialization

Fri Apr 14 09:14:59 PDT 2006

Chris,

On Fri, 2006-04-14 at 12:05, Chris Worley wrote:
> Hal,
> 
> Note that I got an /etc/init.d/openibd script that's getting
> everything running (I still don't have IPoIB or MVAPICH2... but I can
> live without both).
> 
> Now, I'm running Opensm with -V, and it looks as I expected.

So what's the cap mask change being indicated ?

Are you sure there's no embedded SM running on the switch ?

-- Hal

> 
> This cluster is simple: 9 nodes in one switch.
> 
> Thanks,
> 
> Chris
> On 14 Apr 2006 11:38:21 -0400, Hal Rosenstock <halr at voltaire.com> wrote:
> > Hi again Chris,
> >
> > On Fri, 2006-04-14 at 11:29, Chris Worley wrote:
> > > Hal,
> > >
> > > It looks like 1 per GUID.  I don't see a capability mask.  An example is:
> > >
> > > Apr 14 07:28:18 879428 [40602960] -> __osm_trap_rcv_process_request:
> > > Received Generic Notice type:0x04 num:144 Producer:1 f
> > > rom LID:0x0007 TID:0x0000000000000001
> > > Apr 14 07:28:18 879513 [40602960] -> osm_report_notice: Reporting
> > > Generic Notice type:4 num:144 from LID:0x0007 GID:0xfe800
> > > 00000000000,0x0002c9020020c3b6
> >
> > Are you running with verbose (-V) ? You only see that extra info then.
> >
> > Just out of curiousity, how big is your subnet and what is the topology
> > ?
> >
> > -- Hal
> >
> > > Thanks,
> > >
> > > Chris
> > > On 14 Apr 2006 10:55:00 -0400, Hal Rosenstock <halr at voltaire.com> wrote:
> > > > Hi again Chris,
> > > >
> > > > On Fri, 2006-04-14 at 10:39, Chris Worley wrote:
> > > > > Hal,
> > > > >
> > > > > You're correct... the results of the scans are in /var/log/osm.log.  I
> > > > > was expecting the "-console" mode to show more.
> > > > >
> > > > > In looking at the /var/log/osm.log I'm seeing a lot of:
> > > > >
> > > > > Reporting Generic Notice type:4 num:144
> > > > >
> > > > > For different GUIDs.
> > > >
> > > > What's a lot ? One for each GUID ? What's the capability mask indicated
> > > > ?
> > > >
> > > > >   Is there a place to look these up?
> > > >
> > > > Yes, the IBA spec (volume 1). Trap 144 indicates that the capability
> > > > mask at the indicated LID has changed.
> > > >
> > > > > I still don't have IPoIB running, and ibv_devinfo says I'm not setup
> > > > > right either (couldn't open a device).
> > > >
> > > > I'm not sure why not.
> > > >
> > > > -- Hal
> > > >
> > > > > Thanks,
> > > > >
> > > > > Chris
> > > > > On 14 Apr 2006 10:22:22 -0400, Hal Rosenstock <halr at voltaire.com> wrote:
> > > > > > Hi Chris,
> > > > > >
> > > > > > On Fri, 2006-04-14 at 10:19, Chris Worley wrote:
> > > > > > > I installed the SuSE 10 OpenIB RC2 RPMS.
> > > > > > >
> > > > > > > The installation went well, but I'm stuck at the startup.
> > > > > > >
> > > > > > > As an IBGD user, I'm used to an init file in /etc/init.d... but there was none.
> > > > > > >
> > > > > > > >From the wiki, I was able to glean:
> > > > > > >
> > > > > > >             Make the udev file:
> > > > > > >
> > > > > > > # cat > /etc/udev/rules.d/40-infiniband.rules
> > > > > > > KERNEL="umad*", NAME="infiniband/%k"
> > > > > > > KERNEL="issm*", NAME="infiniband/%k"
> > > > > > >
> > > > > > >              Install some modules:
> > > > > > >
> > > > > > > modprobe ib_ucm
> > > > > > > modprobe ib_cm
> > > > > > > modprobe ib_uverbs
> > > > > > > modprobe ib_umad
> > > > > > >
> > > > > > > And make sure udev is running, and start the opensm.
> > > > > > >
> > > > > > > I've done this on all nodes, and ibstat shows I have a link up and
> > > > > > > running on every node.  Opensm doesn't show any scanning.  It's been
> > > > > > > hung all night at:
> > > > > > >
> > > > > > > # opensm --console
> > > > > > > -------------------------------------------------
> > > > > > > OpenSM Rev:openib-1.2.0
> > > > > > > Based on OpenIB svn Exported revision
> > > > > > > Command Line Arguments:
> > > > > > >  Enabling OpenSM interactive console
> > > > > > >  Log File: /var/log/osm.log
> > > > > > > -------------------------------------------------
> > > > > > > OpenSM Rev:openib-1.2.0 OpenIB svn Exported revision
> > > > > > >
> > > > > > > Using default guid 0x2c9020020c3ce
> > > > > > >
> > > > > > > OpenSM Console
> > > > > > >
> > > > > > > $ Entering MASTER state
> > > > > > >
> > > > > > > SUBNET UP
> > > > > >
> > > > > > Looks like everything is fine from the OpenSM standpoint.
> > > > > >
> > > > > > I see no indication that OpenSM is hung. You are in the console.
> > > > > >
> > > > > > Also, why do you say OpenSM isn't "scanning" ?
> > > > > >
> > > > > > What is in /var/log/osm.log ? Any errors ?
> > > > > >
> > > > > > If you want more verbose messages start OpenSM with -V.
> > > > > >
> > > > > > -- Hal
> > > > > >
> > > > > > > IPoIB isn't up.  ibv_rc_pingpong doesn't work.  Neither does ibv_devinfo.
> > > > > > >
> > > > > > > Is there a definitive guide on the initialization of the drivers and fabric?
> > > > > > >
> > > > > > > Also, is there an MVAPICH2 for SuSE 10 RPM?
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Chris
> > > > > > > _______________________________________________
> > > > > > > openib-general mailing list
> > > > > > > openib-general at openib.org
> > > > > > > http://openib.org/mailman/listinfo/openib-general
> > > > > > >
> > > > > > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
> > > > > >
> > > > > >
> > > > > _______________________________________________
> > > > > openib-general mailing list
> > > > > openib-general at openib.org
> > > > > http://openib.org/mailman/listinfo/openib-general
> > > > >
> > > > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
> > > >
> > > >
> > > _______________________________________________
> > > openib-general mailing list
> > > openib-general at openib.org
> > > http://openib.org/mailman/listinfo/openib-general
> > >
> > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
> >
> >