[ewg] OPENSM cONFIGURATION

Atul Yadav atulyadavtech at gmail.com
Sun Apr 13 04:20:22 PDT 2014


Hi Hal,

Thanks for providing knowledgeable info and diagnosing the issue.
I am  sharing the full details of the cluster with the purpose.

Lustre

   - mds1
   - mds2
   - oss1
   - oss2

Compute Node

   - Nalanda
   - compute-0-1 to compute-0-34

Topology
Ftree is configured with the help of yours.

So, we are using common infiniband cable for Lustre and MPI traffic.

Can i make sure my Lustre traffic work without any congestion.

Thanks in advance.
Atul Yadav





On Sat, Apr 12, 2014 at 9:08 PM, Hal Rosenstock <hal at dev.mellanox.co.il>wrote:

> On 4/12/2014 11:29 AM, Atul Yadav wrote:
> > Hi,
> >
> > Yes, i am able to ping all the nodes connected with Infiniband switch
> > For more details please go through the attachment.
>
> OpenSM looks fine although it is very old (3.3.5). Is this SM host based
> or embedded in one of your switches ?
>
> I didn't see any output related to showing pings working but I'll take
> your word for this. If pings work, I have no theory why this wouldn't work.
>
> -- Hal
>
> >
> >
> >
> > Thanks
> > Atul Yadav
> >
> >
> > On Sat, Apr 12, 2014 at 7:28 PM, Hal Rosenstock <hal at dev.mellanox.co.il
> > <mailto:hal at dev.mellanox.co.il>> wrote:
> >
> >     On 4/12/2014 6:59 AM, Atul Yadav wrote:
> >     > HI,
> >     >
> >     > Thanks for replying
> >     > In this artectuire, when we are doing ibv_rc_pingpong between two
> >     nodes
> >     > connected with same switch we are getting result. But when we use
> two
> >     > nodes with 2 switches we are getting error.
> >     >
> >     > Success:-
> >     > [root at oss1 ~]# ibv_rc_pingpong
> >     >   local address:  LID 0x001e, QPN 0x2c004a, PSN 0x554863, GID ::
> >     >   remote address: LID 0x0022, QPN 0x20004a, PSN 0x7c9dc2, GID ::
> >     > 8192000 bytes in 0.01 seconds = 6992.74 Mbit/sec
> >     > 1000 iters in 0.01 seconds = 9.37 usec/iter
> >     > [root at oss1 ~]#
> >     >
> >     > [root at mds1 ~]# ibv_rc_pingpong 173.16.1.52
> >     >   local address:  LID 0x0022, QPN 0x20004a, PSN 0x7c9dc2, GID ::
> >     >   remote address: LID 0x001e, QPN 0x2c004a, PSN 0x554863, GID ::
> >     > 8192000 bytes in 0.01 seconds = 7084.97 Mbit/sec
> >     > 1000 iters in 0.01 seconds = 9.25 usec/iter
> >     > [root at mds1 ~]#
> >     >
> >     >
> >     >
> >     >
> >     > Error
> >     > [root at nalanda mvapich2-1.9]# ibv_rc_pingpong
> >     >   local address:  LID 0x0001, QPN 0x56004e, PSN 0x704d51
> >     >   remote address: LID 0x0022, QPN 0x1c004a, PSN 0x07a0b2
> >     >
> >     > [root at mds1 ~]# ibv_rc_pingpong 173.16.1.1
> >     >   local address:  LID 0x0022, QPN 0x1c004a, PSN 0x07a0b2, GID ::
> >     > client read: Success
> >     > Couldn't read remote address
> >     > [root at mds1 ~]#
> >
> >     Looking at libibverbs/examples/rc_pingpong.c:
> >
> >     static struct pingpong_dest *pp_client_exch_dest(const char
> >     *servername, int port,
> >                                                      const struct
> >     pingpong_dest *my_dest)
> >     {
> >     ...
> >             gid_to_wire_gid(&my_dest->gid, gid);
> >             sprintf(msg, "%04x:%06x:%06x:%s", my_dest->lid, my_dest->qpn,
> >
> >     my_dest->psn, gid);
> >             if (write(sockfd, msg, sizeof msg) != sizeof msg) {
> >                     fprintf(stderr, "Couldn't send local address\n");
> >                     goto out;
> >             }
> >
> >
> >             if (read(sockfd, msg, sizeof msg) != sizeof msg) {
> >                     perror("client read");
> >                     fprintf(stderr, "Couldn't read remote address\n");
> >                     goto out;
> >             }
> >
> >     This read is failing for some reason. This is some message exchange
> >     over some IP network (for example, IPoIB or ethernet).
> >
> >     >
> >     > And how we test our ftree topology is working fine.
> >     >
> >     > Please go through the attachment.
> >
> >     Looks like LIDs are assigned but can't tell about routing from info
> >     supplied but topology looks relatively simple (5 switches,
> >     homogenous 4x QDR links). Is the OpenSM log clean ? Any fat tree
> >     related messages. This is likely not SM issue.
> >
> >     The next issues are end node related (probably with IPoIB
> >     configuration). Can you ping between the nodes which fail
> >     rc_pingpong ? If not,
> >
> >     -- Hal
> >
> >     >
> >     > Thank You
> >     > Atul Yadav
> >     >
> >     >
> >     > On Sat, Apr 12, 2014 at 12:14 AM, Hal Rosenstock
> >     <hal at dev.mellanox.co.il <mailto:hal at dev.mellanox.co.il>
> >     > <mailto:hal at dev.mellanox.co.il <mailto:hal at dev.mellanox.co.il>>>
> >     wrote:
> >     >
> >     >     On 4/11/2014 2:21 PM, Atul Yadav wrote:
> >     >     > Dear Team,
> >     >     >
> >     >     > We are trying to build Fat tree topology.
> >     >     > The details are given below:
> >     >     > Unmanaged switches 36 port  quantity 5
> >     >     > As per the some blog we need to modify the opensm.conf file
> >     >     > But we are unable to identify some parameter like:-
> >     >     >  root_guid_file    ???????
> >     >
> >     >     Fat tree routing will try to autodetect the roots but this may
> not
> >     >     work and it is better to specify the root GUIDs. In your case,
> >     they
> >     >     are the GUIDs for switches A and B.
> >     >
> >     >     The root GUID file is then provided to OpenSM either via the
> conf
> >     >     file or command line parameters. The command line parameter is
> >     [-a |
> >     >            --root_guid_file <path to file>]
> >     >
> >     >     OpenSM man page says:
> >     >
> >     >            -a, --root_guid_file <file name>
> >     >                   Set the root nodes for the Up/Down or Fat-Tree
> >     routing
> >     >     algorithm
> >     >                   to the guids provided in the given file (one to
> >     a line).
> >     >
> >     >     It also says:
> >     >
> >     >            If the root guid file  is  not  provided  (?-a?  or
> >     >      ?--root_guid_file?
> >     >            options),  the  topology has to be pure fat-tree that
> >     >     complies with the
> >     >            following rules:
> >     >              - Tree rank should be between two and eight
> (inclusively)
> >     >              - Switches of the same rank should have the same
> number
> >     >                of UP-going port groups*, unless they are root
> >     switches,
> >     >                in which case the shouldn?t have UP-going ports at
> all.
> >     >              - Switches of the same rank should have the same
> number
> >     >                of DOWN-going port groups, unless they are leaf
> >     switches.
> >     >              - Switches of the same rank should have the same
> number
> >     >                of ports in each UP-going port group.
> >     >              - Switches of the same rank should have the same
> number
> >     >                of ports in each DOWN-going port group.
> >     >              - All the CAs have to be at the same tree level
> (rank).
> >     >
> >     >            If the root guid file is provided, the topology doesn?t
> >     have
> >     >     to be pure
> >     >            fat-tree, and it should only comply with the following
> >     rules:
> >     >              - Tree rank should be between two and eight
> (inclusively)
> >     >              - All the Compute Nodes** have to be at the same tree
> >     level
> >     >     (rank).
> >     >                Note that non-compute node CAs are allowed here to
> >     be at
> >     >     different
> >     >                tree ranks.
> >     >
> >     >            *  ports that are connected to the same remote switch
> are
> >     >     referenced as
> >     >            port group.
> >     >
> >     >            **  list  of  compute  nodes  (CNs)  can  be  specified
>  by
> >     >     -u   or
> >     >            --cn_guid_file OpenSM options.
> >     >
> >     >     -- Hal
> >     >
> >     >     >
> >     >     > Need your input for this ?
> >     >     >
> >     >     >
> >     >     >
> >     >     >
> >     >     > Thank You
> >     >     > Atul Yadav
> >     >     >
> >     >     >
> >     >     >
> >     >     >
> >     >     > _______________________________________________
> >     >     > ewg mailing list
> >     >     > ewg at lists.openfabrics.org <mailto:ewg at lists.openfabrics.org>
> >     <mailto:ewg at lists.openfabrics.org <mailto:ewg at lists.openfabrics.org
> >>
> >     >     > http://lists.openfabrics.org/mailman/listinfo/ewg
> >     >
> >     >
> >
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/ewg/attachments/20140413/e93efe68/attachment.html>


More information about the ewg mailing list