[openib-general] Re: Solaris 10 with OpenIB OpenSM

[Hal Rosenstock]

On Fri, 2005-02-11 at 20:08, Tom Duffy wrote:
> [ Putting on OpenIB ]
> 
> On Fri, 2005-02-11 at 19:43 -0500, Hal Rosenstock wrote:
> > Hey Tom,
> > 
> > On Fri, 2005-02-11 at 19:35, Tom Duffy wrote:
> > > Solaris, you need to "ifconfig ibd0 plumb" to see it in ifconfig -a. You
> > > are further than me since you see /dev/ibd0.  You might even be there...
> > 
> > That did it :-) Thanks.
> > 
> > # ifconfig ibd0 plumb
> > # ifconfig -a
> > lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu
> > 8232 index 1
> >         inet 127.0.0.1 netmask ff000000 
> > e1000g0: flags=1004843<UP,BROADCAST,RUNNING,MULTICAST,DHCP,IPv4> mtu
> > 1500 index 2
> >         inet 10.0.2.56 netmask ffffff00 broadcast 10.0.2.255
> >         ether 0:30:48:27:21:2c 
> > ibd0: flags=1000842<BROADCAST,RUNNING,MULTICAST,IPv4> mtu 252 index 3
> >         inet 0.0.0.0 netmask 0 
> >         ipib 0:0:4:6:fe:80:0:0:0:0:0:0:0:8:f1:4:3:96:4:e 
> > 
> > Not sure why the MTU comes up as 252 (other than it is the 256 minimum
> > minus 4 bytes overhead).
> 
> That is odd.  I don't see this when using IBSRM to manage the network:
> 
> ibd1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 2044 index 3
>         inet 192.168.0.78 netmask ffffff00 broadcast 192.168.0.255
>         ipib 0:0:0:18:0:0:0:0:0:0:12:34:0:2:c9:1:9:76:56:31 
> 
> 
> > It also doesn't like me changing the MTU.
> > ifconfig ibd0 mtu 2044
> > ifconfig: setifmtu: SIOCSLIFMTU: ibd0: Invalid argument
> 
> Huh, Nitin, any ideas?

The log indicates that the MTU is 4 which is 2048. I also saw this in
the IB trace.

Feb 11 19:39:47 [B57AEA40] -> MCMember Record dump:
                               
MGID....................0xff12401bffff0000 : 0x0000000000000001
                               
PortGid.................0xfe80000000000000 : 0x0008f1040396040e
                                qkey....................0xB1B
                                Mlid....................0xC001
                                ScopeState..............0x21
                                Rate....................0x3
                                Mtu.....................0x4
                                TClass..................0x0
                                SLFlowLabelHopLimit.....0x0


> > But when I lower the MTU on the OpenIB side, I have connectivity:
> > 
> > >From OpenIB -> Solaris
> > ping 192.168.0.2
> > PING 192.168.0.2 (192.168.0.2) 56(84) bytes of data.
> > 64 bytes from 192.168.0.2: icmp_seq=1 ttl=255 time=8.57 ms
> > 64 bytes from 192.168.0.2: icmp_seq=2 ttl=255 time=0.118 ms
> > 64 bytes from 192.168.0.2: icmp_seq=3 ttl=255 time=0.127 ms
> > 
> > >From Solaris -> OpenIB
> > ping -s 192.168.0.1
> > PING 192.168.0.1: 56 data bytes
> > 64 bytes from 192.168.0.1: icmp_seq=0. time=0.306 ms
> > 64 bytes from 192.168.0.1: icmp_seq=1. time=0.232 ms
> > 64 bytes from 192.168.0.1: icmp_seq=2. time=0.191 ms
> > 64 bytes from 192.168.0.1: icmp_seq=3. time=0.180 ms
> 
> Awesome!!
> 
> > Let me know if you want the additional change to what is currently in
> > the tree. (I forget whether this matters or not for Solaris). I am
> > waiting to hear back from Eitan on this but am pretty sure it's correct.
> 
> Yes, I would like the patch.


Index: osm_sa_mcmember_record.c
===================================================================
--- osm_sa_mcmember_record.c	(revision 1745)
+++ osm_sa_mcmember_record.c	(working copy)
@@ -176,8 +176,8 @@
   p_recvd_mcmember_rec = p_ctxt->p_mcmember_rec;
   p_rcv = p_ctxt->p_rcv;
 
-  /* Why not comparing all the MGID ???? */
-  /* different scope can sneek in for the same MGID ? */
+  /* Why not compare the entire MGID ???? */
+  /* different scope can sneak in for the same MGID ? */
   /* EZ: I changed it to full compare ! */
   if (cl_memcmp(&p_mgrp->mcmember_rec.mgid,
                 &p_recvd_mcmember_rec->mgid,
@@ -732,7 +732,7 @@
 
   OSM_LOG_ENTER( p_rcv->p_log, __validate_requested_mgid );
 
-  /* 14-a : mcaut GUID must start with 0xFF */
+  /* 14-a : mcast GID must start with 0xFF */
   if (p_mcm_rec->mgid.multicast.header[0] != 0xFF)
   {
     osm_log( p_rcv->p_log, OSM_LOG_ERROR,
@@ -1009,8 +1009,8 @@
     {
 
       /*
-       * acording to the same o15-0.1.14 we must get the stored
JoinState and
-       * the request JoinState be oposite to leave - other wise just
update it
+       * according to the same o15-0.1.14 we must get the stored
JoinState and
+       * the request JoinState be opposite to leave - otherwise just
update it
        */
       port_join_state = p_mcm_port->scope_state & 0x0F;
       new_join_state = 
@@ -1419,7 +1419,7 @@
   osm_mcmr_recv_t*            const p_rcv  = p_ctxt->p_rcv;
   const ib_member_rec_t*      p_rcvd_rec   = p_ctxt->p_mcmember_rec;
   const osm_physp_t*          p_req_physp  = p_ctxt->p_req_physp;
-  ib_member_rec_t             match_rec; // since we might change
scope_state
+  ib_member_rec_t             match_rec; /* since we might change
scope_state */
   ib_net64_t                  comp_mask    = p_ctxt->comp_mask;
   osm_mcm_port_t*             p_mcm_port;
   ib_net64_t portguid = p_rcvd_rec->port_gid.unicast.interface_id;
@@ -1433,7 +1433,7 @@
           "__osm_sa_mcm_by_comp_mask_cb: "
           "Checking mlid:0x%X\n",
           cl_ntoh16(p_mgrp->mlid ));
-  /* first try to eliminate the group by MGID, MLID or P_Key */
+  /* first try to eliminate the group by MGID, MLID, or P_Key */
   if ((IB_MCR_COMPMASK_MGID & comp_mask) &&
       cl_memcmp(&p_rcvd_rec->mgid, &p_mgrp->mcmember_rec.mgid,
sizeof(ib_gid_t))) {
     goto Exit;
@@ -1490,10 +1490,10 @@
     sl_flow_hop_mask = sl_flow_hop_mask | 0x000F;
 
   if (IB_MCR_COMPMASK_FLOW & comp_mask)
-    sl_flow_hop_mask = sl_flow_hop_mask | 0x0FF0;
+    sl_flow_hop_mask = sl_flow_hop_mask | 0xFFFFFF0;
 
   if (IB_MCR_COMPMASK_HOP & comp_mask)
-    sl_flow_hop_mask = sl_flow_hop_mask | 0xF000;
+    sl_flow_hop_mask = sl_flow_hop_mask | 0xFF000000;
 
   if ((p_rcvd_rec->sl_flow_hop & sl_flow_hop_mask) !=
       (p_mgrp->mcmember_rec.sl_flow_hop & sl_flow_hop_mask)) goto Exit;



> Thanks,
> 
> -tduffy