[ofa-general] Installing SDP on existing OFED 1.3.1 install - DRBD SDP/Infiniband Support

Tue Sep 1 23:46:24 PDT 2009

On Tuesday 01 September 2009 16:24, Robert Dunklewrote:
> Hi Jack,
> 
> Thanks for the tip.  (*Embarassed*)
>  
> I think a Kernel upgrade since might have broken the source RPM. Is
> there any way for me to fix this? (I have an identical hardware server
> that was originally installed with SDP, details of that below, it seems
> the system without sdp that I'm trying to add it to got a minor kernel
> upgrade)
> 
> Thanks again,
> 
> Rob 
You need to see where the kernel is taking ib_sdp from.
Do "modinfo ib_sdp" to see the ib_sdp.ko file which gets loaded.

Note that on the working system, you have:
 /lib/modules/2.6.18-92.1.6.el5xen/updates/kernel/drivers/infiniband/ulp/sdp/ib_sdp.ko
Note **updates** in the above directory path -- indicating that this
sdp was installed as part of your OFED install.  You have no such path for ib_sdp.ko in
your broken system.

1. Was OFED installed on the broken system with your xen kernel?
  (check this by seeing if directory /lib/modules/2.6.18-92.1.13.el5xen/updates
   exists).

2. Where did you place the ib_sdp.ko module that you built?
3. The version disagreement below indicates that the sdp module
   expects different versions of ib_core.ko and rdma_cm.ko than are
   currently loaded.

-Jack
> 
> Error on system I'm trying to add to ("Broken System"):
> ib_sdp: disagrees about version of symbol ib_unregister_client
> ib_sdp: Unknown symbol ib_unregister_client
> ib_sdp: disagrees about version of symbol ib_create_cq
> ib_sdp: Unknown symbol ib_create_cq
> ib_sdp: disagrees about version of symbol rdma_resolve_addr
> ib_sdp: Unknown symbol rdma_resolve_addr
> ib_sdp: disagrees about version of symbol ib_dereg_mr
> ib_sdp: Unknown symbol ib_dereg_mr
> ib_sdp: disagrees about version of symbol rdma_reject
> ib_sdp: Unknown symbol rdma_reject
> ib_sdp: disagrees about version of symbol rdma_disconnect
> ib_sdp: Unknown symbol rdma_disconnect
> ib_sdp: disagrees about version of symbol rdma_resolve_route
> ib_sdp: Unknown symbol rdma_resolve_route
> ib_sdp: disagrees about version of symbol rdma_bind_addr
> ib_sdp: Unknown symbol rdma_bind_addr
> ib_sdp: disagrees about version of symbol ib_register_client
> ib_sdp: Unknown symbol ib_register_client
> ib_sdp: disagrees about version of symbol rdma_create_qp
> ib_sdp: Unknown symbol rdma_create_qp
> ib_sdp: disagrees about version of symbol ib_destroy_cq
> ib_sdp: Unknown symbol ib_destroy_cq
> ib_sdp: disagrees about version of symbol rdma_create_id
> ib_sdp: Unknown symbol rdma_create_id
> ib_sdp: disagrees about version of symbol rdma_notify
> ib_sdp: Unknown symbol rdma_notify
> ib_sdp: disagrees about version of symbol rdma_listen
> ib_sdp: Unknown symbol rdma_listen
> ib_sdp: disagrees about version of symbol ib_get_dma_mr
> ib_sdp: Unknown symbol ib_get_dma_mr
> ib_sdp: disagrees about version of symbol ib_alloc_pd
> ib_sdp: Unknown symbol ib_alloc_pd
> ib_sdp: disagrees about version of symbol rdma_connect
> ib_sdp: Unknown symbol rdma_connect
> ib_sdp: disagrees about version of symbol rdma_destroy_id
> ib_sdp: Unknown symbol rdma_destroy_id
> ib_sdp: disagrees about version of symbol rdma_accept
> ib_sdp: Unknown symbol rdma_accept
> ib_sdp: disagrees about version of symbol ib_destroy_qp
> ib_sdp: Unknown symbol ib_destroy_qp
> ib_sdp: disagrees about version of symbol ib_dealloc_pd
> ib_sdp: Unknown symbol ib_dealloc_pd
> 
> Working system files:
> /lib/modules/2.6.18-92.1.6.el5xen/kernel/drivers/infiniband/ulp/sdp
> /lib/modules/2.6.18-92.1.6.el5xen/kernel/drivers/infiniband/ulp/sdp/ib_s
> dp.ko
> /lib/modules/2.6.18-92.1.6.el5xen/updates/kernel/drivers/infiniband/ulp/
> sdp
> /lib/modules/2.6.18-92.1.6.el5xen/updates/kernel/drivers/infiniband/ulp/
> sdp/ib_s
> dp.ko
> /lib/modules/2.6.18-92.el5/kernel/drivers/infiniband/ulp/sdp
> /lib/modules/2.6.18-92.el5/kernel/drivers/infiniband/ulp/sdp/ib_sdp.ko
> 
> Broken System Files:
> # locate sdp | more
> /etc/libsdp.conf
> /lib/modules/2.6.18-92.1.13.el5xen/kernel/drivers/infiniband/ulp/sdp
> /lib/modules/2.6.18-92.1.13.el5xen/kernel/drivers/infiniband/ulp/sdp/ib_
> sdp.ko
> /lib/modules/2.6.18-92.el5/kernel/drivers/infiniband/ulp/sdp
> /lib/modules/2.6.18-92.el5/kernel/drivers/infiniband/ulp/sdp/ib_sdp.ko
> 
> Uname -r on Broken:
> 2.6.18-92.1.13.el5xen
> 
> Uname -r on Works:
> 2.6.18-92.1.6.el5xen
> 
> 
> -----Original Message-----
> From: Jack Morgenstein [mailto:jackm at dev.mellanox.co.il] 
> Sent: 01 September 2009 14:05
> To: Robert Dunkley
> Cc: general at lists.openfabrics.org
> Subject: Re: [ofa-general] Installing SDP on existing OFED 1.3.1 install
> - DRBD SDP/Infiniband Support
> 
> On Tuesday 01 September 2009 14:52, Robert Dunkley wrote:
> > Hi Jack,
> > 
> > Thanks for the reply, it now tries to load the ib_sdp module but
> fails:
> > 
> > # /etc/rc.d/init.d/openibd restart
> > Unloading HCA driver:                                      [  OK  ]
> > Loading HCA driver and Access Layer:                       [  OK  ]
> > Setting up InfiniBand network interfaces:
> > Bringing up interface ib0:                                 [  OK  ]
> > Setting up service network . . .                           [  done  ]
> > Loading ib_sdp                                             [FAILED]
> > 
> > Where does the full log for this go? Am I missing some sort of
> > dependency? (Loaded modules shown below)
> > 
> do "dmesg" from a console window to see what the failure is.
> - Jack
> > # /etc/rc.d/init.d/openibd status
> > 
> >   HCA driver loaded
> > 
> > Configured devices:
> > ib0
> > 
> > Currently active devices:
> > ib0
> > 
> > The following OFED modules are loaded:
> > 
> >   rdma_ucm
> >   rdma_cm
> >   ib_addr
> >   ib_ipoib
> >   mlx4_core
> >   mlx4_ib
> >   ib_mthca
> >   ib_uverbs
> >   ib_umad
> >   ib_sa
> >   ib_cm
> >   ib_mad
> >   ib_core
> >   iw_cxgb3
> > 
> > 
> > Thanks,
> > 
> > Rob
> >  
> > 
> > -----Original Message-----
> > From: Jack Morgenstein [mailto:jackm at dev.mellanox.co.il] 
> > Sent: 01 September 2009 12:17
> > To: general at lists.openfabrics.org
> > Cc: Robert Dunkley
> > Subject: Re: [ofa-general] Installing SDP on existing OFED 1.3.1
> install
> > - DRBD SDP/Infiniband Support
> > 
> > On Tuesday 01 September 2009 13:44, Robert Dunkley wrote:
> > > Hi everyone,
> > > 
> > > A DRBD release candidate with specific SDP/Infiniband support was
> > > released last week.
> > > 
> > > I have an existing OFED 1.3.1 install without the SDP protocol
> loaded,
> > I
> > > need to add it. I still have the original source I installed with
> and
> > > found what looked like a suitable SRPM, I built the SRPM, installed
> > the
> > > resulting RPM and then restarted OpenSM and OpenIBD but OpenIBD does
> > not
> > > seem to have loaded "ib_sdp". I don't want to reboot this server.
> Does
> > > anyone know where I am going wrong?
> > > 
> > Try adding the lines:
> > 
> > # Load SDP module
> > SDP_LOAD=yes
> > 
> > to file /etc/infiniband/openib.conf
> > 
> > and then restart the driver
> > 
> > (if SDP_LOAD is already in the file, and set to "no", just change it
> to
> > "yes").
> > 
> > -Jack
> > 
> > > Thanks,
> > > 
> > > Rob
> > > 
> > > The SAQ Group
> > > 
> > > Registered Office: 18 Chapel Street, Petersfield, Hampshire GU32 3DZ
> > > SAQ is the trading name of SEMTEC Limited. Registered in England &
> > Wales
> > > Company Number: 06481952
> > > 
> > > http://www.saqnet.co.uk AS29219
> > > 
> > > SAQ Group Delivers high quality, honestly priced communication and
> > I.T. services to UK Business.
> > > 
> > > Broadband : Domains : Email : Hosting : CoLo : Servers : Racks :
> > Transit : Backups : Managed Networks : Remote Support.
> > > 
> > > ISPA Member
> > > 
> > > _______________________________________________
> > > general mailing list
> > > general at lists.openfabrics.org
> > > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
> > > 
> > > To unsubscribe, please visit
> > http://openib.org/mailman/listinfo/openib-general
> > > 
> > 
> 