[ofa-general]SRP target

[Marty Schlining]

Try running srp_daemon with the -n option. This adds a unique initiator_ext identifier to the each SRP target port. Also, opensm should be running on each port of the initiator.

Example:

 For each active umad device:

srp_daemon -o -e -n -d /dev/infiniband/$umad

Example script to reload srp drivers and add targets.:

#!/bin/bash

declare -a UMAD
declare -a HCA_DEVICE
declare -a PORTS
declare -a SPEEDS
declare -a WIDTHS
declare -a STATE
declare -a PORT_GUID
let PORT_COUNT=0

#---------------------------------------------------------------
# Reload all important infiniband drivers
#---------------------------------------------------------------
echo
echo "Unloading ib_srp"
modprobe -r ib_srp
echo "Loading ib_srp"
modprobe ib_srp

#---------------------------------------------------------------
# Kill all instances of opensm
#---------------------------------------------------------------
echo "Stopping opensm"
killall opensm
sleep 2
echo "REALLY stopping opensm"
killall -9 opensm

#---------------------------------------------------------------
# Run opensm on each port (even if there is no link)
#---------------------------------------------------------------
echo
ibstat | grep "Port GUID" | awk '{print $3}' | \
while read a
do
   echo "Starting OpenSM on port $a"
   opensm -g $a > /dev/null 2> /dev/null &
done

#--------------------------------------------------------------
# Delay before opensm settles down
#--------------------------------------------------------------
echo
echo "Waiting 20 seconds for everything to stabilize"
sleep 20

#--------------------------------------------------------------
# Gather and print information about IB ports
#--------------------------------------------------------------
printf "\n";
echo "UMAD  Dev:Port Speed     STATE   Port GUID"
echo "------------------------------------------------------"

INDEX=0
for a in /sys/class/infiniband_mad/umad*
do
   UMAD[$INDEX]=`basename $a` 

   HCA_DEVICE[$INDEX]=`cat $a/ibdev`

   PORTS[$INDEX]=`cat $a/port`

   SPEEDS[$INDEX]=`cat /sys/class/infiniband/${HCA_DEVICE[$INDEX]}/ports/${PORTS[$INDEX]}/rate | awk '{print $1 $3 $4}'`

   STATE[$INDEX]=`cat /sys/class/infiniband/${HCA_DEVICE[$INDEX]}/ports/${PORTS[$INDEX]}/state | awk '{print $2}'`

   PORT_GUID[$INDEX]=`ibstat ${HCA_DEVICE[$INDEX]} ${PORTS[$INDEX]} | grep "Port GUID" | awk '{print $3}'`
 
   printf "%s %s:%d %-9s %-7s %s\n" ${UMAD[$INDEX]} \
                                 ${HCA_DEVICE[$INDEX]} \
                                 ${PORTS[$INDEX]} \
                                 ${SPEEDS[$INDEX]} \
                                 ${STATE[$INDEX]} \
                                 ${PORT_GUID[$INDEX]} ;
   let INDEX=INDEX+1 
done

let PORT_COUNT=INDEX+1

echo

#---------------------------------------------------------------
# for each umad device, assign all srp targets detected on that
# device to its port. Assumes there will be an SRP target on
# each umad device that has an ACTIVE status. ibsrpdm hangs if 
# no SRP targets are on the
# specified mad device.
#---------------------------------------------------------------
INDEX=0
for ((i=1; i < ${PORT_COUNT} ; i++))
do
   umad=${UMAD[$INDEX]}
   ibdev=${HCA_DEVICE[$INDEX]}
   port=${PORTS[$INDEX]}
   state=${STATE[$INDEX]}

   if [ "$state" = "ACTIVE" ];
   then
      echo "Looking for devices on $ibdev port $port ($umad)"

      # This is the preferred method for adding SRP targets, Uses the contents
      # of the default config file /etc/srp_daemon.conf
      srp_daemon -o -e -n -d /dev/infiniband/$umad

#--------------------------------------------------------------
# Old (but still used) methods for adding SRP targets.
#--------------------------------------------------------------
#      ibsrpdm -d /dev/infiniband/$umad -c | awk '{ORS="";print $1",max_sect=65535"}' > /sys/class/infiniband_srp/srp-$ibdev-$port/add_target
#   ibsrpdm -d /dev/infiniband/$umad -c | awk '{print $1",max_sect=4096"}'
#   ibsrpdm -d /dev/infiniband/$umad -c | awk '{ORS="";print $1",max_sect=65535"}' > /sys/class/infiniband_srp/srp-$ibdev-$port/add_target
   fi

#--------------------------------------------------------------

   let INDEX=INDEX+1
done

#--------------------------------------------------------------
# List all LUN targets
#--------------------------------------------------------------
echo
echo "Target LUN listing"
lsscsi -gl 
echo

#--------------------------------------------------------------
# List all SCSI hosts
#--------------------------------------------------------------
echo "Host listing"
lsscsi -Hl
echo


-----Original Message-----
From: general-bounces at lists.openfabrics.org [mailto:general-bounces at lists.openfabrics.org] On Behalf Of Celine Bourde
Sent: Tuesday, January 20, 2009 12:43 PM
To: general at lists.openfabrics.org; celine.bourde at bull.net
Subject: [ofa-general]SRP target 


I've trouble running srp tools.

I try to setup the following configuration.

The first machine,``the initiator'', has a 2-port Connectx IB card.
Each port is connected to the a second machine, ``the target'',
which has a 2-port Connectx IB card too.

+-----------------+                     +---------------+
|                 |- - - - - - - - - - -|               |
|   initiator     |                     |     target    |
|                 |- - - - - - - - - - -|               |
+-----------------+                     +---------------+

the ibstat utility shows that both link are up.

My goal is to setup a failover connection. On the target machine,
both ports expose the the same storage device.

on my "target machine", I see 2 targets :
# cat /sys/class/infiniband_srpt/srpt-mlx4_0/login_info
tid_ext=0002c90300001d2c,ioc_guid=0002c90300001d2c,pkey=ffff,dgid=fe800000000000000002c90300001d2d,service_id=0002c90300001d2c
tid_ext=0002c90300001d2c,ioc_guid=0002c90300001d2c,pkey=ffff,dgid=fe800000000000000002c90300001d2e,service_id=0002c90300001d2c

But when I launch ibsrpdm -c on "initiator machine", the tool
discovers only one
target corresponding to the first port of my distant HCA card.

# ibsrpdm -c
id_ext=0002c90300001d2c,ioc_guid=0002c90300001d2c,dgid=fe800000000000000002c90300001d2d,pkey=ffff,service_id=0002c90300001d2c
# srp_daemon
IO Unit Info:
   port LID:        0002
   port GID:        fe800000000000000002c90300001d2d
   change ID:       0100
   max controllers: 0x10

   controller[  1]
       GUID:      0002c90300001d2c
       vendor ID: 000002
       device ID: 00634a
       IO class : 0100
       ID:        Mellanox OFED SRP target
       service entries: 1
           service[  0]: 0002c90300001d2c / SRP.T10:0002c90300001d2c

so i see only one of the targets; any hints why i don't see both
targets?
    
Thanks,

Céline Bourde.

_______________________________________________
general mailing list
general at lists.openfabrics.org
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general