[openib-general] [ANNOUNCE] Initial checkin of SRP initiator

Fri Jul 8 22:13:22 PDT 2005

I just checked in a first version of a SCSI RDMA Protocol (SRP)
initiator under infiniband/ulp/srp.  Error handling is generously
described as minimal, but normal operation is reasonably solid.  The
initiator has been tested against the Topspin/Cisco FC/SRP gateway as
well as another vendor's native IB storage.

The initiator does not do any device management queries to discover
targets in the kernel.  Instead, userspace needs to pass in target
information through an add_target file under /sys/class/infiniband_srp/.
See below for an example.

I'll be on vacation Monday through Wednesday of next week, and in
Ottawa for the Kernel Summit and Ottawa Linux Symposium the whole week
after that, but more documentation and better code will appear once
I'm back.  In the meantime, please fill my mailbox with feedback.

I'm attaching some (very hacky) python scripts to this email that can
be used as a device management client to discover storage.  Just untar
the package and follow the README to use it.

Finally, here's an example SRP in action: I discover a target and then
connect to it and do some I/O.  Don't take the performance numbers
seriously, since this is a single old 36 GB drive in a JBOD.  I see
_much_ higher numbers talking to a fancy native IB RAID box.

    # dmcli -t /sys/class/infiniband/mthca1/ports/1 -d /dev/infiniband/umad1 2
    IO Unit Info:
        max controllers:  1

        controller[  1]
        GUID:      0005ad00000015dd
        vendor ID: 0005ad
        device ID: 0005ad
        ID:         Topspin SRP/FC TCA
        service entries:  11
            service[  0]: 0000000000000066 / SRP.T10:21000004CF758882
            service[  1]: 0000000000000066 / SRP.T10:21000004CF92BDDA
            service[  2]: 0000000000000066 / SRP.T10:21000004CF92BE07
            service[  3]: 0000000000000066 / SRP.T10:21000004CF92BE12
            service[  4]: 0000000000000066 / SRP.T10:21000004CF92BE15
            service[  5]: 0000000000000066 / SRP.T10:21000004CF92BE24
            service[  6]: 0000000000000066 / SRP.T10:21000004CF92BE33
            service[  7]: 0000000000000066 / SRP.T10:21000004CF92C110
            service[  8]: 0000000000000066 / SRP.T10:21000004CF92C354
            service[  9]: 0000000000000066 / SRP.T10:21000004CF92C360
            service[ 10]: 0000000000000066 / SRP.T10:21000004CFE7A949

    # modprobe ib_srp
    # echo id_ext=21000004CFE7A949,ioc_gguid=0005ad00000015dd,dgid=fe800000000000000005ad00000015dd,pkey=ffff,service_id=0000000000000066 > /sys/class/infiniband_srp/srp-mthca1-1/add_target
    # dmesg
    [45839.392438] ib_srp: new target: id_ext 21000004cfe7a949 ioc_guid 0005ad00000015dd pkey ffff service_id 0000000000000066 dgid fe80:0000:0000:0000:0005:ad00:0000:15dd
    [45839.393309] ib_srp: Topspin/Cisco initiator port ID workaround activated for target GUID 0005ad00000015dd
    [45839.393814] ib_srp: REJ received
    [45839.393821] ib_srp: Topspin/Cisco redirect to target port GID fe800000000000000005ad0000012371
    [45839.394428] ib_srp: Topspin/Cisco initiator port ID workaround activated for target GUID 0005ad00000015dd
    [45839.394675] ib_srp: MRA received
    [45839.405305] ib_srp: REP received
    [45839.405309]   req_lim_delta 32
    [45839.405311]   max_it_iu_len 384
    [45839.405312]   max_ti_iu_len 128
    [45839.405483] scsi2 : SRP.T10:21000004CFE7A949
    [45839.420978]   Vendor: SEAGATE   Model: ST336607FC        Rev: 0004
    [45839.442096]   Type:   Direct-Access                      ANSI SCSI revision: 03
    [45839.468006] SCSI device sdb: 71687372 512-byte hdwr sectors (36704 MB)
    [45839.490459] SCSI device sdb: drive cache: write through
    [45839.508436] SCSI device sdb: 71687372 512-byte hdwr sectors (36704 MB)
    [45839.530869] SCSI device sdb: drive cache: write through
    [45839.548097]  sdb: sdb1 sdb2 sdb3 sdb4
    [45839.571821] Attached scsi disk sdb at scsi2, channel 0, id 0, lun 0
    # hdparm -tT /dev/sdb

    /dev/sdb:
     Timing cached reads:   2312 MB in  2.00 seconds = 1156.18 MB/sec
     Timing buffered disk reads:  194 MB in  3.02 seconds =  64.16 MB/sec

    # mount /dev/sdb1 /mnt
    # cd /mnt
    # iozone -s 512m -r64 -i0 -i1         Iozone: Performance Test of File I/O
                    Version $Revision: 3.221 $                 Compiled for 32 bit mode.
                    Build: linux

            Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
                         Al Slater, Scott Rhine, Mike Wisner, Ken Goss
                         Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
                         Randy Dunlap, Mark Montague, Dan Million,
                         Jean-Marc Zucconi, Jeff Blomberg,
                         Erik Habbinga, Kris Strecker.

            Run began: Fri Jul  8 22:05:24 2005

            File size set to 524288 KB
            Record Size 64 KB
            Command line used: iozone -s 512m -r64 -i0 -i1
            Output is in Kbytes/sec
            Time Resolution = 0.000001 seconds.
            Processor cache size set to 1024 Kbytes.
            Processor cache line size set to 32 bytes.
            File stride size set to 17 * record size.
                                                                random  random    bkwd  record  stride 
                  KB  reclen   write rewrite    read    reread    read   write    read rewrite    read  fwrite frewrite   fread  freread
              524288      64   46894   52908    27904    28061

 - R.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: PyOpenIB-1.0.tar.gz
Type: application/x-compressed-tar
Size: 4925 bytes
Desc: not available
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050708/3e37162d/attachment.bin>