[ofa-general] Update (Re: open iSCSI over iSER target RPM ...)
Joe Landman
landman at scalableinformatics.com
Thu Feb 7 08:05:03 PST 2008
Update:
[root at woody etc]# dd if=/dev/zero of=/big/local.file bs=256k count=100000
100000+0 records in
100000+0 records out
26214400000 bytes (26 GB) copied, 58.7484 seconds, 446 MB/s
Better. I rebuilt OFED 1.2.5.5. Are there specific recommended tuning
guides for iSER? Backing store in this case are real disks, and we can
sink/source >750 MB/s on them, so I am not worried about disk IO
bottlenecks, more worried about bad config of iSCSI/iSER.
BTW: the 2TB LUN limit I asked about is still here in this code. Same
machines (initiator and target) used for SRP reported correct LUN sizes.
Here we are using the -868 open-iscsi initiator, and the tgt RPM
announced. I would like to dig into this.
This is what I am getting in dmesg for this iSER target:
iscsi: registered transport (tcp)
iscsi: registered transport (iser)
iser: iser_connect:connecting to: 10.2.1.2, port 0xbc0c
iser: iser_cma_handler:event 0 conn ffff81024b9f69c0 id ffff810209748c00
iser: iser_cma_handler:event 2 conn ffff81024b9f69c0 id ffff810209748c00
iser: iser_create_ib_conn_res:setting conn ffff81024b9f69c0 cma_id
ffff810209748c00: fmr_pool ffff81024bfb32c0 qp ffff8101cb16d600
iser: iser_cma_handler:event 9 conn ffff81024b9f69c0 id ffff810209748c00
iser: iscsi_iser_ep_poll:ib conn ffff81024b9f69c0 rc = 1
scsi13 : iSCSI Initiator over iSER, v.0.1
iser: iscsi_iser_conn_bind:binding iscsi conn ffff81021b65fa90 to
iser_conn ffff81024b9f69c0
Vendor: IET Model: Controller Rev: 0001
Type: RAID ANSI SCSI revision: 05
scsi 13:0:0:0: Attached scsi generic sg2 type 12
Vendor: IET Model: VIRTUAL-DISK Rev: 0001
Type: Direct-Access ANSI SCSI revision: 05
sdc : very big device. try to use READ CAPACITY(16).
sdc : READ CAPACITY(16) failed.
sdc : status=1, message=00, host=0, driver=08
sdc : use 0xffffffff as device size
SCSI device sdc: 4294967296 512-byte hdwr sectors (2199023 MB)
sdc: Write Protect is off
sdc: Mode Sense: 79 00 00 08
SCSI device sdc: drive cache: write back
sdc : very big device. try to use READ CAPACITY(16).
sdc : READ CAPACITY(16) failed.
sdc : status=1, message=00, host=0, driver=08
sdc : use 0xffffffff as device size
SCSI device sdc: 4294967296 512-byte hdwr sectors (2199023 MB)
sdc: Write Protect is off
sdc: Mode Sense: 79 00 00 08
SCSI device sdc: drive cache: write back
sdc: unknown partition table
sd 13:0:0:1: Attached scsi disk sdc
sd 13:0:0:1: Attached scsi generic sg3 type 0
and this is what we get in SRP
scsi6 : SRP.T10:0008F104039862A4
Vendor: SCST_BIO Model: vdisk0 Rev: 096
Type: Direct-Access ANSI SCSI revision: 04
sdc : very big device. try to use READ CAPACITY(16).
SCSI device sdc: 12693355130 512-byte hdwr sectors (6498998 MB)
sdc: Write Protect is off
sdc: Mode Sense: 6b 00 10 08
SCSI device sdc: drive cache: write back w/ FUA
This looks suspiciously like a 2^32 limit somewhere.
Our exported device is
[root at jr1 ~]# parted /dev/sdb print
Model: Areca jrvs1 (scsi)
Disk /dev/sdb: 6500GB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Number Start End Size File system Flags
1 0.00kB 6500GB 6500GB xfs
and this is what tgtadm reports
[root at jr1 ~]# tgtadm --lld iscsi --op show --mode target
Target 1: iqn.2001-04.com.jr1-jackrabbit.small
System information:
Driver: iscsi
Status: running
I_T nexus information:
I_T nexus: 4
Initiator: iqn.1996-04.voltaire.com:01:dfa8888a3fd
Connection: 0
RDMA IP Address: 10.2.1.1
LUN information:
LUN: 0
Type: controller
SCSI ID: deadbeaf1:0
SCSI SN: beaf10
Size: 0
Online: No
Poweron/Reset: Yes
Removable media: No
Backing store: No backing store
LUN: 1
Type: disk
SCSI ID: deadbeaf1:1
SCSI SN: beaf11
Size: 5T
Online: Yes
Poweron/Reset: No
Removable media: No
Backing store: /dev/sdb
Account information:
ACL information:
10.2.1.1
So it looks like the LUN 1 is approximately correct (5T ???) on the
target, and incorrect when the initiator asks for it.
Please note that I have successfully used the full 6+TB as an iSCSI
target using the SCST-iscsi code, so I do know that the initiator works
correctly.
Is there a source RPM/tree for this target?
Joe Landman wrote:
> Hi Erez
>
> Erez Zilber wrote:
>> stgt (SCSI target) is an open-source framework for storage target
>> drivers. It supports iSCSI over iSER among other storage target drivers.
>>
>> Voltaire added a git tree for stgt that will be added to OFED 1.4:
>> http://www2.openfabrics.org/git/?p=~dorons/tgt.git;a=summary
>>
>> Until OFED 1.4 gets released, it is possible to install the stgt RPM on
>> top of OFED 1.3. For more details about how to install and use stgt,
>> please refer to
>> https://wiki.openfabrics.org/tiki-index.php?page=ISER-target
>>
>> Some performance numbers that were measured by OSC (using SDR cards):
>
> Is there a 2TB limit on this target? It turns our 6TB partition into a
> 2TB lun.
>
>> * READ: 920 MB/sec
>> * WRITE: 850 MB/sec
>
> Not getting anything even remotely close to this. Are there more
> details on configuration somewhere? I followed the web page as indicated.
>
> Joe
>
>>
>> We hope to have DDR measurements numbers soon.
>>
>
>
--
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics LLC,
email: landman at scalableinformatics.com
web : http://www.scalableinformatics.com
http://jackrabbit.scalableinformatics.com
phone: +1 734 786 8423
fax : +1 866 888 3112
cell : +1 734 612 4615
More information about the general
mailing list