[ofa-general] IO Size more than 48K
chas williams - CONTRACTOR
chas at cmf.nrl.navy.mil
Fri Nov 30 10:18:43 PST 2007
increase the srp_sg_tablesize isnt going to help much. you want to
make the rdma segments bigger, and with the default sg tablesize
is big enough for large i/o if you get the segment size up. increasing
the srp_sg_tablesize leads to a bigger iu message size and some
devices have some limitations in this area.
checking with our analyzer here on sles10, i cannot get dd to issue
a read with an rdma segment larger than 512k. however:
echo 256 > /sys/block/<device>/queue/max_phys_segments
echo 256 > /sys/block/<device>/queue/max_hw_segments
echo 1 > /proc/scsi/sg/allow_dio
sg_dd if=/dev/<device> of=/tmp/file dio=1 bpt=2048 count=10240 blk_sgio=1
does produce 5 scsi reads with a single rdma segment of 1MB.
In message <01B9E81EECACE94DBBD0A556E768FB8A01E3C7A0 at NAMAIL2.ad.lsil.com>,"Batw
ara, Ashish" writes:
>This is what I did as suggested by Vu and it seems to be working.
>However, when I send 2MB IO, it gets broken into 512K+1MB+512K by SRP as
>seen on analyzer. I am just wondering what the logic is? On the other
>side, when we increase the srp_sg_tablesize beyond 256, we are seeing
>following message in /var/log/messages "Nov 29 21:17:50 p50 kernel:
>REJ reason 0x3" which indicates "IB_CM_REJ_NO_RESOURCES", so not sure
>how to get around to this problem to send larger IO than 1MB in one
>shot.
>
>=20
>
>=20
>
>modprobe ib_srp srp_sg_tablesize=3D256
>
>echo
>id_ext=3D200600A0B81138C9,max_sect=3D4096,ioc_guid=3D00a0b81112da0003,dgi=
>d=3Dfe8
>000000000000000a0b81112da0001,pkey=3Dffff,service_id=3D200600a0b81138c9>
>/sys/class/infiniband_srp/srp-mthca0-1/add_target
>
>=20
>
>-----Original Message-----
>From: chas williams - CONTRACTOR [mailto:chas at cmf.nrl.navy.mil]=20
>Sent: Friday, November 30, 2007 11:43 AM
>To: Kevin Harms
>Cc: Vu Pham; openib-general at openib.org; Batwara, Ashish
>Subject: Re: [ofa-general] IO Size more than 48K=20
>
>=20
>
>addtionally, you might need to echo 'blocks' >
>
>/sys/block/<device/queue/max_hw_segments to increase the size of the
>
>rdma segments.
>
>=20
>
>max_hw_segments doesnt exist on all kernels i think.
>
>=20
>
>In message <3A453CF1-5FFC-44BF-8F72-7E3EF5AA6E41 at alcf.anl.gov>,Kevin
>Harms writ
>
>es:
>
>>=20
>
>> you may also have to go to /sys/block/sdX/queue and echo 1024 > =20
>
>>max_sectors_kb
>
>> if you use the srp_daemon you can also add:
>
>> a max_sect=3D2048 to /etc/srp_daemon.conf
>
>>=20
>
>>kevin
>
>>=20
>
>>On Nov 29, 2007, at 11:08 AM, Vu Pham wrote:
>
>>=20
>
>>>=20
>
>>>> Hi,
>
>>>> We are using OFED-1.2, and using xdd and some other tools, and =20
>
>>>> trying to
>
>>>> send 1/2MB IOs, but what we are seeing in analyzer traces, that =20
>
>>>> memory
>
>>>> descriptor in SRP command shows max. 48K which means 1MB I/Os has =20
>
>>>> broken
>
>>>> into smaller SRP request from initiator.
>
>>>> How can I have this I/O directly going to target? What parameter I =20
>
>>>> need
>
>>>> to change?
>
>>>>=20
>
>>>>=20
>
>>>=20
>
>>> module param srp_sg_tablesize (default is 12 ie. 12 x 4K =3D 48K)
>
>>> and/or
>
>>> max_sect=3Dyyy in echo id_ext=3Dxxx,...,max_sect=3D1024,service_id=3D =
>> /sys/
>
>
>>> class/infiniband_srp/...
>
>>>=20
>
>>> -vu
>
>>>=20
>
>>>> Thanks
>
>>>> Ashish
>
>>>> _______________________________________________
>
>>>> general mailing list
>
>>>> general at lists.openfabrics.org
>
>>>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>
>>>>=20
>
>>>> To unsubscribe, please visit
>http://openib.org/mailman/listinfo/openib-gene
>
>>ral
>
>>>>=20
>
>>>=20
>
>>> _______________________________________________
>
>>> general mailing list
>
>>> general at lists.openfabrics.org
>
>>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>
>>>=20
>
>>> To unsubscribe, please visit
>http://openib.org/mailman/listinfo/openib-gener
>
>>al
>
>>>=20
>
>>=20
>
>>_______________________________________________
>
>>general mailing list
>
>>general at lists.openfabrics.org
>
>>http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>
>>=20
>
>>To unsubscribe, please visit
>http://openib.org/mailman/listinfo/openib-general
>
>>=20
>
>
>------_=_NextPart_001_01C83379.3F66B657
>Content-Type: text/html;
> charset="us-ascii"
>Content-Transfer-Encoding: quoted-printable
>
><html xmlns:o=3D"urn:schemas-microsoft-com:office:office" =
>xmlns:w=3D"urn:schemas-microsoft-com:office:word" =
>xmlns:st1=3D"urn:schemas-microsoft-com:office:smarttags" =
>xmlns=3D"http://www.w3.org/TR/REC-html40">
>
><head>
><META HTTP-EQUIV=3D"Content-Type" CONTENT=3D"text/html; =
>charset=3Dus-ascii">
><meta name=3DGenerator content=3D"Microsoft Word 11 (filtered medium)">
><o:SmartTagType =
>namespaceuri=3D"urn:schemas-microsoft-com:office:smarttags"
> name=3D"PersonName"/>
><!--[if !mso]>
><style>
>st1\:*{behavior:url(#default#ieooui) }
></style>
><![endif]-->
><style>
><!--
> /* Style Definitions */
> p.MsoNormal, li.MsoNormal, div.MsoNormal
> {margin:0in;
> margin-bottom:.0001pt;
> font-size:12.0pt;
> font-family:"Times New Roman";}
>a:link, span.MsoHyperlink
> {color:blue;
> text-decoration:underline;}
>a:visited, span.MsoHyperlinkFollowed
> {color:purple;
> text-decoration:underline;}
>p.MsoPlainText, li.MsoPlainText, div.MsoPlainText
> {margin:0in;
> margin-bottom:.0001pt;
> font-size:10.0pt;
> font-family:"Courier New";}
>@page Section1
> {size:8.5in 11.0in;
> margin:1.0in 77.95pt 1.0in 77.95pt;}
>div.Section1
> {page:Section1;}
>-->
></style>
>
></head>
>
><body lang=3DEN-US link=3Dblue vlink=3Dpurple>
>
><div class=3DSection1>
>
><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span =
>style=3D'font-size:
>10.0pt'>This is what I did as suggested by Vu and it seems to be =
>working.
>However, when I send 2MB IO, it gets broken into 512K+1MB+512K by SRP as =
>seen
>on analyzer. I am just wondering what the logic is? On the other side, =
>when we
>increase the srp_sg_tablesize beyond 256, we are seeing following =
>message in
>/var/log/messages “Nov 29 21:17:50 p50 kernel: REJ =
>reason 0x3”
>which indicates “IB_CM_REJ_NO_RESOURCES”, so not sure how to =
>get
>around to this problem to send larger IO than 1MB in one =
>shot.<o:p></o:p></span></font></p>
>
><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span =
>style=3D'font-size:
>10.0pt'><o:p> </o:p></span></font></p>
>
><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span =
>style=3D'font-size:
>10.0pt'><o:p> </o:p></span></font></p>
>
><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span =
>style=3D'font-size:
>10.0pt'>modprobe ib_srp <b><font color=3Dred><span =
>style=3D'color:red;font-weight:
>bold'>srp_sg_tablesize=3D256</span></font></b><o:p></o:p></span></font></=
>p>
>
><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span =
>style=3D'font-size:
>10.0pt'>echo id_ext=3D200600A0B81138C9,<b><font color=3Dred><span =
>style=3D'color:
>red;font-weight:bold'>max_sect=3D4096</span></font></b>,ioc_guid=3D00a0b8=
>1112da0003,dgid=3Dfe8000000000000000a0b81112da0001,pkey=3Dffff,service_id=
>=3D200600a0b81138c9>
>/sys/class/infiniband_srp/srp-mthca0-1/add_target<o:p></o:p></span></font=
>></p>
>
><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span =
>style=3D'font-size:
>10.0pt'><o:p> </o:p></span></font></p>
>
><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span =
>style=3D'font-size:
>10.0pt'>-----Original Message-----<br>
>From: chas williams - CONTRACTOR [mailto:chas at cmf.nrl.navy.mil] <br>
>Sent: Friday, November 30, 2007 11:43 AM<br>
>To: Kevin Harms<br>
>Cc: Vu Pham; openib-general at openib.org; <st1:PersonName =
>w:st=3D"on">Batwara,
> Ashish</st1:PersonName><br>
>Subject: Re: [ofa-general] IO Size more than 48K </span></font></p>
>
><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span =
>style=3D'font-size:
>10.0pt'><o:p> </o:p></span></font></p>
>
><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span =
>style=3D'font-size:
>10.0pt'>addtionally, you might need to echo 'blocks' =
>><o:p></o:p></span></font></p>
>
><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span =
>style=3D'font-size:
>10.0pt'>/sys/block/<device/queue/max_hw_segments to increase the size =
>of the<o:p></o:p></span></font></p>
>
><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span =
>style=3D'font-size:
>10.0pt'>rdma segments.<o:p></o:p></span></font></p>
>
><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span =
>style=3D'font-size:
>10.0pt'><o:p> </o:p></span></font></p>
>
><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span =
>style=3D'font-size:
>10.0pt'>max_hw_segments doesnt exist on all kernels i =
>think.<o:p></o:p></span></font></p>
>
><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span =
>style=3D'font-size:
>10.0pt'><o:p> </o:p></span></font></p>
>
><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span =
>style=3D'font-size:
>10.0pt'>In message
><3A453CF1-5FFC-44BF-8F72-7E3EF5AA6E41 at alcf.anl.gov>,Kevin Harms =
>writ<o:p></o:p></span></font></p>
>
><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span =
>style=3D'font-size:
>10.0pt'>es:<o:p></o:p></span></font></p>
>
><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span =
>style=3D'font-size:
>10.0pt'>><o:p> </o:p></span></font></p>
>
><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span =
>style=3D'font-size:
>10.0pt'>> you may also have to go to =
>/sys/block/sdX/queue and echo 1024
>> <o:p></o:p></span></font></p>
>
><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span =
>style=3D'font-size:
>10.0pt'>>max_sectors_kb<o:p></o:p></span></font></p>
>
><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span =
>style=3D'font-size:
>10.0pt'>> if you use the srp_daemon you can =
>also add:<o:p></o:p></span></font></p>
>
><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span =
>style=3D'font-size:
>10.0pt'>> a max_sect=3D2048 to =
>/etc/srp_daemon.conf<o:p></o:p></span></font></p>
>
><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span =
>style=3D'font-size:
>10.0pt'>><o:p> </o:p></span></font></p>
>
><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span =
>style=3D'font-size:
>10.0pt'>>kevin<o:p></o:p></span></font></p>
>
><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span =
>style=3D'font-size:
>10.0pt'>><o:p> </o:p></span></font></p>
>
><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span =
>style=3D'font-size:
>10.0pt'>>On Nov 29, 2007, at 11:08 AM, Vu Pham =
>wrote:<o:p></o:p></span></font></p>
>
><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span =
>style=3D'font-size:
>10.0pt'>><o:p> </o:p></span></font></p>
>
><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span =
>style=3D'font-size:
>10.0pt'>>><o:p> </o:p></span></font></p>
>
><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span =
>style=3D'font-size:
>10.0pt'>>>> Hi,<o:p></o:p></span></font></p>
>
><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span =
>style=3D'font-size:
>10.0pt'>>>> We are using OFED-1.2, and using xdd and some other =
>tools,
>and <o:p></o:p></span></font></p>
>
><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span =
>style=3D'font-size:
>10.0pt'>>>> trying to<o:p></o:p></span></font></p>
>
><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span =
>style=3D'font-size:
>10.0pt'>>>> send 1/2MB IOs, but what we are seeing in analyzer =
>traces,
>that <o:p></o:p></span></font></p>
>
><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span =
>style=3D'font-size:
>10.0pt'>>>> memory<o:p></o:p></span></font></p>
>
><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span =
>style=3D'font-size:
>10.0pt'>>>> descriptor in SRP command shows max. 48K which =
>means 1MB
>I/Os has <o:p></o:p></span></font></p>
>
><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span =
>style=3D'font-size:
>10.0pt'>>>> broken<o:p></o:p></span></font></p>
>
><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span =
>style=3D'font-size:
>10.0pt'>>>> into smaller SRP request from =
>initiator.<o:p></o:p></span></font></p>
>
><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span =
>style=3D'font-size:
>10.0pt'>>>> How can I have this I/O directly going to target? =
>What
>parameter I <o:p></o:p></span></font></p>
>
><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span =
>style=3D'font-size:
>10.0pt'>>>> need<o:p></o:p></span></font></p>
>
><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span =
>style=3D'font-size:
>10.0pt'>>>> to change?<o:p></o:p></span></font></p>
>
><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span =
>style=3D'font-size:
>10.0pt'>>>><o:p> </o:p></span></font></p>
>
><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span =
>style=3D'font-size:
>10.0pt'>>>><o:p> </o:p></span></font></p>
>
><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span =
>style=3D'font-size:
>10.0pt'>>><o:p> </o:p></span></font></p>
>
><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span =
>style=3D'font-size:
>10.0pt'>>> module param srp_sg_tablesize (default is 12 ie. 12 x =
>4K =3D
>48K)<o:p></o:p></span></font></p>
>
><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span =
>style=3D'font-size:
>10.0pt'>>> and/or<o:p></o:p></span></font></p>
>
><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span =
>style=3D'font-size:
>10.0pt'>>> max_sect=3Dyyy in echo =
>id_ext=3Dxxx,...,max_sect=3D1024,service_id=3D
>> /sys/ <o:p></o:p></span></font></p>
>
><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span =
>style=3D'font-size:
>10.0pt'>>> class/infiniband_srp/...<o:p></o:p></span></font></p>
>
><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span =
>style=3D'font-size:
>10.0pt'>>><o:p> </o:p></span></font></p>
>
><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span =
>style=3D'font-size:
>10.0pt'>>> -vu<o:p></o:p></span></font></p>
>
><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span =
>style=3D'font-size:
>10.0pt'>>><o:p> </o:p></span></font></p>
>
><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span =
>style=3D'font-size:
>10.0pt'>>>> Thanks<o:p></o:p></span></font></p>
>
><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span =
>style=3D'font-size:
>10.0pt'>>>> Ashish<o:p></o:p></span></font></p>
>
><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span =
>style=3D'font-size:
>10.0pt'>>>> =
>_______________________________________________<o:p></o:p></span></font><=
>/p>
>
><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span =
>style=3D'font-size:
>10.0pt'>>>> general mailing list<o:p></o:p></span></font></p>
>
><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span =
>style=3D'font-size:
>10.0pt'>>>> =
>general at lists.openfabrics.org<o:p></o:p></span></font></p>
>
><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span =
>style=3D'font-size:
>10.0pt'>>>>
>http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general<o:p></o:p><=
>/span></font></p>
>
><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span =
>style=3D'font-size:
>10.0pt'>>>><o:p> </o:p></span></font></p>
>
><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span =
>style=3D'font-size:
>10.0pt'>>>> To unsubscribe, please visit
>http://openib.org/mailman/listinfo/openib-gene<o:p></o:p></span></font></=
>p>
>
><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span =
>style=3D'font-size:
>10.0pt'>>ral<o:p></o:p></span></font></p>
>
><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span =
>style=3D'font-size:
>10.0pt'>>>><o:p> </o:p></span></font></p>
>
><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span =
>style=3D'font-size:
>10.0pt'>>><o:p> </o:p></span></font></p>
>
><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span =
>style=3D'font-size:
>10.0pt'>>> =
>_______________________________________________<o:p></o:p></span></font><=
>/p>
>
><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span =
>style=3D'font-size:
>10.0pt'>>> general mailing list<o:p></o:p></span></font></p>
>
><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span =
>style=3D'font-size:
>10.0pt'>>> =
>general at lists.openfabrics.org<o:p></o:p></span></font></p>
>
><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span =
>style=3D'font-size:
>10.0pt'>>> =
>http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general<o:p></o:p><=
>/span></font></p>
>
><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span =
>style=3D'font-size:
>10.0pt'>>><o:p> </o:p></span></font></p>
>
><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span =
>style=3D'font-size:
>10.0pt'>>> To unsubscribe, please visit =
>http://openib.org/mailman/listinfo/openib-gener<o:p></o:p></span></font><=
>/p>
>
><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span =
>style=3D'font-size:
>10.0pt'>>al<o:p></o:p></span></font></p>
>
><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span =
>style=3D'font-size:
>10.0pt'>>><o:p> </o:p></span></font></p>
>
><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span =
>style=3D'font-size:
>10.0pt'>><o:p> </o:p></span></font></p>
>
><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span =
>style=3D'font-size:
>10.0pt'>>_______________________________________________<o:p></o:p></s=
>pan></font></p>
>
><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span =
>style=3D'font-size:
>10.0pt'>>general mailing list<o:p></o:p></span></font></p>
>
><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span =
>style=3D'font-size:
>10.0pt'>>general at lists.openfabrics.org<o:p></o:p></span></font></p>
>
><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span =
>style=3D'font-size:
>10.0pt'>>http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general=
><o:p></o:p></span></font></p>
>
><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span =
>style=3D'font-size:
>10.0pt'>><o:p> </o:p></span></font></p>
>
><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span =
>style=3D'font-size:
>10.0pt'>>To unsubscribe, please visit =
>http://openib.org/mailman/listinfo/openib-general<o:p></o:p></span></font=
>></p>
>
><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span =
>style=3D'font-size:
>10.0pt'>><o:p> </o:p></span></font></p>
>
></div>
>
></body>
>
></html>
>
>------_=_NextPart_001_01C83379.3F66B657--
>
More information about the general
mailing list