<html>
<body>
<font size=3><br>
PCIe supports a Max_Payload_Size (MPS) of 128B-4KB (128B is
mandatory). However, nearly all x86 chipsets I've seen implement
only 128-256B within the Root Complex with generally only servers
supporting the larger of the two values. As a result, the PCIe side
is limited to 256B for DMA Writes and due to the way completions are
accomplished, will return 1-4 completions of 64B cache lines for DMA Read
completions with many tending to the 1-2 cache line sized
completion. <br><br>
Unclear what you are trying to ascertain. The PCIe protocol
efficiency will, of course, vary quite a bit by the traffic patterns for
reads and writes. You can find various presentations showing
small MPS is sufficient for many workloads primarily because the
workloads are small transaction dominated, e.g. graphics. For
server workloads, the traffic patterns also vary quite a bit so the
argument has been successfully put forth that with slight
over-provisioning of the PCIe link, there should not be any bottlenecks
(other than the usual ones when it comes to I/O-to--memory bandwidth
especially if there are multiple high-speed I/O devices very active in
the system). One might argue that Moore's Law should help out
here as well by allowing for more buffers to be cost effectively
provisioned but some people prefer to spend those gates on other purposes
so things will likely remain largely unchanged for the foreseeable future
and people will need to use higher signaling rates / wider links to
obtain the associated PCIe bandwidth. <br><br>
One thing to note is the number of PCIe lanes and wider link PCIe Root
Ports hasn't gone up that much over time. As a result, the
push will be to drive to higher signaling rates to meet workload needs
which will necessitate a broader interoperability matrix. The
PCIe 2.0 CEM specification also mandates these interoperability
requirements for the silicon on each side of the link to avoid customer
confusion on what can be plugged into what slots and deliver the best
possible performance as a function of link width and signaling
rate. <br><br>
Mike<br><br>
<br>
At 05:47 AM 3/31/2008, Koen Segers wrote:<br>
<blockquote type=cite class=cite cite="">Hi Dotan,<br><br>
You are refering to the MTU of InfiniBand. This parameter defines
how<br>
much data a single packet can contain when transferred over an<br>
Infiniband network. <br><br>
I want to know the MTU of PCI-Express. PCI-Express is used to
transfer<br>
data from memory to HCA and vice versa. As far as I know, this MTU
can<br>
differ. Our SAS HBA, has a PCI-Express MTU of 512 bytes. <br><br>
Regards,<br><br>
Koen<br><br>
<br>
On Mon, 2008-03-31 at 15:05 +0300, Dotan Barak wrote:<br>
> Hi.<br>
> <br>
> You can check what is the maximum supported MTU of your HCA using
<br>
> ibv_devinfo.<br>
> Most of the HCAs supports 2KB MTU and some of them 4KB MTU.<br>
> <br>
> The MTU is the maximum payload size that can be sent/received<br>
> (the space for the IB headers are not part of the MTU)<br>
> <br>
> <br>
> Every connected QP (RC/UC) can define which MTU value he is using
<br>
> (according to the<br>
> used path).<br>
> <br>
> <br>
> I hope that is the info you asked for ..<br>
> Dotan<br>
> <br>
> Koen Segers wrote:<br>
> > Hi,<br>
> ><br>
> > I'm interested in computing the overhead coming from
PCI-Express to an <br>
> > IB HCA. Therefore I need to know the payload size of different
types <br>
> > of HCA.<br>
> ><br>
> > We have InfiniHost III Mellanox cards of 4x SDR and DDR IB.
According <br>
> > the lspci -vvv command on SLES 10 SP1, the PCI-e payload of
these <br>
> > cards is maximum 128 bytes.<br>
> ><br>
> > Can someone give my the payload size for a ConnectX 4x SDR and
DDR IB <br>
> > card?<br>
> ><br>
> > Thank you,<br>
> ><br>
> > Koen<br>
> ><br>
> > This is my output:<br>
> > 41:00.0 InfiniBand: Mellanox Technologies MT25208 InfiniHost
III Ex <br>
> > (rev a0)<br>
> > Subsystem:
Mellanox Technologies MT25208 InfiniHost III Ex<br>
> > Control: I/O-
Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- <br>
> > ParErr+ Stepping- SERR- FastB2B-<br>
> > Status: Cap+
66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <br>
> > <TAbort- <MAbort- >SERR- <PERR-<br>
> > Latency: 0,
Cache Line Size: 64 bytes<br>
> > Interrupt: pin
A routed to IRQ 209<br>
> > Region 0:
Memory at e1700000 (64-bit, non-prefetchable) [size=1M]<br>
> > Region 2:
Memory at e1800000 (64-bit, prefetchable) [size=8M]<br>
> > Capabilities:
[40] Power Management version 2<br>
>
>
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA <br>
> > PME(D0-,D1-,D2-,D3hot-,D3cold-)<br>
>
>
Status: D0 PME-Enable- DSel=0 DScale=0 PME-<br>
> > Capabilities:
[48] Vital Product Data<br>
> > Capabilities:
[90] Message Signalled Interrupts: Mask- 64bit+ <br>
> > Queue=0/5 Enable-<br>
>
>
Address: 0000000000000000 Data: 0000<br>
> > Capabilities:
[84] MSI-X: Enable+ Mask- TabSize=32<br>
>
>
Vector table: BAR=0 offset=00082000<br>
>
>
PBA: BAR=0 offset=00082200<br>
> > Capabilities:
[60] Express Endpoint IRQ 0<br>
> >
*
Device: Supported: MaxPayload 128 bytes, PhantFunc 0, <br>
> > ExtTag+*<br>
>
>
Device: Latency L0s <64ns, L1 unlimited<br>
>
>
Device: AtnBtn- AtnInd- PwrInd-<br>
>
>
Device: Errors: Correctable- Non-Fatal+ Fatal+ <br>
> > Unsupported-<br>
>
>
Device: RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-<br>
>
>
Device: MaxPayload 128 bytes, MaxReadReq 4096 bytes<br>
>
>
Link: Supported Speed 2.5Gb/s, Width x8, ASPM L0s, Port 8<br>
>
>
Link: Latency L0s unlimited, L1 unlimited<br>
>
>
Link: ASPM Disabled RCB 128 bytes CommClk- ExtSynch-<br>
>
>
Link: Speed 2.5Gb/s, Width x8<br>
> ><br>
> > <br>
> > *** Disclaimer ***<br>
> ><br>
> > Vlaamse Radio- en Televisieomroep<br>
> > Auguste Reyerslaan 52, 1043 Brussel<br>
> ><br>
> > nv van publiek recht<br>
> > BTW BE 0244.142.664<br>
> > RPR Brussel<br>
> >
<a href="http://www.vrt.be/disclaimer" eudora="autourl">
http://www.vrt.be/disclaimer</a><br>
> >
------------------------------------------------------------------------<br>
> ><br>
> > _______________________________________________<br>
> > general mailing list<br>
> > general@lists.openfabrics.org<br>
> >
<a href="http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general" eudora="autourl">
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general</a><br>
> ><br>
> > To unsubscribe, please visit
<a href="http://openib.org/mailman/listinfo/openib-general" eudora="autourl">
http://openib.org/mailman/listinfo/openib-general</a><br>
> <br>
*** Disclaimer ***<br><br>
Vlaamse Radio- en Televisieomroep<br>
Auguste Reyerslaan 52, 1043 Brussel<br><br>
nv van publiek recht<br>
BTW BE 0244.142.664<br>
RPR Brussel<br>
<a href="http://www.vrt.be/disclaimer" eudora="autourl">
http://www.vrt.be/disclaimer</a><br>
_______________________________________________<br>
general mailing list<br>
general@lists.openfabrics.org<br>
<a href="http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general" eudora="autourl">
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general</a><br><br>
To unsubscribe, please visit
<a href="http://openib.org/mailman/listinfo/openib-general" eudora="autourl">
http://openib.org/mailman/listinfo/openib-general</a>
</font></blockquote></body>
</html>