<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 TRANSITIONAL//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; CHARSET=UTF-8">
<META NAME="GENERATOR" CONTENT="GtkHTML/3.3.2">
</HEAD>
<BODY LINK="#0000ff">
lspci -vv : ( 02:00.0 works fine)<BR>
02:00.0 PCI bridge: Mellanox Technologies MT23108 PCI Bridge (rev a1) (prog-if 00 [Normal decode])<BR>
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-<BR>
Status: Cap+ 66Mhz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-<BR>
Latency: 32, Cache Line Size 08<BR>
Bus: primary=02, secondary=03, subordinate=03, sec-latency=32<BR>
Memory behind bridge: e0000000-efffffff<BR>
Secondary status: 66Mhz+ FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- <SERR- <PERR-<BR>
BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-<BR>
Capabilities: [70] PCI-X bridge device.<BR>
Secondary Status: 64bit+, 133MHz+, SCD-, USC-, SCO-, SRD- Freq=3<BR>
Status: Bus=2 Dev=0 Func=0 64bit+ 133MHz+ SCD- USC-, SCO-, SRD-<BR>
: Upstream: Capacity=512, Commitment Limit=512<BR>
: Downstream: Capacity=128, Commitment Limit=128<BR>
<BR>
03:00.0 InfiniBand: Mellanox Technologies MT23108 InfiniHost (rev a1)<BR>
Subsystem: Mellanox Technologies MT23108 InfiniHost<BR>
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-<BR>
Status: Cap+ 66Mhz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-<BR>
Latency: 32, Cache Line Size 08<BR>
Interrupt: pin A routed to IRQ 193<BR>
Region 0: Memory at e8800000 (64-bit, non-prefetchable) [size=1M]<BR>
Region 2: Memory at e8000000 (64-bit, prefetchable) [size=8M]<BR>
Region 4: Memory at e0000000 (64-bit, prefetchable) [size=128M]<BR>
Capabilities: [40] MSI-X: Enable- Mask- TabSize=32<BR>
Vector table: BAR=0 offset=00082000<BR>
PBA: BAR=0 offset=00082200<BR>
Capabilities: [50] Vital Product Data<BR>
Capabilities: [60] Message Signalled Interrupts: 64bit+ Queue=0/5 Enable-<BR>
Address: 0000000000000000 Data: 0000<BR>
Capabilities: [70] PCI-X non-bridge device.<BR>
Command: DPERE- ERO- RBC=3 OST=1<BR>
Status: Bus=3 Dev=0 Func=0 64bit+ 133MHz+ SCD- USC-, DC=simple, DMMRBC=3, DMOST=1, DMCRS=0, RSCEM-<BR>
<BR>
<BR>
on failed nodes: lspci -vv<BR>
<BR>
<FONT COLOR="#ff0000">02:01.0</FONT> PCI bridge: Mellanox Technologies MT23108 PCI Bridge (rev a1) (prog-if 00 [Normal decode])<BR>
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-<BR>
Status: Cap+ 66Mhz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-<BR>
Latency: 32, Cache Line Size 08<BR>
Bus: primary=02, secondary=03, subordinate=03, sec-latency=32<BR>
Memory behind bridge: e0000000-efffffff<BR>
Secondary status: 66Mhz+ FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- <SERR- <PERR-<BR>
BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-<BR>
Capabilities: [70] PCI-X bridge device.<BR>
Secondary Status: 64bit+, 133MHz+, SCD-, USC-, SCO-, SRD- Freq=3<BR>
Status: Bus=2 Dev=1 Func=0 64bit+ 133MHz+ SCD- USC-, SCO-, SRD-<BR>
: Upstream: Capacity=512, Commitment Limit=512<BR>
: Downstream: Capacity=128, Commitment Limit=128<BR>
<BR>
03:00.0 InfiniBand: Mellanox Technologies MT23108 InfiniHost (rev a1)<BR>
Subsystem: Mellanox Technologies MT23108 InfiniHost<BR>
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-<BR>
Status: Cap+ 66Mhz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-<BR>
Latency: 32, Cache Line Size 08<BR>
Interrupt: pin A routed to IRQ 201<BR>
Region 0: Memory at e8800000 (64-bit, non-prefetchable) [size=1M]<BR>
Region 2: Memory at e8000000 (64-bit, prefetchable) [size=8M]<BR>
Region 4: Memory at e0000000 (64-bit, prefetchable) [size=128M]<BR>
Capabilities: [40] MSI-X: Enable- Mask- TabSize=32<BR>
Vector table: BAR=0 offset=00082000<BR>
PBA: BAR=0 offset=00082200<BR>
Capabilities: [50] Vital Product Data<BR>
Capabilities: [60] Message Signalled Interrupts: 64bit+ Queue=0/5 Enable-<BR>
Address: 0000000000000000 Data: 0000<BR>
Capabilities: [70] PCI-X non-bridge device.<BR>
Command: DPERE- ERO- RBC=3 OST=1<BR>
Status: Bus=3 Dev=0 Func=0 64bit+ 133MHz+ SCD- USC-, DC=simple, DMMRBC=3, DMOST=1, DMCRS=0, RSCEM-<BR>
<BR>
<BR>
<BR>
On Sun, 2005-09-18 at 11:22 +0300, Guy Shevet wrote:<BR>
<BLOCKQUOTE TYPE=CITE>
<FONT SIZE="2"><FONT COLOR="#000080">Hi , </FONT></FONT><BR>
<BR>
<FONT SIZE="2"><FONT COLOR="#000080"> Can you please send “lspci –vv “ for all Mellanox devices on the system ( the virtual bridge and the simple device ) .</FONT></FONT><BR>
<BR>
<FONT SIZE="2"><FONT COLOR="#000080"> </FONT></FONT><BR>
<BR>
<FONT SIZE="2"><FONT COLOR="#000080">--- Guy</FONT></FONT><BR>
<BR>
<BR>
<DIV ALIGN=center>
<HR>
</DIV><BR>
<BR>
<B><FONT SIZE="2"><FONT COLOR="#000000">From:</FONT></FONT></B><FONT COLOR="#000000"><FONT SIZE="2"> QiWang, Chen [mailto:QiWang.Chen@Clustars.CN] </FONT></FONT><BR>
<B><FONT SIZE="2"><FONT COLOR="#000000">Sent:</FONT></FONT></B><FONT COLOR="#000000"><FONT SIZE="2"> Sunday, September 18, 2005 9:56 AM</FONT></FONT><BR>
<B><FONT SIZE="2"><FONT COLOR="#000000">To:</FONT></FONT></B><FONT COLOR="#000000"><FONT SIZE="2"> Grant Grundler</FONT></FONT><BR>
<B><FONT SIZE="2"><FONT COLOR="#000000">Cc:</FONT></FONT></B><FONT COLOR="#000000"><FONT SIZE="2"> openib-general@openib.org</FONT></FONT><BR>
<B><FONT SIZE="2"><FONT COLOR="#000000">Subject:</FONT></FONT></B><FONT COLOR="#000000"><FONT SIZE="2"> Re: [openib-general] could not add HCA InfiniHost0</FONT></FONT><BR>
<BR>
<BR>
<FONT COLOR="#000000"> </FONT><BR>
<BR>
<TT><FONT SIZE="2"><FONT COLOR="#000000">Hi, Grant</FONT></FONT></TT><BR>
<TT><FONT SIZE="2"><FONT COLOR="#000000"> I can not change the slot, I only have one slot (BladeServer), </FONT></FONT></TT><BR>
<BR>
<TT><FONT SIZE="2"><FONT COLOR="#000000">--------------------------------------------------------------------</FONT></FONT></TT><BR>
<TT><FONT SIZE="2"><FONT COLOR="#000000">lspci -vvs 02:00.0 :</FONT></FONT></TT><BR>
<BR>
<TT><FONT SIZE="2"><FONT COLOR="#000000">02:00.0 PCI bridge: Mellanox Technologies MT23108 PCI Bridge (rev a1)</FONT></FONT></TT><BR>
<TT><FONT SIZE="2"><FONT COLOR="#000000">(prog-if 00 [Normal decode])</FONT></FONT></TT><BR>
<TT><FONT SIZE="2"><FONT COLOR="#000000"> Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-</FONT></FONT></TT><BR>
<TT><FONT SIZE="2"><FONT COLOR="#000000">ParErr- Stepping- SERR+ FastB2B-</FONT></FONT></TT><BR>
<TT><FONT SIZE="2"><FONT COLOR="#000000"> Status: Cap+ 66Mhz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort-</FONT></FONT></TT><BR>
<TT><FONT SIZE="2"><FONT COLOR="#000000"><TAbort- <MAbort- >SERR- <PERR-</FONT></FONT></TT><BR>
<TT><FONT SIZE="2"><FONT COLOR="#000000"> Latency: 32, Cache Line Size 08</FONT></FONT></TT><BR>
<TT><FONT SIZE="2"><FONT COLOR="#000000"> Bus: primary=02, secondary=03, subordinate=03, sec-latency=32</FONT></FONT></TT><BR>
<TT><FONT SIZE="2"><FONT COLOR="#000000"> Memory behind bridge: e0000000-efffffff</FONT></FONT></TT><BR>
<TT><FONT SIZE="2"><FONT COLOR="#000000"> Secondary status: 66Mhz+ FastB2B- ParErr- DEVSEL=medium >TAbort-</FONT></FONT></TT><BR>
<TT><FONT SIZE="2"><FONT COLOR="#000000"><TAbort- <MAbort- <SERR- <PERR-</FONT></FONT></TT><BR>
<TT><FONT SIZE="2"><FONT COLOR="#000000"> BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-</FONT></FONT></TT><BR>
<TT><FONT SIZE="2"><FONT COLOR="#000000"> Capabilities: [70] PCI-X bridge device.</FONT></FONT></TT><BR>
<TT><FONT SIZE="2"><FONT COLOR="#000000"> Secondary Status: 64bit+, 133MHz+, SCD-, USC-, SCO-,</FONT></FONT></TT><BR>
<TT><FONT SIZE="2"><FONT COLOR="#000000">SRD- Freq=3</FONT></FONT></TT><BR>
<TT><FONT SIZE="2"><FONT COLOR="#000000"> Status: Bus=2 Dev=0 Func=0 64bit+ 133MHz+ SCD- USC-,</FONT></FONT></TT><BR>
<TT><FONT SIZE="2"><FONT COLOR="#000000">SCO-, SRD-</FONT></FONT></TT><BR>
<TT><FONT SIZE="2"><FONT COLOR="#000000"> : Upstream: Capacity=512, Commitment Limit=512</FONT></FONT></TT><BR>
<TT><FONT SIZE="2"><FONT COLOR="#000000"> : Downstream: Capacity=128, Commitment Limit=128</FONT></FONT></TT><BR>
<BR>
<TT><FONT SIZE="2"><FONT COLOR="#000000">---------------------------------------------------------------------------------------------------</FONT></FONT></TT><BR>
<BR>
<TT><FONT SIZE="2"><FONT COLOR="#000000">lspci -vvs 02:01.0</FONT></FONT></TT><BR>
<BR>
<BR>
<TT><FONT SIZE="2"><FONT COLOR="#000000">02:01.0 PCI bridge: Mellanox Technologies MT23108 PCI Bridge (rev a1)</FONT></FONT></TT><BR>
<TT><FONT SIZE="2"><FONT COLOR="#000000">(prog-if 00 [Normal decode])</FONT></FONT></TT><BR>
<TT><FONT SIZE="2"><FONT COLOR="#000000"> Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-</FONT></FONT></TT><BR>
<TT><FONT SIZE="2"><FONT COLOR="#000000">ParErr- Stepping- SERR+ FastB2B-</FONT></FONT></TT><BR>
<TT><FONT SIZE="2"><FONT COLOR="#000000"> Status: Cap+ 66Mhz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort-</FONT></FONT></TT><BR>
<TT><FONT SIZE="2"><FONT COLOR="#000000"><TAbort- <MAbort- >SERR- <PERR-</FONT></FONT></TT><BR>
<TT><FONT SIZE="2"><FONT COLOR="#000000"> Latency: 32, Cache Line Size 08</FONT></FONT></TT><BR>
<TT><FONT SIZE="2"><FONT COLOR="#000000"> Bus: primary=02, secondary=03, subordinate=03, sec-latency=32</FONT></FONT></TT><BR>
<TT><FONT SIZE="2"><FONT COLOR="#000000"> Memory behind bridge: e0000000-efffffff</FONT></FONT></TT><BR>
<TT><FONT SIZE="2"><FONT COLOR="#000000"> Secondary status: 66Mhz+ FastB2B- ParErr- DEVSEL=medium >TAbort-</FONT></FONT></TT><BR>
<TT><FONT SIZE="2"><FONT COLOR="#000000"><TAbort- <MAbort- <SERR- <PERR-</FONT></FONT></TT><BR>
<TT><FONT SIZE="2"><FONT COLOR="#000000"> BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-</FONT></FONT></TT><BR>
<TT><FONT SIZE="2"><FONT COLOR="#000000"> Capabilities: [70] PCI-X bridge device.</FONT></FONT></TT><BR>
<TT><FONT SIZE="2"><FONT COLOR="#000000"> Secondary Status: 64bit+, 133MHz+, SCD-, USC-, SCO-,</FONT></FONT></TT><BR>
<TT><FONT SIZE="2"><FONT COLOR="#000000">SRD- Freq=3</FONT></FONT></TT><BR>
<TT><FONT SIZE="2"><FONT COLOR="#000000"> Status: Bus=2 Dev=1 Func=0 64bit+ 133MHz+ SCD- USC-,</FONT></FONT></TT><BR>
<TT><FONT SIZE="2"><FONT COLOR="#000000">SCO-, SRD-</FONT></FONT></TT><BR>
<TT><FONT SIZE="2"><FONT COLOR="#000000"> : Upstream: Capacity=512, Commitment Limit=512</FONT></FONT></TT><BR>
<TT><FONT SIZE="2"><FONT COLOR="#000000"> : Downstream: Capacity=128, Commitment Limit=128</FONT></FONT></TT><BR>
<BR>
<TT><FONT SIZE="2"><FONT COLOR="#000000">-----------------------------------------------------------------------------------------------------</FONT></FONT></TT><BR>
<BR>
<TT><FONT SIZE="2"><FONT COLOR="#000000">Thx</FONT></FONT></TT><BR>
<BR>
<FONT COLOR="#000000">On Thu, 2005-09-15 at 08:50 -0700, Grant Grundler wrote: </FONT><BR>
<BR>
<PRE>
<FONT SIZE="2"><FONT COLOR="#000000"> </FONT></FONT>
<FONT SIZE="2"><FONT COLOR="#000000">On Thu, Sep 15, 2005 at 06:11:26PM +0800, QiWang, Chen wrote:</FONT></FONT>
<FONT SIZE="2"><FONT COLOR="#000000">> there are some diff:</FONT></FONT>
<FONT SIZE="2"><FONT COLOR="#000000">> 02:00.0 --> work</FONT></FONT>
<FONT SIZE="2"><FONT COLOR="#000000">> 02:01.0 --> failed</FONT></FONT>
<FONT SIZE="2"><FONT COLOR="#000000">> </FONT></FONT>
<FONT SIZE="2"><FONT COLOR="#000000">> and first time I install the ib-verbs on node1, It also failed, because </FONT></FONT>
<FONT SIZE="2"><FONT COLOR="#000000">> lspci= 02:01.0, an I don not know how i change 02:01.0 to 02:00.0, and </FONT></FONT>
<FONT SIZE="2"><FONT COLOR="#000000">> it works fine for me.</FONT></FONT>
<FONT SIZE="2"><FONT COLOR="#000000"> </FONT></FONT>
<FONT SIZE="2"><FONT COLOR="#000000">You can only change it by removing the Mellanox card and re-installing</FONT></FONT>
<FONT SIZE="2"><FONT COLOR="#000000">in the other slot.</FONT></FONT>
<FONT SIZE="2"><FONT COLOR="#000000"> </FONT></FONT>
<FONT SIZE="2"><FONT COLOR="#000000">Can you post "lspci -vvs 02:01.0" output from the machine that failed?</FONT></FONT>
<FONT SIZE="2"><FONT COLOR="#000000">Can you post "lspci -vvs 02:00.0" output from the machine that worked?</FONT></FONT>
<FONT SIZE="2"><FONT COLOR="#000000"> </FONT></FONT>
<FONT SIZE="2"><FONT COLOR="#000000">grant</FONT></FONT>
</PRE>
<TABLE CELLSPACING="0" CELLPADDING="0" WIDTH="100%">
<TR>
<TD>
<PRE>
<FONT SIZE="2"> </FONT>
<FONT SIZE="2">-- </FONT>
<FONT SIZE="2">QiWang, Chen <<A HREF="mailto:QiWang.Chen@Clustars.CN">QiWang.Chen@Clustars.CN</A>></FONT>
<FONT SIZE="2">Clustars Supercomputing Technology corp.</FONT>
<FONT SIZE="2"><A HREF="http://www.Clustars.CN">http://www.Clustars.CN</A></FONT>
<FONT SIZE="2">TEL:+86-0816-2546345-815</FONT>
<FONT SIZE="2">FAX:+86-0816-2546370</FONT>
<FONT SIZE="2">Mobile</FONT>:+86-13096497499
</PRE>
</TD>
</TR>
</TABLE>
<BR>
<FONT COLOR="#000000"> </FONT><BR>
<BR>
<BR>
</BLOCKQUOTE>
<TABLE CELLSPACING="0" CELLPADDING="0" WIDTH="100%">
<TR>
<TD>
<PRE>
--
QiWang, Chen <<A HREF="mailto:QiWang.Chen@Clustars.CN">QiWang.Chen@Clustars.CN</A>>
Clustars Supercomputing Technology corp.
<A HREF="http://www.Clustars.CN">http://www.Clustars.CN</A>
TEL:+86-0816-2546345-815
FAX:+86-0816-2546370
Mobile:+86-13096497499
</PRE>
</TD>
</TR>
</TABLE>
</BODY>
</HTML>