<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
<title></title>
</head>
<body bgcolor="#ffffff" text="#000000">
As joe requested, here's before reloading the openIB stuff<br>
<br>
[root@h2o01 ~]# lsmod | grep ib_<br>
ib_srp 38281 0 <br>
ib_sdp 54785 0 <br>
rdma_cm 39381 2 ib_sdp,rdma_ucm<br>
ib_addr 11081 1 rdma_cm<br>
ib_mthca 158357 0 <br>
ib_ipoib 96673 0 <br>
ib_umad 20969 0 <br>
ib_ucm 20937 0 <br>
ib_uverbs 43377 2 rdma_ucm,ib_ucm<br>
ib_cm 42217 4 ib_srp,rdma_cm,ib_ipoib,ib_ucm<br>
ib_sa 48841 4 ib_srp,rdma_cm,ib_ipoib,ib_cm<br>
ib_mad 43497 4 ib_mthca,ib_umad,ib_cm,ib_sa<br>
ib_core 69825 13
ib_srp,ib_sdp,rdma_ucm,rdma_cm,iw_cm,ib_mthca,ib_ipoib,ib_umad,ib_ucm,ib_uverbs,ib_cm,ib_sa,ib_mad<br>
ipv6 285729 29 ib_ipoib<br>
scsi_mod 145425 3 ib_srp,libata,sd_mod<br>
<br>
[root@h2o01 ~]# /etc/init.d/openibd restart<br>
Unloading OpenIB kernel modules: [ OK ]<br>
Loading OpenIB kernel modules: [ OK ]<br>
[root@h2o01 ~]# <br>
[root@h2o01 ~]# lsmod | grep ib_<br>
ib_srp 38281 0 <br>
ib_sdp 54785 0 <br>
ib_ipoib 96673 0 <br>
rdma_cm 39381 2 ib_sdp,rdma_ucm<br>
ib_addr 11081 1 rdma_cm<br>
ib_mthca 158357 0 <br>
ib_umad 20969 0 <br>
ib_ucm 20937 0 <br>
ib_uverbs 43377 2 rdma_ucm,ib_ucm<br>
ib_cm 42217 4 ib_srp,ib_ipoib,rdma_cm,ib_ucm<br>
ib_sa 48841 4 ib_srp,ib_ipoib,rdma_cm,ib_cm<br>
ib_mad 43497 4 ib_mthca,ib_umad,ib_cm,ib_sa<br>
ib_core 69825 13
ib_srp,ib_sdp,ib_ipoib,rdma_ucm,rdma_cm,iw_cm,ib_mthca,ib_umad,ib_ucm,ib_uverbs,ib_cm,ib_sa,ib_mad<br>
ipv6 285729 29 ib_ipoib<br>
scsi_mod 145425 3 ib_srp,libata,sd_mod<br>
<br>
[root@h2o01 ~]# ifconfig ib0 up<br>
[root@h2o01 ~]# ifconfig ib0<br>
ib0 Link encap:UNSPEC HWaddr
80-00-04-04-FE-80-00-00-00-00-00-00-00-00-00-00 <br>
inet addr:192.168.2.1 Bcast:192.168.2.255 Mask:255.255.255.0<br>
UP BROADCAST MULTICAST MTU:2044 Metric:1<br>
RX packets:0 errors:0 dropped:0 overruns:0 frame:0<br>
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0<br>
collisions:0 txqueuelen:256 <br>
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)<br>
<br>
<br>
As you can see that the ib0 interface does come up and routing seems to
be setup properly<br>
<br>
Kernel IP routing table<br>
Destination Gateway Genmask Flags MSS Window
irtt Iface<br>
192.168.2.0 * 255.255.255.0 U 0 0
0 ib0<br>
10.84.4.0 * 255.255.255.0 U 0 0
0 eth0<br>
192.168.1.0 * 255.255.255.0 U 0 0
0 eth1<br>
169.254.0.0 * 255.255.0.0 U 0 0
0 ib0<br>
224.0.0.0 * 240.0.0.0 U 0 0
0 eth1<br>
default 10.84.4.1 0.0.0.0 UG 0 0 <br>
<br>
But if i ping a node, i get nothing:<br>
<br>
[root@h2o01 ~]# ping h2oi05.cluster<br>
PING h2oi05.cluster (192.168.2.5) 56(84) bytes of data.<br>
>From h2oi01.cluster (192.168.2.1) icmp_seq=0 Destination Host
Unreachable<br>
>From h2oi01.cluster (192.168.2.1) icmp_seq=1 Destination Host
Unreachable<br>
>From h2oi01.cluster (192.168.2.1) icmp_seq=2 Destination Host
Unreachable<br>
>From h2oi01.cluster (192.168.2.1) icmp_seq=4 Destination Host
Unreachable<br>
>From h2oi01.cluster (192.168.2.1) icmp_seq=5 Destination Host
Unreachable<br>
>From h2oi01.cluster (192.168.2.1) icmp_seq=6 Destination Host
Unreachable<br>
<br>
--- h2oi05.cluster ping statistics ---<br>
8 packets transmitted, 0 received, +6 errors, 100% packet loss, time
7000ms<br>
, pipe 4<br>
<br>
I did ping myself and i get :<br>
<br>
[root@h2o01 ~]# ping h2oi01.cluster<br>
PING h2oi01.cluster (192.168.2.1) 56(84) bytes of data.<br>
64 bytes from h2oi01.cluster (192.168.2.1): icmp_seq=0 ttl=64
time=0.018 ms<br>
64 bytes from h2oi01.cluster (192.168.2.1): icmp_seq=1 ttl=64
time=0.010 ms<br>
64 bytes from h2oi01.cluster (192.168.2.1): icmp_seq=2 ttl=64
time=0.011 ms<br>
64 bytes from h2oi01.cluster (192.168.2.1): icmp_seq=3 ttl=64
time=0.011 ms<br>
64 bytes from h2oi01.cluster (192.168.2.1): icmp_seq=4 ttl=64
time=0.015 ms<br>
64 bytes from h2oi01.cluster (192.168.2.1): icmp_seq=5 ttl=64
time=0.008 ms<br>
<br>
--- h2oi01.cluster ping statistics ---<br>
6 packets transmitted, 6 received, 0% packet loss, time 4999ms<br>
rtt min/avg/max/mdev = 0.008/0.012/0.018/0.004 ms, pipe 2<br>
<br>
<br>
It appears that the Ip stack over IB is up and installed, just not
talking on the wire or passing thru the switch.<br>
<br>
jeff<br>
<br>
<br>
Joe Landman wrote:
<blockquote cite="mid:49BFCA36.3050104@scalableinformatics.com"
type="cite">
<pre wrap="">jeffrey Lang wrote:
</pre>
<blockquote type="cite">
<pre wrap="">First let me say, I hope this is the right list for this email, if not
please forgive me.
I have a small 16 node compute cluster. The university where I work
at recently opened a new Datacenter. My cluster was moved from the old
Datacenter. Before the move the inifiniband was working properly,
after the move the ipoib has stopped working.
</pre>
</blockquote>
<pre wrap=""><!---->
[...]
</pre>
<blockquote type="cite">
<pre wrap="">I've reset the sm on the switch, but nothing seems to work.
Any ideas of where to look for whats causing the problem?
</pre>
</blockquote>
<pre wrap=""><!---->
Could you do an
lsmod | grep ib_
I assume you did an
/etc/init.d/openibd restart
If not, now is a good time ... then rerun the lsmod above.
If you don't see ib_ipoib, then you might try this
ifconfig ib0 up
then send the output of
lsmod | grep ib_
ifconfig ib0
If these still don't work, try
modprobe ib_ipoib
ifconfig ib0 up
ifconfig ib0
</pre>
</blockquote>
</body>
</html>