<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">
<div>All:</div><div><br></div><div>I have two test machines, ib0-test and ib1-test, connect together by a pair of <span class="Apple-style-span" style="color: rgb(20, 20, 20); ">QLogic InfiniPath QLE7140 HCAs. There is no router involved. My goal is to test DHCP provisioning onver infiniband on this simple topology. Unfortunately I am unable to get the client to accept the offered address.</span></div><div><font class="Apple-style-span" color="#141414"><br></font></div><div><font class="Apple-style-span" color="#141414">I have the DHCP daemon running on ib0-test. I have given the ib0 interface the following configuration:</font></div><div><font class="Apple-style-span" color="#141414"><br></font></div><div><font class="Apple-style-span" color="#141414"><div><font class="Apple-style-span" face="Monaco" size="2"><span class="Apple-style-span" style="font-size: 10px;"><span class="Apple-tab-span" style="white-space:pre"> </span>DEVICE=ib0</span></font></div><div><font class="Apple-style-span" face="Monaco" size="2"><span class="Apple-style-span" style="font-size: 10px;"><span class="Apple-tab-span" style="white-space:pre"> </span>ONBOOT=yes</span></font></div><div><font class="Apple-style-span" face="Monaco" size="2"><span class="Apple-style-span" style="font-size: 10px;"><span class="Apple-tab-span" style="white-space:pre"> </span>BOOTPROTO=static</span></font></div><div><font class="Apple-style-span" face="Monaco" size="2"><span class="Apple-style-span" style="font-size: 10px;"><span class="Apple-tab-span" style="white-space:pre"> </span>IPADDR=172.10.10.2</span></font></div><div><font class="Apple-style-span" face="Monaco" size="2"><span class="Apple-style-span" style="font-size: 10px;"><span class="Apple-tab-span" style="white-space:pre"> </span>NETMASK=255.255.255.0</span></font></div><div><br></div><div>If I set the ib0 interface on ib1-test to a static IP similar to this one, everything works well. However, if I try DHCP, it will not accept the offered IP address. This is the configuration on ib1-test:</div><div><br></div><div><span class="Apple-style-span" style="color: rgb(0, 0, 0); "><div><span class="Apple-style-span" style="font-family: Monaco; font-size: 10px; "><span class="Apple-tab-span" style="white-space:pre"> </span>DEVICE=ib0</span></div><div><div><font class="Apple-style-span" face="Monaco" size="2"><span class="Apple-tab-span" style="white-space:pre"> </span>ONBOOT=yes</font></div><div><font class="Apple-style-span" face="Monaco" size="2"><span class="Apple-tab-span" style="white-space:pre"> </span>BOOTPROTO=dhcp</font></div><div><font class="Apple-style-span" face="Monaco" size="2"><span class="Apple-style-span" style="font-size: 10px;"><br></span></font></div></div></span></div></font></div><div>If I do a tcpdump on both nodes during the DHCP request, I get very similar answers. On the dhcpd host, ib0-test, I get:</div><div><br></div><div><div><font class="Apple-style-span" face="Monaco" size="2"><span class="Apple-tab-span" style="white-space:pre"> </span>18:40:32.982491 IP ib1-test.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request, length: 300</font></div><div><font class="Apple-style-span" face="Monaco" size="2"><span class="Apple-tab-span" style="white-space:pre"> </span>18:40:32.984083 arp who-has 172.10.10.158 tell 172.10.10.2 hardware #32</font></div><div><font class="Apple-style-span" face="Monaco" size="2"><span class="Apple-tab-span" style="white-space:pre"> </span>18:40:33.001142 IP 172.10.10.2.bootps > 255.255.255.255.bootpc: BOOTP/DHCP, Reply, length: 309</font></div><div><font class="Apple-style-span" face="Monaco" size="2"><span class="Apple-tab-span" style="white-space:pre"> </span>18:40:33.984028 arp who-has 172.10.10.158 tell 172.10.10.2 hardware #32</font></div><div><font class="Apple-style-span" face="Monaco" size="2"><span class="Apple-tab-span" style="white-space:pre"> </span>18:40:34.983974 arp who-has 172.10.10.158 tell 172.10.10.2 hardware #32</font></div><div><font class="Apple-style-span" face="Monaco" size="2"><span class="Apple-tab-span" style="white-space:pre"> </span>18:40:37.991935 IP ib1-test.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request, length: 300</font></div><div><font class="Apple-style-span" face="Monaco" size="2"><span class="Apple-tab-span" style="white-space:pre"> </span>18:40:37.992125 IP 172.10.10.2.bootps > 255.255.255.255.bootpc: BOOTP/DHCP, Reply, length: 309</font></div><div><font class="Apple-style-span" face="Monaco" size="2"><span class="Apple-tab-span" style="white-space:pre"> </span>18:40:52.992217 IP ib1-test.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request, length: 300</font></div><div><font class="Apple-style-span" face="Monaco" size="2"><span class="Apple-tab-span" style="white-space:pre"> </span>18:40:52.992428 IP 172.10.10.2.bootps > 255.255.255.255.bootpc: BOOTP/DHCP, Reply, length: 309</font></div><div><font class="Apple-style-span" face="Monaco" size="2"><span class="Apple-tab-span" style="white-space:pre"> </span>18:41:13.991209 IP ib1-test.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request, length: 300</font></div><div><font class="Apple-style-span" face="Monaco" size="2"><span class="Apple-tab-span" style="white-space:pre"> </span>18:41:13.991384 IP 172.10.10.2.bootps > 255.255.255.255.bootpc: BOOTP/DHCP, Reply, length: 309</font></div><div><font class="Apple-style-span" face="Monaco" size="2"><span class="Apple-tab-span" style="white-space:pre"> </span>18:41:23.991705 IP ib1-test.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request, length: 300</font></div><div><font class="Apple-style-span" face="Monaco" size="2"><span class="Apple-tab-span" style="white-space:pre"> </span>18:41:23.991909 IP 172.10.10.2.bootps > 255.255.255.255.bootpc: BOOTP/DHCP, Reply, length: 309</font></div><div><font class="Apple-style-span" face="Monaco" size="2"><span class="Apple-style-span" style="font-size: 10px;"><br></span></font></div></div><div>Whereas on the client host, ib1-test, I get:</div><div><br></div><div><div><font class="Apple-style-span" face="Monaco" size="2"><span class="Apple-tab-span" style="white-space:pre"> </span>18:40:32.991272 IP ib1-test.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request, length: 300</font></div><div><font class="Apple-style-span" face="Monaco" size="2"><span class="Apple-tab-span" style="white-space:pre"> </span>18:40:32.992963 arp who-has 172.10.10.158 tell 172.10.10.2 hardware #32</font></div><div><font class="Apple-style-span" face="Monaco" size="2"><span class="Apple-tab-span" style="white-space:pre"> </span>18:40:33.010013 IP 172.10.10.2.bootps > 255.255.255.255.bootpc: BOOTP/DHCP, Reply, length: 309</font></div><div><font class="Apple-style-span" face="Monaco" size="2"><span class="Apple-tab-span" style="white-space:pre"> </span>18:40:33.992895 arp who-has 172.10.10.158 tell 172.10.10.2 hardware #32</font></div><div><font class="Apple-style-span" face="Monaco" size="2"><span class="Apple-tab-span" style="white-space:pre"> </span>18:40:34.992846 arp who-has 172.10.10.158 tell 172.10.10.2 hardware #32</font></div><div><font class="Apple-style-span" face="Monaco" size="2"><span class="Apple-tab-span" style="white-space:pre"> </span>18:40:38.000761 IP ib1-test.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request, length: 300</font></div><div><font class="Apple-style-span" face="Monaco" size="2"><span class="Apple-tab-span" style="white-space:pre"> </span>18:40:38.001014 IP 172.10.10.2.bootps > 255.255.255.255.bootpc: BOOTP/DHCP, Reply, length: 309</font></div><div><font class="Apple-style-span" face="Monaco" size="2"><span class="Apple-tab-span" style="white-space:pre"> </span>18:40:53.001075 IP ib1-test.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request, length: 300</font></div><div><font class="Apple-style-span" face="Monaco" size="2"><span class="Apple-tab-span" style="white-space:pre"> </span>18:40:53.001363 IP 172.10.10.2.bootps > 255.255.255.255.bootpc: BOOTP/DHCP, Reply, length: 309</font></div><div><font class="Apple-style-span" face="Monaco" size="2"><span class="Apple-tab-span" style="white-space:pre"> </span>18:41:14.000117 IP ib1-test.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request, length: 300</font></div><div><font class="Apple-style-span" face="Monaco" size="2"><span class="Apple-tab-span" style="white-space:pre"> </span>18:41:14.000366 IP 172.10.10.2.bootps > 255.255.255.255.bootpc: BOOTP/DHCP, Reply, length: 309</font></div><div><font class="Apple-style-span" face="Monaco" size="2"><span class="Apple-tab-span" style="white-space:pre"> </span>18:41:24.000642 IP ib1-test.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request, length: 300</font></div><div><font class="Apple-style-span" face="Monaco" size="2"><span class="Apple-tab-span" style="white-space:pre"> </span>18:41:24.000917 IP 172.10.10.2.bootps > 255.255.255.255.bootpc: BOOTP/DHCP, Reply, length: 309</font></div></div><div><br></div><div>There are no substantive differences between these two transfers. Having said, that, there are differences at the packet level. I will not insert all of the packet data for brevity's sake. However I can offer an example: one packet may have the following data:</div><div><br></div><div><div><span class="Apple-tab-span" style="white-space:pre"> </span>< 0000 00 00 00 20 00 00 00 00 00 00 00 00 00 00 08 00</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>> 0000 00 04 00 20 00 00 00 00 00 00 00 00 00 00 08 00</div></div><div><br></div><div>while a subsequent packet may have:</div><div><br></div><div><div><span class="Apple-tab-span" style="white-space:pre"> </span>< 0000 00 04 00 20 00 00 00 00 00 00 00 00 00 00 08 06</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>> 0000 00 00 00 20 00 00 00 00 00 00 00 00 00 00 08 06</div></div><div><br></div><div>I will not claim to understand enough about TCP/IP transport to say whether this is normal. From a pattern perspective, it appears to be in line with what I would expect. </div><div><br></div><div>The TCP dumps are in line with the output I get in /var/log/messages. From the dhcpd host machine I get:</div><div><br></div><div><div><font class="Apple-style-span" face="Monaco"><font class="Apple-style-span" size="2"><span class="Apple-tab-span" style="white-space:pre"> </span>Jul 18 17:41:35 ib0-test dhcpd: DHCPDISCOVER from via ib0</font></font></div><div><font class="Apple-style-span" face="Monaco"><font class="Apple-style-span" size="2"><span class="Apple-tab-span" style="white-space:pre"> </span>Jul 18 17:41:36 ib0-test dhcpd: DHCPOFFER on 172.10.10.158 to via ib0</font></font></div><div><font class="Apple-style-span" face="Monaco"><font class="Apple-style-span" size="2"><span class="Apple-tab-span" style="white-space:pre"> </span>Jul 18 17:41:40 ib0-test dhcpd: DHCPDISCOVER from via ib0</font></font></div><div><font class="Apple-style-span" face="Monaco"><font class="Apple-style-span" size="2"><span class="Apple-tab-span" style="white-space:pre"> </span>Jul 18 17:41:40 ib0-test dhcpd: DHCPOFFER on 172.10.10.158 to via ib0</font></font></div><div><font class="Apple-style-span" face="Monaco"><font class="Apple-style-span" size="2"><span class="Apple-tab-span" style="white-space:pre"> </span>Jul 18 17:41:54 ib0-test dhcpd: DHCPDISCOVER from via ib0</font></font></div><div><font class="Apple-style-span" face="Monaco"><font class="Apple-style-span" size="2"><span class="Apple-tab-span" style="white-space:pre"> </span>Jul 18 17:41:54 ib0-test dhcpd: DHCPOFFER on 172.10.10.158 to via ib0</font></font></div><div><font class="Apple-style-span" face="Monaco"><font class="Apple-style-span" size="2"><span class="Apple-tab-span" style="white-space:pre"> </span>Jul 18 17:42:13 ib0-test dhcpd: DHCPDISCOVER from via ib0</font></font></div><div><font class="Apple-style-span" face="Monaco"><font class="Apple-style-span" size="2"><span class="Apple-tab-span" style="white-space:pre"> </span>Jul 18 17:42:13 ib0-test dhcpd: DHCPOFFER on 172.10.10.158 to via ib0</font></font></div><div><font class="Apple-style-span" face="Monaco"><font class="Apple-style-span" size="2"><span class="Apple-tab-span" style="white-space:pre"> </span>Jul 18 17:42:23 ib0-test dhcpd: DHCPDISCOVER from via ib0</font></font></div><div><font class="Apple-style-span" face="Monaco"><font class="Apple-style-span" size="2"><span class="Apple-tab-span" style="white-space:pre"> </span>Jul 18 17:42:23 ib0-test dhcpd: DHCPOFFER on 172.10.10.158 to via ib0</font></font></div><div><font class="Apple-style-span" face="Monaco" size="2"><span class="Apple-style-span" style="font-size: 10px;"><br></span></font></div></div><div>while the client machine has:</div><div><br></div><div><div><font class="Apple-style-span" face="Monaco" size="2"><span class="Apple-tab-span" style="white-space:pre"> </span>Jul 18 17:41:35 ib1-test dhclient: Sending on Socket/fallback</font></div><div><font class="Apple-style-span" face="Monaco" size="2"><span class="Apple-tab-span" style="white-space:pre"> </span>Jul 18 17:41:35 ib1-test dhclient: DHCPDISCOVER on ib0 to 255.255.255.255 port 67 interval 6</font></div><div><font class="Apple-style-span" face="Monaco" size="2"><span class="Apple-tab-span" style="white-space:pre"> </span>Jul 18 17:41:41 ib1-test dhclient: DHCPDISCOVER on ib0 to 255.255.255.255 port 67 interval 14</font></div><div><font class="Apple-style-span" face="Monaco" size="2"><span class="Apple-tab-span" style="white-space:pre"> </span>Jul 18 17:41:54 ib1-test dhclient: DHCPDISCOVER on ib0 to 255.255.255.255 port 67 interval 19</font></div><div><font class="Apple-style-span" face="Monaco" size="2"><span class="Apple-tab-span" style="white-space:pre"> </span>Jul 18 17:42:14 ib1-test dhclient: DHCPDISCOVER on ib0 to 255.255.255.255 port 67 interval 10</font></div><div><font class="Apple-style-span" face="Monaco" size="2"><span class="Apple-tab-span" style="white-space:pre"> </span>Jul 18 17:42:24 ib1-test dhclient: DHCPDISCOVER on ib0 to 255.255.255.255 port 67 interval 12</font></div><div><font class="Apple-style-span" face="Monaco" size="2"><span class="Apple-tab-span" style="white-space:pre"> </span>Jul 18 17:42:35 ib1-test dhclient: No DHCPOFFERS received.</font></div></div><div><br></div><div>What baffles me is that the offer is made but never accepted. I have tried any number of changes to the dhclient.conf file to avoid rejections, to no avail.</div><div><br></div><div>Finally, I will leave you with the relevant configuration files...</div><div><br></div><div>/etc/dhcpd.conf on ib0-test:</div><div><br></div><div><div><font class="Apple-style-span" face="Monaco" size="2"><span class="Apple-style-span" style="font-size: 10px;"><span class="Apple-tab-span" style="white-space:pre"> </span>ddns-update-style interim;</span></font></div><div><font class="Apple-style-span" face="Monaco" size="2"><span class="Apple-style-span" style="font-size: 10px;"><span class="Apple-tab-span" style="white-space:pre"> </span>ignore client-updates;</span></font></div><div><font class="Apple-style-span" face="Monaco" size="2"><span class="Apple-style-span" style="font-size: 10px;"><br></span></font></div><div><font class="Apple-style-span" face="Monaco" size="2"><span class="Apple-style-span" style="font-size: 10px;"><span class="Apple-tab-span" style="white-space:pre"> </span>subnet 172.10.10.0 netmask 255.255.255.0 {</span></font></div><div><font class="Apple-style-span" face="Monaco" size="2"><span class="Apple-style-span" style="font-size: 10px;"> <span class="Apple-tab-span" style="white-space:pre"> </span>range dynamic-bootp 172.10.10.100 172.10.10.200;</span></font></div><div><font class="Apple-style-span" face="Monaco" size="2"><span class="Apple-style-span" style="font-size: 10px;"><span class="Apple-tab-span" style="white-space:pre"> </span> option domain-name "univaud.com";</span></font></div><div><font class="Apple-style-span" face="Monaco" size="2"><span class="Apple-style-span" style="font-size: 10px;"><span class="Apple-tab-span" style="white-space:pre"> </span> option domain-name-servers 192.168.31.10, 10.10.0.12, 10.10.0.13;</span></font></div><div><font class="Apple-style-span" face="Monaco" size="2"><span class="Apple-style-span" style="font-size: 10px;"><span class="Apple-tab-span" style="white-space:pre"> </span> #option routers 192.168.31.1;</span></font></div><div><font class="Apple-style-span" face="Monaco" size="2"><span class="Apple-style-span" style="font-size: 10px;"><span class="Apple-tab-span" style="white-space:pre"> </span> option subnet-mask 255.255.255.0;</span></font></div><div><font class="Apple-style-span" face="Monaco" size="2"><span class="Apple-style-span" style="font-size: 10px;"><span class="Apple-tab-span" style="white-space:pre"> </span> option time-offset -21600; # Central Standard Time</span></font></div><div><font class="Apple-style-span" face="Monaco" size="2"><span class="Apple-style-span" style="font-size: 10px;"><span class="Apple-tab-span" style="white-space:pre"> </span> option ntp-servers 64.113.32.5, 65.111.164.223, 72.52.190.26;</span></font></div><div><font class="Apple-style-span" face="Monaco" size="2"><span class="Apple-style-span" style="font-size: 10px;"><span class="Apple-tab-span" style="white-space:pre"> </span> default-lease-time 21600;</span></font></div><div><font class="Apple-style-span" face="Monaco" size="2"><span class="Apple-style-span" style="font-size: 10px;"><span class="Apple-tab-span" style="white-space:pre"> </span> max-lease-time 43200;</span></font></div><div><font class="Apple-style-span" face="Monaco" size="2"><span class="Apple-style-span" style="font-size: 10px;"><span class="Apple-tab-span" style="white-space:pre"> </span>}</span></font></div><div><br></div></div>/etc/dhclient.conf on ib1-test:<div><br></div><div><div><font class="Apple-style-span" face="Monaco" size="2"><span class="Apple-style-span" style="font-size: 10px;"><span class="Apple-tab-span" style="white-space:pre"> </span>interface "ib0" {</span></font></div><div><font class="Apple-style-span" face="Monaco" size="2"><span class="Apple-style-span" style="font-size: 10px;"><span class="Apple-tab-span" style="white-space:pre"> </span> send dhcp-client-identifier 20:00:55:00:01:FE:80:00:00:00:00:00:00:00:11:75:00:00:ff:94:fd;</span></font></div><div><font class="Apple-style-span" face="Monaco" size="2"><span class="Apple-style-span" style="font-size: 10px;"><span class="Apple-tab-span" style="white-space:pre"> </span>}</span></font></div><div><br></div><div>I created the client identifier from the following information: </div><div><br></div><div><span class="Apple-tab-span" style="white-space:pre"><font class="Apple-style-span" face="Monaco" size="2"><span class="Apple-style-span" style="font-size: 10px;"> </span></font></span><font class="Apple-style-span" face="Monaco" size="2"><span class="Apple-style-span" style="font-size: 10px;">20:<4 byte QP Number><8-byte subnet prefix><8 byte GUID>. </span></font></div><div><br></div><div>This seems to be in line with the patch that we applied to the DHCP software to make it work with infiniband.</div><div><br></div><div>Lastly, an excerpt from /etc/modules.conf on ip1-test:</div><div><br></div><div><div><font class="Apple-style-span" face="Monaco" size="2"><span class="Apple-style-span" style="font-size: 10px;"><span class="Apple-tab-span" style="white-space:pre"> </span>alias ib0 ib_ipoib</span></font></div><div><div><br></div></div></div></div> <span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0; "><div>I tried to use ipath_ether but qlogic's website only has the RHEL4 release (despite having a RHEL5 label).</div><div><br></div><div>Cheers!!!</div><div><br></div><div>--</div><div>Roderick Flores</div></span><div><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div>Solutions Architect</div><div>Univa UD</div><div><a href="mailto:rflores@univaud.com">rflores@univaud.com</a></div></span></span> </div><br></body></html>