[openib-general] question regarding GRH flag in ib_ah_attr

[Jason Gunthorpe]

On Thu, May 11, 2006 at 10:21:08AM -0700, Sean Hefty wrote:
> Hal Rosenstock wrote:
> >Anytime the send is off the local subnet (as well as multicast), a GRH
> >is required. Also, there is a management response rule for responding
> >when the request contained a GRH that require a GRH (13.5.4.4 p. 769).
 
> Reading through the responses, I think my problems are worse.  Now I'm not 
> even sure how I determine which remote node I'm trying to talk to short of 
> hard-coding the DGID...

> We currently use ARP to resolve an IP address to a DGID, which I don't 
> believe will work across a router.  Does an app even know enough to be able 
> to get a path record?

The only wrinkles I could see you having is how to choose between
multiple DGID's when generating the ARP response. I don't think that
is a serious issue though since any GID to any GID should be routable
on the subnet.

I haven't looked at the ARP code, but based on the RFCs the IPv4 ARP
process would be more or less:

1) Send ARP datagram to the broadcast multicast group LID w/ GRH. The
   ARP packet includes the IPv4 address of the sender and the
   GID/QPN (hardware address) of the sender, asking for the hardware
   address of the target IPv4.

   A router must support multicast routing so that the ARP request is
   forwarded to the remote subnet. It has a GRH of course so this is
   OK. The SM and router work together to make this happen.
2) ARP responder matches the target IP address, gets the IP of the
   requestor, and the GID/QPN from the ARP packet's sender fields

   We are still OK since the GID in the ARP packet's sender fields is
   global.
3) ARP responder produces a unicast packet to the IPv4 requestor
   address:
   - The sender's GID/QPN is converted into a path either from a local cache
     or via a SA query. The sender's GID combined with any of the target's
     GID's should be sufficient to ask the SA for a path.
     [Note: that you must use the _hardware_ address here and you
      cannot just lookup the IPv4 sender address in the neighbor
      cache. This is needed to support ARP tricks like zeroconf that
      use null source IPs]
   - This query results in a path record for communication with the
     sender. [Some implementations will learn based on ARP requests
     and will update the neighbor cache here]
   - The path record is used to generate the unicast headers, GRH and
     all - if necessary.
   - The same SGID that was used in the path record query above is returned
     in the ARP response as the target's address.
  
   Since the SA specifies the path to get back to the requestor based
   only on the GID in the ARP request it can produce a path that
   crosses the router.

4) The ARP requestor now gets the respondor's GID/QPN from the unicast
   ARP response and does the same path lookup that the ARP requestor
   did to get the 'reverse' path.

   Again, since the SA is now involved the resulting path can cross
   the router.

IPv6 is similar, but the packet format is different and the 'ARP'
(NS packet) request is sent to a multicast address chosen by 'hashing'
the IPv6 address.

Hope this helps,
Jason