[ofw] Patch: swap the src and dst gids to allow mad replies to return

[Sean Hefty]

>I don't think that the lids are being treated in a similar way, but there is a
>very similar code on the function spl_qp_alias_recv_cb (al_smi.c line 3325)
>
>	if( p_mad_request->grh_valid )
>	{
>		cl_memcpy( &av_attr.grh, p_mad_request->p_grh1, sizeof( ib_grh_t
)
>);
>		av_attr.grh.src_gid	 = p_mad_request->p_grh1->dest_gid;
>		av_attr.grh.dest_gid = p_mad_request->p_grh1->src_gid;
>		av_attr.grh_valid = TRUE;
>	}

This is different in that it is initializing an address vector that will be used
to send a response mad.  The actual received dgid/sgid referenced by the MAD are
not changed.

>> >+ if (p_mad_element->grh_valid) {
>> >+  // swap the src and dst gids to allow replys to return  ib_gid_t
>> >+ tmp_gid;  tmp_gid = p_mad_element->p_grh1->src_gid;
>> >+ p_mad_element->p_grh1->src_gid = p_mad_element->p_grh1->dest_gid;
>> >+ p_mad_element->p_grh1->dest_gid = tmp_gid; }

This changes the actual dgid/sgid reported in the mad, such that the user must
now know that the 'source' or sending of the mad is now listed as the dgid.

It's not clear to me if this swapping ends up breaking the swapping done in
spl_qp_alias_recv_cb, since the dgid/sgid could be swapped twice in that case.

I think we want the received MAD sgid to reference the actual source gid of the
sender when reporting the receive.  That seems to make the most sense to me.  Is
this what the code reports today, or does it report the receiver's gid in the
src_gid field?

We also need to be careful about this sort of change.  ib_mad_element_t is
exposed to user space applications, so any change such as this will break those
applications that depend on the current behavior.

- Sean