[ofa-general] 20% latency increase between UD to RC latency

[Or Gerlitz]

OK,its always good to start with facts on the ground... before
commiting this test, my original thinking was that for messages
whose size=X is less then the IB Link level MTU it holds that:

	latency(X,UD) <= latency(X,UC) <= latency(X,RC)

Running the latency test provided with the perftest package on my systems (*)
I get the below results. Does anyone has insight why the --minimal-- and typical
UD latency is 1us ( = 20%) worse then the --minimal-- and typical RC latency???

Or.

(*) the system spec is:
HW    : 4 way Intel Xeon 1.6GHz 4GB RAM
IB  HW: Arbel memfull (25208) DDR running in SDR mode
HCA FW: 4.8.200
IB  SW: OFED 1.2
OS     : RH4 U3 i386 smp

[root at rain5 ~]# /usr/bin/ib_send_lat -c RC -n 100000 172.30.8.61
------------------------------------------------------------------
                    Send Latency Test
Inline data is used up to 400 bytes message
Connection type : RC
   local address: LID 0x26 QPN 0x330407 PSN 0xf6ba57
  remote address: LID 0x28 QPN 0x40407 PSN 0xc2c9f9
Mtu : 2048
------------------------------------------------------------------
 #bytes #iterations    t_min[usec]    t_max[usec]  t_typical[usec]
      2        100000           4.75          38.53             4.82
------------------------------------------------------------------

[root at rain5 ~]# /usr/bin/ib_send_lat -c UC -n 100000 172.30.8.61
------------------------------------------------------------------
                    Send Latency Test
Inline data is used up to 400 bytes message
Connection type : UC
   local address: LID 0x26 QPN 0x340407 PSN 0xbb4a0e
  remote address: LID 0x28 QPN 0x50407 PSN 0xb916a9
Mtu : 2048
------------------------------------------------------------------
 #bytes #iterations    t_min[usec]    t_max[usec]  t_typical[usec]
      2        100000           4.71          42.03             4.77
------------------------------------------------------------------

[root at rain5 ~]# /usr/bin/ib_send_lat -c UD -n 100000 172.30.8.61
------------------------------------------------------------------
                    Send Latency Test
Inline data is used up to 400 bytes message
Connection type : UD
   local address: LID 0x26 QPN 0x350407 PSN 0xfdc2c0
  remote address: LID 0x28 QPN 0x60407 PSN 0x63c30e
------------------------------------------------------------------
 #bytes #iterations    t_min[usec]    t_max[usec]  t_typical[usec]
      2        100000           5.71          44.51             5.81
------------------------------------------------------------------