[ofa-general] Re: 20% latency increase between UD to RC latency

[Or Gerlitz]

On Mon, 23 Jul 2007, Or Gerlitz wrote:

> my original thinking was that for messages
> whose size=X is less then the IB Link level MTU it holds that:
>
> 	latency(X,UD) <= latency(X,UC) <= latency(X,RC)
>
> Running the latency test provided with the perftest package on my systems (*)
> I get the below results. Does anyone has insight why the --minimal-- and typical
> UD latency is 1us ( = 20%) worse then the --minimal-- and typical RC latency???

running the latecy test on a similar system (RH5 smp / four way Xeon 1.9GHz /
4GB RAM) but this time with the memfree --Hermon-- HCA (25418 / FW 2.1.0) the minimal
AND typical UD latecy is very much the same as the RC and UC ones which is ~1.5us

So it both fixes the UD issue on Arbel and improves the latency from 4.5us to 1.5us

nice,

Or.

root at iris6 ~]# ib_send_lat -c RC 172.30.3.252
------------------------------------------------------------------
                    Send Latency Test
Inline data is used up to 400 bytes message
Connection type : RC
   local address: LID 0x06 QPN 0xb004a PSN 0xe6014a
  remote address: LID 0x05 QPN 0xb004a PSN 0xbe437f
Mtu : 2048
------------------------------------------------------------------
 #bytes #iterations    t_min[usec]    t_max[usec]  t_typical[usec]
      2        1000           1.39           8.08             1.42
------------------------------------------------------------------
[root at iris6 ~]# ib_send_lat -c UC 172.30.3.252
------------------------------------------------------------------
                    Send Latency Test
Inline data is used up to 400 bytes message
Connection type : UC
   local address: LID 0x06 QPN 0xc004a PSN 0xb4281e
  remote address: LID 0x05 QPN 0xc004a PSN 0xb14013
Mtu : 2048
------------------------------------------------------------------
 #bytes #iterations    t_min[usec]    t_max[usec]  t_typical[usec]
      2        1000           1.37           8.17             1.42
------------------------------------------------------------------
[root at iris6 ~]# ib_send_lat -c UD 172.30.3.252
------------------------------------------------------------------
                    Send Latency Test
Inline data is used up to 400 bytes message
Connection type : UD
   local address: LID 0x06 QPN 0xd004a PSN 0xf63264
  remote address: LID 0x05 QPN 0xd004a PSN 0xf7821
------------------------------------------------------------------
 #bytes #iterations    t_min[usec]    t_max[usec]  t_typical[usec]
      2        1000           1.47           7.66             1.51
------------------------------------------------------------------

[root at iris6 ~]# ibv_devinfo
hca_id: mlx4_0
        fw_ver:                         2.1.000
        node_guid:                      0002:c903:0000:0434
        sys_image_guid:                 0002:c903:0000:0437
        vendor_id:                      0x02c9
        vendor_part_id:                 25418
        hw_ver:                         0xA0
        board_id:                       MT_04A0110002
        phys_port_cnt:                  2
                port:   1
                        state:                  PORT_ACTIVE (4)
                        max_mtu:                2048 (4)
                        active_mtu:             2048 (4)
                        sm_lid:                 1
                        port_lid:               6
                        port_lmc:               0x00

                port:   2
                        state:                  PORT_DOWN (1)
                        max_mtu:                2048 (4)
                        active_mtu:             2048 (4)
                        sm_lid:                 0
                        port_lid:               0
                        port_lmc:               0x00