[openib-general] Re: [PATCH] perftest/rdma_bw; add support for RDMA read and starting PSN

[Arlin Davis]

Michael S. Tsirkin wrote:

>Quoting r. Arlin Davis <arlin.r.davis at intel.com>:
>  
>
>>Subject: [PATCH] perftest/rdma_bw; add support for RDMA read and starting PSN
>>
>>Michael,
>>
>>The patch adds command line options for RDMA reads and starting PSN. I
>>used these modifications to
>>help isolate the RDMA read performance degradation with 4.6.2 firmware. 
>>
>>-arlin
>>    
>>
>
>Thanks Arlin. I plan to look into integrating this.
>One question: for which psn values do you see performance drop on 4.6.0 FW?
>
>
>  
>

A quick run at 1 and then 0x100000 dropped from 682MB/s to 49MB/s for 
32KB buffers.
What is really strange is that it takes a couple runs to start seeing 
the drop in performance.

PSN=1 no problems...

[ardavis at iclust-20 perftest]$  ./rdma_bw -P 0x1 -s 32768 -r iclust-19
  local address:  LID 0x02, QPN 0x20406, PSN 0x0001 RKey 0x0c0032 VAddr 
0x00000000514000 RDMA_READ
  remote address: LID 0x05, QPN 0x20406, PSN 0x0001 RKey 0x0c0032 VAddr 
0x00000000513000 RDMA_READ
Bandwidth peak (#0 to #999): 682.504 MB/sec
Bandwidth average: 682.501 MB/sec
Service Demand peak (#0 to #999): 5138 cycles/KB
Service Demand Avg  : 5138 cycles/KB

[ardavis at iclust-20 perftest]$  ./rdma_bw -P 0x1 -s 32768 -r iclust-19
  local address:  LID 0x02, QPN 0x30406, PSN 0x0001 RKey 0x120032 VAddr 
0x00000000514000 RDMA_READ
  remote address: LID 0x05, QPN 0x30406, PSN 0x0001 RKey 0x120032 VAddr 
0x00000000513000 RDMA_READ
Bandwidth peak (#0 to #990): 682.496 MB/sec
Bandwidth average: 682.496 MB/sec
Service Demand peak (#0 to #990): 5138 cycles/KB
Service Demand Avg  : 5138 cycles/KB

[ardavis at iclust-20 perftest]$  ./rdma_bw -P 0x1 -s 32768 -r iclust-19
  local address:  LID 0x02, QPN 0x40406, PSN 0x0001 RKey 0x180032 VAddr 
0x00000000514000 RDMA_READ
  remote address: LID 0x05, QPN 0x40406, PSN 0x0001 RKey 0x180032 VAddr 
0x00000000513000 RDMA_READ
Bandwidth peak (#0 to #990): 682.5 MB/sec
Bandwidth average: 682.499 MB/sec
Service Demand peak (#0 to #990): 5138 cycles/KB
Service Demand Avg  : 5138 cycles/KB

PSN=0x100000  (start to see problems after first run)

[ardavis at iclust-20 perftest]$  ./rdma_bw -P 0x100000 -s 32768 -r iclust-19
  local address:  LID 0x02, QPN 0xb0406, PSN 0x100000 RKey 0x420032 
VAddr 0x00000000514000 RDMA_READ
  remote address: LID 0x05, QPN 0x90406, PSN 0x100000 RKey 0x360032 
VAddr 0x00000000513000 RDMA_READ
Bandwidth peak (#0 to #996): 682.5 MB/sec
Bandwidth average: 682.499 MB/sec
Service Demand peak (#0 to #996): 5138 cycles/KB
Service Demand Avg  : 5138 cycles/KB

[ardavis at iclust-20 perftest]$  ./rdma_bw -P 0x100000 -s 32768 -r iclust-19
  local address:  LID 0x02, QPN 0xc0406, PSN 0x100000 RKey 0x480032 
VAddr 0x00000000514000 RDMA_READ
  remote address: LID 0x05, QPN 0xa0406, PSN 0x100000 RKey 0x3c0032 
VAddr 0x00000000513000 RDMA_READ
Bandwidth peak (#0 to #0): 48.5441 MB/sec
Bandwidth average: 47.4502 MB/sec
Service Demand peak (#0 to #0): 72244 cycles/KB
Service Demand Avg  : 73909 cycles/KB

[ardavis at iclust-20 perftest]$  ./rdma_bw -P 0x100000 -s 32768 -r iclust-19
  local address:  LID 0x02, QPN 0xd0406, PSN 0x100000 RKey 0x4e0032 
VAddr 0x00000000514000 RDMA_READ
  remote address: LID 0x05, QPN 0xb0406, PSN 0x100000 RKey 0x420032 
VAddr 0x00000000513000 RDMA_READ
Bandwidth peak (#0 to #0): 48.4803 MB/sec
Bandwidth average: 47.4501 MB/sec
Service Demand peak (#0 to #0): 72339 cycles/KB
Service Demand Avg  : 73909 cycles/KB

PSN = 1 (first run is bad, and then it is back to normal)

[ardavis at iclust-20 perftest]$  ./rdma_bw -P 0x1 -s 32768 -r iclust-19
  local address:  LID 0x02, QPN 0xe0406, PSN 0x0001 RKey 0x540032 VAddr 
0x00000000514000 RDMA_READ
  remote address: LID 0x05, QPN 0xc0406, PSN 0x0001 RKey 0x480032 VAddr 
0x00000000513000 RDMA_READ
Bandwidth peak (#0 to #0): 48.5798 MB/sec
Bandwidth average: 47.4502 MB/sec
Service Demand peak (#0 to #0): 72190 cycles/KB
Service Demand Avg  : 73909 cycles/KB

[ardavis at iclust-20 perftest]$  ./rdma_bw -P 0x1 -s 32768 -r iclust-19
  local address:  LID 0x02, QPN 0xf0406, PSN 0x0001 RKey 0x5a0032 VAddr 
0x00000000514000 RDMA_READ
  remote address: LID 0x05, QPN 0xd0406, PSN 0x0001 RKey 0x4e0032 VAddr 
0x00000000513000 RDMA_READ
Bandwidth peak (#0 to #990): 682.492 MB/sec
Bandwidth average: 682.49 MB/sec
Service Demand peak (#0 to #990): 5138 cycles/KB
Service Demand Avg  : 5138 cycles/KB

-arlin