[libfabric-users] How to retrieve the source address of an incoming datagram with prov/verbs?

[Yilong Li]

On Fri, Aug 16, 2019 at 10:35 AM Ilango, Arun <arun.ilango at intel.com> wrote:

> The verbs provider doesn’t support FI_SOURCE for the FI_EP_DGRAM endpoint
> type today but it can be added.
>

It would be really nice to have the verbs provider support FI_SOURCE.
Otherwise, I have to embed the source address in the datagram on my own.


> Btw, you’re using the verbs interface of hfi1. Is there a reason to use
> verbs instead of psm2? What’s the use case here?
>

Actually, I am mostly using the psm2 provider on this Intel OPA cluster.
But I have other IB clusters to work with. On the other hand, if Intel OPA
has a kernel-bypass verbs library, I would prefer that over psm2 because
psm2 doesn't seem to provide a real unreliable datagram endpoint, which is
the only type of communication I care about. The unnecessary abstraction of
psm2 probably means dropping performance on the table for me.

Thanks,
Yilong


>
>
> Thanks,
>
> Arun.
>
>
>
> *From:* Libfabric-users [mailto:
> libfabric-users-bounces at lists.openfabrics.org] *On Behalf Of *Yilong Li
> *Sent:* Friday, August 16, 2019 12:08 AM
> *To:* libfabric-users at lists.openfabrics.org
> *Subject:* [libfabric-users] How to retrieve the source address of an
> incoming datagram with prov/verbs?
>
>
>
> Hi,
>
>
>
> I notice that the verbs provider doesn't advertise the FI_SOURCE
> capability. So how can I retrieve the source address of an incoming
> datagram? My application has a client-server architecture: the server
> cannot know the address of a client until the client tries to contact the
> server.
>
>
>
> I am pasting the output of `fi_info -t FI_EP_DGRAM -p verbs -v` on my
> machine below.
>
>
>
> Thanks,
>
> Yilong
>
>
>
> ---------------------------------------------------------------------
>
> fi_info:
>     caps: [ FI_MSG, FI_RECV, FI_SEND, FI_LOCAL_COMM, FI_REMOTE_COMM ]
>     mode: [ FI_MSG_PREFIX, FI_RX_CQ_DATA ]
>     addr_format: FI_ADDR_IB_UD
>     src_addrlen: 32
>     dest_addrlen: 0
>     src_addr: fi_addr_ib_ud://:::0/0/0/0
>     dest_addr: (null)
>     handle: (nil)
>     fi_tx_attr:
>         caps: [ FI_MSG, FI_RECV, FI_SEND, FI_LOCAL_COMM, FI_REMOTE_COMM ]
>         mode: [  ]
>         op_flags: [  ]
>         msg_order: [ FI_ORDER_RAR, FI_ORDER_RAW, FI_ORDER_RAS,
> FI_ORDER_WAW, FI_ORDER_WAS, FI_ORDER_SAW, FI_ORDER_SAS, FI_ORDER_RMA_RAR,
> FI_ORDER_RMA_RAW, FI_ORDER_RMA_WAW, FI_ORDER_ATOMIC_RAR,
> FI_ORDER_ATOMIC_RAW, FI_ORDER_ATOMIC_WAW ]
>         comp_order: [ FI_ORDER_STRICT ]
>         inject_size: 0
>         size: 384
>         iov_limit: 4
>         rma_iov_limit: 1
>     fi_rx_attr:
>         caps: [ FI_MSG, FI_RECV, FI_SEND, FI_LOCAL_COMM, FI_REMOTE_COMM ]
>         mode: [ FI_MSG_PREFIX, FI_RX_CQ_DATA ]
>         op_flags: [  ]
>         msg_order: [ FI_ORDER_RAR, FI_ORDER_RAW, FI_ORDER_RAS,
> FI_ORDER_WAW, FI_ORDER_WAS, FI_ORDER_SAW, FI_ORDER_SAS, FI_ORDER_RMA_RAR,
> FI_ORDER_RMA_RAW, FI_ORDER_RMA_WAW, FI_ORDER_ATOMIC_RAR,
> FI_ORDER_ATOMIC_RAW, FI_ORDER_ATOMIC_WAW ]
>         comp_order: [ FI_ORDER_STRICT, FI_ORDER_DATA ]
>         total_buffered_recv: 0
>         size: 384
>         iov_limit: 4
>     fi_ep_attr:
>         type: FI_EP_DGRAM
>         protocol: FI_PROTO_IB_UD
>         protocol_version: 1
>         max_msg_size: 4096
>         msg_prefix_size: 40
>         max_order_raw_size: 4096
>         max_order_war_size: 0
>         max_order_waw_size: 4096
>         mem_tag_format: 0x0000000000000000
>         tx_ctx_cnt: 1
>         rx_ctx_cnt: 1
>         auth_key_size: 0
>     fi_domain_attr:
>         domain: 0x0
>         name: hfi1_0-dgram
>         threading: FI_THREAD_SAFE
>         control_progress: FI_PROGRESS_AUTO
>         data_progress: FI_PROGRESS_AUTO
>         resource_mgmt: FI_RM_ENABLED
>         av_type: FI_AV_UNSPEC
>         mr_mode: [ FI_MR_LOCAL, FI_MR_ALLOCATED ]
>         mr_key_size: 4
>         cq_data_size: 4
>         cq_cnt: 131071
>         ep_cnt: 32768
>         tx_ctx_cnt: 1024
>         rx_ctx_cnt: 1024
>         max_ep_tx_ctx: 1024
>         max_ep_rx_ctx: 1024
>         max_ep_stx_ctx: 0
>         max_ep_srx_ctx: 1024
>         cntr_cnt: 0
>         mr_iov_limit: 1
>     caps: [ FI_LOCAL_COMM, FI_REMOTE_COMM ]
>     mode: [  ]
>         auth_key_size: 0
>         max_err_data: 255
>         mr_cnt: 0
>     fi_fabric_attr:
>         name: IB-0xfe80000000000000
>         prov_name: verbs
>         prov_version: 1.0
>         api_version: 1.8
>
>     fid_nic:
>         fi_device_attr:
>             name: hfi1_0
>             device_id: 9456
>             device_version: 17
>             vendor_id: 4469
>             driver: (null)
>             firmware: 1.26.1
>         fi_bus_attr:
>             fi_bus_type: FI_BUS_UNKNOWN
>         fi_link_attr:
>             address: (null)
>             mtu: 4096
>             speed: 104857600
>             state: FI_LINK_UP
>             network_type: InfiniBand
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/libfabric-users/attachments/20190816/ec2eb571/attachment.html>