<div dir="ltr"><div dir="ltr">On Fri, Aug 16, 2019 at 10:35 AM Ilango, Arun <<a href="mailto:arun.ilango@intel.com">arun.ilango@intel.com</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div lang="EN-US">
<div class="gmail-m_758398437894414219WordSection1">
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">The verbs provider doesn’t support FI_SOURCE for the FI_EP_DGRAM endpoint type today but it can be added. </span></p></div></div></blockquote><div><br></div><div>It would be really nice to have the verbs provider support FI_SOURCE. Otherwise, I have to embed the source address in the datagram on my own.<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div lang="EN-US"><div class="gmail-m_758398437894414219WordSection1"><p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Btw, you’re using the verbs interface of hfi1. Is there
a reason to use verbs instead of psm2? What’s the use case here?</span></p></div></div></blockquote><div><br></div><div>Actually, I am mostly using the psm2 provider on this Intel OPA cluster. But I have other IB clusters to work with. On the other hand, if Intel OPA has a kernel-bypass verbs library, I would prefer that over psm2 because psm2 doesn't seem to provide a real unreliable datagram endpoint, which is the only type of communication I care about. The unnecessary abstraction of psm2 probably means dropping performance on the table for me.</div><div><br></div><div>Thanks,</div><div>Yilong</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div lang="EN-US"><div class="gmail-m_758398437894414219WordSection1"><p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Thanks,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Arun.<u></u><u></u></span></p>
<p class="MsoNormal"><a name="m_758398437894414219__MailEndCompose"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></a></p>
<div style="border-top:none;border-right:none;border-bottom:none;border-left:1.5pt solid blue;padding:0in 0in 0in 4pt">
<div>
<div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb(225,225,225);padding:3pt 0in 0in">
<p class="MsoNormal"><a name="m_758398437894414219______replyseparator"></a><b><span style="font-size:11pt;font-family:Calibri,sans-serif">From:</span></b><span style="font-size:11pt;font-family:Calibri,sans-serif"> Libfabric-users [mailto:<a href="mailto:libfabric-users-bounces@lists.openfabrics.org" target="_blank">libfabric-users-bounces@lists.openfabrics.org</a>]
<b>On Behalf Of </b>Yilong Li<br>
<b>Sent:</b> Friday, August 16, 2019 12:08 AM<br>
<b>To:</b> <a href="mailto:libfabric-users@lists.openfabrics.org" target="_blank">libfabric-users@lists.openfabrics.org</a><br>
<b>Subject:</b> [libfabric-users] How to retrieve the source address of an incoming datagram with prov/verbs?<u></u><u></u></span></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">Hi,<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I notice that the verbs provider doesn't advertise the FI_SOURCE capability. So how can I retrieve the source address of an incoming datagram? My application has a client-server architecture: the server cannot know the address of a client
until the client tries to contact the server.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I am pasting the output of `fi_info -t FI_EP_DGRAM -p verbs -v` on my machine below.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Thanks,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Yilong<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">---------------------------------------------------------------------<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">fi_info:<br>
caps: [ FI_MSG, FI_RECV, FI_SEND, FI_LOCAL_COMM, FI_REMOTE_COMM ]<br>
mode: [ FI_MSG_PREFIX, FI_RX_CQ_DATA ]<br>
addr_format: FI_ADDR_IB_UD<br>
src_addrlen: 32<br>
dest_addrlen: 0<br>
src_addr: fi_addr_ib_ud://:::0/0/0/0<br>
dest_addr: (null)<br>
handle: (nil)<br>
fi_tx_attr:<br>
caps: [ FI_MSG, FI_RECV, FI_SEND, FI_LOCAL_COMM, FI_REMOTE_COMM ]<br>
mode: [ ]<br>
op_flags: [ ]<br>
msg_order: [ FI_ORDER_RAR, FI_ORDER_RAW, FI_ORDER_RAS, FI_ORDER_WAW, FI_ORDER_WAS, FI_ORDER_SAW, FI_ORDER_SAS, FI_ORDER_RMA_RAR, FI_ORDER_RMA_RAW, FI_ORDER_RMA_WAW, FI_ORDER_ATOMIC_RAR, FI_ORDER_ATOMIC_RAW, FI_ORDER_ATOMIC_WAW ]<br>
comp_order: [ FI_ORDER_STRICT ]<br>
inject_size: 0<br>
size: 384<br>
iov_limit: 4<br>
rma_iov_limit: 1<br>
fi_rx_attr:<br>
caps: [ FI_MSG, FI_RECV, FI_SEND, FI_LOCAL_COMM, FI_REMOTE_COMM ]<br>
mode: [ FI_MSG_PREFIX, FI_RX_CQ_DATA ]<br>
op_flags: [ ]<br>
msg_order: [ FI_ORDER_RAR, FI_ORDER_RAW, FI_ORDER_RAS, FI_ORDER_WAW, FI_ORDER_WAS, FI_ORDER_SAW, FI_ORDER_SAS, FI_ORDER_RMA_RAR, FI_ORDER_RMA_RAW, FI_ORDER_RMA_WAW, FI_ORDER_ATOMIC_RAR, FI_ORDER_ATOMIC_RAW, FI_ORDER_ATOMIC_WAW ]<br>
comp_order: [ FI_ORDER_STRICT, FI_ORDER_DATA ]<br>
total_buffered_recv: 0<br>
size: 384<br>
iov_limit: 4<br>
fi_ep_attr:<br>
type: FI_EP_DGRAM<br>
protocol: FI_PROTO_IB_UD<br>
protocol_version: 1<br>
max_msg_size: 4096<br>
msg_prefix_size: 40<br>
max_order_raw_size: 4096<br>
max_order_war_size: 0<br>
max_order_waw_size: 4096<br>
mem_tag_format: 0x0000000000000000<br>
tx_ctx_cnt: 1<br>
rx_ctx_cnt: 1<br>
auth_key_size: 0<br>
fi_domain_attr:<br>
domain: 0x0<br>
name: hfi1_0-dgram<br>
threading: FI_THREAD_SAFE<br>
control_progress: FI_PROGRESS_AUTO<br>
data_progress: FI_PROGRESS_AUTO<br>
resource_mgmt: FI_RM_ENABLED<br>
av_type: FI_AV_UNSPEC<br>
mr_mode: [ FI_MR_LOCAL, FI_MR_ALLOCATED ]<br>
mr_key_size: 4<br>
cq_data_size: 4<br>
cq_cnt: 131071<br>
ep_cnt: 32768<br>
tx_ctx_cnt: 1024<br>
rx_ctx_cnt: 1024<br>
max_ep_tx_ctx: 1024<br>
max_ep_rx_ctx: 1024<br>
max_ep_stx_ctx: 0<br>
max_ep_srx_ctx: 1024<br>
cntr_cnt: 0<br>
mr_iov_limit: 1<br>
caps: [ FI_LOCAL_COMM, FI_REMOTE_COMM ]<br>
mode: [ ]<br>
auth_key_size: 0<br>
max_err_data: 255<br>
mr_cnt: 0<br>
fi_fabric_attr:<br>
name: IB-0xfe80000000000000<br>
prov_name: verbs<br>
prov_version: 1.0<br>
api_version: 1.8<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> fid_nic:<br>
fi_device_attr:<br>
name: hfi1_0<br>
device_id: 9456<br>
device_version: 17<br>
vendor_id: 4469<br>
driver: (null)<br>
firmware: 1.26.1<br>
fi_bus_attr:<br>
fi_bus_type: FI_BUS_UNKNOWN<br>
fi_link_attr:<br>
address: (null)<br>
mtu: 4096<br>
speed: 104857600<br>
state: FI_LINK_UP<br>
network_type: InfiniBand<u></u><u></u></p>
</div>
</div>
</div>
</div>
</div>
</blockquote></div></div>