[libfabric-users] udp packets and jumbo frames

[Martin Pokorny]

Hi, Jianxin;

Thanks for your reply. I'm sorry that it's taken me so long to 
respond, but I've only now found the time to think about trying to 
implement this feature myself.

"Xiong, Jianxin" <jianxin.xiong at intel.com> writes:
> The hard coded value 1472 I believe is based on MTU size of 
> 1500. For jumbo frames, the value can be increased 
> accordingly. E.g., 8972 for MTU size 9000.

These look like MTU - UDP header - IPv4 header. Minor question: 
does libfabric define any constants for the header sizes?

> I can see three options to do this:
> (1) Add a configure option for the default MTU size and initiate 
> the attributes accordingly.
> (2) Add a runtime parameter to set the MTU size and modify the 
> attributes accordingly.
> (3) Detect the MTU size of the interface being used and modify 
> the attributes accordingly.

I prefer either option 2 or 3. After reviewing some of the 
libfabric code, I imagine option 2 would be easier for me to 
implement, but option 3 might be better for users. In fact, option 
2 seems like it's probably relatively straightforward, and I would 
likely start with that.

For option 3, I have some questions. Determining the MTU of an 
interface, at least on linux, I believe requires an ioctl call 
with a socket file descriptor, and a device name in a ifreq 
value. I don't know where that would be best implemented: in 
udpx_domain_open(), or elsewhere? It would have to occur before 
either udpx_tx_attr of udpx_ep_attr values are created, since 
those are the types that reflect MTU values.

> Either (2) and (3) alone should be sufficient. But (1) will need 
> to be combined with (2) or (3) so that the new default can be 
> overridden.
>
> You are welcome to take the task to make this enhancement, or 
> you can open an issue here so we can have someone working on 
> that: https://github.com/ofiwg/libfabric/issues

I'll go ahead and try an implementation of option 2, and then I 
will submit a PR if all goes well. If people prefer option 3, I'm 
happy to have that discussion in the PR, and make any desired 
changes (based on guidance I hope would be offered in that 
discussion).

Regards,
Martin

>> -----Original Message-----
>> From: Libfabric-users 
>> <libfabric-users-bounces at lists.openfabrics.org> On Behalf Of 
>> Martin Pokorny
>> Sent: Tuesday, July 30, 2024 1:30 PM
>> To: libfabric-users at lists.openfabrics.org
>> Subject: [libfabric-users] udp packets and jumbo frames
>>
>> Hello,
>>
>> The application I'm working on needs to support an Ethernet 
>> network configured with jumbo frames to capture data in UDP 
>> packets, where
>> each UDP packet fits into a single Ethernet jumbo frame. I'd 
>> like to use libfabric for this configuration, as I've already 
>> got an implementation
>> based on libfabric for other providers. However, it seems that 
>> the value of `udpx_ep_attr.max_msg_size` defined in 
>> `prov/udp/src/udpx_attr.c`
>> is hard-coded to 1472, which prevents using UDP packets with a 
>> larger payload. I don't know that this value could be 
>> determined from the
>> Ethernet configuration at runtime, but as I'd be happy to have 
>> it set at compile time, would a possible approach to supporting 
>> jumbo frames of
>> a known size be to increase that value appropriately? I don't 
>> think that `udpx_tx_attr.size`, `udpx_tx_attr.inject_size`, and 
>> `udpx_rx_attr.size`
>> would need to change, but I'm less certain about that. In any 
>> case, I'm not trying to implement the most general, "correct" 
>> solution at this
>> time, and I'm happy to patch some source files specifically for 
>> my needs. If anyone could offer any advice about what I'm 
>> trying to do, I'd
>> appreciate it. TIA.
>>
>> --
>> Martin
>> _______________________________________________
>> Libfabric-users mailing list
>> Libfabric-users at lists.openfabrics.org
>> https://lists.openfabrics.org/mailman/listinfo/libfabric-users


--
Martin