[ofa-general] Re: [PATCH 1/10 Rev4] [Doc] HOWTO Documentation for batching

[Randy Dunlap]

On Wed, 22 Aug 2007 13:58:58 +0530 Krishna Kumar wrote:

> Add Documentation describing batching skb xmit capability.
> 
> Signed-off-by: Krishna Kumar <krkumar2 at in.ibm.com>
> ---
>  batching_skb_xmit.txt |   78 ++++++++++++++++++++++++++++++++++++++++++++++++++
>  1 files changed, 78 insertions(+)
> 
> diff -ruNp org/Documentation/networking/batching_skb_xmit.txt new/Documentation/networking/batching_skb_xmit.txt
> --- org/Documentation/networking/batching_skb_xmit.txt	1970-01-01 05:30:00.000000000 +0530
> +++ new/Documentation/networking/batching_skb_xmit.txt	2007-08-22 10:21:19.000000000 +0530
> @@ -0,0 +1,78 @@
> +		 HOWTO for batching skb xmit support
> +		 -----------------------------------
> +
> +Section 1: What is batching skb xmit
> +Section 2: How batching xmit works vs the regular xmit
> +Section 3: How drivers can support batching
> +Section 4: How users can work with batching
> +
> +
> +Introduction: Kernel support for batching skb
> +----------------------------------------------
> +
> +A new capability to support xmit of multiple skbs is provided in the netdevice
> +layer. Drivers which enable this capability should be able to process multiple
> +skbs in a single call to their xmit handler.
> +
> +
> +Section 1: What is batching skb xmit
> +-------------------------------------
> +
> +	This capability is optionally enabled by a driver by setting the
> +	NETIF_F_BATCH_SKBS bit in dev->features. The pre-requisite for a

                                                     prerequisite

> +	driver to use this capability is that it should have a reasonably

	I would say "reasonably-sized".

> +	sized hardware queue that can process multiple skbs.
> +
> +
> +Section 2: How batching xmit works vs the regular xmit
> +-------------------------------------------------------
> +
> +	The network stack gets called from upper layer protocols with a single
> +	skb to transmit. This skb is first enqueue'd and an attempt is made to

                                           enqueued

> +	transmit it immediately (via qdisc_run). However, events like tx lock
> +	contention, tx queue stopped, etc, can result in the skb not getting

                                      etc.,

> +	sent out and it remains in the queue. When the next xmit is called or
> +	when the queue is re-enabled, qdisc_run could potentially find
> +	multiple packets in the queue, and iteratively send them all out
> +	one-by-one.
> +
> +	Batching skb xmit is a mechanism to exploit this situation where all
> +	skbs can be passed in one shot to the device. This reduces driver
> +	processing, locking at the driver (or in stack for ~LLTX drivers)
> +	gets amortized over multiple skbs, and in case of specific drivers
> +	where every xmit results in a completion processing (like IPoIB) -
> +	optimizations can be made in the driver to request a completion for
> +	only the last skb that was sent which results in saving interrupts
> +	for every (but the last) skb that was sent in the same batch.
> +
> +	Batching can result in significant performance gains for systems that
> +	have multiple data stream paths over the same network interface card.
> +
> +
> +Section 3: How drivers can support batching
> +---------------------------------------------
> +
> +	Batching requires the driver to set the NETIF_F_BATCH_SKBS bit in
> +	dev->features.
> +
> +	The driver's xmit handler should be modified to process multiple skbs
> +	instead of one skb. The driver's xmit handler is called either with a

                                                                           an

> +	skb to transmit or NULL skb, where the latter case should be handled
> +	as a call to xmit multiple skbs. This is done by sending out all skbs
> +	in the dev->skb_blist list (where it was added by the core stack).
> +
> +
> +Section 4: How users can work with batching
> +---------------------------------------------
> +
> +	Batching can be disabled for a particular device, e.g. on desktop
> +	systems if only one stream of network activity for that device is
> +	taking place, since performance could be slightly affected due to
> +	extra processing that batching adds (unless packets are getting
> +	sent fast resulting in stopped queue's). Batching can be enabled if

                                       queues).

> +	more than one stream of network activity per device is being done,
> +	e.g. on servers; or even desktop usage with multiple browser, chat,
> +	file transfer sessions, etc.
> +
> +	Per device batching can be enabled/disabled by passing 'on' or 'off'
> +	respectively to ethtool.

	with what other parameter(s), e.g.,

	ethtool <dev> batching on/off ?

---
~Randy
*** Remember to use Documentation/SubmitChecklist when testing your code ***