[ofa-general] Re: [PATCH 2/3][NET_BATCH] net core use batching

[Jeff Garzik]

jamal wrote:
> Ok, so the "concurency" aspect is what worries me. What i am saying is
> that sooner or later you have to serialize (which is anti-concurency)
> For example, consider CPU0 running a high prio queue and CPU1 running
> the low prio queue of the same netdevice.
> Assume CPU0 is getting a lot of interupts or other work while CPU1
> doesnt (so as to create a condition that CPU1 is slower). Then as long
> as there packets and there is space on the drivers rings, CPU1 will send
> more packets per unit time than CPU0.
> This contradicts the strict prio scheduler which says higher priority
> packets ALWAYS go out first regardless of the presence of low prio
> packets.  I am not sure i made sense.

You made sense.  I think it is important to note simply that the packet 
scheduling algorithm itself will dictate the level of concurrency you 
can achieve.

Strict prio is fundamentally an interface to a big imaginary queue, with 
multiple packet insertion points (the individual bands/rings for each 
prio band).

If you assume a scheduler implementation where each prio band is mapped 
to a separate CPU, you can certainly see where some CPUs could be 
substantially idle while others are overloaded, largely depending on the 
data workload (and priority contained within).

Moreover, you increase L1/L2 cache traffic, not just because of locks, 
but because of data dependencies:

	user	prio	packet		NIC TX ring
	process	band	scheduler

	cpu7	1	cpu1		1
	cpu5	1	cpu1		1
	cpu2	0	cpu0		0

At that point, it is probably more cache- and lock-friendly to keep the 
current TX softirq scheme.

In contrast, a pure round-robin approach is more friendly to concurrency.

	Jeff