[ofw] [PATCH] Limit time spent at DISPATCH_LEVEL when processingMADs

[Tzachi Dar]

Hi Fab,

There are some points in what you are saying, but there are also some
drawbacks.
I believe that the main problem is that if your DPC is long than you
should give us 
Your DPC object. On the other hand if your DPC is short and there are
many CQs to serve 
than creating many dpcs is waste of CPU.

Please also note that mixing of these solutions is not easy. The reason
for that is
that on the call back solution you go over the EQ at DPC level while on
the DPC object
solution you go over it in an interrupt level. Mixing this two makes the
code very complicated.

The solution that we thought to overcome this problem in mlx4 was to let
clients 
create their own Eqs and manage them.

It is possibale to do things differently:
Create one EQ for people who give functions and another one for people
who wants 
A DPC created. This will make our code more general, and live handling
the EQ to 
One place. Once you register your CQ you will be moved to the correct EQ
based on
the function/DPC that you will give.

So if my idea sounds reasonable, we should decide in what time frame it
will be 
implemented.

Thanks
Tzachi

> -----Original Message-----
> From: Fab Tillier [mailto:ftillier at windows.microsoft.com] 
> Sent: Monday, July 14, 2008 8:45 PM
> To: Tzachi Dar; ofw at lists.openfabrics.org
> Subject: RE: [ofw] [PATCH] Limit time spent at DISPATCH_LEVEL 
> when processingMADs
> 
> Hi Tzachi,
> 
> > From: Tzachi Dar [mailto:tzachid at mellanox.co.il]
> > Sent: Sunday, July 13, 2008 12:23 AM
> >
> > Hi Fab,
> >
> > For the long term, we should probably change the way that we work:
> >
> > Today's machines always come with more than one core. I'm quit sure 
> > that in the situation that you are facing, there are other 
> cores that 
> > are not doing anything, while other cores are competing on 
> one core. 
> > The natural solution is to use more than one core for this requests.
> >
> > I believe that once we will move to MSI on server 2008, this should 
> > become more natural.
> 
> I agree spreading the load on more cores would be beneficial, 
> however we still want to limit how long we spend in DPCs.
> 
> I think a worthwhile change to CQ creation would be for the 
> client to pass in the DPC object they want to use for 
> notifications.  This would allow the client to set the 
> affinity and priority of the DPC, not to mention clients that 
> stay too long in their DPC would get the blame rather than 
> the HCA driver.
> 
> -Fab
>