[Openib-windows] RE: [PATCH] WPP support in IPoIB

[Yossi Leybovich]

> -----Original Message-----
> From: Fab Tillier [mailto:ftillier at silverstorm.com] 
> Sent: Tuesday, January 17, 2006 10:26 AM
> To: Yossi Leybovich
> Cc: openib-windows at openib.org
> Subject: RE: [PATCH] WPP support in IPoIB
> 
> Hi Yossi,
> 
> > From: Yossi Leybovich [mailto:sleybo at mellanox.co.il]
> > Sent: Tuesday, December 20, 2005 7:12 AM
> > 
> > Fab
> > 
> > Attached is patch that enable WPP in the IPoIB module
> > 
> > The WPP is controlled in compilation 
> (ENABLING_EVENT_TRACING) If  this 
> > Macro is enable then IPoIB will use IPOIB_PRINT to print to the WPP.
> > otherwise, in check build it will use the the regular prints and in 
> > free will print nothing.
> 
> I think that we want to have WPP tracing enabled all the 
> time, but I do want to be a little sparing about it, since 
> even if it's off, it's still a conditional branch and putting 
> it in every function entry/exit is going to be detrimental to 
> performance.  For IPoIB especially, performance is low enough 
> as it is that we should not deliberately reduce it.
> 
> I also don't really like merging debug output with WPP.  
> Reading through the docs, I saw where one could redirect a 
> WPP session to the debugger, but getting this to work at 
> boot-time seems like a lot more setup than just setting the 
> debug level to a given value and being done with it.  I'm 
> still evaluating this, so it may well be much easier in 
> practice than what the docs make it out to be.

I don't think that using 2 mechanisim for printing is good idea 
1. It increase the code and reduce its readability.
2. If we will use 2 mechanisms there is good chance that one will add
printing in one mechanism and forget to add it to the second mechanism
.We need to work with the same mechanism that will collect the data in
the customer site. 
As developer you can always work with DebugPrints and in the customer
site you will get the same priting that you use to work with.
3. We tested WPP on SDP and with no consumer of the WPP, we saw 2-4%
degradation in the performance. I think we can live with this
penalty.(moreover we can use compiling macros to remove the enter/exit
printing in release mode in case it will show performance issue)
We also found it very useful for debugging because its enable you to
reproduce racing condition more easily.
4. As for boot time event for now we can use the DebugOutput. 
 



> 
> > BTW WPP can redirect its prints to the Kernel debugger or 
> to WinDbg so 
> > even with traces we can use these WinDbg .
> > (we used traceview and found it very powerful) For the patch I kept 
> > the regular IPOIB_TRACE I hope to replace them with IPOIB_PRINT
> 
> I think WPP and debug output are both quite useful, and we'll 
> want to use both as they serve different purposes.  I would 
> like to see us use WPP tracing for things that might be 
> useful in the field.  Following the code path (function
> entry/exit) really isn't that useful except during actual 
> debug of the driver, for which regular debug output is 
> adequate.  WPP traces however should contain information that 
> would help diagnose issues in the field, so that a trace 
> gathered in the field would give us enough information to 
> analyze issues without needing customers to attach a 
> debugger.  To that extent, I want to use WPP tracing to 
> replace the cl_perf calls throughout the code, as well as 
> tracking the main entry points into the drivers (and upcalls 
> when calling a system port
> driver) and any error paths.
> 

Again I think that we should consolidate the WPP and the debug traces
and one should use one of the mechanisms.
The WPP build with two dimension of filters (level and flags)
So you can use the appropriate level for getting the amount of messages
you need (and not all the traces).
And as I mention earlier we can use compiling macros to remove the
enter/exit printing in free version.
There are also bugs that accured only with heavy traffic and we found it
useful to debug them with WPP (with all the prints).
And I see option thath in futur debuging of big cluster we simply ask
the customer to increase the debug level and send us logs.


> > I suggest to move all our macros (IPOIB_TRACE ,AL_TRACE....) to use 
> > WPP, We can do it with simple script that will remove the 
> parentheses 
> > from the macros and maybe change some formats type 
> mismatches (the WPP 
> > check format string more carefully then print). I did it 
> for the SDP 
> > and its not too much work.
> 
> I'm a bit weary of getting rid of the standard debug output.  
> Maybe we can have two different types of WPP output, some 
> that's always compiled in, and some that's conditionally 
> compiled and can be changed to regular debug output (or do 
> both WPP and debug output).  Getting the output redirected 
> from WPP to the kernel debugger, especially at boot time, 
> seems to be more work than simply setting the debug level.  I 
> still need to play around with it to get first hand 
> experience, and if it's straight forward then I'll buy into a 
> full conversion to WPP and elimination of debug prints.
> 

I also agree that its not rtivial to redirect boot time events to kernel
debbuger.
But for boot time debuging you can use the debugOutput (same as current
code,in the current code in free version there is no prints at all so
just compile the check version without events and you will get the
regular printing)
I will further investigate that ( I recall that was enabale to log the
traces to the debugger)

 
> Thanks for sending this.  I definitely want to enable WPP in 
> the different drivers - having the ability to collect from 
> multiple data sources and correlate the results is very 
> valuable, and WPP allows that to be done without the impact 
> on performance that debug output imposes.
> 
> - Fab
> 
>