[openib-general] how to turn on kernel level tracing (mtl_log)

[mark kowalski]

I've been trying, without success unfortunately to turn on  tracing
within the kernel components of openib.  I've used the logset
mtl_log_dbg_print command to toggle the "debug_print" variable used in
the mtl_log command. I've also used logset to set the list of
severities to be printed to 8 (12345678).  I've also turned on
printing for any debug or error messages (MTL_DEBUG and MTL_ERROR).  I
even went so far as to add print info structure records for every
module_name I could grep in the source code (VIP, HCA, VIPKL, etc). 
None of this had any affect on getting any kind of trace records
printed out of the kernel.  the only messages I got were sev 1
messages when I shut the system down.

 <4>***** mtl_log:DEBUG: layer 'VIPKL', type 2, sev '1'
<4>***** mtl_log:DEBUG: Found layer 'VIPKL', Name="error", sev="12345678"
<4>***** mtl_log:DEBUG: print string
<1> VIPKL(1): var/tmp/IBGD/lib/modules/2.6.4-52-smp/build/drivers/infiniband/hw/mellanox-hca/vipkl/em.c[87]:
EM delete:found unreleased async object

<4>***** mtl_log:DEBUG: layer 'VIPKL', type 2, sev '1'
<4>***** mtl_log:DEBUG: Found layer 'VIPKL', Name="error", sev="12345678"
<4>***** mtl_log:DEBUG: print string
<1> VIPKL(1): var/tmp/IBGD/lib/modules/2.6.4-52-smp/build/drivers/infiniband/hw/mellanox-hca/vipkl/em.c[87]:
EM delete:found unreleased async object

Is the code originally compiled with the MAX_TRACE variable set to 1? 
Is there a way this could be changed  or bypassed without recompiling
the source?  I also added the MTL_LOG environment variable to get
non-kernel messages but that didn't produce any messages either.

any help would be appreciated.
Thanks
Mark