<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<html>
<head>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=us-ascii">
<meta name=Generator content="Microsoft Word 10 (filtered)">
<title>RE: [Openib-windows] Re: Complib changes to support OpenSM in OpenIB</title>
<style>
<!--
/* Font Definitions */
@font-face
{font-family:"MS Mincho";
panose-1:2 2 6 9 4 2 5 8 3 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:"\@MS Mincho";
panose-1:2 2 6 9 4 2 5 8 3 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman";}
a:link, span.MsoHyperlink
{color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{color:blue;
text-decoration:underline;}
p
{margin-right:0in;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman";}
span.emailstyle18
{font-family:Arial;
color:navy;}
span.EmailStyle19
{font-family:Arial;
color:navy;}
@page Section1
{size:8.5in 11.0in;
margin:1.0in 1.25in 1.0in 1.25in;}
div.Section1
{page:Section1;}
-->
</style>
</head>
<body lang=EN-US link=blue vlink=blue>
<div class=Section1>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>What is the problem with keeping qlockpool
in opensm?</span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'> </span></font></p>
<div style='border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt'>
<p class=MsoNormal><font size=2 face=Tahoma><span style='font-size:10.0pt;
font-family:Tahoma'>-----Original Message-----<br>
<b><span style='font-weight:bold'>From:</span></b> </span></font><font size=2
face=Tahoma><span style='font-size:10.0pt;font-family:Tahoma'>Tzachi Dar</span></font><font
size=2 face=Tahoma><span style='font-size:10.0pt;font-family:Tahoma'>
[mailto:tzachid@mellanox.co.il] <br>
<b><span style='font-weight:bold'>Sent:</span></b> </span></font><font size=2 face=Tahoma><span style='font-size:10.0pt;font-family:Tahoma'>Sunday, August
28, 2005</span></font><font size=2 face=Tahoma><span style='font-size:10.0pt;
font-family:Tahoma'> </span></font><font size=2 face=Tahoma><span
style='font-size:10.0pt;font-family:Tahoma'>2:15 AM</span></font><font size=2
face=Tahoma><span style='font-size:10.0pt;font-family:Tahoma'><br>
<b><span style='font-weight:bold'>To:</span></b> </span></font><font size=2
face=Tahoma><span style='font-size:10.0pt;font-family:Tahoma'>Tillier, Fabian</span></font><font
size=2 face=Tahoma><span style='font-size:10.0pt;font-family:Tahoma'>; </span></font><font
size=2 face=Tahoma><span style='font-size:10.0pt;font-family:Tahoma'>Tzachi
Dar</span></font><font size=2 face=Tahoma><span style='font-size:10.0pt;
font-family:Tahoma'>; </span></font><font size=2 face=Tahoma><span
style='font-size:10.0pt;font-family:Tahoma'>Eitan Zahavi</span></font><font
size=2 face=Tahoma><span style='font-size:10.0pt;font-family:Tahoma'><br>
<b><span style='font-weight:bold'>Cc:</span></b> </span></font><font size=2
face=Tahoma><span style='font-size:10.0pt;font-family:Tahoma'>Aviram Gutman</span></font><font
size=2 face=Tahoma><span style='font-size:10.0pt;font-family:Tahoma'>; </span></font><font
size=2 face=Tahoma><span style='font-size:10.0pt;font-family:Tahoma'>openib-windows@openib.org</span></font><font
size=2 face=Tahoma><span style='font-size:10.0pt;font-family:Tahoma'><br>
<b><span style='font-weight:bold'>Subject:</span></b> RE: [Openib-windows] Re:
Complib changes to support OpenSM in OpenIB</span></font></p>
<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'> </span></font></p>
<p><font size=2 face="Times New Roman"><span style='font-size:10.0pt'>Hi fab, </span></font></p>
<p><font size=2 face="Times New Roman"><span style='font-size:10.0pt'>I read
the answer that you have sent me and I still disagree. </span></font><br>
<font size=2><span style='font-size:10.0pt'>Locking of objects and maintaining
their life cycle is a very complicated task no matter what approach is used. I
hope that we can agree at least on this.</span></font></p>
<p><font size=2 face="Times New Roman"><span style='font-size:10.0pt'>As a
result there are also many "valid" approaches on how to solve these
problems. Deciding which approach is better has to take the following
considerations: 1) simplicity of the code. 2) Performance of the code. 3) Time
that is available for developing the code. 4) Existing code already. 5) Does
the code has to be portable or not (and to where), and probably many other
issues. Please also note, that when trying to decide on one of this issues, it
is very hard to tell what are the "sub-considerations" that should be
argued for example let's take performance: Are we measuring performance on a
single processor or on a multiprocessor system? Are we measuring the
performance when the system is loaded or not. Are we measuring the worst time
complicity or the average complexity?</span></font></p>
<p><font size=2 face="Times New Roman"><span style='font-size:10.0pt'>So far,
many questions and not a single answer.</span></font> <br>
<font size=2><span style='font-size:10.0pt'>I have wrote all this questions
just to show that since the question is very complicated, it is hard to believe
that it will have a simple answer. (I guess that this is the nice thing about
computer since: although this "science" exists for more than 50
years, there is not one implementation of an array that is agreed on everyone).</span></font></p>
<p><font size=2 face="Times New Roman"><span style='font-size:10.0pt'>The
answer that you gave me bellow, if I understand it correctly, tells that it is
possible to do better than always taking the lock (at some situations).
Although I agree that it is some times true, I would like to suggest three
other approaches to solving a programming problem in which this is not the
case. What I mean by that is that using these approaches, one can write a
"good" (what is really good, was not yet defined) program. Under this
approaches one need a "multi-thread safe" container.</span></font></p>
<p><font size=2 face="Times New Roman"><span style='font-size:10.0pt'>These
approaches are:</span></font> <br>
<font size=2><span style='font-size:10.0pt'>1) Use reference counting to mange
the life time of the container, and use it from many threads. Personally I
believe that when the project is big enough, this is the only thing that works,
but this is just my opinion.</span></font></p>
<p><font size=2 face="Times New Roman"><span style='font-size:10.0pt'>2) Use
reference counting to manage the life time of more than one container and use
them all.</span></font> <br>
<font size=2><span style='font-size:10.0pt'>3) Have the container(s) accessed
by more than one thread simultaneously and destroy it only when that thread is
dead.</span></font> </p>
<p><font size=2 face="Times New Roman"><span style='font-size:10.0pt'>Again,
I'm not saying that any of this approaches are "always better", but I
do say that they might be used. As a result, I suggest that we will add the
locked-pool to the complib implementation.</span></font></p>
<p><font size=2 face="Times New Roman"><span style='font-size:10.0pt'>Thanks</span></font>
<br>
<font size=2><span style='font-size:10.0pt'>Tzachi</span></font> </p>
<p><font size=2 face="Times New Roman"><span style='font-size:10.0pt'>One other
small point that I want to mention is this: weather to use inline or not inline
is an implementation detail - although we might have to close this detail one
day, I'm not sure that this is the time. </span></font></p>
<p><font size=2 face="Times New Roman"><span style='font-size:10.0pt'>>-----Original
Message-----</span></font> <br>
<font size=2><span style='font-size:10.0pt'>>From: </span></font><font
size=2><span style='font-size:10.0pt'>Fab Tillier</span></font><font size=2><span
style='font-size:10.0pt'> [<a href="mailto:ftillier@silverstorm.com">mailto:ftillier@silverstorm.com</a>]</span></font>
<br>
<font size=2><span style='font-size:10.0pt'>>Sent: Friday, August 26, 2005
8:56 PM</span></font> <br>
<font size=2><span style='font-size:10.0pt'>>To: '</span></font><font
size=2><span style='font-size:10.0pt'>Tzachi Dar</span></font><font size=2><span
style='font-size:10.0pt'>'; </span></font><font size=2><span style='font-size:
10.0pt'>Eitan Zahavi</span></font> <br>
<font size=2><span style='font-size:10.0pt'>>Cc: </span></font><font size=2><span
style='font-size:10.0pt'>Aviram Gutman</span></font><font size=2><span
style='font-size:10.0pt'>; </span></font><font size=2><span style='font-size:
10.0pt'>openib-windows@openib.org</span></font> <br>
<font size=2><span style='font-size:10.0pt'>>Subject: RE: [Openib-windows]
Re: Complib changes to support OpenSM in</span></font> <br>
<font size=2><span style='font-size:10.0pt'>>OpenIB</span></font> <br>
<font size=2><span style='font-size:10.0pt'>></span></font> <br>
<font size=2><span style='font-size:10.0pt'>>> From: </span></font><font
size=2><span style='font-size:10.0pt'>Tzachi Dar</span></font><font size=2><span
style='font-size:10.0pt'> [<a href="mailto:tzachid@mellanox.co.il">mailto:tzachid@mellanox.co.il</a>]</span></font>
<br>
<font size=2><span style='font-size:10.0pt'>>> Sent: Friday, August 26,
2005 6:37 AM</span></font> <br>
<font size=2><span style='font-size:10.0pt'>>></span></font> <br>
<font size=2><span style='font-size:10.0pt'>>> Can you please explain in
more detail what is wrong with the</span></font> <br>
<font size=2><span style='font-size:10.0pt'>>cl_qlockpool.c ?</span></font> <br>
<font size=2><span style='font-size:10.0pt'>>> It seems to me like a very
common programming way that there is a</span></font> <br>
<font size=2><span style='font-size:10.0pt'>>container</span></font> <br>
<font size=2><span style='font-size:10.0pt'>>> that is single threaded,
and in order to make it "multi-threaded save"</span></font> <br>
<font size=2><span style='font-size:10.0pt'>>one</span></font> <br>
<font size=2><span style='font-size:10.0pt'>>> adds a lock around it.</span></font>
<br>
<font size=2><span style='font-size:10.0pt'>></span></font> <br>
<font size=2><span style='font-size:10.0pt'>>I have generally found that
locking is required at a higher level than just</span></font> <br>
<font size=2><span style='font-size:10.0pt'>>the</span></font> <br>
<font size=2><span style='font-size:10.0pt'>>pool or list operations.
There must be extra logic to prevent the pool</span></font> <br>
<font size=2><span style='font-size:10.0pt'>>from</span></font> <br>
<font size=2><span style='font-size:10.0pt'>>being destroyed while items are
out, and locking at such a fine level</span></font> <br>
<font size=2><span style='font-size:10.0pt'>>doesn't</span></font> <br>
<font size=2><span style='font-size:10.0pt'>>help at all.</span></font> <br>
<font size=2><span style='font-size:10.0pt'>></span></font> <br>
<font size=2><span style='font-size:10.0pt'>>Even looking in the OSM code
base, the few cases that I tried to figure out</span></font> <br>
<font size=2><span style='font-size:10.0pt'>>held</span></font> <br>
<font size=2><span style='font-size:10.0pt'>>other locks while accessing the
pools. If there are other locks held, then</span></font> <br>
<font size=2><span style='font-size:10.0pt'>>locking at the pool level is a
waste of resources as the lock will never</span></font> <br>
<font size=2><span style='font-size:10.0pt'>>see</span></font> <br>
<font size=2><span style='font-size:10.0pt'>>contention. Having the
lock hidden in the pool operations makes it much</span></font> <br>
<font size=2><span style='font-size:10.0pt'>>less</span></font> <br>
<font size=2><span style='font-size:10.0pt'>>obvious when the lock is
unnecessary. If you see explicit locking, it's</span></font> <br>
<font size=2><span style='font-size:10.0pt'>>much</span></font> <br>
<font size=2><span style='font-size:10.0pt'>>easier to find out if the lock
is needed or not, and if not, eliminated.</span></font> <br>
<font size=2><span style='font-size:10.0pt'>></span></font> <br>
<font size=2><span style='font-size:10.0pt'>>Lastly, if an operation does
something like:</span></font> <br>
<font size=2><span style='font-size:10.0pt'>></span></font> <br>
<font size=2><span style='font-size:10.0pt'>>get item from pool</span></font>
<br>
<font size=2><span style='font-size:10.0pt'>>try something</span></font> <br>
<font size=2><span style='font-size:10.0pt'>>if failed, put item in pool</span></font>
<br>
<font size=2><span style='font-size:10.0pt'>></span></font> <br>
<font size=2><span style='font-size:10.0pt'>>If "try something" is
a relatively quick operation, you'll get better</span></font> <br>
<font size=2><span style='font-size:10.0pt'>>performance taking the lock up
front and releasing it after the failure</span></font> <br>
<font size=2><span style='font-size:10.0pt'>>test.</span></font> <br>
<font size=2><span style='font-size:10.0pt'>></span></font> <br>
<font size=2><span style='font-size:10.0pt'>>Also, as I mentioned before,
having a function call just to mask a couple</span></font> <br>
<font size=2><span style='font-size:10.0pt'>>function calls doesn't seem
efficient. The qlockpool functions should be</span></font> <br>
<font size=2><span style='font-size:10.0pt'>>inline.</span></font> <br>
<font size=2><span style='font-size:10.0pt'>></span></font> <br>
<font size=2><span style='font-size:10.0pt'>>- Fab</span></font> </p>
</div>
</div>
</body>
</html>