<html>
<body>
<font size=3>At 07:46 AM 8/31/2005, James Lentini wrote:<br><br>
<br>
<blockquote type=cite class=cite cite="">On Tue, 30 Aug 2005, Roland
Dreier wrote:<br><br>
> I just committed this SRP fix, which should make sure we don't use
a<br>
> device after it's gone. And it actually simplifies the code a
teeny bit...<br><br>
The device could still be used after it's gone. For example:<br><br>
- the user is configuring SRP via sysfs. The thread in <br>
srp_create_target() has just called ib_sa_path_rec_get()
<br>
[srp.c line 1209] and is waiting for the path <br>
record query to complete in wait_for_completion()<br>
- the SA callback, srp_path_rec_completion(), is called. This <br>
callback thread will make several verb calls (ib_create_cq,
<br>
ib_req_notify_cq, ib_create_qp, ...) without any
coordination with <br>
the hotplug device removal callback, srp_remove_one
<br><br>
Notice that if the SA client's hotplug removal function, <br>
ib_sa_remove_one(), ensured that all callbacks had completed before <br>
returning the problem would be fixed. This would protect all ULPs from
<br>
having to deal with hotplug races in their SA callback function. The
<br>
fix belongs in the SA client (the core stack), not in SRP.<br><br>
All the ULPs are deficient with respect to their hotplug <br>
synchronization. Given that there is a common problem, doesn't it make
<br>
sense to try and solve it in a generic way instead of in each
ULP?</blockquote><br>
There are two approaches to device removal to consider - both are
required to have a credible solution:<br><br>
(1) Inform all entities that a planned device removal is to occur and
allow them to close gracefully or migrate to alternatives. Ideally,
the OS comprehends whether the removal will result in the loss of any
critical resources and not inform or take action unless it knows the
removal is something that the system can survive. Doing this
requires the ULP to register interest with the OS in a particular
hardware resource. This also allows the OS to construct a resource
analysis tool to determine whether the removal of a device will be a good
idea or not. This is really outside the scope of an RDMA
infrastructure and should be done by the OS through an OS defined API
which is applicable to all types of hardware resources and
sub-systems.<br><br>
(2) Design all ULP to handle surprise removal, e.g. device failure, from
the start and allow them to close gracefully or migrate to
alternatives. The OS would inform the device driver of the failure
if the device driver has not already discovered the problem. The OS
would also inform interested parties of the device failure. The
device driver would simply error out all users of the device instance -
there are already error codes defined for IB and iWARP for this
purpose. The associated verbs resources should be released as the
ULP closes out its resources through the verbs API (we did define the
verbs to clean up resources that the infrastructure may allocate on
behalf of the ULP). Activities such as listen entries would be
released just like what is done for Sockets, etc. today. <br><br>
<br>
Device addition is simply a matter of informing policy or whatever
service management within the OS that determines what services should be
available on a given device. The device driver really does not need
to do anything special. One area to consider is whether a planned
migration of a service needs to be supported. This is generally
best handled by the ULP with only a small set of services required of the
infrastructure, e.g. get / set of QP / LLP context and then coordinating
any other aspects with the appropriate SM or network services such
updating address vectors or fabric management / configuration.<br><br>
In general, the ULP should already be designed to handle the error
condition and whether they support a managed / planned removal or
migration is perhaps the only potential area of deficiency.<br><br>
Mike </font></body>
</html>