[ewg] Re: Removing the transport module before iscsid

[Erez Zilber]

Mike Christie wrote:
> Erez Zilber wrote:
>   
>> Mike Christie wrote:
>>     
>>> Erez Zilber wrote:
>>>   
>>>       
>>>> Mike,
>>>>
>>>> We're trying to solve a problem, but first the motivation: a user may
>>>> want to stop the InfiniBand modules (/etc/init.d/openibd stop) while
>>>> open-iscsi is running. In order to do that, we must remove ib_iser.
>>>>
>>>> Now, the question: is it ok to modprobe -r ib_iser (or iscsi_tcp) before
>>>>     
>>>>         
>>> It is and it isn't :)
>>>
>>>   
>>>       
>>>> stopping iscsid? From what I see in the init.d scripts in open-iscsi, we
>>>> kill iscsid first. Is there a reason for that?
>>>>
>>>>     
>>>>         
>>> We kill iscsid first because it has a ref to scsi_transport_iscsi (load 
>>> that module and run iscsid and then do a lsmod). So if you ran iscsid, 
>>> then just modprobe -r ib_iser, it would leave scsi_transport_iscsi loaded.
>>>
>>>   
>>>       
>> If I run /etc/init.d/openibd stop, it will (with a patch that I'm going
>> to submit) call iscsi_conn_failure for all iSER connections, and then
>> remove ib_iser. scsi_transport_iscsi will be removed later when
>>     
>
>
> This does not sound right or even possible with current kernels. If a 
> session is runing and scsi_devices exists for it and if a user has a 
> device open then there are references to the ib_iser module.
>
> If the user does a close then it needs the scsi_host_tempalte that was 
> allocated in the ib_iser module. For libiscsi/scsi_transport_iscsi there 
> is similar problems with the those modules accessing the iscsi_transport 
> struct allocated in the module.
>
> The thing is that scsi-ml should be incrementing the ib_iser refcount if 
> a device is open. And open-iscsi should be incrementing the ib_iser 
> refctoun if there is a session. So you should not be able to remove the 
> ib_iser module if there are connections.
>
>   

I'm adding Vlad & openfabrics-ewg to this discussion, as it seems
relevant for OFED.

Mike - you're right. I guess that I missed something here. My patch
(http://groups.google.com/group/open-iscsi/browse_frm/thread/2c99cce2956ac4d7)
only allows the removal of the HCA driver (e.g. ib_mthca). This is
better than the BUG() call that we had there until now.

The motivation was to allow users to stop openibd even while open-iscsi
is running. The problem is that even after removing the HCA driver,
ib_iser is still referenced and cannot be unloaded (unless
/etc/init.d/open-iscsi stop is called). Therefore, most of the IB stack
cannot be unloaded (rdma_cm, ib_cm, ib_core etc).

Here's my suggestion: in addition to this patch (which is good if a user
decides to remove only ib_mthca), we should add some code to
/etc/init.d/openibd that will check if open-iscsi is loaded and if there
are iSER sessions. If yes, openibd will not stop until the user logs out
from the iSER sessions. When there are no more iSER sessions, openibd
will stop and will also remove ib_iser which is required for the removal
of other IB modules. Mike - the removal of ib_iser will require the
other patch that we talked about (deleting the iSER transport entry from
the transport list).

If everyone's ok with this, I will make the required changes and send a
patch.

Erez