[ofa-general] [PATCH 0/ 8] XRC patch series (including xrc receive-only QPs)

[Jack Morgenstein]

On Friday 01 February 2008 00:50, Tang, Changqing wrote:
> Jack:
>         In order to open a new XRC domain, all processes on a node open a file descriptor using the
> same pathname, and pass the fd to ibv_open_xrc_domain().
> 
>         When can I close the fd ? when can I remove the temp file ?  Can I close the fd and unlink the
> temp file right after ibv_open_xrc_domain() returns ?
> 
>         Does ibv_open_xrc_domain() increase the fd reference count and ibv_close_xrc_domain() decrease the
> fd reference count ?
> 

I don't know what you mean by "unlink the temp file".
However, the following is true:

1. The first time ibv_open_xrc_domain() is used with a temp file, in kernel space the temp file
   descriptor is used to access the file's inode entry.

   a. An xrc_domain "object" is created in kernel space, with a reference count of 1.
   b. A reference on the inode is taken (just once, at xrc_domain object creation time, 
      for all ibv_open_xrc_domain() calls which arrive at that same inode).
   c. Once you have an xrc_domain handle, you may close the temp file -- you've already
      got the xrc handle you need, and no longer need to go through the inode.

   d. All subsequent calls to ibv_open_xrc_domain() for that inode will not increment the
      inode's reference count.  However, each such call WILL increment the xrc_domain object's
      reference count. (a 2-layer reference counting system) -- and each such call must
      have a corresponding ibv_close_xrc_domain() call to decrement the xrc_domain object's ref count.

2. ibv_close_xrc_domain()
   a. decrements the xrc_domain object's ref count.  If that count is then zero, the inode
      reference count is also decremented (see step 1.b above), and the xrc_domain object
      is destroyed.

Note:  NO extra reference counts are taken on the fd -- only a single extra ref count is taken
       on the inode itself. (Thus, even if the file is removed, the inode associated with the
       file will still be kept, until all xrc_domain users have closed the xrc domain).

Note 2:  Even if you "remove" the file and create a new file with the same filename before ALL
        users of the previous domain have released (i.e., closed) the xrc domain, the new file
        will get a different inode entry, so there should be no xrc domain collisions.

- Jack