[ofa-general] Is IBIS only for querying OpenSM?

[Philippe Gregoire]

terry watson a e'crit :
> Hi all,
>
> I will be performing some testing of partitioning used as a security control. Am I right in believing that IBIS will be able to set partition table values of the local compute node I am logged on to, even though they are not using OpenSM, but rather a SM on a switch? Could I then attempt to access a partition that I was originally excluded from accessing?
>
> I am new to Infiniband technology and would also appreciate a response from an expert who has views on the strength of the security that partitioning provides in separating two clusters that should have no interaction whatsoever.
>
> Thanks,
> Dave
> _________________________________________________________________
> Discover the new Windows Vista
> http://search.msn.com/results.aspx?q=windows+vista&mkt=en-US&form=QBRE_______________________________________________
> general mailing list
> general at lists.openfabrics.org
> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
>
>   
The partitions are only managed by the subnet manager - either opensm 
running on a node into the fabric or an embedded subnet manager on a switch.
For opensm , partitions are defined into a configuration file 
/etc/opensm/partitions.conf, for a embedded subnet manager, you have to 
configure the partitions using the CLI or GUI provided by the switch.
Defining a partition is mainly choosing a pkey and ports nodes with 
their membership (limited or not).

The subnet manager assigned the pkeys to the ports of the node when ib 
kernel modules are loaded. You can see the partitions the IB port belong 
to by ( I mean those defined by the subnet manager) :
# grep -v 0x0000 /sys/class/infiniband/mthca0/ports/1/pkeys/*
/sys/class/infiniband/mthca0/ports/1/pkeys/0:0xffff
/sys/class/infiniband/mthca0/ports/1/pkeys/1:0x8001
/sys/class/infiniband/mthca0/ports/1/pkeys/2:0x8002
/sys/class/infiniband/mthca0/ports/1/pkeys/3:0x8003
/sys/class/infiniband/mthca0/ports/1/pkeys/4:0x8010

A port may belong to many partitions. Nodes (ports) may have different 
partitions configurations. Partitions order for a port is not always the 
same ( it may depend on the chronology of partition declarations in the 
subnet manager)

Over these partitions, you can define new IP (IP over IB) interfaces by 
creating files like /etc/sysconfig/network-scripts/ifcfg-ib0.8002 :
# cat /etc/sysconfig/network-scripts/ifcfg-ib0.8002
DEVICE=ib0.8002
BOOTPROTO=static
IPADDR=XXX.YYY.ZZZ.TTT
NETMASK=255.255.255.0
NETWORK=255.255.255.0
ONBOOT=yes

The openibd script create the child interface and configure it at system 
startup using some special devices to do that :
echo $pkey > /sys/class/net/ib0/create_child

But this command creates only a child interface on the node, but 
communications on this interface will not work until you add the port 
node to the corresponding partition into the subnet manager 
configuration. Then you will see the pkey appearing automatically  into 
files /sys/class/infiniband/mthca0/ports/1/pkeys/*  on the node.

[root at cors118 ~]# echo 0x8009 >  /sys/class/net/ib0/create_child
[root at cors118 ~]# dmesg | grep 8009
divert: not allocating divert_blk for non-ethernet device ib0.8009
[root at cors118 ~]# grep -v 0x0000 
/sys/class/infiniband/mthca0/ports/1/pkeys/*
/sys/class/infiniband/mthca0/ports/1/pkeys/0:0xffff
/sys/class/infiniband/mthca0/ports/1/pkeys/1:0x8001
/sys/class/infiniband/mthca0/ports/1/pkeys/2:0x8002
/sys/class/infiniband/mthca0/ports/1/pkeys/3:0x8003
/sys/class/infiniband/mthca0/ports/1/pkeys/4:0x8010
[root at cors118 ~]# ifconfig -a | grep 8009
ib0.8009  Link encap:UNSPEC  HWaddr 
00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
[root at cors118 ~]# echo 0x8009 >  /sys/class/net/ib0/delete_child
[root at cors118 ~]# dmesg | grep 8009
divert: not allocating divert_blk for non-ethernet device ib0.8009
divert: no divert_blk to free, ib0.8009 not ethernet

To use MPI with partitions, you have also to configure it (in the 
configuration file) . For MVAPICH you must use  VIADEV_DEFAULT_PKEY_IX 
or VIADEV_DEFAULT_PKEY in the config file : 
/usr/mpi/gcc/mvapich-1.0.0/etc/mvapich.conf . AT CEA, I'm using  
VIADEV_DEFAULT_PKEY (pkey value)
as we have nodes with different partitions configurations.


Hoping this will help you.
Regards
Philippe Gregoire CEA/DAM


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20080418/534584d2/attachment.html>