[ofw] [PATCH] partition
Yiftah Shahar
yiftahs at voltaire.com
Thu Apr 3 03:07:10 PDT 2008
Fab,
I trust them but the problem is that organization (at list as we have
encountered) tend not to "trust" different groups.
The server system administrators are not the network system guys and
they do not want that anything will happen in the server without their
specific approval/action.
This is not related to IB and also for Ethernet VLAN tagging they insist
to set the VLAN also in the nodes and not only in the switches ...
In any case we can discuss more about this next week (like adding
work-mode auto or manual that will be set during install time...).
Yiftah
> -----Original Message-----
> From: Fab Tillier [mailto:ftillier at windows.microsoft.com]
> Sent: Thursday, April 03, 2008 08:14
> To: Yiftah Shahar; Slava Strebkov; ofw at lists.openfabrics.org
> Subject: RE: [ofw] [PATCH] partition
>
> If you don't trust whoever controls the SM you have a serious problem
on
> your hands. A single point of control is much preferable to thousands
of
> them. A thousand duplicated points of control is error prone.
>
> -Fab
>
> -----Original Message-----
> From: Yiftah Shahar [mailto:yiftahs at voltaire.com]
> Sent: Wednesday, April 02, 2008 10:06 PM
> To: Fab Tillier; Slava Strebkov; ofw at lists.openfabrics.org
> Subject: RE: [ofw] [PATCH] partition
>
>
> Hi Fab,
>
> > This way all configuration is automatic, rather than requiring
registry
> > modification. A single point of configuration (the SM) takes care
of
> > it all.
>
> From our experience with many different types of customers the
requirement
> is clear: whenever someone does something on the switch, NOTHING
should
> automatically happen on the host.
> This is a HUGE security hole and sys admins will complain that the
> networking group who controls the switches is now able to screw with
host
> configuration from the switch without explicitly having been permitted
to
> do so by someone on the windows sysadmin security team.
>
> Maybe we need to have other control mechanisms but for first version
> registry key is reasonable way to go.
>
>
> Do remember that we still have this completion with error bug that
with
> out it this feature is not relevant ( ;-( ) so if anyone can help us
fix
> it this will be great.
>
>
> Yiftah
>
>
> ________________________________________
> From: ofw-bounces at lists.openfabrics.org [mailto:ofw-
> bounces at lists.openfabrics.org] On Behalf Of Fab Tillier
> Sent: Wednesday, April 02, 2008 17:30
> To: Slava Strebkov; ofw at lists.openfabrics.org
> Subject: RE: [ofw] [PATCH] partition
>
> Why not have ibbus create a PDO for each valid pkey in the pkey table
> automatically, and then report the pkey as part of the
ipoib_ifc_data_t
> structure returned when IPoIB queries for the IBAL_INTERFACE_STANDARD?
> Note that a similar change could then be done to the other interfaces
> reported to allow partition management for all IB attached devices.
>
> This way all configuration is automatic, rather than requiring
registry
> modification. A single point of configuration (the SM) takes care of
it
> all.
>
> Also, it might be clearer if you renamed the port_guid structure to
> port_info or similar, so you don't have port_guid.port_guid and
> port_guid.pkey.
>
> -Fab
>
> From: ofw-bounces at lists.openfabrics.org [mailto:ofw-
> bounces at lists.openfabrics.org] On Behalf Of Slava Strebkov
> Sent: Wednesday, April 02, 2008 5:39 AM
> To: ofw at lists.openfabrics.org
> Subject: [ofw] [PATCH] partition
>
> Hi,
> The following patch implements partition feature. It's still not
working;
> CQ completes WR's with IB_WCS_WR_FLUSHED_ERR.
> Pkey value is written into registry under
> HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\ibbus\Parameters.
> To use partition, specify valid pkey value in IbPkey.
> Ibbus driver creates additional PDO for each Pkey value read from
registry.
>
> Index: core/bus/kernel/bus_driver.c
> ===================================================================
> --- core/bus/kernel/bus_driver.c (revision 1026)
> +++ core/bus/kernel/bus_driver.c (working copy)
> @@ -56,6 +56,8 @@
> #else
> #define DEFAULT_NODE_DESC "OpenIB Windows(r) Host"
> #endif
> +/* Pkey read from registry */
> +pkey_array g_ib_pkey_array;
>
>
> char node_desc[IB_NODE_DESCRIPTION_SIZE];
> @@ -185,7 +187,100 @@
> BUS_ENTER( BUS_DBG_DRV );
> }
>
> +static void __prepare_pKey_array(const UNICODE_STRING *str)
> +{
> + uint16_t i;
> + NTSTATUS status;
> + ANSI_STRING ansi_str;
> + g_ib_pkey_array.num_pKeys = 0;
>
> + status = RtlUnicodeStringToAnsiString(&ansi_str,str,TRUE);
> + if(! NT_SUCCESS(status))
> + return;
> +
> + for (i = 0; (i < ansi_str.MaximumLength) &&
> (g_ib_pkey_array.num_pKeys < MAX_NUM_PKEY) ; i++)
> + {
> + switch(ansi_str.Buffer[i])
> + {
> + case '0':
> + if (((i+1) < ansi_str.Length) &&
> ( ( ansi_str.Buffer[i+1] == 'x') || ( ansi_str.Buffer[i+1] == 'X')))
> + break;
> + else
> + {
> +
> g_ib_pkey_array.pKey[g_ib_pkey_array.num_pKeys] = \
> +
> (g_ib_pkey_array.pKey[g_ib_pkey_array.num_pKeys] << 4)| 0;
> + break;
> + }
> +
> + case 'x':
> + case 'X':
> + break;
> +
> + case ',':
> + g_ib_pkey_array.num_pKeys++;
> + break;
> +
> + case 'A':
> + case 'a':
> +
> g_ib_pkey_array.pKey[g_ib_pkey_array.num_pKeys] = \
> +
> (g_ib_pkey_array.pKey[g_ib_pkey_array.num_pKeys] << 4)| 0xA;
> + break;
> +
> + case 'B':
> + case 'b':
> +
> g_ib_pkey_array.pKey[g_ib_pkey_array.num_pKeys] = \
> +
> (g_ib_pkey_array.pKey[g_ib_pkey_array.num_pKeys] << 4)| 0xB;
> + break;
> +
> + case 'C':
> + case 'c':
> +
> g_ib_pkey_array.pKey[g_ib_pkey_array.num_pKeys] = \
> +
> (g_ib_pkey_array.pKey[g_ib_pkey_array.num_pKeys] << 4)| 0xC;
> + break;
> +
> + case 'D':
> + case 'd':
> +
> g_ib_pkey_array.pKey[g_ib_pkey_array.num_pKeys] = \
> +
> (g_ib_pkey_array.pKey[g_ib_pkey_array.num_pKeys] << 4)| 0xD;
> + break;
> +
> + case 'E':
> + case 'e':
> +
> g_ib_pkey_array.pKey[g_ib_pkey_array.num_pKeys] = \
> +
> (g_ib_pkey_array.pKey[g_ib_pkey_array.num_pKeys] << 4)| 0xE;
> + break;
> +
> + case 'F':
> + case 'f':
> +
> g_ib_pkey_array.pKey[g_ib_pkey_array.num_pKeys] = \
> +
> (g_ib_pkey_array.pKey[g_ib_pkey_array.num_pKeys] << 4)| 0xF;
> + break;
> +
> + case '1':
> + case '2':
> + case '3':
> + case '4':
> + case '5':
> + case '6':
> + case '7':
> + case '8':
> + case '9':
> +
> g_ib_pkey_array.pKey[g_ib_pkey_array.num_pKeys] = \
> +
> (g_ib_pkey_array.pKey[g_ib_pkey_array.num_pKeys] << 4)|
> (ansi_str.Buffer[i] - '0');
> + break;
> +
> + case '\0':
> + g_ib_pkey_array.num_pKeys++;
> + break;
> +
> + default:
> + break;
> +
> + }
> + }
> +
> + RtlFreeAnsiString(&ansi_str);
> +}
> static NTSTATUS
> __read_registry(
> IN
UNICODE_STRING*
> const p_registry_path )
> @@ -194,6 +289,7 @@
> /* Remember the terminating entry in the table below. */
> RTL_QUERY_REGISTRY_TABLE table[9];
> UNICODE_STRING
> param_path;
> + UNICODE_STRING
> pkeyString;
>
> BUS_ENTER( BUS_DBG_DRV );
>
> @@ -212,7 +308,18 @@
>
> RtlAppendUnicodeStringToString( ¶m_path,
> p_registry_path );
> RtlAppendUnicodeToString( ¶m_path, L"\\Parameters" );
> + RtlInitUnicodeString(&pkeyString,NULL);
> + pkeyString.MaximumLength = 1024*sizeof(WCHAR);
> + pkeyString.Buffer = cl_zalloc( pkeyString.MaximumLength );
> + if( !pkeyString.Buffer )
> + {
> + cl_free( param_path.Buffer );
>
> + BUS_TRACE_EXIT( BUS_DBG_ERROR,
> + ("Failed to allocate parameters
path
> pkeyString.\n") );
> + return STATUS_INSUFFICIENT_RESOURCES;
> + }
> +
> /*
> * Clear the table. This clears all the query callback
> pointers,
> * and sets up the terminating table entry.
> @@ -277,9 +384,17 @@
> table[7].DefaultData = &g_ioc_poll_interval;
> table[7].DefaultLength = sizeof(ULONG);
>
> + table[8].Flags = RTL_QUERY_REGISTRY_DIRECT;
> + table[8].Name = L"IbPkey";
> + table[8].EntryContext = &pkeyString;
> + table[8].DefaultType = REG_SZ;
> + table[8].DefaultData = NULL;
> + table[8].DefaultLength = 1024*sizeof(WCHAR);
> /* Have at it! */
> status = RtlQueryRegistryValues( RTL_REGISTRY_ABSOLUTE,
> param_path.Buffer, table, NULL, NULL );
> + if (NT_SUCCESS(status))
> + __prepare_pKey_array(&pkeyString);
>
> #if DBG
> if( g_al_dbg_flags & AL_DBG_ERR )
> @@ -291,6 +406,7 @@
> g_al_dbg_level,
> g_al_dbg_flags));
>
> + cl_free( pkeyString.Buffer );
> cl_free( param_path.Buffer );
> BUS_EXIT( BUS_DBG_DRV );
> return status;
> Index: core/bus/kernel/bus_driver.h
> ===================================================================
> --- core/bus/kernel/bus_driver.h (revision 1026)
> +++ core/bus/kernel/bus_driver.h (working copy)
> @@ -203,5 +203,11 @@
>
> extern bus_globals_t bus_globals;
>
> +#define MAX_NUM_PKEY 10
> +typedef struct
> +{
> + uint16_t pKey[MAX_NUM_PKEY];
> + uint8_t num_pKeys;
> +}pkey_array;
>
> #endif /* !defined _BUS_DRIVER_H_ */
> Index: core/bus/kernel/bus_port_mgr.c
> ===================================================================
> --- core/bus/kernel/bus_port_mgr.c (revision 1026)
> +++ core/bus/kernel/bus_port_mgr.c (working copy)
> @@ -54,6 +54,8 @@
> DEFINE_GUID(GUID_BUS_TYPE_IBA,
> 0x5a9649f4, 0x101, 0x4a7c, 0x83, 0x37, 0x79, 0x6c, 0x48, 0x8, 0x2d,
> 0xa2);
>
> +/* IB pKey array from registry */
> +extern pkey_array g_ib_pkey_array ;
>
> /*
> * Device extension for IPoIB port PDOs.
> @@ -62,7 +64,7 @@
> {
> bus_pdo_ext_t pdo;
>
> - net64_t
> port_guid;
> + port_guid_pkey port_guid;
> uint32_t n_port;
>
> /* Number of references on the upper interface. */
> @@ -512,8 +514,7 @@
> p_pdo_ext = PARENT_STRUCT( p_list_item,
> bus_pdo_ext_t, list_item );
> p_port_ext = (bus_port_ext_t*)p_pdo_ext;
>
> - if( p_pdo_ext->b_present && p_pdo_ext-
> >b_hibernating &&
> - (p_port_ext->port_guid ==
p_pnp_rec-
> >p_port_attr->port_guid) )
> + if( p_pdo_ext->b_present && p_pdo_ext-
> >b_hibernating && (p_port_ext->port_guid.port_guid == p_pnp_rec-
> >p_port_attr->port_guid))
> {
> n_devs++;
> break;
> @@ -523,7 +524,7 @@
> ("Skipped PDO for %s: PDO %p, ext
%p,
> present %d, missing %d, hibernating %d, port_guid %I64x.\n",
> p_pdo_ext->cl_ext.vfptr_pnp_po-
> >identity, p_pdo_ext->cl_ext.p_self_do,
> p_pdo_ext, p_pdo_ext->b_present,
> p_pdo_ext->b_reported_missing,
> - p_pdo_ext->b_hibernating,
p_port_ext-
> >port_guid ) );
> + p_pdo_ext->b_hibernating,
p_port_ext-
> >port_guid.port_guid ) );
> }
>
> if (n_devs)
> @@ -546,7 +547,7 @@
> ("Found PDO for %s:
PDO %p,
> ext %p, present %d, missing %d, hibernating %d, port_guid %I64x.\n",
> p_pdo_ext-
> >cl_ext.vfptr_pnp_po->identity, p_pdo_ext->cl_ext.p_self_do,
> p_pdo_ext, p_pdo_ext-
> >b_present, p_pdo_ext->b_reported_missing,
> -
p_pdo_ext->b_hibernating,
> p_port_ext->port_guid ) );
> +
p_pdo_ext->b_hibernating,
> p_port_ext->port_guid.port_guid ) );
> }
> }
> else
> @@ -567,9 +568,9 @@
> IN
> ib_pnp_port_rec_t* p_pnp_rec )
> {
> NTSTATUS status;
> - DEVICE_OBJECT *p_pdo;
> bus_port_ext_t *p_port_ext;
> -
> + DEVICE_OBJECT *p_pdo[MAX_NUM_PKEY + 1];
> + uint8_t num_pdo;
> BUS_ENTER( BUS_DBG_PNP );
>
> if( !bus_globals.b_report_port_nic )
> @@ -588,11 +589,15 @@
> return status;
> }
>
> + p_port_ext = NULL;
> +
> + for (num_pdo = 0; num_pdo < g_ib_pkey_array.num_pKeys + 1;
> num_pdo++)
> + {
> /* Create the PDO for the new port device. */
> status = IoCreateDevice( bus_globals.p_driver_obj,
> sizeof(bus_port_ext_t),
> NULL, FILE_DEVICE_CONTROLLER,
> FILE_DEVICE_SECURE_OPEN |
> FILE_AUTOGENERATED_DEVICE_NAME,
> - FALSE, &p_pdo );
> + FALSE, &p_pdo[num_pdo] );
> if( !NT_SUCCESS( status ) )
> {
> BUS_TRACE_EXIT( BUS_DBG_ERROR,
> @@ -601,13 +606,13 @@
> }
>
> /* Initialize the device extension. */
> - cl_init_pnp_po_ext( p_pdo, NULL, p_pdo,
bus_globals.dbg_lvl,
> + cl_init_pnp_po_ext( p_pdo[num_pdo], NULL, p_pdo[num_pdo],
> bus_globals.dbg_lvl,
> &vfptr_port_pnp, &vfptr_port_query_txt );
>
> /* Set the DO_BUS_ENUMERATED_DEVICE flag to mark it as a
PDO.
> */
> - p_pdo->Flags |= DO_BUS_ENUMERATED_DEVICE;
> + p_pdo[num_pdo]->Flags |= DO_BUS_ENUMERATED_DEVICE;
>
> - p_port_ext = p_pdo->DeviceExtension;
> + p_port_ext = p_pdo[num_pdo]->DeviceExtension;
> p_port_ext->pdo.dev_po_state.DeviceState = PowerDeviceD0;
> p_port_ext->pdo.p_parent_ext = bus_globals.p_bus_ext;
> p_port_ext->pdo.b_present = TRUE;
> @@ -615,22 +620,33 @@
> p_port_ext->pdo.b_hibernating = FALSE;
> p_port_ext->pdo.p_po_work_item = NULL;
> BUS_TRACE( BUS_DBG_PNP, ("Created device for %s: PDO
%p,ext %p,
> present %d, missing %d .\n",
> - p_port_ext->pdo.cl_ext.vfptr_pnp_po->identity,
> p_pdo, p_port_ext, p_port_ext->pdo.b_present,
> + p_port_ext->pdo.cl_ext.vfptr_pnp_po->identity,
> p_pdo[num_pdo], p_port_ext, p_port_ext->pdo.b_present,
> p_port_ext->pdo.b_reported_missing ) );
>
> /* Cache the CA GUID. */
> p_port_ext->pdo.ca_guid = p_pnp_rec->p_ca_attr->ca_guid;
>
> /* Take a reference on the parent HCA. */
> - p_port_ext->pdo.h_ca = acquire_ca( p_pnp_rec->p_ca_attr-
> >ca_guid );
> + if(num_pdo > 0)
> + {
> + p_port_ext->pdo.h_ca =
((bus_port_ext_t*)p_pdo[0]-
> >DeviceExtension)->pdo.h_ca;
> + }
> + else
> + p_port_ext->pdo.h_ca = acquire_ca( p_pnp_rec-
> >p_ca_attr->ca_guid );
> if( !p_port_ext->pdo.h_ca )
> {
> BUS_TRACE( BUS_DBG_PNP, ("Deleted device:
> PDO %p\n", p_pdo ));
> - IoDeleteDevice( p_pdo );
> + IoDeleteDevice( p_pdo[num_pdo] );
> BUS_TRACE_EXIT( BUS_DBG_ERROR, ("acquire_ca
failed
> to find CA.\n") );
> return IB_INVALID_GUID;
> }
> - p_port_ext->port_guid = p_pnp_rec->p_port_attr->port_guid;
> + p_port_ext->port_guid.port_guid = p_pnp_rec->p_port_attr-
> >port_guid;
> + p_port_ext->port_guid.pkey = IB_DEFAULT_PKEY;
> +
> + if(num_pdo > 0)
> + {
> + p_port_ext->port_guid.pkey =
> g_ib_pkey_array.pKey[num_pdo -1];
> + }
> p_port_ext->n_port = p_pnp_rec->p_port_attr->port_num;
>
> /* Store the device extension in the port vector for
future
> queries. */
> @@ -643,8 +659,11 @@
> * Set the context of the PNP event. The context is
passed in
> for future
> * events on the same port.
> */
> - p_pnp_rec->pnp_rec.context = p_port_ext;
> + if(num_pdo == 0)
> + p_pnp_rec->pnp_rec.context = p_port_ext;
>
> + }
> +
> /* Tell the PnP Manager to rescan for the HCA's bus
relations.
> */
> IoInvalidateDeviceRelations(
>
p_port_ext->pdo.h_ca->obj.p_ci_ca->verbs.p_hca_dev,
> BusRelations );
> Index: core/bus/kernel/ib_bus.inf
> ===================================================================
> --- core/bus/kernel/ib_bus.inf (revision 1026)
> +++ core/bus/kernel/ib_bus.inf (working copy)
> @@ -183,6 +183,7 @@
> HKR,"Parameters","IocQueryTimeout",%REG_DWORD_NO_CLOBBER%,250
> HKR,"Parameters","IocQueryRetries",%REG_DWORD_NO_CLOBBER%,4
> HKR,"Parameters","IocPollInterval",%REG_DWORD_NO_CLOBBER%,30000
> +HKR,"Parameters","IbPkey",%REG_SZ%,"0xBABA"
>
> [Iou.ParamsReg]
> HKR,"Parameters","DebugLevel",%REG_DWORD%,2
> Index: ulp/ipoib/kernel/ipoib_adapter.c
> ===================================================================
> --- ulp/ipoib/kernel/ipoib_adapter.c (revision 1026)
> +++ ulp/ipoib/kernel/ipoib_adapter.c (working copy)
> @@ -171,7 +171,7 @@
>
> IPOIB_PRINT( TRACE_LEVEL_INFORMATION, IPOIB_DBG_INIT,
> ("Port %016I64x (CA %016I64x port
%d)
> initializing\n",
> - p_adapter->guids.port_guid,
p_adapter-
> >guids.ca_guid,
> +
p_adapter->guids.port_guid.port_guid,
> p_adapter->guids.ca_guid,
> p_adapter->guids.port_num) );
>
> cl_status = cl_obj_init( &p_adapter->obj, CL_DESTROY_SYNC,
> @@ -343,7 +343,7 @@
>
> /* Validate the port GUID and generate the MAC address. */
> status =
> - ipoib_mac_from_guid(
p_adapter->guids.port_guid,
> &p_adapter->mac );
> + ipoib_mac_from_guid( p_adapter-
> >guids.port_guid.port_guid, &p_adapter->mac );
> if( status != IB_SUCCESS )
> {
> IPOIB_PRINT_EXIT( TRACE_LEVEL_ERROR,
> IPOIB_DBG_ERROR,
> @@ -530,7 +530,7 @@
> case IB_PNP_PORT_ADD:
> CL_ASSERT( !p_pnp_rec->context );
> /* Only process our port GUID. */
> - if( p_pnp_rec->guid != p_adapter-
> >guids.port_guid )
> + if( p_pnp_rec->guid != p_adapter-
> >guids.port_guid.port_guid )
> {
> status = IB_NOT_DONE;
> break;
> Index: ulp/ipoib/kernel/ipoib_driver.c
> ===================================================================
> --- ulp/ipoib/kernel/ipoib_driver.c (revision 1026)
> +++ ulp/ipoib/kernel/ipoib_driver.c (working copy)
> @@ -2336,10 +2336,10 @@
> cl_memset( &ib_service, 0, sizeof(ib_service) );
>
> /* BUGBUG Only register local subnet GID prefix for now */
> - ib_gid_set_default( &port_gid, p_adapter->guids.port_guid
);
> + ib_gid_set_default( &port_gid, p_adapter-
> >guids.port_guid.port_guid );
> ib_service.svc_rec.service_gid = port_gid;
>
> - ib_service.svc_rec.service_pkey =
IB_DEFAULT_PKEY;
> + ib_service.svc_rec.service_pkey = p_adapter-
> >guids.port_guid.pkey;
> ib_service.svc_rec.service_lease =
> IB_INFINITE_SERVICE_LEASE;
>
> /* Must cast here because the service name is an array of
> unsigned chars but
> @@ -2347,7 +2347,7 @@
> strcpy( (char *)ib_service.svc_rec.service_name, ATS_NAME
);
>
> /* IP Address in question will be put in below */
> - ib_service.port_guid = p_adapter-
> >guids.port_guid;
> + ib_service.port_guid = p_adapter-
> >guids.port_guid.port_guid;
> ib_service.timeout_ms = p_adapter-
> >params.sa_timeout;
> ib_service.retry_cnt = p_adapter-
> >params.sa_retry_cnt;
>
> Index: ulp/ipoib/kernel/ipoib_endpoint.c
> ===================================================================
> --- ulp/ipoib/kernel/ipoib_endpoint.c (revision 1026)
> +++ ulp/ipoib/kernel/ipoib_endpoint.c (working copy)
> @@ -312,13 +312,13 @@
>
> cl_memclr( &path, sizeof(ib_path_rec_t) );
> path.dgid = p_endpt->dgid;
> - ib_gid_set_default( &path.sgid, p_port->p_adapter-
> >guids.port_guid );
> + ib_gid_set_default( &path.sgid, p_port->p_adapter-
> >guids.port_guid.port_guid );
> path.num_path = 0x1;
>
> cl_memclr( &query, sizeof(ib_query_req_t) );
> query.query_type = IB_QUERY_USER_DEFINED;
> query.p_query_input = &info;
> - query.port_guid = p_port->p_adapter->guids.port_guid;
> + query.port_guid =
p_port->p_adapter->guids.port_guid.port_guid;
> query.timeout_ms = p_port->p_adapter->params.sa_timeout;
> query.retry_cnt = p_port->p_adapter->params.sa_retry_cnt;
>
> Index: ulp/ipoib/kernel/ipoib_ibat.c
> ===================================================================
> --- ulp/ipoib/kernel/ipoib_ibat.c (revision 1026)
> +++ ulp/ipoib/kernel/ipoib_ibat.c (working copy)
> @@ -140,7 +140,7 @@
> {
> pAdapter = CONTAINING_RECORD( pItem,
> ipoib_adapter_t, entry );
> pOut->Ports[pOut->NumPorts].CaGuid = pAdapter-
> >guids.ca_guid;
> - pOut->Ports[pOut->NumPorts].PortGuid =
pAdapter-
> >guids.port_guid;
> + pOut->Ports[pOut->NumPorts].PortGuid =
pAdapter-
> >guids.port_guid.port_guid;
> pOut->Ports[pOut->NumPorts].PortNum =
pAdapter-
> >guids.port_num;
> pOut->NumPorts++;
>
> @@ -210,7 +210,7 @@
> pItem = cl_qlist_next( pItem ) )
> {
> pAdapter = CONTAINING_RECORD( pItem,
> ipoib_adapter_t, entry );
> - if( PortGuid && pAdapter->guids.port_guid !=
> PortGuid )
> + if( PortGuid && pAdapter-
> >guids.port_guid.port_guid != PortGuid )
> continue;
>
> cl_obj_lock( &pAdapter->obj );
> @@ -300,7 +300,7 @@
> pItem = cl_qlist_next( pItem ) )
> {
> pAdapter = CONTAINING_RECORD( pItem,
> ipoib_adapter_t, entry );
> - if( pIn->PortGuid != pAdapter->guids.port_guid
)
> + if( pIn->PortGuid != pAdapter-
> >guids.port_guid.port_guid )
> continue;
>
> /* Found the port - lookup the MAC. */
> Index: ulp/ipoib/kernel/ipoib_port.c
> ===================================================================
> --- ulp/ipoib/kernel/ipoib_port.c (revision 1026)
> +++ ulp/ipoib/kernel/ipoib_port.c (working copy)
> @@ -793,6 +793,8 @@
> uint64_t vaddr;
> net32_t rkey;
> ib_qp_attr_t qp_attr;
> + ib_ca_attr_t *ca_attr;
> + uint32_t ca_size;
>
> IPOIB_ENTER( IPOIB_DBG_INIT );
>
> @@ -810,6 +812,44 @@
> return status;
> }
>
> + /* Query the CA for Pkey table */
> + status =
p_port->p_adapter->p_ifc->query_ca(p_port->ib_mgr.h_ca,
> NULL, &ca_size);
> + if(status != IB_INSUFFICIENT_MEMORY)
> + {
> + IPOIB_PRINT_EXIT( TRACE_LEVEL_ERROR,
> IPOIB_DBG_ERROR,
> + ("ib_query_ca failed\n"));
> + return status;
> + }
> +
> + ca_attr = (ib_ca_attr_t*)cl_zalloc(ca_size);
> + if (!ca_attr)
> + {
> + IPOIB_PRINT_EXIT( TRACE_LEVEL_ERROR,
> IPOIB_DBG_ERROR,
> + ("cl_zalloc can't
> allocate %d\n",ca_size));
> + return IB_INSUFFICIENT_RESOURCES;
> + }
> +
> + status =
p_port->p_adapter->p_ifc->query_ca(p_port->ib_mgr.h_ca,
> ca_attr,&ca_size);
> + if( status != IB_SUCCESS )
> + {
> + IPOIB_PRINT_EXIT( TRACE_LEVEL_ERROR,
> IPOIB_DBG_ERROR,
> + ("ib_query_ca returned %s\n",
> + p_port->p_adapter->p_ifc-
> >get_err_str( status )) );
> + return status;
> + }
> + if( ca_attr->p_port_attr->link_state == IB_LINK_ACTIVE)
> + {
> + uint16_t index;
> + CL_ASSERT(ca_attr->p_port_attr->p_pkey_table[0]
==
> IB_DEFAULT_PKEY);
> + for(index = 0; index < ca_attr->p_port_attr-
> >num_pkeys; index++)
> + {
> + if(p_port->p_adapter-
> >guids.port_guid.pkey == ca_attr->p_port_attr->p_pkey_table[index])
> + break;
> + }
> + p_port->pkey_index = index;
> + }
> + cl_free(ca_attr);
> +
> /* Allocate the PD. */
> status = p_port->p_adapter->p_ifc->alloc_pd(
> p_port->ib_mgr.h_ca, IB_PDT_UD, p_port,
&p_port-
> >ib_mgr.h_pd );
> @@ -3309,7 +3349,7 @@
>
> CL_ASSERT( p_cid[1] == 21 );
> p_cid[23]= DHCP_OPT_END;
> - ib_gid_set_default( &gid, p_port->p_adapter-
> >guids.port_guid );
> + ib_gid_set_default( &gid, p_port->p_adapter-
> >guids.port_guid.port_guid );
> cl_memcpy( &p_cid[7], &gid, sizeof(ib_gid_t)
);
> cl_memcpy( &p_cid[3], &p_port->ib_mgr.qpn,
> sizeof(p_port->ib_mgr.qpn) );
> p_ib_dhcp->htype = DHCP_HW_TYPE_IB;
> @@ -3409,7 +3449,7 @@
> p_ib_arp->op = p_arp->op;
> p_ib_arp->src_hw.flags_qpn = p_port->ib_mgr.qpn;
> ib_gid_set_default( &p_ib_arp->src_hw.gid,
> - p_port->p_adapter->guids.port_guid );
> + p_port->p_adapter->guids.port_guid.port_guid );
> p_ib_arp->src_ip = p_arp->src_ip;
> if( cl_memcmp( &p_arp->dst_hw, &null_hw,
> sizeof(mac_addr_t) ) )
> {
> @@ -3584,7 +3624,7 @@
> p_desc->wr.dgrm.ud.remote_qp = p_desc->p_endpt->qpn;
> p_desc->wr.dgrm.ud.remote_qkey = p_port-
> >ib_mgr.bcast_rec.qkey;
> p_desc->wr.dgrm.ud.h_av = p_desc->p_endpt->h_av;
> - p_desc->wr.dgrm.ud.pkey_index = 0;
> + p_desc->wr.dgrm.ud.pkey_index = p_port->pkey_index;
> p_desc->wr.dgrm.ud.rsvd = NULL;
>
> /* Store context in our reserved area of the packet. */
> @@ -4683,7 +4723,7 @@
> cl_memclr( &query, sizeof(ib_query_req_t) );
> query.query_type = IB_QUERY_USER_DEFINED;
> query.p_query_input = &info;
> - query.port_guid = p_port->p_adapter->guids.port_guid;
> + query.port_guid =
p_port->p_adapter->guids.port_guid.port_guid;
> query.timeout_ms = p_port->p_adapter->params.sa_timeout;
> query.retry_cnt = p_port->p_adapter->params.sa_retry_cnt;
> query.query_context = p_port;
> @@ -4721,7 +4761,7 @@
>
> IPOIB_ENTER( IPOIB_DBG_INIT );
>
> - ib_gid_set_default( &gid, p_port->p_adapter-
> >guids.port_guid );
> + ib_gid_set_default( &gid, p_port->p_adapter-
> >guids.port_guid.port_guid );
> p_endpt = ipoib_endpt_create(
> &gid, p_port_info->base_lid,
p_port->ib_mgr.qpn );
> if( !p_endpt )
> @@ -4894,10 +4934,13 @@
> cl_memclr( &member_rec, sizeof(ib_member_rec_t) );
> member_rec.mgid = bcast_mgid_template;
>
> + member_rec.mgid.raw[4] = (uint8_t)(p_port->p_adapter-
> >guids.port_guid.pkey >> 8);
> + member_rec.mgid.raw[5] = (uint8_t) p_port->p_adapter-
> >guids.port_guid.pkey;
> + member_rec.pkey = p_port->p_adapter->guids.port_guid.pkey;
> cl_memclr( &query, sizeof(ib_query_req_t) );
> query.query_type = IB_QUERY_USER_DEFINED;
> query.p_query_input = &info;
> - query.port_guid = p_port->p_adapter->guids.port_guid;
> + query.port_guid =
p_port->p_adapter->guids.port_guid.port_guid;
> query.timeout_ms = p_port->p_adapter->params.sa_timeout;
> query.retry_cnt = p_port->p_adapter->params.sa_retry_cnt;
> query.query_context = p_port;
> @@ -5031,13 +5074,13 @@
> /* We specify our port GID for the join operation. */
> mcast_req.member_rec.port_gid.unicast.prefix =
> IB_DEFAULT_SUBNET_PREFIX;
> mcast_req.member_rec.port_gid.unicast.interface_id =
> - p_port->p_adapter->guids.port_guid;
> + p_port->p_adapter->guids.port_guid.port_guid;
>
> mcast_req.mcast_context = p_port;
> mcast_req.pfn_mcast_cb = __bcast_cb;
> mcast_req.timeout_ms =
p_port->p_adapter->params.sa_timeout;
> mcast_req.retry_cnt =
p_port->p_adapter->params.sa_retry_cnt;
> - mcast_req.port_guid = p_port->p_adapter->guids.port_guid;
> + mcast_req.port_guid = p_port->p_adapter-
> >guids.port_guid.port_guid;
> mcast_req.pkey_index = 0;
>
> if( ib_member_get_state( mcast_req.member_rec.scope_state
) !=
> @@ -5086,7 +5129,9 @@
> */
> mcast_req.member_rec.mgid = bcast_mgid_template;
> ib_gid_set_default( &mcast_req.member_rec.port_gid,
> - p_port->p_adapter->guids.port_guid );
> + p_port->p_adapter->guids.port_guid.port_guid );
> + mcast_req.member_rec.mgid.raw[4] = (uint8_t) (p_port-
> >p_adapter->guids.port_guid.pkey >> 8);
> + mcast_req.member_rec.mgid.raw[5] = (uint8_t)
p_port->p_adapter-
> >guids.port_guid.pkey;
> /*
> * IPOIB spec requires that the QKEY have the MSb set so
that
> the QKEY
> * from the QP is used rather than the QKEY in the send
WR.
> @@ -5096,7 +5141,7 @@
> mcast_req.member_rec.mtu =
> (IB_PATH_SELECTOR_EXACTLY << 6) |
IB_MTU_LEN_2048;
>
> - mcast_req.member_rec.pkey = IB_DEFAULT_PKEY;
> + mcast_req.member_rec.pkey = p_port->p_adapter-
> >guids.port_guid.pkey;
>
> mcast_req.member_rec.sl_flow_hop =
> ib_member_set_sl_flow_hop( 0, 0, 0 );
> mcast_req.member_rec.scope_state =
> @@ -5106,8 +5151,8 @@
> mcast_req.pfn_mcast_cb = __bcast_cb;
> mcast_req.timeout_ms =
p_port->p_adapter->params.sa_timeout;
> mcast_req.retry_cnt =
p_port->p_adapter->params.sa_retry_cnt;
> - mcast_req.port_guid = p_port->p_adapter->guids.port_guid;
> - mcast_req.pkey_index = 0;
> + mcast_req.port_guid = p_port->p_adapter-
> >guids.port_guid.port_guid;
> + mcast_req.pkey_index = p_port->pkey_index;
>
> /* reference the object for the multicast join request. */
> ipoib_port_ref( p_port, ref_join_bcast );
> @@ -5362,9 +5407,9 @@
> }
>
> /* Move the QP to RTS. */
> - dgrm_info.port_guid = p_port->p_adapter->guids.port_guid;
> + dgrm_info.port_guid = p_port->p_adapter-
> >guids.port_guid.port_guid;
> dgrm_info.qkey = p_port->ib_mgr.bcast_rec.qkey;
> - dgrm_info.pkey_index = 0;
> + dgrm_info.pkey_index = p_port->pkey_index;
> status = p_port->p_adapter->p_ifc->init_dgrm_svc( p_port-
> >ib_mgr.h_qp, &dgrm_info );
> if( status != IB_SUCCESS )
> {
> @@ -5464,9 +5509,9 @@
> mcast_req.pfn_mcast_cb = __mcast_cb;
> mcast_req.timeout_ms =
p_port->p_adapter->params.sa_timeout;
> mcast_req.retry_cnt =
p_port->p_adapter->params.sa_retry_cnt;
> - mcast_req.port_guid = p_port->p_adapter->guids.port_guid;
> - mcast_req.pkey_index = 0;
> -
> + mcast_req.port_guid = p_port->p_adapter-
> >guids.port_guid.port_guid;
> + mcast_req.pkey_index = p_port->pkey_index;
> + mcast_req.member_rec.pkey = p_port->p_adapter-
> >guids.port_guid.pkey;
> /*
> * Create the endpoint and insert it in the port. Since
we
> don't wait for
> * the mcast SA operations to complete before returning
from
> the multicast
> Index: ulp/ipoib/kernel/ipoib_port.h
> ===================================================================
> --- ulp/ipoib/kernel/ipoib_port.h (revision 1026)
> +++ ulp/ipoib/kernel/ipoib_port.h (working copy)
> @@ -506,6 +506,7 @@
> atomic32_t
> endpt_rdr;
>
> atomic32_t
hdr_idx;
> + uint16_t pkey_index;
> ipoib_hdr_t
hdr[1];
> /* Must be last! */
>
> } ipoib_port_t;
>
> Slava Strebkov
> SW Engineer
> Voltaire
> 099718750
>
More information about the ofw
mailing list