[PATCH] opensm: Special Case the IPv6 Solicited Node Multicast address to use a single Mcast (WAS: Re: [ofa-general] IPoIB, OFED 1.2.5, and multicast groups.)
Ira Weiny
weiny2 at llnl.gov
Sat Jan 12 00:01:17 PST 2008
And to further answer my question...[*]
This seems to fix the problem for us, however I know that it could be better.
For example it only takes care of partition 0xFFFF, and I think Jason's idea of
having say 16 Mcast Groups and some hash of these into them would be nice. But
is this on the right track? Am I missing some other place in the code?
Thanks,
Ira
[*] Again I apologize for the spam but we were in a bit of a panic as we only
have the big system for the weekend and IB was not part of the test... ;-)
>From 35e35a9534bd49147886ac93ab1601acadcdbe26 Mon Sep 17 00:00:00 2001
From: Ira K. Weiny <weiny2 at llnl.gov>
Date: Fri, 11 Jan 2008 22:58:19 -0800
Subject: [PATCH] Special Case the IPv6 Solicited Node Multicast address to use a single Mcast
Group.
Signed-off-by: root <weiny2 at llnl.gov>
---
opensm/opensm/osm_sa_mcmember_record.c | 30 +++++++++++++++++++++++++++++-
opensm/opensm/osm_sa_path_record.c | 31 ++++++++++++++++++++++++++++++-
2 files changed, 59 insertions(+), 2 deletions(-)
diff --git a/opensm/opensm/osm_sa_mcmember_record.c b/opensm/opensm/osm_sa_mcmember_record.c
index 8eb97ad..6bcc124 100644
--- a/opensm/opensm/osm_sa_mcmember_record.c
+++ b/opensm/opensm/osm_sa_mcmember_record.c
@@ -124,9 +124,37 @@ __search_mgrp_by_mgid(IN cl_map_item_t * const p_map_item, IN void *context)
/* compare entire MGID so different scope will not sneak in for
the same MGID */
if (memcmp(&p_mgrp->mcmember_rec.mgid,
- &p_recvd_mcmember_rec->mgid, sizeof(ib_gid_t)))
+ &p_recvd_mcmember_rec->mgid, sizeof(ib_gid_t))) {
+
+ /* Special Case IPV6 Multicast Loopback addresses */
+ /* 0xff12601bffff0000 : 0x00000001ffXXXXXX */
+#define SPEC_PREFIX (0xff12601bffff0000)
+#define INT_ID_MASK (0x00000001ff000000)
+ uint64_t g_prefix = cl_ntoh64(p_mgrp->mcmember_rec.mgid.unicast.prefix);
+ uint64_t g_interface_id = cl_ntoh64(p_mgrp->mcmember_rec.mgid.unicast.interface_id);
+ uint64_t rcv_prefix = cl_ntoh64(p_recvd_mcmember_rec->mgid.unicast.prefix);
+ uint64_t rcv_interface_id = cl_ntoh64(p_recvd_mcmember_rec->mgid.unicast.interface_id);
+
+ if (rcv_prefix == SPEC_PREFIX
+ &&
+ (rcv_interface_id & INT_ID_MASK) == INT_ID_MASK) {
+
+ if ((g_prefix == rcv_prefix)
+ &&
+ (g_interface_id & INT_ID_MASK) ==
+ (rcv_interface_id & INT_ID_MASK)
+ ) {
+ osm_log(sa->p_log, OSM_LOG_INFO,
+ "Special Case Mcast Join for MGID "
+ " MGID 0x%016"PRIx64" : 0x%016"PRIx64"\n",
+ rcv_prefix, rcv_interface_id);
+ goto match;
+ }
+ }
return;
+ }
+match:
if (p_ctxt->p_mgrp) {
osm_log(sa->p_log, OSM_LOG_ERROR,
"__search_mgrp_by_mgid: ERR 1B03: "
diff --git a/opensm/opensm/osm_sa_path_record.c b/opensm/opensm/osm_sa_path_record.c
index 749a936..469773a 100644
--- a/opensm/opensm/osm_sa_path_record.c
+++ b/opensm/opensm/osm_sa_path_record.c
@@ -1536,8 +1536,37 @@ __search_mgrp_by_mgid(IN cl_map_item_t * const p_map_item, IN void *context)
/* compare entire MGID so different scope will not sneak in for
the same MGID */
- if (memcmp(&p_mgrp->mcmember_rec.mgid, p_recvd_mgid, sizeof(ib_gid_t)))
+ if (memcmp(&p_mgrp->mcmember_rec.mgid, p_recvd_mgid, sizeof(ib_gid_t))) {
+
+ /* Special Case IPV6 Multicast Loopback addresses */
+ /* 0xff12601bffff0000 : 0x00000001ffXXXXXX */
+#define SPEC_PREFIX (0xff12601bffff0000)
+#define INT_ID_MASK (0x00000001ff000000)
+ uint64_t g_prefix = cl_ntoh64(p_mgrp->mcmember_rec.mgid.unicast.prefix);
+ uint64_t g_interface_id = cl_ntoh64(p_mgrp->mcmember_rec.mgid.unicast.interface_id);
+ uint64_t rcv_prefix = cl_ntoh64(p_recvd_mgid->unicast.prefix);
+ uint64_t rcv_interface_id = cl_ntoh64(p_recvd_mgid->unicast.interface_id);
+
+ if (rcv_prefix == SPEC_PREFIX
+ &&
+ (rcv_interface_id & INT_ID_MASK) == INT_ID_MASK) {
+
+ if ((g_prefix == rcv_prefix)
+ &&
+ (g_interface_id & INT_ID_MASK) ==
+ (rcv_interface_id & INT_ID_MASK)
+ ) {
+ osm_log(sa->p_log, OSM_LOG_INFO,
+ "Special Case Mcast Join for MGID "
+ " MGID 0x%016"PRIx64" : 0x%016"PRIx64"\n",
+ rcv_prefix, rcv_interface_id);
+ goto match;
+ }
+ }
return;
+ }
+
+match:
#if 0
for (i = 0;
--
1.5.1
On Fri, 11 Jan 2008 22:04:56 -0800
Ira Weiny <weiny2 at llnl.gov> wrote:
> Ok,
>
> I found my own answer. Sorry for the spam.
>
> http://lists.openfabrics.org/pipermail/general/2006-November/029617.html
>
> Sorry,
> Ira
>
>
> On Fri, 11 Jan 2008 19:36:57 -0800
> Ira Weiny <weiny2 at llnl.gov> wrote:
>
> > I don't really understand the innerworkings of IPoIB so forgive me if this is a
> > really stupid question but:
> >
> > Is it a bug that there is a Multicast group created for every node in our
> > clusters?
> >
> > If not a bug why is this done? We just tried to boot on a 1151 node cluster
> > and opensm is complaining there are not enough multicast groups.
> >
> > Jan 11 18:30:42 728984 [40C05960] -> __get_new_mlid: ERR 1B23: All available:1024 mlids are taken
> > Jan 11 18:30:42 729050 [40C05960] -> osm_mcmr_rcv_create_new_mgrp: ERR 1B19: __get_new_mlid failed
> > Jan 11 18:30:42 730647 [40401960] -> __get_new_mlid: ERR 1B23: All available:1024 mlids are taken
> > Jan 11 18:30:42 730691 [40401960] -> osm_mcmr_rcv_create_new_mgrp: ERR 1B19: __get_new_mlid failed
> >
> >
> > Here is the output from my small test cluster: (ibnodesinmcast uses saquery a
> > couple of times to print this nice report.)
> >
> >
> > 19:17:24 > whatsup
> > up: 9: wopr[0-7],wopri
> > down: 0:
> > root at wopri:/tftpboot/images
> > 19:25:03 > ibnodesinmcast -g
> > 0xC000 (0xff12401bffff0000 : 0x00000000ffffffff)
> > In 9: wopr[0-7],wopri
> > Out 0: 0
> > 0xC001 (0xff12401bffff0000 : 0x0000000000000001)
> > In 9: wopr[0-7],wopri
> > Out 0: 0
> > 0xC002 (0xff12601bffff0000 : 0x00000001ff2265ed)
> > In 1: wopr3
> > Out 8: wopr[0-2,4-7],wopri
> > 0xC003 (0xff12601bffff0000 : 0x0000000000000001)
> > In 9: wopr[0-7],wopri
> > Out 0: 0
> > 0xC004 (0xff12601bffff0000 : 0x00000001ff222729)
> > In 1: wopr4
> > Out 8: wopr[0-3,5-7],wopri
> > 0xC005 (0xff12601bffff0000 : 0x00000001ff219e65)
> > In 1: wopri
> > Out 8: wopr[0-7]
> > 0xC006 (0xff12601bffff0000 : 0x00000001ff00232d)
> > In 1: wopr6
> > Out 8: wopr[0-5,7],wopri
> > 0xC007 (0xff12601bffff0000 : 0x00000001ff002325)
> > In 1: wopr7
> > Out 8: wopr[0-6],wopri
> > 0xC008 (0xff12601bffff0000 : 0x00000001ff228d35)
> > In 1: wopr1
> > Out 8: wopr[0,2-7],wopri
> > 0xC009 (0xff12601bffff0000 : 0x00000001ff2227f1)
> > In 1: wopr2
> > Out 8: wopr[0-1,3-7],wopri
> > 0xC00A (0xff12601bffff0000 : 0x00000001ff219ef1)
> > In 1: wopr0
> > Out 8: wopr[1-7],wopri
> > 0xC00B (0xff12601bffff0000 : 0x00000001ff0021e9)
> > In 1: wopr5
> > Out 8: wopr[0-4,6-7],wopri
> >
> >
> > Each of these MGIDS of the prefix (0xff12601bffff0000) have just one node in
> > them and represent an ipv6 address. Could you turn off ipv6 with the latest
> > IPoIB?
> >
> > In a bind,
> > Ira
> > _______________________________________________
> > general mailing list
> > general at lists.openfabrics.org
> > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
> >
> > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-Special-Case-the-IPv6-Solicited-Node-Multicast-addre.patch
Type: application/octet-stream
Size: 3637 bytes
Desc: not available
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20080112/9472fd05/attachment.obj>
More information about the general
mailing list