Re: [TCLUG:15591] MrNet sucking yet again

[Mike Hicks <hick0088@tc.umn.edu>]

Scott Dier - dieman wrote:
> 
> plah-eeeze.  Lets wait until the post-op comes off of this one before we
> say someone at mr.net fudged hardcore.

Okay, here's a post-op:

--- BEGIN ---

Date: Thu, 6 Apr 2000 02:46:57 -0500 (CDT)
From: Dan Boehlke <dboehlke@MR.Net>
To: tech-announce@MR.Net
Subject: NETWORK-STATUS: Onvoy core routing problems and congestion
issues
        addressed.

Greetings,

We have found and resolved several major issues with Onvoy's core
network.  These all contributed to what we consider to be a catastrophic
failure of our network infrastructure.

Thanks to help from many of our customers we have tracked down many of
the
routing problems, including routing loops within the core
infrastructure.
We have closely examined our BGP routing configurations and believe that
we have resolved any architectural issues with its design.  We are still
tracing down some isolated issues with individual customers.  Many of
these have been long standing issues and complexities, that were brought
to light because of the configuration changes this last weekend that
were
intended to increase the performance of the network by retiring the
Cisco
7507 routers from core service, and allowing the Juniper routers to
fully
replace them.

We discovered that traffic between the Juniper routers was bursting in
such as way as to overwhelm the output buffers on the OC3 ports which
run
the inter-hub-room OC3 lines on our ATM switches.  The Juniper routers
have OC12 ports and while they were not sending more than an OC3's worth
of data, they were doing so in such a short interval to cause cell loss
in
the ATM part of the core network.  We have applied traffic shaping to
each
virtual circuit to keep its traffic within the bandwidth allocated in
the
network.  This resolved the bursting issue and restored our confidence
in
the ATM network core.  The cell loss issue in the core has been building
over the last few weeks as traffic increased.  Shaping on the edge is
one
of the best ways to deal with the management of Internet traffic in an
ATM
network.  We also moved a major customer from a port on the ATM switch
at
MSC to an ATM router port so that we could be sure that their traffic
was
not part of the problem, it turns out that their connection did not
contribute to the problem.

Once the traffic shaping was in place things got a little better,
however
we were still suffering cell loss at MSC.  Investigation revealed that
the
OC12 port on the Juniper router needed to be replaced.  Upon replacement
the error cleared and so did all the packet loss and latency issues with
the MSC routers.

We apologize for the length of this outage.  We called in our best
people
to work on it.  They have spent many hours under great pressure to get
this
resolved.  It was not been easy to get to the bottom of the issues,
the combination of issues hid the true cause of the problems so that
only
worked to find the problems.

Thanks to our customers who supplied us with crucial information on
their
view of the problems.  Without this we would not be as sure as we are
that
the issues are resolved.  I was impressed by the patients shown by our
contacts and their willingness to help.

We are still dealing with some isolated routing issues.  Many of these
are
due to minor configuration issues.  We are still working to find them,
however do not hesitate to report any problems that you may still see
with
the network.  Thanks.

        Onvvoy Technical Support
         trouble@onvoy.com
         612-362-5890

Please contact your customer care representative or sales person with
any
non-technical issues.  Thanks.

--
Dan Boehlke, Senior Network Engineer                              Onvoy
Internet:  dboehlke@onvoy.com                  Formerly MEANS and MRNet
Phone:  612-362-5890                  2829 SE University Ave. Suite 200
WWW: http://www.onvoy.com/~dboehlke/             Minneapolis, MN  55414

--- END ---

-- 
 _  _  _  _ _  ___    _ _  _  ___ _ _  __   Zen T-Shirt: 
/ \/ \(_)| ' // ._\  / - \(_)/ ./| ' /(__   Enlightenment Available  
\_||_/|_||_|_\\___/  \_-_/|_|\__\|_|_\ __)  -- Enquire Within. 
 [ Mike Hicks | http://umn.edu/~hick0088/ | mailto:hick0088@umn.edu ]