On 1/27/06, Torleiv Ringer <tringer at consumption.net> wrote:
> Hello,
>
> We have a situation at work where someone misconfigured a switch and it
> caused some major failures that were hard to trace.

Shoot them.

> The point of contention is that someone thought they were doing the
> right thing and jumped in where they were not asked to by installing a
> new 3560.

Shoot them.

>   *) This person set the switch up "hot" (on the network)
>   *) They used two uplink ports, intending on ganging them togther
>   *) They did not properly set the ports into a channel-group

Also shoot whoever let this idiot anywhere near the switches.

> So here is my question:
>
> I am being pushed by the higher-ups to come up with a software solution
> for this problem, which I feel is a process problem.

Correct.  When you're getting to the point of routing proprietary
protocols, anyone setting up switches should realize that there's some
config that needs to be done.  I'm not sure that software ANYTHING
would help you here.

> Should I bow to the pressure and force our vendor to "fix" their
> software to be able to function in an abnormal network setup?

At least investigate it, but vendors (and their supplied protocols)
can be a bear to work with.  I doubt you'll get anywhere with the
vendor.  Based on your use of the word "custom", I'm guessing that
there's only one vendor who provides this particular software that
serves a specific function so dropping them isn't an option.    Isn't
vendor locking wonderful? :-P

> Or
> Should I instill a process such that this would never happen again and
> put the lock-down on people who configure devices in/on this network?

That's part of it.

> This involves disallowing the people who are supposed to be the
> networking specialists from configuring the "custom" network.

Or train, train, and train some more.  Does the idiot in question
understand that there is a proper way to configure your custom
enviroment?  What about the other admins who probably didn't know
either?

> Is there a Cisco configuration that can be used to disallow "unknown"
> routers on the VLAN? This seems unlikely to me.

Not that I know of.  IF your custom protocol supports it, you might be
able to prevent such a thing using 802.1p priority queueing.  If a new
switch comes online, it won't be part of the 802.1p scheme which MIGHT
keep nasty packets from propogating (or, at the very least, shoving
them to the lowest priority).  Of course an ACL permitting only
specific MACs could be used, but that might be more work than its
worth when you're talking 200+ devices.

> It's one or the other at this point, as we have lost a lot of
> credibility in this situation, and we must move forward with
> implementation. This is the second time now that a misconfigured switch
> has been setup hot on the "custom" network.

Playing devil's advocate: first time, shame on them, second time,
shame on you.  What happened to the idiot in the first instance?