Spanning Tree Loop Problem (I think)

Discussion in 'Cisco' started by Michael, Jan 11, 2005.

  1. Michael

    Michael Guest

    Hey guys...

    We have a strange issue in of our datacenters and I was hoping you guys
    could help. We have five racks, each with a 3750 in each cabinet. The
    basic wiring is like this:

    http://www.twentyten.org/images/switches.gif

    As you see, we have the Internet link from our co-lo facility going into
    switch 1, and the other cabinets are just chained off of each other. Over
    the past couple weeks, a problem has arisen. Whenever a server is rebooted,
    as the server comes up the entire network goes down for about 2 minutes.
    After the 2 minutes pass, everything starts working and our applications
    begin to breathe again.

    The guys I've talked to say that it's a spanning tree loop issue, but there
    are no redundant links to any switch. What is the likely cause here? How
    do I diagnose this issue and find the solution?

    This is a production datacenter that isn't our most important stuff, but it
    is "live" enough to where I can't go re-proing the issue over and over just
    to investigate.

    Thanks in advance.


    Michael
     
    Michael, Jan 11, 2005
    #1
    1. Advertisements

  2. :We have a strange issue in of our datacenters and I was hoping you guys
    :could help. We have five racks, each with a 3750 in each cabinet. The
    :basic wiring is like this:

    :As you see, we have the Internet link from our co-lo facility going into
    :switch 1, and the other cabinets are just chained off of each other.

    I do see that in your diagram. My first thought upon seeing it was
    that ideally it might be better to stack the 3750's into one
    logical switch... that might, though, be too much of a pain from
    the point of view of getting the cables over to the right place.

    : Over
    :the past couple weeks, a problem has arisen. Whenever a server is rebooted,
    :as the server comes up the entire network goes down for about 2 minutes.
    :After the 2 minutes pass, everything starts working and our applications
    :begin to breathe again.

    :The guys I've talked to say that it's a spanning tree loop issue, but there
    :are no redundant links to any switch. What is the likely cause here? How
    :do I diagnose this issue and find the solution?

    If it were about 1 minute then it could be explained not by spanning
    tree loops but merely by the need to recalculate the spanning tree
    to take into account the missing port. In the case of a server
    going down, the spanning tree is going to end up the same, but
    the spanning tree algorithm doesn't know that. What I would
    suggest you try, if you haven't already, is to set spanning-tree
    portfast on all of the links -except- the ones leading to other
    routers. portfast tells those ports not to go into blocking state
    as is normal with spanning tree.
     
    Walter Roberson, Jan 12, 2005
    #2
    1. Advertisements

  3. Michael

    Michael Guest

    [snip]
    Hi Walter.

    No I haven't tried this but....

    First, why would this start happening all of the sudden? Well, it wasn't
    really all of the sudden but we added our latest cabinet about 2 months ago.
    This problem has shown up in I'm guessing the past 3 weeks? Maybe past 4?
    It's completely possible that we just haven't rebooted anything since we
    added the last switch, but I don't know....

    Second, would this lack of portfast really cause the entire network to
    freeze like it does? Just from adding a port? I'm willing to try
    spanning-tree portfast on the ports, but it all seems suspect to me.


    Michael
     
    Michael, Jan 12, 2005
    #3
  4. [portfast]

    :No I haven't tried this but....

    :First, why would this start happening all of the sudden? Well, it wasn't
    :really all of the sudden but we added our latest cabinet about 2 months ago.
    :This problem has shown up in I'm guessing the past 3 weeks? Maybe past 4?
    :It's completely possible that we just haven't rebooted anything since we
    :added the last switch, but I don't know....

    Your diagram showed 5 switches chained together, and also showed
    a connection to your ISP. Is it possible that your connection to
    the ISP is going through a few switches before it gets to the router?
    If so then you might perhaps be hitting the spanning tree depth
    limit of 7 -- there are only 7 spaces reserved in the appropriate
    data structure in the packets, so if you had 3 switches chained at the
    ISP then you would run into spanning tree problems.


    :Second, would this lack of portfast really cause the entire network to
    :freeze like it does? Just from adding a port? I'm willing to try
    :spanning-tree portfast on the ports, but it all seems suspect to me.

    In theory, yes. When the spanning tree algorithm is run, it has to be
    run on -all- ports that are up, because of the possibility that the
    link that went down was providing the chosen connection to something
    beyond that there is a different topological way to get to. Spanning
    tree can be dumb that way.
     
    Walter Roberson, Jan 12, 2005
    #4
  5. Michael

    Michael Guest

    [snip]

    Portfast is clearly my friend.

    Problem solved. Thanks, Walter!


    Michael
     
    Michael, Jan 14, 2005
    #5
    1. Advertisements

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.