Mystery Network Problem

Discussion in 'Cisco' started by Aaron, Feb 18, 2008.

  1. Aaron

    Aaron Guest

    So I'm tossing this out there to see if anyone might have some
    thoughts on where to look because I just find this baffling.

    We had a Cisco 3750 Catalyst switch set up with a number of hosts
    connected to it. All of these hosts were configured to do NIC teaming
    with etherchannels set up on the switch. We then added a second 3750
    to create a stack. We re-configured the etherchannels on the existing
    servers so that each one had one connection to each switch. We then
    added a number of new servers with each one using etherchannels (one
    connection to each switch). Everything seemed fine. All of the
    servers were available on the network, etc. But, we noticed that of
    the 12 new servers, two of them had this odd issue. All of the
    servers are in the same VLAN and IP's in the same subnet. All of them
    can get out to the internet and can be reached by *most* computers in
    the same VLAN and in other VLANS. BUT...the two oddballs can't talk
    to the other servers connected to the same 3750.

    I turned off the NIC teaming and disabled the secondary NIC port. We
    removed the switch ports from the etherchannel so it was just a
    normal, plain jane port. Still had the problem. Moving the server IP
    information from the primary to the secondary NIC port cleared up the
    problem...at first. While trying various permutations of the NIC
    teaming (trying to track down the problem) things got the point where
    if the server was configured to use NIC1, it could get out to the
    internet and other various servers, but couldn't talk (no ping, no
    thing..they don't even see ARP requests) to the other servers on the
    switch (EXCEPT those in a different VLAN) or some other random
    machines on that same VLAN.). Setting the server to use NIC2 resulted
    in it being able to talk to servers it previously couldn't talk to,
    but it suddenly couldn't talk to the gateway and so couldn't reach
    anything else. It was previously able to do so.

    What this FEELS like is that the switch has somehow declared these two
    server's MAC addresses as persona non-grata and won't route traffic
    from or to it except to seemingly random exceptions.

    Anyone have any thoughts on where to look as I'm pretty much stumped
    at this point.
    Aaron, Feb 18, 2008
    #1
    1. Advertising

  2. Aaron

    Thrill5 Guest

    The 3750 only has a limited amount of TCAM that must be used for routing
    (CEF entries) switching (for MAC entries), multicast and qos queues.
    Sometimes the "default" allocation of TCAM isn't right and you have to
    adjust it.

    Do the MAC entries for the servers exist in "show mac-address-table", if
    not you probably have an TCAM issue. Also do a "show mac-address-table
    count" and see how many entries are left for MAC addresses (this will be the
    last line of the output). If you are doing layer 3 on the switch then you
    probably don't have enough TCAM allocated for CEF entries, and the reason
    you are having issues.

    The "show sdm prefer" command will show you what is currently configured.
    The default is "default" but we have had to change this on 3750's that we
    are doing layer 3 switching on. To change it, issue a "sdm prefer
    <template>" command, and you will need to reboot for it to take effect.


    See this article for more information on TCAM and the 3750.
    http://www.enterprisenetworkingplanet.com/nethub/article.php/3527301

    "Aaron" <> wrote in message
    news:...
    > So I'm tossing this out there to see if anyone might have some
    > thoughts on where to look because I just find this baffling.
    >
    > We had a Cisco 3750 Catalyst switch set up with a number of hosts
    > connected to it. All of these hosts were configured to do NIC teaming
    > with etherchannels set up on the switch. We then added a second 3750
    > to create a stack. We re-configured the etherchannels on the existing
    > servers so that each one had one connection to each switch. We then
    > added a number of new servers with each one using etherchannels (one
    > connection to each switch). Everything seemed fine. All of the
    > servers were available on the network, etc. But, we noticed that of
    > the 12 new servers, two of them had this odd issue. All of the
    > servers are in the same VLAN and IP's in the same subnet. All of them
    > can get out to the internet and can be reached by *most* computers in
    > the same VLAN and in other VLANS. BUT...the two oddballs can't talk
    > to the other servers connected to the same 3750.
    >
    > I turned off the NIC teaming and disabled the secondary NIC port. We
    > removed the switch ports from the etherchannel so it was just a
    > normal, plain jane port. Still had the problem. Moving the server IP
    > information from the primary to the secondary NIC port cleared up the
    > problem...at first. While trying various permutations of the NIC
    > teaming (trying to track down the problem) things got the point where
    > if the server was configured to use NIC1, it could get out to the
    > internet and other various servers, but couldn't talk (no ping, no
    > thing..they don't even see ARP requests) to the other servers on the
    > switch (EXCEPT those in a different VLAN) or some other random
    > machines on that same VLAN.). Setting the server to use NIC2 resulted
    > in it being able to talk to servers it previously couldn't talk to,
    > but it suddenly couldn't talk to the gateway and so couldn't reach
    > anything else. It was previously able to do so.
    >
    > What this FEELS like is that the switch has somehow declared these two
    > server's MAC addresses as persona non-grata and won't route traffic
    > from or to it except to seemingly random exceptions.
    >
    > Anyone have any thoughts on where to look as I'm pretty much stumped
    > at this point.
    Thrill5, Feb 18, 2008
    #2
    1. Advertising

  3. Aaron

    Aaron Guest

    We ended up tracking down the problem to how the servers NIC teaming
    was set up. The company that did the initial build and install set
    the NIC teaming up to use Broadcom's proprietary "Smart Load
    Balancing". Apparently, this was butting heads something fierce with
    the Fast Etherchannel settings on the switch. Changing that NIC
    teaming type to just "Generic Trunking" cleared up all the problems!

    On Feb 18, 6:14 pm, "Thrill5" <> wrote:
    > The 3750 only has a limited amount of TCAM that must be used for routing
    > (CEF entries) switching (for MAC entries), multicast and qos queues.
    > Sometimes the "default" allocation of TCAM isn't right and you have to
    > adjust it.
    >
    > Do the MAC entries for the servers exist in "show mac-address-table", if
    > not you probably have an TCAM issue. Also do a "show mac-address-table
    > count" and see how many entries are left for MAC addresses (this will be the
    > last line of the output). If you are doing layer 3 on the switch then you
    > probably don't have enough TCAM allocated for CEF entries, and the reason
    > you are having issues.
    >
    > The "show sdm prefer" command will show you what is currently configured.
    > The default is "default" but we have had to change this on 3750's that we
    > are doing layer 3 switching on. To change it, issue a "sdm prefer
    > <template>" command, and you will need to reboot for it to take effect.
    >
    > See this article for more information on TCAM and the 3750.http://www.enterprisenetworkingplanet.com/nethub/article.php/3527301
    >
    Aaron, Feb 19, 2008
    #3
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Brad

    Mystery Problem with Compaq Presario 1060 Laptop

    Brad, Sep 19, 2006, in forum: Computer Information
    Replies:
    0
    Views:
    447
  2. Brad
    Replies:
    3
    Views:
    433
    Travis
    Sep 29, 2006
  3. Brad
    Replies:
    0
    Views:
    391
  4. Ockerr

    ZAP 0.0.0.0/0.0.0.0 mystery network

    Ockerr, May 28, 2006, in forum: Computer Support
    Replies:
    3
    Views:
    381
    Ralph Wade Phillips
    May 29, 2006
  5. Tony Sperling

    Network Config Mystery

    Tony Sperling, Apr 3, 2006, in forum: Windows 64bit
    Replies:
    24
    Views:
    1,245
    Jud Hendrix
    May 1, 2006
Loading...

Share This Page