trunk connection dies when 6509 reboots.

Discussion in 'Cisco' started by news8080@yahoo.com, Sep 16, 2007.

  1. Guest

    we have an interesting problem with 6509 routes with 50 or so 3750
    switches (all running same ios) in closets. Whenever the 6509 reboots,
    we loose connectivity to some (always the same bunch) switches, its
    almost weird.

    The port just won't connect unless we power cycle the switch on the
    other end, I've rules out the following.

    - STP (we don't run it)
    - VTP (all closet switches are transparent and 6509 is server)
    - its not 'err disabled' thing because when I login to the router, I
    don't see the port as err-disabled, its just sitting dead as if no
    cable is connected on the other side, when we reboot the switch on the
    other side, the trunk come up and it all works fine. I HAVE to bring
    up switches more or less right away since we run a 24x7 environment
    that can't go down for more then 10 min or so.

    anyone care to take a poke. I've posted the trunk definitions of
    problem switches and 6509 here. I am also going to call cisco to find
    out too to see if they have anything to say about this..

    always the same bunch!!!

    ==============================================
    switch:
    interface GigabitEthernet2/0/12
    description Uplink to 6509
    switchport access vlan 6
    switchport trunk encapsulation dot1q
    switchport trunk allowed vlan 6,320
    switchport mode trunk
    end

    runs CDP

    6509:

    interface GigabitEthernet2/15
    no ip address
    switchport
    switchport trunk encapsulation dot1q
    switchport trunk allowed vlan 6,320
    switchport mode trunk
    end

    ==============================================

    switch:

    interface GigabitEthernet2/10
    description DnLink
    no ip address
    logging event link-status
    switchport
    switchport trunk encapsulation dot1q
    switchport trunk allowed vlan 2,322
    switchport mode trunk
    spanning-tree bpdufilter enable
    end

    doesn't run CDP

    6509:

    interface GigabitEthernet1/0/1
    description Uplink
    switchport trunk encapsulation dot1q
    switchport trunk allowed vlan 2,322,323
    switchport mode trunk
    end
    ==============================================

    switch:

    interface GigabitEthernet1/0/1
    description Uplink
    switchport trunk encapsulation dot1q
    switchport trunk allowed vlan 2,101,322,323
    switchport mode trunk
    end

    6509:

    interface GigabitEthernet2/13
    description DnLink
    no ip address
    switchport
    switchport trunk encapsulation dot1q
    switchport trunk allowed vlan 2,101,322
    switchport mode trunk
    spanning-tree bpdufilter enable
    end

    ==============================================
    switch:

    interface GigabitEthernet1/0/1
    description UpLink 6509
    switchport trunk encapsulation dot1q
    switchport mode trunk
    end

    runs CDP

    6509:
    interface GigabitEthernet2/3
    description DnLink
    no ip address
    switchport
    switchport trunk encapsulation dot1q
    switchport trunk allowed vlan 2,322,2104
    switchport mode trunk
    end
    ==============================================
    switch:

    interface GigabitEthernet1/0/1
    switchport trunk encapsulation dot1q
    switchport mode trunk
    end

    runs CDP

    interface GigabitEthernet2/2
    description DnLink
    no ip address
    logging event link-status
    switchport
    switchport trunk encapsulation dot1q
    switchport trunk allowed vlan 2,322
    switchport mode trunk
    end
    ==============================================
    switch:

    interface GigabitEthernet1/0/1
    description Uplink-6509
    switchport trunk encapsulation dot1q
    switchport mode trunk
    end

    runs CDP

    6509:
    interface GigabitEthernet2/1
    description DnLink
    no ip address
    logging event link-status
    switchport
    switchport trunk encapsulation dot1q
    switchport trunk allowed vlan 2,322,2104
    switchport mode trunk
    end
    ==============================================
    switch:

    interface GigabitEthernet1/0/1
    description Uplink to 6509
    switchport trunk encapsulation dot1q
    switchport trunk allowed vlan 2,322,323
    switchport mode trunk
    end

    no CDP

    6509:
    interface GigabitEthernet2/11
    description DnLink
    no ip address
    logging event link-status
    switchport
    switchport trunk encapsulation dot1q
    switchport trunk allowed vlan 2,322
    switchport mode trunk
    end

    ==============================================
     
    , Sep 16, 2007
    #1
    1. Advertising

  2. Merv Guest

    Post output of

    show version

    show interface status

    show module

    show diag

    show spanning-tree - for both the 6509 and a troubled switch



    Are all troubled ports on one 6509 module ?

    How many troubled ports are there in total ?
     
    Merv, Sep 16, 2007
    #2
    1. Advertising

  3. TravisT Guest

    I've seen a similar problem on a 3750 switch, but it would happen when
    plugging PoE IP phones into the ports. The phones would come up most
    of the time, but sometimes if we connected/disconnected the phone a
    couple of times the port would not come up at all. If we did a shut/
    no shut on the port in question it would come back up and run
    normally. I have not figured out what was causing this problem yet
    either. Maybe the next time you see the problem you could try the
    shut/no shut and see if that brings the trunk back up.
     
    TravisT, Sep 16, 2007
    #3
  4. Guest

    On Sep 16, 5:53 am, Merv <> wrote:
    > Post output of
    >
    > show version
    >
    > show interface status
    >
    > show module
    >
    > show diag
    >
    > show spanning-tree - for both the 6509 and a troubled switch
    >
    > Are all troubled ports on one 6509 module ?


    3
    > How many troubled ports are there in total ?

    7

    problem switch:
    =================

    sh ver
    Cisco IOS Software, C3750 Software (C3750-IPBASEK9-M), Version
    12.2(25)SEC, RELEASE SOFTWARE (fc2)
    Copyright (c) 1986-2005 by Cisco Systems, Inc.
    Compiled Fri 15-Jul-05 02:03 by antonino

    ROM: Bootstrap program is C3750 boot loader
    BOOTLDR: C3750 Boot Loader (C3750-HBOOT-M) Version 12.2(25r)SEC,
    RELEASE SOFTWARE (fc4)

    MDF uptime is 16 hours, 15 minutes
    System returned to ROM by power-on
    System restarted at 18:24:24 EDT Sat Sep 15 2007
    System image file is "flash:c3750-ipbasek9-mz.122-25.SEC/c3750-
    ipbasek9-mz.122-25.SEC.bin"


    This product contains cryptographic features and is subject to United
    States and local country laws governing import, export, transfer and
    use. Delivery of Cisco cryptographic products does not imply
    third-party authority to import, export, distribute or use encryption.
    Importers, exporters, distributors and users are responsible for
    compliance with U.S. and local country laws. By using this product you
    agree to comply with applicable laws and regulations. If you are
    unable
    to comply with U.S. and local laws, return this product immediately.

    A summary of U.S. laws governing Cisco cryptographic products may be
    found at:
    http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

    If you require further assistance please contact us by sending email
    to
    .

    cisco WS-C3750G-12S (PowerPC405) processor (revision N0) with 118784K/
    12280K bytes of memory.
    Processor board ID CAT0938R5D2
    Last reset from power-on
    3 Virtual Ethernet interfaces
    12 Gigabit Ethernet interfaces
    The password-recovery mechanism is enabled.

    512K bytes of flash-simulated non-volatile configuration memory.
    Base ethernet MAC Address : 00:15:C6:0B:5B:00
    Motherboard assembly number : 73-9678-05
    Power supply part number : 341-0048-03
    Motherboard serial number : CAT09390DRH
    Power supply serial number : DTH09312UUS
    Model revision number : N0
    Motherboard revision number : A0
    Model number : WS-C3750G-12S-S
    System serial number : CAT0938R5D2
    Top Assembly Part Number : 800-25856-02
    Top Assembly Revision Number : C0
    Version ID : V04
    CLEI Code Number : CNM8ZV0GRB
    Hardware Board Revision Number : 0x06


    Switch Ports Model SW Version SW
    Image
    ------ ----- ----- ----------
    ----------
    * 2 12 WS-C3750G-12S 12.2(25)SEC C3750-
    IPBASEK9-M


    Configuration register is 0xF

    MDF# sh int status

    Port Name Status Vlan Duplex Speed
    Type
    Gi2/0/1 xxxxxx connected trunk a-full a-1000
    1000BaseSX SFP
    Gi2/0/2 xxxxxx connected trunk a-full a-1000
    1000BaseSX SFP
    Gi2/0/3 xxxxxx connected trunk a-full a-1000
    1000BaseSX SFP
    Gi2/0/4 xxxxxx connected trunk a-full a-1000
    1000BaseSX SFP
    Gi2/0/5 xxxxxx connected trunk a-full a-1000
    1000BaseSX SFP
    Gi2/0/6 notconnect 6 auto auto Not
    Present
    Gi2/0/7 notconnect 6 auto auto Not
    Present
    Gi2/0/8 xxxxx connected trunk a-full a-1000
    1000BaseSX SFP
    Gi2/0/9 notconnect 6 auto auto Not
    Present
    Gi2/0/10 notconnect 6 auto auto Not
    Present
    Gi2/0/11 notconnect 6 auto auto Not
    Present
    Gi2/0/12 Uplink to 6509 connected trunk a-full a-1000
    1000BaseLX SFP
    MDF# sh mod
    ^
    % Invalid input detected at '^' marker.

    MDF#sh module
    ^
    % Invalid input detected at '^' marker.

    Elliman-MDF#sh diag
    ^
    % Invalid input detected at '^' marker.

    MDF#sh span
    MDF#sh spanning-tree

    No spanning tree instance exists.


    6509
    ======
    cat6509#sh inter Gi2/15 status

    Port Name Status Vlan Duplex Speed Type
    Gi2/15 DnLink MDF connected trunk full 1000
    1000BaseLH
    cat6509#sh ha
    Cisco Internetwork Operating System Software
    IOS (tm) s72033_rp Software (s72033_rp-JK9O3SV-M), Version
    12.2(17d)SXB6, RELEASE SOFTWARE (fc1)
    Technical Support: http://www.cisco.com/techsupport
    Copyright (c) 1986-2004 by cisco Systems, Inc.
    Compiled Fri 17-Dec-04 01:07 by hqluong
    Image text-base: 0x40020FBC, data-base: 0x422F0000

    ROM: System Bootstrap, Version 12.2(14r)S9, RELEASE SOFTWARE (fc1)
    BOOTLDR: s72033_rp Software (s72033_rp-JK9O3SV-M), Version
    12.2(17d)SXB6, RELEASE SOFTWARE (fc1)

    cat6509 uptime is 18 hours, 1 minute
    Time since cat6509 switched to active is 18 hours, 0 minutes
    System returned to ROM by power-on (SP by power-on)
    System restarted at 16:43:40 EDS Sat Sep 15 2007
    System image file is "sup-bootflash:s72033-jk9o3sv-mz.
    122-17d.SXB6.bin"


    This product contains cryptographic features and is subject to United
    States and local country laws governing import, export, transfer and
    use. Delivery of Cisco cryptographic products does not imply
    third-party authority to import, export, distribute or use encryption.
    Importers, exporters, distributors and users are responsible for
    compliance with U.S. and local country laws. By using this product you
    agree to comply with applicable laws and regulations. If you are
    unable
    to comply with U.S. and local laws, return this product immediately.

    A summary of U.S. laws governing Cisco cryptographic products may be
    found at:
    http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

    If you require further assistance please contact us by sending email
    to
    .

    cisco WS-C6509 (R7000) processor (revision 3.0) with 458752K/65536K
    bytes of memory.
    Processor board ID SAL0808UUZJ
    SR71000 CPU at 600Mhz, Implementation 0x504, Rev 1.2, 512KB L2 Cache
    Last reset from power-on
    Bridging software.
    X.25 software, Version 3.0.0.
    SuperLAT software (copyright 1990 by Meridian Technology Corp).
    TN3270 Emulation software.
    2141 Virtual Ethernet/IEEE 802.3 interface(s)
    116 Gigabit Ethernet/IEEE 802.3 interface(s)
    1917K bytes of non-volatile configuration memory.
    8192K bytes of packet buffer memory.

    65536K bytes of Flash internal SIMM (Sector size 512K).
    Standby is up
    Standby has 458752K/65536K bytes of memory.

    Configuration register is 0x2102

    cat6509# sh mod
    Mod Ports Card Type Model
    Serial No.
    --- ----- -------------------------------------- ------------------
    -----------
    1 16 Pure SFM-mode 16 port 1000mb GBIC WS-X6816-GBIC
    XXXXXXXXXXX
    2 16 Pure SFM-mode 16 port 1000mb GBIC WS-X6816-GBIC
    XXXXXXXXXXX
    3 48 CEF720 48 port 10/100/1000mb Ethernet WS-X6748-GE-TX
    XXXXXXXXXXX
    4 24 CEF720 24 port 1000mb SFP WS-X6724-SFP
    XXXXXXXXXXX
    5 2 Supervisor Engine 720 (Active) WS-SUP720-BASE
    XXXXXXXXXXX
    6 2 Supervisor Engine 720 (Warm) WS-SUP720-BASE
    XXXXXXXXXXX
    7 8 Intrusion Detection System WS-SVC-IDSM-2
    XXXXXXXXXXX

    Mod MAC addresses Hw Fw
    Sw Status
    --- ---------------------------------- ------ ------------
    ------------ -------
    1 000e.38c4.3cb8 to 000e.38c4.3cc7 1.7 12.2(14r)S3
    12.2(17d)SXB Ok
    2 000e.3828.e7a0 to 000e.3828.e7af 1.7 12.2(14r)S3
    12.2(17d)SXB Ok
    3 000f.8f3e.fa98 to 000f.8f3e.fac7 1.5 12.2(14r)S5
    12.2(17d)SXB Ok
    4 0012.7f51.4504 to 0012.7f51.451b 2.1 12.2(14r)S5
    12.2(17d)SXB Ok
    5 000d.6536.32a8 to 000d.6536.32ab 3.1 7.7(1)
    12.2(17d)SXB Ok
    6 000d.bda3.e9b0 to 000d.bda3.e9b3 3.1 7.7(1)
    12.2(17d)SXB Ok
    7 0014.a9bc.f788 to 0014.a9bc.f78f 5.0 7.2(1)
    6.0(2)E1 Ok

    Mod Sub-Module Model Serial
    Hw Status
    --- --------------------------- ------------------ ------------
    ------- -------
    1 Distributed Forwarding Card WS-F6K-DFC3A XXXXXXXXXXX
    2.0 Ok
    2 Distributed Forwarding Card WS-F6K-DFC3A XXXXXXXXXXX
    2.0 Ok
    3 Distributed Forwarding Card WS-F6700-DFC3A XXXXXXXXXXX
    2.1 Ok
    4 Distributed Forwarding Card WS-F6700-DFC3B XXXXXXXXXXX
    2.0 Ok
    5 Policy Feature Card 3 WS-F6K-PFC3A XXXXXXXXXXX
    2.2 Ok
    5 MSFC3 Daughterboard WS-SUP720 XXXXXXXXXXX
    2.2 Ok
    6 Policy Feature Card 3 WS-F6K-PFC3A XXXXXXXXXXX
    2.2 Ok
    6 MSFC3 Daughterboard WS-SUP720 XXXXXXXXXXX
    2.2 Ok
    7 IDS 2 accelerator board WS-SVC-IDSUPG XXXXXXXXXXX
    2.3 Ok

    Mod Online Diag Status
    --- -------------------
    1 Pass
    2 Pass
    3 Pass
    4 Pass
    5 Pass
    6 Pass
    7 Pass
    cat6509# sh diag

    cat6509#sh span

    No spanning tree instance exists.

    cat6509#
     
    , Sep 16, 2007
    #4
  5. Merv Guest

    The IOS software being used on the 6509 is quite dated. Not saying it
    is the cause of your issue
    but I would definitely start evaluating if an IOS upgrade should be
    considered. I know it can be
    a daunting task to pick a new IOS release for a 6500. I am running
    12.2(18)SXF6 on a SUP720,
    however, it is a lab environment.


    Cisco has yanked 3750 IOS images 12.2(25)SE and 12.2(25)SEA, it looks
    like you have 12.2(25)SEC.


    I would upgrade ONE of the troubled 3750 switches to say 12.2(35)SE5
    to see it it makes any difference


    Question - if you move a troubled 3750 switch to a different port on
    the 6509 does the problem follow ?
     
    Merv, Sep 16, 2007
    #5
  6. Merv Guest

    another thing you could try is disabling DTP on the troubled trunks -
    BOTH ends

    int type x/y
    switchport
    switchport trunk encapsulation dot1q
    switchport mode trunk
    switchport nonegotiate
     
    Merv, Sep 16, 2007
    #6
  7. Merv Guest

    OBTW does the 6509 need to be powered cycled oftern ?

    If so what is the situation that requires the power cycling?
     
    Merv, Sep 17, 2007
    #7
  8. Guest

    On Sep 17, 5:36 am, Merv <> wrote:
    > OBTW does the 6509 need to be powered cycled oftern ?
    >
    > If so what is the situation that requires the power cycling?


    6509 doesn't need power cycling often. the room that its in, has some
    AC issues so it rebooted on its own last week and that is when we saw
    the switch failures (again). It happened a month ago too but that was
    a scheduled reboot of the router.

    I guess I can try to turn DTP off, that won't hurt anything but if it
    is DTP then that would fail ALL switches and not just some and whats
    weird is that its the same set of switches all the time.

    I did a side by side comparison of a working and a problem switch and
    found no diff. (other then IP) in config. both on switches and their
    trunk definition on 6509. I do have err-disabled recovery off on both
    6509 and switches so I am going to see if I turn that on (recovery),
    does that make any diff?

    Question: If the switch side gets 'err-disabled', would the 6509 side
    also show 'err-disabled'? or would it show disconnected. we tried shut/
    no shut on 6509 when this happened and that never borough the
    connection up. I am wondering if that is the problem here.
     
    , Sep 17, 2007
    #8
  9. Don't overlook the fact this this may be a bug. IOS code is sometimes crap.

    Just curious - what are your thoughts behind STP being shut off? I'm
    assuming a star topology?

    On your uplinks, have you played with the UDLD settings to see if
    there's any change?

    Are the GBICs you bought directly from Cisco? If not, there are many,
    many counterfeits that will act flakey. They will not always throw an
    error in the syslog.



    wrote:
    > On Sep 17, 5:36 am, Merv <> wrote:
    >> OBTW does the 6509 need to be powered cycled oftern ?
    >>
    >> If so what is the situation that requires the power cycling?

    >
    > 6509 doesn't need power cycling often. the room that its in, has some
    > AC issues so it rebooted on its own last week and that is when we saw
    > the switch failures (again). It happened a month ago too but that was
    > a scheduled reboot of the router.
    >
    > I guess I can try to turn DTP off, that won't hurt anything but if it
    > is DTP then that would fail ALL switches and not just some and whats
    > weird is that its the same set of switches all the time.
    >
    > I did a side by side comparison of a working and a problem switch and
    > found no diff. (other then IP) in config. both on switches and their
    > trunk definition on 6509. I do have err-disabled recovery off on both
    > 6509 and switches so I am going to see if I turn that on (recovery),
    > does that make any diff?
    >
    > Question: If the switch side gets 'err-disabled', would the 6509 side
    > also show 'err-disabled'? or would it show disconnected. we tried shut/
    > no shut on 6509 when this happened and that never borough the
    > connection up. I am wondering if that is the problem here.
    >
    >
     
    fugettaboutit, Sep 17, 2007
    #9
  10. Merv Guest

    I suggest do the following:

    1. upgrade ONE to troubled 3750 to the IOS that I suggested.

    2. disable DTP on the two or three of the trunsk to the the troubled
    3750's

    3. Move one of the troubled 3750 to a different 6509 port to see if
    problem follows.

    Then schedule a reboot to observe what occurs or does not occur.

    Capture the output of "show interface trunk"


    OBTW I ALWAYS disable DTP on trunks as it is just one more thing that
    can go wrong.
    If a link is a trunk then it is a trunk ...
     
    Merv, Sep 17, 2007
    #10
  11. Merv Guest


    > I am running 12.2(18)SXF6 on a SUP720, however, it is a lab environment.



    Now using 12.2(18)SXF11 as Cisco junked SXF, SXF1 thru SXF9
     
    Merv, Sep 19, 2007
    #11
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Dmitry Melekhov

    connection trunk, FXS-FXS

    Dmitry Melekhov, Nov 10, 2003, in forum: Cisco
    Replies:
    1
    Views:
    1,917
    aab-telecom
    Nov 22, 2012
  2. alex

    Echo Reply dies at pix

    alex, Jul 28, 2004, in forum: Cisco
    Replies:
    3
    Views:
    2,401
  3. root
    Replies:
    1
    Views:
    3,931
  4. Jim

    Quickcam just dies???

    Jim, Oct 4, 2003, in forum: Computer Support
    Replies:
    4
    Views:
    418
  5. Replies:
    1
    Views:
    2,318
    Brian V
    Jun 27, 2007
Loading...

Share This Page