Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Computing > Cisco > trunk connection dies when 6509 reboots.

Reply
Thread Tools

trunk connection dies when 6509 reboots.

 
 
news8080@yahoo.com
Guest
Posts: n/a
 
      09-15-2007
we have an interesting problem with 6509 routes with 50 or so 3750
switches (all running same ios) in closets. Whenever the 6509 reboots,
we loose connectivity to some (always the same bunch) switches, its
almost weird.

The port just won't connect unless we power cycle the switch on the
other end, I've rules out the following.

- STP (we don't run it)
- VTP (all closet switches are transparent and 6509 is server)
- its not 'err disabled' thing because when I login to the router, I
don't see the port as err-disabled, its just sitting dead as if no
cable is connected on the other side, when we reboot the switch on the
other side, the trunk come up and it all works fine. I HAVE to bring
up switches more or less right away since we run a 24x7 environment
that can't go down for more then 10 min or so.

anyone care to take a poke. I've posted the trunk definitions of
problem switches and 6509 here. I am also going to call cisco to find
out too to see if they have anything to say about this..

always the same bunch!!!

==============================================
switch:
interface GigabitEthernet2/0/12
description Uplink to 6509
switchport access vlan 6
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 6,320
switchport mode trunk
end

runs CDP

6509:

interface GigabitEthernet2/15
no ip address
switchport
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 6,320
switchport mode trunk
end

==============================================

switch:

interface GigabitEthernet2/10
description DnLink
no ip address
logging event link-status
switchport
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 2,322
switchport mode trunk
spanning-tree bpdufilter enable
end

doesn't run CDP

6509:

interface GigabitEthernet1/0/1
description Uplink
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 2,322,323
switchport mode trunk
end
==============================================

switch:

interface GigabitEthernet1/0/1
description Uplink
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 2,101,322,323
switchport mode trunk
end

6509:

interface GigabitEthernet2/13
description DnLink
no ip address
switchport
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 2,101,322
switchport mode trunk
spanning-tree bpdufilter enable
end

==============================================
switch:

interface GigabitEthernet1/0/1
description UpLink 6509
switchport trunk encapsulation dot1q
switchport mode trunk
end

runs CDP

6509:
interface GigabitEthernet2/3
description DnLink
no ip address
switchport
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 2,322,2104
switchport mode trunk
end
==============================================
switch:

interface GigabitEthernet1/0/1
switchport trunk encapsulation dot1q
switchport mode trunk
end

runs CDP

interface GigabitEthernet2/2
description DnLink
no ip address
logging event link-status
switchport
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 2,322
switchport mode trunk
end
==============================================
switch:

interface GigabitEthernet1/0/1
description Uplink-6509
switchport trunk encapsulation dot1q
switchport mode trunk
end

runs CDP

6509:
interface GigabitEthernet2/1
description DnLink
no ip address
logging event link-status
switchport
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 2,322,2104
switchport mode trunk
end
==============================================
switch:

interface GigabitEthernet1/0/1
description Uplink to 6509
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 2,322,323
switchport mode trunk
end

no CDP

6509:
interface GigabitEthernet2/11
description DnLink
no ip address
logging event link-status
switchport
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 2,322
switchport mode trunk
end

==============================================

 
Reply With Quote
 
 
 
 
Merv
Guest
Posts: n/a
 
      09-16-2007
Post output of

show version

show interface status

show module

show diag

show spanning-tree - for both the 6509 and a troubled switch



Are all troubled ports on one 6509 module ?

How many troubled ports are there in total ?



 
Reply With Quote
 
 
 
 
TravisT
Guest
Posts: n/a
 
      09-16-2007
I've seen a similar problem on a 3750 switch, but it would happen when
plugging PoE IP phones into the ports. The phones would come up most
of the time, but sometimes if we connected/disconnected the phone a
couple of times the port would not come up at all. If we did a shut/
no shut on the port in question it would come back up and run
normally. I have not figured out what was causing this problem yet
either. Maybe the next time you see the problem you could try the
shut/no shut and see if that brings the trunk back up.

 
Reply With Quote
 
news8080@yahoo.com
Guest
Posts: n/a
 
      09-16-2007
On Sep 16, 5:53 am, Merv <(E-Mail Removed)> wrote:
> Post output of
>
> show version
>
> show interface status
>
> show module
>
> show diag
>
> show spanning-tree - for both the 6509 and a troubled switch
>
> Are all troubled ports on one 6509 module ?


3
> How many troubled ports are there in total ?

7

problem switch:
=================

sh ver
Cisco IOS Software, C3750 Software (C3750-IPBASEK9-M), Version
12.2(25)SEC, RELEASE SOFTWARE (fc2)
Copyright (c) 1986-2005 by Cisco Systems, Inc.
Compiled Fri 15-Jul-05 02:03 by antonino

ROM: Bootstrap program is C3750 boot loader
BOOTLDR: C3750 Boot Loader (C3750-HBOOT-M) Version 12.2(25r)SEC,
RELEASE SOFTWARE (fc4)

MDF uptime is 16 hours, 15 minutes
System returned to ROM by power-on
System restarted at 18:24:24 EDT Sat Sep 15 2007
System image file is "flash:c3750-ipbasek9-mz.122-25.SEC/c3750-
ipbasek9-mz.122-25.SEC.bin"


This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are
unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be
found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email
to
http://www.velocityreviews.com/forums/(E-Mail Removed).

cisco WS-C3750G-12S (PowerPC405) processor (revision N0) with 118784K/
12280K bytes of memory.
Processor board ID CAT0938R5D2
Last reset from power-on
3 Virtual Ethernet interfaces
12 Gigabit Ethernet interfaces
The password-recovery mechanism is enabled.

512K bytes of flash-simulated non-volatile configuration memory.
Base ethernet MAC Address : 00:15:C6:0B:5B:00
Motherboard assembly number : 73-9678-05
Power supply part number : 341-0048-03
Motherboard serial number : CAT09390DRH
Power supply serial number : DTH09312UUS
Model revision number : N0
Motherboard revision number : A0
Model number : WS-C3750G-12S-S
System serial number : CAT0938R5D2
Top Assembly Part Number : 800-25856-02
Top Assembly Revision Number : C0
Version ID : V04
CLEI Code Number : CNM8ZV0GRB
Hardware Board Revision Number : 0x06


Switch Ports Model SW Version SW
Image
------ ----- ----- ----------
----------
* 2 12 WS-C3750G-12S 12.2(25)SEC C3750-
IPBASEK9-M


Configuration register is 0xF

MDF# sh int status

Port Name Status Vlan Duplex Speed
Type
Gi2/0/1 xxxxxx connected trunk a-full a-1000
1000BaseSX SFP
Gi2/0/2 xxxxxx connected trunk a-full a-1000
1000BaseSX SFP
Gi2/0/3 xxxxxx connected trunk a-full a-1000
1000BaseSX SFP
Gi2/0/4 xxxxxx connected trunk a-full a-1000
1000BaseSX SFP
Gi2/0/5 xxxxxx connected trunk a-full a-1000
1000BaseSX SFP
Gi2/0/6 notconnect 6 auto auto Not
Present
Gi2/0/7 notconnect 6 auto auto Not
Present
Gi2/0/8 xxxxx connected trunk a-full a-1000
1000BaseSX SFP
Gi2/0/9 notconnect 6 auto auto Not
Present
Gi2/0/10 notconnect 6 auto auto Not
Present
Gi2/0/11 notconnect 6 auto auto Not
Present
Gi2/0/12 Uplink to 6509 connected trunk a-full a-1000
1000BaseLX SFP
MDF# sh mod
^
% Invalid input detected at '^' marker.

MDF#sh module
^
% Invalid input detected at '^' marker.

Elliman-MDF#sh diag
^
% Invalid input detected at '^' marker.

MDF#sh span
MDF#sh spanning-tree

No spanning tree instance exists.


6509
======
cat6509#sh inter Gi2/15 status

Port Name Status Vlan Duplex Speed Type
Gi2/15 DnLink MDF connected trunk full 1000
1000BaseLH
cat6509#sh ha
Cisco Internetwork Operating System Software
IOS (tm) s72033_rp Software (s72033_rp-JK9O3SV-M), Version
12.2(17d)SXB6, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2004 by cisco Systems, Inc.
Compiled Fri 17-Dec-04 01:07 by hqluong
Image text-base: 0x40020FBC, data-base: 0x422F0000

ROM: System Bootstrap, Version 12.2(14r)S9, RELEASE SOFTWARE (fc1)
BOOTLDR: s72033_rp Software (s72033_rp-JK9O3SV-M), Version
12.2(17d)SXB6, RELEASE SOFTWARE (fc1)

cat6509 uptime is 18 hours, 1 minute
Time since cat6509 switched to active is 18 hours, 0 minutes
System returned to ROM by power-on (SP by power-on)
System restarted at 16:43:40 EDS Sat Sep 15 2007
System image file is "sup-bootflash:s72033-jk9o3sv-mz.
122-17d.SXB6.bin"


This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are
unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be
found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email
to
(E-Mail Removed).

cisco WS-C6509 (R7000) processor (revision 3.0) with 458752K/65536K
bytes of memory.
Processor board ID SAL0808UUZJ
SR71000 CPU at 600Mhz, Implementation 0x504, Rev 1.2, 512KB L2 Cache
Last reset from power-on
Bridging software.
X.25 software, Version 3.0.0.
SuperLAT software (copyright 1990 by Meridian Technology Corp).
TN3270 Emulation software.
2141 Virtual Ethernet/IEEE 802.3 interface(s)
116 Gigabit Ethernet/IEEE 802.3 interface(s)
1917K bytes of non-volatile configuration memory.
8192K bytes of packet buffer memory.

65536K bytes of Flash internal SIMM (Sector size 512K).
Standby is up
Standby has 458752K/65536K bytes of memory.

Configuration register is 0x2102

cat6509# sh mod
Mod Ports Card Type Model
Serial No.
--- ----- -------------------------------------- ------------------
-----------
1 16 Pure SFM-mode 16 port 1000mb GBIC WS-X6816-GBIC
XXXXXXXXXXX
2 16 Pure SFM-mode 16 port 1000mb GBIC WS-X6816-GBIC
XXXXXXXXXXX
3 48 CEF720 48 port 10/100/1000mb Ethernet WS-X6748-GE-TX
XXXXXXXXXXX
4 24 CEF720 24 port 1000mb SFP WS-X6724-SFP
XXXXXXXXXXX
5 2 Supervisor Engine 720 (Active) WS-SUP720-BASE
XXXXXXXXXXX
6 2 Supervisor Engine 720 (Warm) WS-SUP720-BASE
XXXXXXXXXXX
7 8 Intrusion Detection System WS-SVC-IDSM-2
XXXXXXXXXXX

Mod MAC addresses Hw Fw
Sw Status
--- ---------------------------------- ------ ------------
------------ -------
1 000e.38c4.3cb8 to 000e.38c4.3cc7 1.7 12.2(14r)S3
12.2(17d)SXB Ok
2 000e.3828.e7a0 to 000e.3828.e7af 1.7 12.2(14r)S3
12.2(17d)SXB Ok
3 000f.8f3e.fa98 to 000f.8f3e.fac7 1.5 12.2(14r)S5
12.2(17d)SXB Ok
4 0012.7f51.4504 to 0012.7f51.451b 2.1 12.2(14r)S5
12.2(17d)SXB Ok
5 000d.6536.32a8 to 000d.6536.32ab 3.1 7.7(1)
12.2(17d)SXB Ok
6 000d.bda3.e9b0 to 000d.bda3.e9b3 3.1 7.7(1)
12.2(17d)SXB Ok
7 0014.a9bc.f788 to 0014.a9bc.f78f 5.0 7.2(1)
6.0(2)E1 Ok

Mod Sub-Module Model Serial
Hw Status
--- --------------------------- ------------------ ------------
------- -------
1 Distributed Forwarding Card WS-F6K-DFC3A XXXXXXXXXXX
2.0 Ok
2 Distributed Forwarding Card WS-F6K-DFC3A XXXXXXXXXXX
2.0 Ok
3 Distributed Forwarding Card WS-F6700-DFC3A XXXXXXXXXXX
2.1 Ok
4 Distributed Forwarding Card WS-F6700-DFC3B XXXXXXXXXXX
2.0 Ok
5 Policy Feature Card 3 WS-F6K-PFC3A XXXXXXXXXXX
2.2 Ok
5 MSFC3 Daughterboard WS-SUP720 XXXXXXXXXXX
2.2 Ok
6 Policy Feature Card 3 WS-F6K-PFC3A XXXXXXXXXXX
2.2 Ok
6 MSFC3 Daughterboard WS-SUP720 XXXXXXXXXXX
2.2 Ok
7 IDS 2 accelerator board WS-SVC-IDSUPG XXXXXXXXXXX
2.3 Ok

Mod Online Diag Status
--- -------------------
1 Pass
2 Pass
3 Pass
4 Pass
5 Pass
6 Pass
7 Pass
cat6509# sh diag

cat6509#sh span

No spanning tree instance exists.

cat6509#

 
Reply With Quote
 
Merv
Guest
Posts: n/a
 
      09-16-2007

The IOS software being used on the 6509 is quite dated. Not saying it
is the cause of your issue
but I would definitely start evaluating if an IOS upgrade should be
considered. I know it can be
a daunting task to pick a new IOS release for a 6500. I am running
12.2(1SXF6 on a SUP720,
however, it is a lab environment.


Cisco has yanked 3750 IOS images 12.2(25)SE and 12.2(25)SEA, it looks
like you have 12.2(25)SEC.


I would upgrade ONE of the troubled 3750 switches to say 12.2(35)SE5
to see it it makes any difference


Question - if you move a troubled 3750 switch to a different port on
the 6509 does the problem follow ?

 
Reply With Quote
 
Merv
Guest
Posts: n/a
 
      09-16-2007
another thing you could try is disabling DTP on the troubled trunks -
BOTH ends

int type x/y
switchport
switchport trunk encapsulation dot1q
switchport mode trunk
switchport nonegotiate


 
Reply With Quote
 
Merv
Guest
Posts: n/a
 
      09-17-2007

OBTW does the 6509 need to be powered cycled oftern ?

If so what is the situation that requires the power cycling?


 
Reply With Quote
 
news8080@yahoo.com
Guest
Posts: n/a
 
      09-17-2007
On Sep 17, 5:36 am, Merv <(E-Mail Removed)> wrote:
> OBTW does the 6509 need to be powered cycled oftern ?
>
> If so what is the situation that requires the power cycling?


6509 doesn't need power cycling often. the room that its in, has some
AC issues so it rebooted on its own last week and that is when we saw
the switch failures (again). It happened a month ago too but that was
a scheduled reboot of the router.

I guess I can try to turn DTP off, that won't hurt anything but if it
is DTP then that would fail ALL switches and not just some and whats
weird is that its the same set of switches all the time.

I did a side by side comparison of a working and a problem switch and
found no diff. (other then IP) in config. both on switches and their
trunk definition on 6509. I do have err-disabled recovery off on both
6509 and switches so I am going to see if I turn that on (recovery),
does that make any diff?

Question: If the switch side gets 'err-disabled', would the 6509 side
also show 'err-disabled'? or would it show disconnected. we tried shut/
no shut on 6509 when this happened and that never borough the
connection up. I am wondering if that is the problem here.


 
Reply With Quote
 
fugettaboutit
Guest
Posts: n/a
 
      09-17-2007
Don't overlook the fact this this may be a bug. IOS code is sometimes crap.

Just curious - what are your thoughts behind STP being shut off? I'm
assuming a star topology?

On your uplinks, have you played with the UDLD settings to see if
there's any change?

Are the GBICs you bought directly from Cisco? If not, there are many,
many counterfeits that will act flakey. They will not always throw an
error in the syslog.



(E-Mail Removed) wrote:
> On Sep 17, 5:36 am, Merv <(E-Mail Removed)> wrote:
>> OBTW does the 6509 need to be powered cycled oftern ?
>>
>> If so what is the situation that requires the power cycling?

>
> 6509 doesn't need power cycling often. the room that its in, has some
> AC issues so it rebooted on its own last week and that is when we saw
> the switch failures (again). It happened a month ago too but that was
> a scheduled reboot of the router.
>
> I guess I can try to turn DTP off, that won't hurt anything but if it
> is DTP then that would fail ALL switches and not just some and whats
> weird is that its the same set of switches all the time.
>
> I did a side by side comparison of a working and a problem switch and
> found no diff. (other then IP) in config. both on switches and their
> trunk definition on 6509. I do have err-disabled recovery off on both
> 6509 and switches so I am going to see if I turn that on (recovery),
> does that make any diff?
>
> Question: If the switch side gets 'err-disabled', would the 6509 side
> also show 'err-disabled'? or would it show disconnected. we tried shut/
> no shut on 6509 when this happened and that never borough the
> connection up. I am wondering if that is the problem here.
>
>

 
Reply With Quote
 
Merv
Guest
Posts: n/a
 
      09-17-2007
I suggest do the following:

1. upgrade ONE to troubled 3750 to the IOS that I suggested.

2. disable DTP on the two or three of the trunsk to the the troubled
3750's

3. Move one of the troubled 3750 to a different 6509 port to see if
problem follows.

Then schedule a reboot to observe what occurs or does not occur.

Capture the output of "show interface trunk"


OBTW I ALWAYS disable DTP on trunks as it is just one more thing that
can go wrong.
If a link is a trunk then it is a trunk ...

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
connection trunk, FXS-FXS Dmitry Melekhov Cisco 1 11-22-2012 07:56 AM
Cisco 6509-E and trunk ports njwhitworth@gmail.com Cisco 1 06-27-2007 05:27 PM
Question: Regarding VoIP - connection trunk - setup between two offices root Cisco 1 10-21-2004 02:46 PM
Net::SNMP dies on any OID error and doesn't return other possibly good values dan byers Perl 1 03-06-2004 09:50 AM
Perl 'system' Creates Program That Dies When First C Program Dies Christopher M. Lusardi Perl Misc 3 10-19-2003 11:53 AM



Advertisments