Anyone know how to Analyze "crashinfo" files from Cisco access points?
 

Hello ...

We have a number of Cisco AP350 wireless access points thoughout our
network (as well as some AP1100s and AP1210s, but these aren't involved
in the problem that's causing me to send this message).

One of them seems to have started periodically crashing and resetting
this past Monday evening (2006/08/28, 21:57:51, according to the
timestamp on the crashinfo file), and is presently apparently looping
between functioning and not. I have 10 crashinfo files and an r0.rcore
file from it, and would like to try and figure out what the problem is.

Can anyone point me to information regarding how these should be used
for crash analysis?

The syslog from this device is presently cycling through the following
messages. I think I'm going to disconnect the device at this point ...

Aug 30 12:40:21 ap-gm401-90 573: Aug 30 12:40:21: %DOT11-2-RADIO_FAILED:
Interface Dot11Radio0, failed - cmd busy
Aug 30 12:40:21 ap-gm401-90 574: -Traceback= 4DC04 4511C4 45110C 4532B0
6AA4DC 6AA778 6AF00C 4503EC 4507C8 450B1C 1529D8 157988
Aug 30 12:40:22 ap-gm401-90 575: Aug 30 12:40:21: %LINK-3-UPDOWN:
Interface Dot11Radio0, changed state to down
Aug 30 12:40:22 ap-gm401-90 576: Aug 30 12:40:22: %LINEPROTO-5-UPDOWN:
Line protocol on Interface Dot11Radio0, changed state to down
Aug 30 12:40:46 ap-gm401-90 577: Aug 30 12:40:45: %LINK-5-CHANGED:
Interface Dot11Radio0, changed state to reset
Aug 30 12:40:48 ap-gm401-90 578: Aug 30 12:40:47: %LINK-3-UPDOWN:
Interface Dot11Radio0, changed state to up
Aug 30 12:40:48 ap-gm401-90 579: Aug 30 12:40:48: %LINEPROTO-5-UPDOWN:
Line protocol on Interface Dot11Radio0, changed state to up


Any ideas would be greatly appreciated ...

--
----------------------------------------------------------------------
Sylvain Robitaille syl@alcor.concordia.ca

Systems and Network analyst Concordia University
Instructional & Information Technology Montreal, Quebec, Canada
----------------------------------------------------------------------

Re: Anyone know how to Analyze "crashinfo" files from Cisco access points?
 

Did you contact Cisco????

Sylvain Robitaille <syl@alcor.concordia.ca> wrote:

>Hello ...
>
>We have a number of Cisco AP350 wireless access points thoughout our
>network (as well as some AP1100s and AP1210s, but these aren't involved
>in the problem that's causing me to send this message).
>
>One of them seems to have started periodically crashing and resetting
>this past Monday evening (2006/08/28, 21:57:51, according to the
>timestamp on the crashinfo file), and is presently apparently looping
>between functioning and not. I have 10 crashinfo files and an r0.rcore
>file from it, and would like to try and figure out what the problem is.
>
>Can anyone point me to information regarding how these should be used
>for crash analysis?
>
>The syslog from this device is presently cycling through the following
>messages. I think I'm going to disconnect the device at this point ...
>
>Aug 30 12:40:21 ap-gm401-90 573: Aug 30 12:40:21: %DOT11-2-RADIO_FAILED:
> Interface Dot11Radio0, failed - cmd busy
>Aug 30 12:40:21 ap-gm401-90 574: -Traceback= 4DC04 4511C4 45110C 4532B0
> 6AA4DC 6AA778 6AF00C 4503EC 4507C8 450B1C 1529D8 157988
>Aug 30 12:40:22 ap-gm401-90 575: Aug 30 12:40:21: %LINK-3-UPDOWN:
> Interface Dot11Radio0, changed state to down
>Aug 30 12:40:22 ap-gm401-90 576: Aug 30 12:40:22: %LINEPROTO-5-UPDOWN:
> Line protocol on Interface Dot11Radio0, changed state to down
>Aug 30 12:40:46 ap-gm401-90 577: Aug 30 12:40:45: %LINK-5-CHANGED:
> Interface Dot11Radio0, changed state to reset
>Aug 30 12:40:48 ap-gm401-90 578: Aug 30 12:40:47: %LINK-3-UPDOWN:
> Interface Dot11Radio0, changed state to up
>Aug 30 12:40:48 ap-gm401-90 579: Aug 30 12:40:48: %LINEPROTO-5-UPDOWN:
> Line protocol on Interface Dot11Radio0, changed state to up
>
>
>Any ideas would be greatly appreciated ...
>
>--
>----------------------------------------------------------------------
>Sylvain Robitaille syl@alcor.concordia.ca
>
>Systems and Network analyst Concordia University
>Instructional & Information Technology Montreal, Quebec, Canada
>----------------------------------------------------------------------

Re: Anyone know how to Analyze "crashinfo" files from Cisco access points?
 

gene martinez wrote:
> Did you contact Cisco????
>
> Sylvain Robitaille <syl@alcor.concordia.ca> wrote:
>
> >Hello ...
> >
> >We have a number of Cisco AP350 wireless access points thoughout our
> >network (as well as some AP1100s and AP1210s, but these aren't involved
> >in the problem that's causing me to send this message).
> >
> >One of them seems to have started periodically crashing and resetting
> >this past Monday evening (2006/08/28, 21:57:51, according to the
> >timestamp on the crashinfo file), and is presently apparently looping
> >between functioning and not. I have 10 crashinfo files and an r0.rcore
> >file from it, and would like to try and figure out what the problem is.
> >
> >Can anyone point me to information regarding how these should be used
> >for crash analysis?
> >
> >The syslog from this device is presently cycling through the following
> >messages. I think I'm going to disconnect the device at this point ...
> >
> >Aug 30 12:40:21 ap-gm401-90 573: Aug 30 12:40:21: %DOT11-2-RADIO_FAILED:
> > Interface Dot11Radio0, failed - cmd busy
> >Aug 30 12:40:21 ap-gm401-90 574: -Traceback= 4DC04 4511C4 45110C 4532B0
> > 6AA4DC 6AA778 6AF00C 4503EC 4507C8 450B1C 1529D8 157988
> >Aug 30 12:40:22 ap-gm401-90 575: Aug 30 12:40:21: %LINK-3-UPDOWN:
> > Interface Dot11Radio0, changed state to down
> >Aug 30 12:40:22 ap-gm401-90 576: Aug 30 12:40:22: %LINEPROTO-5-UPDOWN:
> > Line protocol on Interface Dot11Radio0, changed state to down
> >Aug 30 12:40:46 ap-gm401-90 577: Aug 30 12:40:45: %LINK-5-CHANGED:
> > Interface Dot11Radio0, changed state to reset
> >Aug 30 12:40:48 ap-gm401-90 578: Aug 30 12:40:47: %LINK-3-UPDOWN:
> > Interface Dot11Radio0, changed state to up
> >Aug 30 12:40:48 ap-gm401-90 579: Aug 30 12:40:48: %LINEPROTO-5-UPDOWN:
> > Line protocol on Interface Dot11Radio0, changed state to up
> >
> >
> >Any ideas would be greatly appreciated ...

Ther eare tools on the Cisco web site but I am pretty sure that you
need
a login.

"output interpreter"
https://www.cisco.com/pcgi-bin/Suppo...preter/home.pl

Re: Anyone know how to Analyze "crashinfo" files from Cisco access points?
 

There is a traceback message amongst those you posted.

A traceback almost invariably means a bug.

Is the AP350 run IOS or VXworks ?

Do you have a Cisco support contract for your APs ?

Power cycle the AP if you have not already.

Re: Anyone know how to Analyze "crashinfo" files from Cisco access points?
 

Merv wrote:

> There is a traceback message amongst those you posted.

Yes, and a number of them in the crashinfo files as well.

> A traceback almost invariably means a bug.

Thanks, I imagined as much. Incidentally, I turned the radio off on
this unit, and it appears to have stopped crashing. I'm not sure
whether that helps pinpoint the problem in hardware or software, but I
find it interesting.

> Is the AP350 run IOS or VXworks ?

Cisco IOS Software, C350 Software (C350-K9W7-M),
Version 12.3(8)JA, RELEASE SOFTWARE (fc2)

> Do you have a Cisco support contract for your APs ?

Yes, but I wanted to get an idea of what the problem is before
submitting a problem report.

> Power cycle the AP if you have not already.

I did (via PoE), but it didn't help ...

--
----------------------------------------------------------------------
Sylvain Robitaille syl@alcor.concordia.ca

Systems and Network analyst Concordia University
Instructional & Information Technology Montreal, Quebec, Canada
----------------------------------------------------------------------

Re: Anyone know how to Analyze "crashinfo" files from Cisco access points?
 

gene martinez wrote:

> Did you contact Cisco????

I imagine I'll need to do that, but would like to have some idea of the
problem first. Thanks, though.

--
----------------------------------------------------------------------
Sylvain Robitaille syl@alcor.concordia.ca

Systems and Network analyst Concordia University
Instructional & Information Technology Montreal, Quebec, Canada
----------------------------------------------------------------------

Re: Anyone know how to Analyze "crashinfo" files from Cisco access points?
 

Bod43@hotmail.co.uk wrote:

> Ther eare tools on the Cisco web site but I am pretty sure that you
> need a login.
>
> "output interpreter"
> https://www.cisco.com/pcgi-bin/Suppo...preter/home.pl

Thanks for that pointer. We do have a support contract, and I'll see if
downloading such tools is included. This might help ...

--
----------------------------------------------------------------------
Sylvain Robitaille syl@alcor.concordia.ca

Systems and Network analyst Concordia University
Instructional & Information Technology Montreal, Quebec, Canada
----------------------------------------------------------------------

Re: Anyone know how to Analyze "crashinfo" files from Cisco access points?
 

12.3(8)JA is under software advisory and has been replaced by
12.3(8)JA2.

One of the first things the Cisco TAC will ask you to do is to upgrade
to 12.3(8)JA2

Do you have other AP350 running this release?

How long had the AP350 with the problem been running with 12.3(8)JA?

What release was it running before that ?

Re: Anyone know how to Analyze "crashinfo" files from Cisco access points?
 

Merv wrote:

> One of the first things the Cisco TAC will ask you to do is to upgrade
> to 12.3(8)JA2

Sigh. That doesn't surprise me if the new version is available; there
is some possibility that it will resolve the problem, given the symptom
strongly suggests "software bug".

> Do you have other AP350 running this release?

Another 112 of them, yes.

> How long had the AP350 with the problem been running with 12.3(8)JA?

Months. I don't recall exactly when I performed the upgrade (I do have
that recorded somewhere, though I'm not at work as I write this), but it
was late March.

> What release was it running before that ?

Based on archived configuration files, I can say for sure that it was
running IOS, but prior to upgrading all the access points I had numbers
of access points running different versions of IOS, and some running
different versions of VxWorks (if that doesn't suggest "inherited
installation" to you, allow me to disclaim that "it was like that when I
got it!"). I don't recall having made an exact list then of which
access points were running what software versions (and have already
wished I had ...)

Once I had a good grasp of the size of the installation, and the different
models of equipment we have (AP350s, AP1100s, and AP1210s), with which
software versions we were running, I undertook first to convert all
VxWorks access points to IOS, then to upgrade all access points to the
then-latest version of IOS. I've had very little trouble since (two
access points that were routinely power-cycling prior to upgrades have
still been exhibiting that behaviour since). The problem I'm reporting
here is very sudden.

I'll have a look for 12.3(8)JA2, and peruse release notes ...

Thanks.

--
----------------------------------------------------------------------
Sylvain Robitaille syl@alcor.concordia.ca

Systems and Network analyst Concordia University
Instructional & Information Technology Montreal, Quebec, Canada
----------------------------------------------------------------------

Re: Anyone know how to Analyze "crashinfo" files from Cisco access points?
 

> > Did you contact Cisco????
>
> I imagine I'll need to do that, but would like to have some idea of the
> problem first. Thanks, though.

Where this dlink or some other vendor with "free" support that'd have merit.
But if you've got that many cisco units you'd probably be better off calling
them FIRST instead of waiting for the usenet rabble to chime in...