Anyone know how to Analyze "crashinfo" files from Cisco access points?

Discussion in 'Cisco' started by Sylvain Robitaille, Aug 30, 2006.

  1. Hello ...

    We have a number of Cisco AP350 wireless access points thoughout our
    network (as well as some AP1100s and AP1210s, but these aren't involved
    in the problem that's causing me to send this message).

    One of them seems to have started periodically crashing and resetting
    this past Monday evening (2006/08/28, 21:57:51, according to the
    timestamp on the crashinfo file), and is presently apparently looping
    between functioning and not. I have 10 crashinfo files and an r0.rcore
    file from it, and would like to try and figure out what the problem is.

    Can anyone point me to information regarding how these should be used
    for crash analysis?

    The syslog from this device is presently cycling through the following
    messages. I think I'm going to disconnect the device at this point ...

    Aug 30 12:40:21 ap-gm401-90 573: Aug 30 12:40:21: %DOT11-2-RADIO_FAILED:
    Interface Dot11Radio0, failed - cmd busy
    Aug 30 12:40:21 ap-gm401-90 574: -Traceback= 4DC04 4511C4 45110C 4532B0
    6AA4DC 6AA778 6AF00C 4503EC 4507C8 450B1C 1529D8 157988
    Aug 30 12:40:22 ap-gm401-90 575: Aug 30 12:40:21: %LINK-3-UPDOWN:
    Interface Dot11Radio0, changed state to down
    Aug 30 12:40:22 ap-gm401-90 576: Aug 30 12:40:22: %LINEPROTO-5-UPDOWN:
    Line protocol on Interface Dot11Radio0, changed state to down
    Aug 30 12:40:46 ap-gm401-90 577: Aug 30 12:40:45: %LINK-5-CHANGED:
    Interface Dot11Radio0, changed state to reset
    Aug 30 12:40:48 ap-gm401-90 578: Aug 30 12:40:47: %LINK-3-UPDOWN:
    Interface Dot11Radio0, changed state to up
    Aug 30 12:40:48 ap-gm401-90 579: Aug 30 12:40:48: %LINEPROTO-5-UPDOWN:
    Line protocol on Interface Dot11Radio0, changed state to up


    Any ideas would be greatly appreciated ...

    --
    ----------------------------------------------------------------------
    Sylvain Robitaille

    Systems and Network analyst Concordia University
    Instructional & Information Technology Montreal, Quebec, Canada
    ----------------------------------------------------------------------
    Sylvain Robitaille, Aug 30, 2006
    #1
    1. Advertising

  2. Did you contact Cisco????

    Sylvain Robitaille <> wrote:

    >Hello ...
    >
    >We have a number of Cisco AP350 wireless access points thoughout our
    >network (as well as some AP1100s and AP1210s, but these aren't involved
    >in the problem that's causing me to send this message).
    >
    >One of them seems to have started periodically crashing and resetting
    >this past Monday evening (2006/08/28, 21:57:51, according to the
    >timestamp on the crashinfo file), and is presently apparently looping
    >between functioning and not. I have 10 crashinfo files and an r0.rcore
    >file from it, and would like to try and figure out what the problem is.
    >
    >Can anyone point me to information regarding how these should be used
    >for crash analysis?
    >
    >The syslog from this device is presently cycling through the following
    >messages. I think I'm going to disconnect the device at this point ...
    >
    >Aug 30 12:40:21 ap-gm401-90 573: Aug 30 12:40:21: %DOT11-2-RADIO_FAILED:
    > Interface Dot11Radio0, failed - cmd busy
    >Aug 30 12:40:21 ap-gm401-90 574: -Traceback= 4DC04 4511C4 45110C 4532B0
    > 6AA4DC 6AA778 6AF00C 4503EC 4507C8 450B1C 1529D8 157988
    >Aug 30 12:40:22 ap-gm401-90 575: Aug 30 12:40:21: %LINK-3-UPDOWN:
    > Interface Dot11Radio0, changed state to down
    >Aug 30 12:40:22 ap-gm401-90 576: Aug 30 12:40:22: %LINEPROTO-5-UPDOWN:
    > Line protocol on Interface Dot11Radio0, changed state to down
    >Aug 30 12:40:46 ap-gm401-90 577: Aug 30 12:40:45: %LINK-5-CHANGED:
    > Interface Dot11Radio0, changed state to reset
    >Aug 30 12:40:48 ap-gm401-90 578: Aug 30 12:40:47: %LINK-3-UPDOWN:
    > Interface Dot11Radio0, changed state to up
    >Aug 30 12:40:48 ap-gm401-90 579: Aug 30 12:40:48: %LINEPROTO-5-UPDOWN:
    > Line protocol on Interface Dot11Radio0, changed state to up
    >
    >
    >Any ideas would be greatly appreciated ...
    >
    >--
    >----------------------------------------------------------------------
    >Sylvain Robitaille
    >
    >Systems and Network analyst Concordia University
    >Instructional & Information Technology Montreal, Quebec, Canada
    >----------------------------------------------------------------------
    gene martinez, Aug 30, 2006
    #2
    1. Advertising

  3. Sylvain Robitaille

    Guest

    gene martinez wrote:
    > Did you contact Cisco????
    >
    > Sylvain Robitaille <> wrote:
    >
    > >Hello ...
    > >
    > >We have a number of Cisco AP350 wireless access points thoughout our
    > >network (as well as some AP1100s and AP1210s, but these aren't involved
    > >in the problem that's causing me to send this message).
    > >
    > >One of them seems to have started periodically crashing and resetting
    > >this past Monday evening (2006/08/28, 21:57:51, according to the
    > >timestamp on the crashinfo file), and is presently apparently looping
    > >between functioning and not. I have 10 crashinfo files and an r0.rcore
    > >file from it, and would like to try and figure out what the problem is.
    > >
    > >Can anyone point me to information regarding how these should be used
    > >for crash analysis?
    > >
    > >The syslog from this device is presently cycling through the following
    > >messages. I think I'm going to disconnect the device at this point ...
    > >
    > >Aug 30 12:40:21 ap-gm401-90 573: Aug 30 12:40:21: %DOT11-2-RADIO_FAILED:
    > > Interface Dot11Radio0, failed - cmd busy
    > >Aug 30 12:40:21 ap-gm401-90 574: -Traceback= 4DC04 4511C4 45110C 4532B0
    > > 6AA4DC 6AA778 6AF00C 4503EC 4507C8 450B1C 1529D8 157988
    > >Aug 30 12:40:22 ap-gm401-90 575: Aug 30 12:40:21: %LINK-3-UPDOWN:
    > > Interface Dot11Radio0, changed state to down
    > >Aug 30 12:40:22 ap-gm401-90 576: Aug 30 12:40:22: %LINEPROTO-5-UPDOWN:
    > > Line protocol on Interface Dot11Radio0, changed state to down
    > >Aug 30 12:40:46 ap-gm401-90 577: Aug 30 12:40:45: %LINK-5-CHANGED:
    > > Interface Dot11Radio0, changed state to reset
    > >Aug 30 12:40:48 ap-gm401-90 578: Aug 30 12:40:47: %LINK-3-UPDOWN:
    > > Interface Dot11Radio0, changed state to up
    > >Aug 30 12:40:48 ap-gm401-90 579: Aug 30 12:40:48: %LINEPROTO-5-UPDOWN:
    > > Line protocol on Interface Dot11Radio0, changed state to up
    > >
    > >
    > >Any ideas would be greatly appreciated ...


    Ther eare tools on the Cisco web site but I am pretty sure that you
    need
    a login.

    "output interpreter"
    https://www.cisco.com/pcgi-bin/Support/OutputInterpreter/home.pl
    , Aug 30, 2006
    #3
  4. Sylvain Robitaille

    Merv Guest

    There is a traceback message amongst those you posted.

    A traceback almost invariably means a bug.

    Is the AP350 run IOS or VXworks ?

    Do you have a Cisco support contract for your APs ?

    Power cycle the AP if you have not already.
    Merv, Aug 30, 2006
    #4
  5. Merv wrote:

    > There is a traceback message amongst those you posted.


    Yes, and a number of them in the crashinfo files as well.

    > A traceback almost invariably means a bug.


    Thanks, I imagined as much. Incidentally, I turned the radio off on
    this unit, and it appears to have stopped crashing. I'm not sure
    whether that helps pinpoint the problem in hardware or software, but I
    find it interesting.

    > Is the AP350 run IOS or VXworks ?


    Cisco IOS Software, C350 Software (C350-K9W7-M),
    Version 12.3(8)JA, RELEASE SOFTWARE (fc2)

    > Do you have a Cisco support contract for your APs ?


    Yes, but I wanted to get an idea of what the problem is before
    submitting a problem report.

    > Power cycle the AP if you have not already.


    I did (via PoE), but it didn't help ...

    --
    ----------------------------------------------------------------------
    Sylvain Robitaille

    Systems and Network analyst Concordia University
    Instructional & Information Technology Montreal, Quebec, Canada
    ----------------------------------------------------------------------
    Sylvain Robitaille, Aug 30, 2006
    #5
  6. gene martinez wrote:

    > Did you contact Cisco????


    I imagine I'll need to do that, but would like to have some idea of the
    problem first. Thanks, though.

    --
    ----------------------------------------------------------------------
    Sylvain Robitaille

    Systems and Network analyst Concordia University
    Instructional & Information Technology Montreal, Quebec, Canada
    ----------------------------------------------------------------------
    Sylvain Robitaille, Aug 30, 2006
    #6
  7. wrote:

    > Ther eare tools on the Cisco web site but I am pretty sure that you
    > need a login.
    >
    > "output interpreter"
    > https://www.cisco.com/pcgi-bin/Support/OutputInterpreter/home.pl


    Thanks for that pointer. We do have a support contract, and I'll see if
    downloading such tools is included. This might help ...

    --
    ----------------------------------------------------------------------
    Sylvain Robitaille

    Systems and Network analyst Concordia University
    Instructional & Information Technology Montreal, Quebec, Canada
    ----------------------------------------------------------------------
    Sylvain Robitaille, Aug 30, 2006
    #7
  8. Sylvain Robitaille

    Merv Guest

    12.3(8)JA is under software advisory and has been replaced by
    12.3(8)JA2.

    One of the first things the Cisco TAC will ask you to do is to upgrade
    to 12.3(8)JA2

    Do you have other AP350 running this release?

    How long had the AP350 with the problem been running with 12.3(8)JA?

    What release was it running before that ?
    Merv, Aug 30, 2006
    #8
  9. Merv wrote:

    > One of the first things the Cisco TAC will ask you to do is to upgrade
    > to 12.3(8)JA2


    Sigh. That doesn't surprise me if the new version is available; there
    is some possibility that it will resolve the problem, given the symptom
    strongly suggests "software bug".

    > Do you have other AP350 running this release?


    Another 112 of them, yes.

    > How long had the AP350 with the problem been running with 12.3(8)JA?


    Months. I don't recall exactly when I performed the upgrade (I do have
    that recorded somewhere, though I'm not at work as I write this), but it
    was late March.

    > What release was it running before that ?


    Based on archived configuration files, I can say for sure that it was
    running IOS, but prior to upgrading all the access points I had numbers
    of access points running different versions of IOS, and some running
    different versions of VxWorks (if that doesn't suggest "inherited
    installation" to you, allow me to disclaim that "it was like that when I
    got it!"). I don't recall having made an exact list then of which
    access points were running what software versions (and have already
    wished I had ...)

    Once I had a good grasp of the size of the installation, and the different
    models of equipment we have (AP350s, AP1100s, and AP1210s), with which
    software versions we were running, I undertook first to convert all
    VxWorks access points to IOS, then to upgrade all access points to the
    then-latest version of IOS. I've had very little trouble since (two
    access points that were routinely power-cycling prior to upgrades have
    still been exhibiting that behaviour since). The problem I'm reporting
    here is very sudden.

    I'll have a look for 12.3(8)JA2, and peruse release notes ...

    Thanks.

    --
    ----------------------------------------------------------------------
    Sylvain Robitaille

    Systems and Network analyst Concordia University
    Instructional & Information Technology Montreal, Quebec, Canada
    ----------------------------------------------------------------------
    Sylvain Robitaille, Aug 31, 2006
    #9
  10. Sylvain Robitaille

    Bill Kearney Guest

    > > Did you contact Cisco????
    >
    > I imagine I'll need to do that, but would like to have some idea of the
    > problem first. Thanks, though.


    Where this dlink or some other vendor with "free" support that'd have merit.
    But if you've got that many cisco units you'd probably be better off calling
    them FIRST instead of waiting for the usenet rabble to chime in...
    Bill Kearney, Aug 31, 2006
    #10
  11. Sylvain Robitaille

    Merv Guest

    For 12.3(8)JA2, take a close look at CSCsb00606

    You might be better to drop back to 12.3(7)JA3 on the troubled unit.

    Also note that the first maintenance release usually gets deferred (
    read junked):

    12.3.8-JA2

    12.3.7-JA3
    12.3.7-JA2
    12.3.7-JA1

    12.3.4-JA2
    12.3.4-JA1
    12.3.4-JA

    12.3.2-JA6
    12.3.2-JA5
    12.3.2-JA2

    I normally like to wait for the third or fourth maintenenace release
    depending on the technology.
    Merv, Aug 31, 2006
    #11
  12. Merv wrote:

    > For 12.3(8)JA2, take a close look at CSCsb00606


    Hrmmm... Thanks for that.

    > You might be better to drop back to 12.3(7)JA3 on the troubled unit.


    Will have a look. Thanks again.

    --
    ----------------------------------------------------------------------
    Sylvain Robitaille

    Systems and Network analyst Concordia University
    Instructional & Information Technology Montreal, Quebec, Canada
    ----------------------------------------------------------------------
    Sylvain Robitaille, Aug 31, 2006
    #12
  13. Sylvain Robitaille

    Guest

    Merv wrote:
    > For 12.3(8)JA2, take a close look at CSCsb00606
    >
    > You might be better to drop back to 12.3(7)JA3 on the troubled unit.
    >
    > Also note that the first maintenance release usually gets deferred (
    > read junked):
    >
    > 12.3.8-JA2
    >
    > 12.3.7-JA3
    > 12.3.7-JA2
    > 12.3.7-JA1
    >
    > 12.3.4-JA2
    > 12.3.4-JA1
    > 12.3.4-JA
    >
    > 12.3.2-JA6
    > 12.3.2-JA5
    > 12.3.2-JA2
    >
    > I normally like to wait for the third or fourth maintenenace release
    > depending on the technology.


    I may have the wrong end of the stick here, do you mean for
    example 12.3.2-JA4 is the fourth maint. rel?

    I am sure that Merv has more Cisco experience than I have
    however, I do not like the idea of what amount to unplanned
    releases. ...JAx where x = 1, 2, 3, ... are I understand
    fixes that are applied to software that is supposed to be working.

    I fancy releases that do not need so much fixing.
    12.3.2-JA6 just makes my skin crawl.

    As I say this is only a wild guess with absolutely
    no evidence either way. I just feel that it looks like a Friday
    afternoon product. Maybe it can be fixed, but why not just
    stay away if that is an option?
    , Sep 1, 2006
    #13
  14. Sylvain Robitaille

    Merv Guest

    Well in the "old days" before IOS version "trains" the number in
    between the ( ) was the maintenance release - things were much simpler
    than.

    12.3JA is known as a STED - specific technology early deployment
    release - and as far as I can tell the number after the JA sure looks
    and feels like a maintenance as one can tell by looking carefully at
    the release notes.
    Merv, Sep 1, 2006
    #14
  15. On Thu, 31 Aug 2006 14:51:39 +0000 (UTC), Sylvain Robitaille <> wrote:

    ~ Merv wrote:
    ~
    ~ > For 12.3(8)JA2, take a close look at CSCsb00606
    ~
    ~ Hrmmm... Thanks for that.

    CSCsb00606 was actually filed against 12.3(2)JA. It doesn't look like
    anything's happened with that one in a year. I don't suppose it would
    pertain to current IOS.

    ~ > You might be better to drop back to 12.3(7)JA3 on the troubled unit.
    ~
    ~ Will have a look. Thanks again.

    Both 12.3(7)JA3 and 12.3(8)JA2 are good solid releases. Btw, we just
    posted 12.3(8)JEA today, if you really want to run the latest and
    greatest.

    That said, it's not clear to me (after scanning known bugs) whether
    the condition you encountered that triggered the radio failure is going
    to be addressed by upgrading software.

    Regards,

    Aaron
    Aaron Leonard, Sep 1, 2006
    #15
  16. Merv wrote:

    > Well in the "old days" before IOS version "trains" the number in
    > between the ( ) was the maintenance release - things were much simpler
    > than.
    >
    > 12.3JA is known as a STED - specific technology early deployment
    > release - and as far as I can tell the number after the JA sure looks
    > and feels like a maintenance as one can tell by looking carefully at
    > the release notes.


    To original question - try this website. I know the proprietor is on
    holiday from Monday for a couple of weeks but might respond after this.

    www.net2netsolutions.co.uk

    All the best,

    Martyn (you can tell him I advised you contact him if you like!).
    Martyn Thompson, Sep 2, 2006
    #16
  17. Sylvain Robitaille

    Guest

    Merv wrote:
    > Well in the "old days" before IOS version "trains" the number in
    > between the ( ) was the maintenance release - things were much simpler
    > than.
    >
    > 12.3JA is known as a STED - specific technology early deployment
    > release - and as far as I can tell the number after the JA sure looks
    > and feels like a maintenance as one can tell by looking carefully at
    > the release notes.


    I think that it is like this:-

    a.b(cx)JAy

    JA tells us that it is a special (required for for many purposes)
    Avoid specials of you can
    These releases are based on a mainstream release
    in this case a.b(cx).

    a - Major release
    b - Minor release
    (c) - Maintenance release - in some trains can contain new features
    x - Re-release of a.b(c) [On which JA is based]
    y - Re-release of a.b(c)JA - changes not affecting base a.b(c)


    x is always a lower case letter
    y is always a number


    As I say, I worry about re-releases but have no direct IOS derived
    evidence on which to base it.
    , Sep 2, 2006
    #17
  18. Sylvain Robitaille

    Merv Guest

    Those releases as you call them are maintenance release pure and
    simple.

    Look at the release notes for 12.3(8)JA2 and you will see that DDTS
    listed that are resolved between it and the previosu release.
    Merv, Sep 2, 2006
    #18
  19. Aaron Leonard wrote:

    > Both 12.3(7)JA3 and 12.3(8)JA2 are good solid releases.


    In my experience, that's also been true of 12.3(8)JA, save for a small
    number of units that have been acting up (the one we're discussing here
    that failed only recently, and a couple of others that have been acting
    up since before the upgrade to 12.3(8)JA).

    > Btw, we just posted 12.3(8)JEA today, if you really want to run the
    > latest and greatest.


    No, I'm more interested in running the same version across all our access
    points. Only when it becomes necessary to upgrade (due to serious bugs,
    or specific new features, or as I did earlier this year, to bring all
    our units up to the same version) do I start looking at upgrading to
    the most current version (or ${most_current}-1, if ${most_current}
    hasn't been around long enough to get a bit of a shakedown).

    > That said, it's not clear to me (after scanning known bugs) whether
    > the condition you encountered that triggered the radio failure is going
    > to be addressed by upgrading software.


    You're probably right. We've started discussing replacing that unit
    with a newer (AP1200) one. That's very likely what I'll do.

    --
    ----------------------------------------------------------------------
    Sylvain Robitaille

    Systems and Network analyst Concordia University
    Instructional & Information Technology Montreal, Quebec, Canada
    ----------------------------------------------------------------------
    Sylvain Robitaille, Sep 5, 2006
    #19
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. How To Analyze CSMA/CD

    , Apr 5, 2006, in forum: Cisco
    Replies:
    0
    Views:
    459
  2. How To Analyze CSMA/CD

    , Apr 5, 2006, in forum: Cisco
    Replies:
    1
    Views:
    1,660
  3. Cynthia K.

    Help analyze HijackThis logfile, Please

    Cynthia K., Jul 12, 2004, in forum: Computer Support
    Replies:
    7
    Views:
    529
    ┬░Mike┬░
    Jul 15, 2004
  4. Nate Goulet
    Replies:
    9
    Views:
    1,225
    Nate Goulet
    Dec 7, 2007
  5. Scott Jackson

    ASA5505 and crashinfo file

    Scott Jackson, Jun 2, 2008, in forum: Cisco
    Replies:
    0
    Views:
    1,588
    Scott Jackson
    Jun 2, 2008
Loading...

Share This Page