Lightning and Switches

Discussion in 'Cisco' started by JF Mezei, Jun 9, 2007.

  1. JF Mezei

    JF Mezei Guest

    Lighning hit a 12kv line very near. (about 30-40m away). This was
    downstream from the transformer that feeds me electricity and there was
    a fuse between the point of impact and our transformer.

    A mac with long ethernet cable had its ethernet interface zapped. But
    otherwise the mac still works.

    It was plugged into port 10 of a Cisco 2924XL switch.

    The switch rebooted, but amber lights remained on ports 9, 10, 11 and 12.

    9 and 11 were local devices (printer and one mac)
    10 was the distant imac. All these 3 were turned off at the time of the
    event. These 4 ports are inoperative.

    The other ports seem to function. But the main status light remains amber.

    is it correct to assume that the hardware drives groups of 4 ports so if
    one port goes bezerk, it brings down the other 3 in that group ?

    Out of curiosity, what IOS command would tell me about the error in
    those ports ?

    show interface doesn't give anything of use:
    SWITCH1#show int fa0/10
    FastEthernet0/10 is down, line protocol is down
    Hardware is Fast Ethernet, address is 0004.4dfd.1a8a (bia 0004.4dfd.1a8a)
    Description: IMAC
    MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
    reliability 255/255, txload 1/255, rxload 1/255
    Encapsulation ARPA, loopback not set
    Keepalive not set
    Auto-duplex , Auto Speed , 100BaseTX/FX
    ARP type: ARPA, ARP Timeout 04:00:00
    Last input never, output 02:40:49, output hang never
    Last clearing of "show interface" counters never
    Queueing strategy: fifo
    Output queue 0/40, 0 drops; input queue 0/75, 0 drops
    5 minute input rate 0 bits/sec, 0 packets/sec
    5 minute output rate 0 bits/sec, 0 packets/sec
    0 packets input, 0 bytes
    Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
    0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
    0 watchdog, 0 multicast
    0 input packets with dribble condition detected
    0 packets output, 0 bytes, 0 underruns
    0 output errors, 0 collisions, 1 interface resets
    0 babbles, 0 late collision, 0 deferred
    0 lost carrier, 0 no carrier
    0 output buffer failures, 0 output buffers swapped out
    SWITCH1#

    What commands would show me the errors on the switch ? (I currently
    can't look at the serial port while the switch is booting because my
    cluster freezes during that time).

    Would there be a way to disable those 4 ports so that the unit would
    then report a healthy status ?


    Also, out of curiosity, would nearby lightning induce currents on long
    ethernet cables which would have explained this ? (devices near the
    switch in the basement were not damaged).
     
    JF Mezei, Jun 9, 2007
    #1
    1. Advertising

  2. JF Mezei <> writes:
    >Lighning hit a 12kv line very near. (about 30-40m away). This was
    >downstream from the transformer that feeds me electricity and there was
    >a fuse between the point of impact and our transformer.


    >A mac with long ethernet cable had its ethernet interface zapped. But
    >otherwise the mac still works.


    Lucky. :) You can never predict what a surge will do.

    >It was plugged into port 10 of a Cisco 2924XL switch.


    >The switch rebooted, but amber lights remained on ports 9, 10, 11 and 12.

    ....
    >is it correct to assume that the hardware drives groups of 4 ports so if
    >one port goes bezerk, it brings down the other 3 in that group ?


    Yep I have lots of experience with exactly that failure. On the 2924xl, the
    PHY's are in blocks of 4 chips, same on some of the newer lines. Its
    probably a blown PHY chip. But they are surface mount quad-flat pack
    chips with over a hundred pins. Ie. a surface-mount rework machine is
    going to be needed to swap the chip.

    >Out of curiosity, what IOS command would tell me about the error in
    >those ports ?


    During a reboot, you'll probably get the switch complaining about
    those 4 ports not passing POST in the log. Sometimes not.


    >What commands would show me the errors on the switch ? (I currently
    >can't look at the serial port while the switch is booting because my
    >cluster freezes during that time).


    Once you start doing hardware frying, its not going to be too easy to
    report errors on it, when the hardware doesn't behave like you need it to.


    >Would there be a way to disable those 4 ports so that the unit would
    >then report a healthy status ?


    No, you'll need to get it repaired or replaced. Smartnet if you can
    still get it on this long EOL'd switch was fairly cheap, $125?

    >Also, out of curiosity, would nearby lightning induce currents on long
    >ethernet cables which would have explained this ? (devices near the
    >switch in the basement were not damaged).


    Yep, sure does. Especially the closer ethernet runs are to the
    outside. I had a building once I supported that ran all the ethernet
    under the roofline. Routinely blows a switch every storm. We had input
    as to where the wire should go, but they didn't listen, just went with
    the cheapest wiring contract they coul. Guess that decision was a mistake :)
    At least it wasn't ours.

    If you ever trench ethernet outside, this is exactly why you'd want fiber in
    the ground, instead of copper.
     
    Doug McIntyre, Jun 9, 2007
    #2
    1. Advertising

  3. JF Mezei

    JF Mezei Guest

    Doug McIntyre wrote:
    > Once you start doing hardware frying, its not going to be too easy to
    > report errors on it, when the hardware doesn't behave like you need it to.


    Well, since the switch has amber lights on the block of 4 ports, and
    since its own main status light is also amber, I would have thought the
    software would know about the problem. (and would think that some error
    message would appear when you do an show interface fa0/xx for one of the
    faulty ports.

    > No, you'll need to get it repaired or replaced. Smartnet if you can
    > still get it on this long EOL'd switch was fairly cheap, $125?


    Or buy a "new" one from ebay. But I'll call Cisco on monday to find out.

    > Yep, sure does. Especially the closer ethernet runs are to the
    > outside. I had a building once I supported that ran all the ethernet
    > under the roofline. Routinely blows a switch every storm.


    Thanks. That was probably it then. I guess I can consider ourselves
    lucky we didn't get a direct hit. The neighbours in the back whose
    house is just a few metres from the pole that was hit are probably
    feeling even luckier.

    Half the townhouse complex is still without power. They not only have to
    replace their underground transformer, but also replace the underground
    cable that goes from thetransformer to that pole. (it is that cable that
    was zappep and Hydro guys tols me tha don't splice this type of cable.


    > If you ever trench ethernet outside, this is exactly why you'd want fiber in
    > the ground, instead of copper.


    I hadn't thought about EMI for thunderstorms for that, always tought
    about grounding differences for this issue.
     
    JF Mezei, Jun 9, 2007
    #3
  4. JF Mezei

    Sam Wilson Guest

    In article <4901f$466af408$cef8887a$>,
    JF Mezei <> wrote:

    > Doug McIntyre wrote:
    > > If you ever trench ethernet outside, this is exactly why you'd want fiber in
    > > the ground, instead of copper.

    >
    > I hadn't thought about EMI for thunderstorms for that, always tought
    > about grounding differences for this issue.


    Back when we were doing RS232 and X.25 we had copper runs in one
    particular place that used to get lightning strikes. The X.25 PAD used
    to suffer from, at best, the tops blown off the RS232 driver chip
    packages and at worst holes through the board, like a bullet hole, where
    the chip used to be.

    Sam
     
    Sam Wilson, Jun 21, 2007
    #4
  5. JF Mezei

    w_tom Guest

    On Jun 9, 2:40 pm, JF Mezei <> wrote:
    > Well, since the switch has amber lights on the block of 4 ports, and
    > since its own main status light is also amber, I would have thought the
    > software would know about the problem. (and would think that some error
    > message would appear when you do an show interface fa0/xx for one of the
    > faulty ports.
    > ...
    >
    > Thanks. That was probably it then. I guess I can consider ourselves
    > lucky we didn't get a direct hit. The neighbours in the back whose
    > house is just a few metres from the pole that was hit are probably
    > feeling even luckier.
    > ...
    >
    > I hadn't thought about EMI for thunderstorms for that, always tought
    > about grounding differences for this issue.


    Your description is typical of a "direct hit". How will be
    explained later. From your description, surge electricity flowed
    through port 10 and through the connected computer. Remember, to have
    electricity; first an incoming and outgoing path must exist. Which
    side was the path to earth? Makes no difference whether electronics
    were powered on or off.

    In another example, a plug-in protector earthed a surge through an
    adjacent computer, out via network wire, into network card of another
    powered off computer, out via that computer's modem, to earth ground
    via phone line. We traced that surge by identifying and replacing
    every damaged semiconductor.

    Possible that four ports in the Cisco share a common interface
    chip. ICs for such ports also come in four or eight port versions.
    This is speculation that would explain why surge current through port
    10 would damage ports 9 through 12. Dead body would better explain
    why damage occurred.

    This we do know. It was a direct strike. A second example. A
    nearby tree was struck. Campers sleeping tangent to that tree did not
    suffer. But two campers sleeping pointed to that tree suffered a
    direct strike. Electricity flowed down the tree into earth. Up into
    camper's feet and back to earth via heads. Current flows some miles
    distant to electrical charges. That was a direct strike to campers
    whose body provided a better electrical path.

    Ethernet (communication) connections between buildings so easily
    suffer damage for same reasons. Third example: one building literally
    becomes the lightning rod (or tree) to conduct a surge through
    ethernet cable to be earthed in other building. That path to earth is
    destructive to electronics - powered on or off - because it easily
    overwhelms protection already inside ethernet interfaces.

    Many then assume damage is from induced surges - or EMI. Numbers
    say otherwise. The concept is called GPR. Solution requires single
    point earthing of every incoming wire, short, to a single point
    ground. Any wire of any cable not earthed, short, by hardwire or
    protector may carry destructive surges inside the building. That other
    building simply acts like a lightning rod to carry surges into this
    building. Damage for reasons similar to those campers.

    Also at risk are four legged animals - for same reasons - a fourth
    example. Properly constructed barns install an even better single
    point ground - halo or Ufer ground. Single point earthing that
    encircles a barn means no current will flow up a cows hind legs and
    down its fore legs. This direct strike (when lightning strikes a
    nearby tree) is reason for livestock deaths. Learn from that example
    to install that inexpensive and effective earthing in new buildings.
    However, even that earthing is useless when incoming wires are not
    earthed before entering a barn or network center.

    Back to the original failure. What was the incoming path of the
    surge that was seeking earth ground? Autopsy of 'dead bodies' would
    be useful. But finding someone with sufficient electrical knowledge is
    difficult.

    What we do know: surge found one earthing path via Cisco port 10 and
    a computer's ethernet interface. Surge could have been incoming on
    AC electric through computer, through Cisco port 10, then ... well,
    could ports 9, 11, or 12 conduct to earth? What thing conductive was
    the Cisco mounted on? Even furniture could have been part of an
    earthing path.

    We know a surge entered the building on a wire that was not 'single
    point' earthed via hardware or protector. Fields do not provide
    sufficient energy to overwhelm a many thousand volt protection
    internal in computer and Cisco. How many thousands? Well, IEC
    standards now define interface chips that must withstand 2K or 15K
    volts. Again, we don't know which were inside those 'dead bodies'.
    But we know EMI will not overwhelm that existing and industry standard
    protection. It was a direct strike - if only just like those campers
    beneath a tree.

    How to prevent such damage? Well your telco have been connected to
    virtually every building in town. Every building is a lightning rod
    that connects surges directly into their switching computer? But
    telco installed protection has been standard everywhere since the
    beginning of the 20th Century. Preventing damage has been that well
    understood for that long. Each incoming wire must make a short
    connection to a common earthing electrode. Each wire connects via a
    'whole house' type protector so that damage will never happen. Your
    damage implies that well proven and properly earthed protection does
    not exist in your facilities.

    Little difference whether wires are underground or that lightning
    struck 30 meters distant. Fifth example: an application note from
    Polyphaser:
    http://tinyurl.com/38v2dv
    > Lightning strikes somewhere across the street close
    > to the below grade West cable vault. ... The first line
    > of defense is the telco protection panel, but the panel
    > must be connected to a low resistance / inductance
    > ground. There was no adequate ground available in
    > the telephone room.


    Those quick to speculate may instead blame damage on EMI. Too many
    examples and those numbers are little understood by those who blame
    EMI. It was a direct strike that overhwhelmed protection found on all
    ethernet ports. Power on or off makes no difference. Solution starts
    with what always defines protection - the earthing system.
     
    w_tom, Jun 23, 2007
    #5
  6. JF Mezei

    t0 Guest

    On Jun 9, 10:28 am, JF Mezei <> wrote:
    > Lighning hit a 12kv line very near. (about 30-40m away). This was
    > downstream from the transformer that feeds me electricity and there was
    > a fuse between the point of impact and our transformer.
    >
    > A mac with long ethernet cable had its ethernet interface zapped. But
    > otherwise the mac still works.
    >
    > It was plugged into port 10 of a Cisco 2924XL switch.
    >
    > The switch rebooted, but amber lights remained on ports 9, 10, 11 and 12.
    >
    > 9 and 11 were local devices (printer and one mac)
    > 10 was the distant imac. All these 3 were turned off at the time of the
    > event. These 4 ports are inoperative.
    >
    > The other ports seem to function. But the main status light remains amber.
    >
    > is it correct to assume that the hardware drives groups of 4 ports so if
    > one port goes bezerk, it brings down the other 3 in that group ?
    >
    > Out of curiosity, what IOS command would tell me about the error in
    > those ports ?
    >
    > show interface doesn't give anything of use:
    > SWITCH1#show int fa0/10
    > FastEthernet0/10 is down, line protocol is down
    > Hardware is Fast Ethernet, address is 0004.4dfd.1a8a (bia 0004.4dfd.1a8a)
    > Description: IMAC
    > MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
    > reliability 255/255, txload 1/255, rxload 1/255
    > Encapsulation ARPA, loopback not set
    > Keepalive not set
    > Auto-duplex , Auto Speed , 100BaseTX/FX
    > ARP type: ARPA, ARP Timeout 04:00:00
    > Last input never, output 02:40:49, output hang never
    > Last clearing of "show interface" counters never
    > Queueing strategy: fifo
    > Output queue 0/40, 0 drops; input queue 0/75, 0 drops
    > 5 minute input rate 0 bits/sec, 0 packets/sec
    > 5 minute output rate 0 bits/sec, 0 packets/sec
    > 0 packets input, 0 bytes
    > Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
    > 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
    > 0 watchdog, 0 multicast
    > 0 input packets with dribble condition detected
    > 0 packets output, 0 bytes, 0 underruns
    > 0 output errors, 0 collisions, 1 interface resets
    > 0 babbles, 0 late collision, 0 deferred
    > 0 lost carrier, 0 no carrier
    > 0 output buffer failures, 0 output buffers swapped out
    > SWITCH1#
    >
    > What commands would show me the errors on the switch ? (I currently
    > can't look at the serial port while the switch is booting because my
    > cluster freezes during that time).
    >
    > Would there be a way to disable those 4 ports so that the unit would
    > then report a healthy status ?
    >
    > Also, out of curiosity, would nearby lightning induce currents on long
    > ethernet cables which would have explained this ? (devices near the
    > switch in the basement were not damaged).


    The SHOW HARD or SHOW TECH command will give you all the details, just
    look throught it for the info you are looking for.

    Regards
    Tewfiq
     
    t0, Jun 23, 2007
    #6
  7. JF Mezei

    JF Mezei Guest

    t0 wrote:

    > The SHOW HARD or SHOW TECH command will give you all the details, just
    > look throught it for the info you are looking for.


    Thanks. Unfortunatly, they don't seem to show any errors. SHOW TECH
    produces a list of the "SHOW INTERFACE fa0/xx" commands (which show no
    errors/problems for those 4 ports), yet amber lights are on for those 4
    ports, as well as one of the main status lights on the left of the unit
    which is amber instead of green.

    I am expecting a replacement unit in a couple of days and will then be
    able to do more experiements in this damaged switch. (especially look
    inside to see if there are traces of the smoke I smelled after lightning
    struck).
     
    JF Mezei, Jun 23, 2007
    #7
  8. JF Mezei

    w_tom Guest

    On Jun 23, 6:45 am, JF Mezei <> wrote:
    > I am expecting a replacement unit in a couple of days and will then be
    > able to do more experiements in this damaged switch. (especially look
    > inside to see if there are traces of the smoke I smelled after lightning
    > struck).


    Most all surges leave no visible damage. Visible damage is usually
    due to something called 'follow through current'. IOW a surge creates
    this tiny little damage. A resulting short circuit permits massive
    energy from AC mains to do further damage - create smoke.

    Recording IC part numbers and manufacturer logo for ethernet
    interface chips would also answer more questions such as how many
    ports are controlled by one IC. Also part of a circuit (why ethernet
    ports are so robust) would be a transformer between each ethernet wire
    and that IC. Electrical characteristics of that transformer (does
    galvanic isolation still exist?) may provide further insight.

    If not obvious, autopsy is how experience educates from the
    'experimental' side of knowledge. Useful experience comes from
    repairing things at the component level. Then mate that with
    'theoretical' concepts (the other side of knowledge) detailed in IEEE
    papers, et al. Reason for doing so is to identify where lightning
    entered the facility. Only then is defective earthing identified
    (with confidence) and repaired.
     
    w_tom, Jun 23, 2007
    #8
  9. In article (Dans l'article)
    <8951e$466a64bf$cef8887a$>,
    JF Mezei <> wrote (ecrivait) :

    > is it correct to assume that the hardware drives groups of 4 ports so if
    > one port goes bezerk, it brings down the other 3 in that group ?


    Quite typical for a 2924.

    > Out of curiosity, what IOS command would tell me about the error in
    > those ports ?


    Did you try 'show post' after a reboot, or connect a console during the
    reboot ?
     
    Alain Fontaine, Jun 27, 2007
    #9
  10. JF Mezei

    JF Mezei Guest

    Alain Fontaine wrote:
    > Did you try 'show post' after a reboot, or connect a console during the
    > reboot ?



    Ah ! Merci !

    SWITCH1#show post
    POST FAILED: FastEthernet0/9 failed front-end loopback test
    POST FAILED: FastEthernet0/10 failed front-end loopback test
    POST FAILED: FastEthernet0/11 failed front-end loopback test
    POST FAILED: FastEthernet0/12 failed front-end loopback test
    SWITCH1#


    So that was the magic command to show the errors.

    Replacement switch coming in today, so I should be able to then do a
    good post mortem on the damaged switch !

    (BTW, I also lost serial ports on 2 computers.)
     
    JF Mezei, Jun 27, 2007
    #10
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. avraham shir-el
    Replies:
    4
    Views:
    8,556
    avraham shir-el
    Jul 20, 2004
  2. Rik
    Replies:
    0
    Views:
    676
  3. DVD Verdict
    Replies:
    0
    Views:
    513
    DVD Verdict
    Sep 1, 2005
  4. Replies:
    1
    Views:
    859
    donjohnston
    Dec 30, 2008
  5. Greg
    Replies:
    5
    Views:
    8,803
    Sarcasmus
    Jul 1, 2013
Loading...

Share This Page