Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Computing > Cisco > Lightning and Switches

Reply
Thread Tools

Lightning and Switches

 
 
JF Mezei
Guest
Posts: n/a
 
      06-09-2007
Lighning hit a 12kv line very near. (about 30-40m away). This was
downstream from the transformer that feeds me electricity and there was
a fuse between the point of impact and our transformer.

A mac with long ethernet cable had its ethernet interface zapped. But
otherwise the mac still works.

It was plugged into port 10 of a Cisco 2924XL switch.

The switch rebooted, but amber lights remained on ports 9, 10, 11 and 12.

9 and 11 were local devices (printer and one mac)
10 was the distant imac. All these 3 were turned off at the time of the
event. These 4 ports are inoperative.

The other ports seem to function. But the main status light remains amber.

is it correct to assume that the hardware drives groups of 4 ports so if
one port goes bezerk, it brings down the other 3 in that group ?

Out of curiosity, what IOS command would tell me about the error in
those ports ?

show interface doesn't give anything of use:
SWITCH1#show int fa0/10
FastEthernet0/10 is down, line protocol is down
Hardware is Fast Ethernet, address is 0004.4dfd.1a8a (bia 0004.4dfd.1a8a)
Description: IMAC
MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive not set
Auto-duplex , Auto Speed , 100BaseTX/FX
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output 02:40:49, output hang never
Last clearing of "show interface" counters never
Queueing strategy: fifo
Output queue 0/40, 0 drops; input queue 0/75, 0 drops
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
0 packets input, 0 bytes
Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast
0 input packets with dribble condition detected
0 packets output, 0 bytes, 0 underruns
0 output errors, 0 collisions, 1 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out
SWITCH1#

What commands would show me the errors on the switch ? (I currently
can't look at the serial port while the switch is booting because my
cluster freezes during that time).

Would there be a way to disable those 4 ports so that the unit would
then report a healthy status ?


Also, out of curiosity, would nearby lightning induce currents on long
ethernet cables which would have explained this ? (devices near the
switch in the basement were not damaged).
 
Reply With Quote
 
 
 
 
Doug McIntyre
Guest
Posts: n/a
 
      06-09-2007
JF Mezei <(E-Mail Removed)> writes:
>Lighning hit a 12kv line very near. (about 30-40m away). This was
>downstream from the transformer that feeds me electricity and there was
>a fuse between the point of impact and our transformer.


>A mac with long ethernet cable had its ethernet interface zapped. But
>otherwise the mac still works.


Lucky. You can never predict what a surge will do.

>It was plugged into port 10 of a Cisco 2924XL switch.


>The switch rebooted, but amber lights remained on ports 9, 10, 11 and 12.

....
>is it correct to assume that the hardware drives groups of 4 ports so if
>one port goes bezerk, it brings down the other 3 in that group ?


Yep I have lots of experience with exactly that failure. On the 2924xl, the
PHY's are in blocks of 4 chips, same on some of the newer lines. Its
probably a blown PHY chip. But they are surface mount quad-flat pack
chips with over a hundred pins. Ie. a surface-mount rework machine is
going to be needed to swap the chip.

>Out of curiosity, what IOS command would tell me about the error in
>those ports ?


During a reboot, you'll probably get the switch complaining about
those 4 ports not passing POST in the log. Sometimes not.


>What commands would show me the errors on the switch ? (I currently
>can't look at the serial port while the switch is booting because my
>cluster freezes during that time).


Once you start doing hardware frying, its not going to be too easy to
report errors on it, when the hardware doesn't behave like you need it to.


>Would there be a way to disable those 4 ports so that the unit would
>then report a healthy status ?


No, you'll need to get it repaired or replaced. Smartnet if you can
still get it on this long EOL'd switch was fairly cheap, $125?

>Also, out of curiosity, would nearby lightning induce currents on long
>ethernet cables which would have explained this ? (devices near the
>switch in the basement were not damaged).


Yep, sure does. Especially the closer ethernet runs are to the
outside. I had a building once I supported that ran all the ethernet
under the roofline. Routinely blows a switch every storm. We had input
as to where the wire should go, but they didn't listen, just went with
the cheapest wiring contract they coul. Guess that decision was a mistake
At least it wasn't ours.

If you ever trench ethernet outside, this is exactly why you'd want fiber in
the ground, instead of copper.

 
Reply With Quote
 
 
 
 
JF Mezei
Guest
Posts: n/a
 
      06-09-2007
Doug McIntyre wrote:
> Once you start doing hardware frying, its not going to be too easy to
> report errors on it, when the hardware doesn't behave like you need it to.


Well, since the switch has amber lights on the block of 4 ports, and
since its own main status light is also amber, I would have thought the
software would know about the problem. (and would think that some error
message would appear when you do an show interface fa0/xx for one of the
faulty ports.

> No, you'll need to get it repaired or replaced. Smartnet if you can
> still get it on this long EOL'd switch was fairly cheap, $125?


Or buy a "new" one from ebay. But I'll call Cisco on monday to find out.

> Yep, sure does. Especially the closer ethernet runs are to the
> outside. I had a building once I supported that ran all the ethernet
> under the roofline. Routinely blows a switch every storm.


Thanks. That was probably it then. I guess I can consider ourselves
lucky we didn't get a direct hit. The neighbours in the back whose
house is just a few metres from the pole that was hit are probably
feeling even luckier.

Half the townhouse complex is still without power. They not only have to
replace their underground transformer, but also replace the underground
cable that goes from thetransformer to that pole. (it is that cable that
was zappep and Hydro guys tols me tha don't splice this type of cable.


> If you ever trench ethernet outside, this is exactly why you'd want fiber in
> the ground, instead of copper.


I hadn't thought about EMI for thunderstorms for that, always tought
about grounding differences for this issue.
 
Reply With Quote
 
Sam Wilson
Guest
Posts: n/a
 
      06-21-2007
In article <4901f$466af408$cef8887a$(E-Mail Removed)>,
JF Mezei <(E-Mail Removed)> wrote:

> Doug McIntyre wrote:
> > If you ever trench ethernet outside, this is exactly why you'd want fiber in
> > the ground, instead of copper.

>
> I hadn't thought about EMI for thunderstorms for that, always tought
> about grounding differences for this issue.


Back when we were doing RS232 and X.25 we had copper runs in one
particular place that used to get lightning strikes. The X.25 PAD used
to suffer from, at best, the tops blown off the RS232 driver chip
packages and at worst holes through the board, like a bullet hole, where
the chip used to be.

Sam
 
Reply With Quote
 
w_tom
Guest
Posts: n/a
 
      06-23-2007
On Jun 9, 2:40 pm, JF Mezei <(E-Mail Removed)> wrote:
> Well, since the switch has amber lights on the block of 4 ports, and
> since its own main status light is also amber, I would have thought the
> software would know about the problem. (and would think that some error
> message would appear when you do an show interface fa0/xx for one of the
> faulty ports.
> ...
>
> Thanks. That was probably it then. I guess I can consider ourselves
> lucky we didn't get a direct hit. The neighbours in the back whose
> house is just a few metres from the pole that was hit are probably
> feeling even luckier.
> ...
>
> I hadn't thought about EMI for thunderstorms for that, always tought
> about grounding differences for this issue.


Your description is typical of a "direct hit". How will be
explained later. From your description, surge electricity flowed
through port 10 and through the connected computer. Remember, to have
electricity; first an incoming and outgoing path must exist. Which
side was the path to earth? Makes no difference whether electronics
were powered on or off.

In another example, a plug-in protector earthed a surge through an
adjacent computer, out via network wire, into network card of another
powered off computer, out via that computer's modem, to earth ground
via phone line. We traced that surge by identifying and replacing
every damaged semiconductor.

Possible that four ports in the Cisco share a common interface
chip. ICs for such ports also come in four or eight port versions.
This is speculation that would explain why surge current through port
10 would damage ports 9 through 12. Dead body would better explain
why damage occurred.

This we do know. It was a direct strike. A second example. A
nearby tree was struck. Campers sleeping tangent to that tree did not
suffer. But two campers sleeping pointed to that tree suffered a
direct strike. Electricity flowed down the tree into earth. Up into
camper's feet and back to earth via heads. Current flows some miles
distant to electrical charges. That was a direct strike to campers
whose body provided a better electrical path.

Ethernet (communication) connections between buildings so easily
suffer damage for same reasons. Third example: one building literally
becomes the lightning rod (or tree) to conduct a surge through
ethernet cable to be earthed in other building. That path to earth is
destructive to electronics - powered on or off - because it easily
overwhelms protection already inside ethernet interfaces.

Many then assume damage is from induced surges - or EMI. Numbers
say otherwise. The concept is called GPR. Solution requires single
point earthing of every incoming wire, short, to a single point
ground. Any wire of any cable not earthed, short, by hardwire or
protector may carry destructive surges inside the building. That other
building simply acts like a lightning rod to carry surges into this
building. Damage for reasons similar to those campers.

Also at risk are four legged animals - for same reasons - a fourth
example. Properly constructed barns install an even better single
point ground - halo or Ufer ground. Single point earthing that
encircles a barn means no current will flow up a cows hind legs and
down its fore legs. This direct strike (when lightning strikes a
nearby tree) is reason for livestock deaths. Learn from that example
to install that inexpensive and effective earthing in new buildings.
However, even that earthing is useless when incoming wires are not
earthed before entering a barn or network center.

Back to the original failure. What was the incoming path of the
surge that was seeking earth ground? Autopsy of 'dead bodies' would
be useful. But finding someone with sufficient electrical knowledge is
difficult.

What we do know: surge found one earthing path via Cisco port 10 and
a computer's ethernet interface. Surge could have been incoming on
AC electric through computer, through Cisco port 10, then ... well,
could ports 9, 11, or 12 conduct to earth? What thing conductive was
the Cisco mounted on? Even furniture could have been part of an
earthing path.

We know a surge entered the building on a wire that was not 'single
point' earthed via hardware or protector. Fields do not provide
sufficient energy to overwhelm a many thousand volt protection
internal in computer and Cisco. How many thousands? Well, IEC
standards now define interface chips that must withstand 2K or 15K
volts. Again, we don't know which were inside those 'dead bodies'.
But we know EMI will not overwhelm that existing and industry standard
protection. It was a direct strike - if only just like those campers
beneath a tree.

How to prevent such damage? Well your telco have been connected to
virtually every building in town. Every building is a lightning rod
that connects surges directly into their switching computer? But
telco installed protection has been standard everywhere since the
beginning of the 20th Century. Preventing damage has been that well
understood for that long. Each incoming wire must make a short
connection to a common earthing electrode. Each wire connects via a
'whole house' type protector so that damage will never happen. Your
damage implies that well proven and properly earthed protection does
not exist in your facilities.

Little difference whether wires are underground or that lightning
struck 30 meters distant. Fifth example: an application note from
Polyphaser:
http://tinyurl.com/38v2dv
> Lightning strikes somewhere across the street close
> to the below grade West cable vault. ... The first line
> of defense is the telco protection panel, but the panel
> must be connected to a low resistance / inductance
> ground. There was no adequate ground available in
> the telephone room.


Those quick to speculate may instead blame damage on EMI. Too many
examples and those numbers are little understood by those who blame
EMI. It was a direct strike that overhwhelmed protection found on all
ethernet ports. Power on or off makes no difference. Solution starts
with what always defines protection - the earthing system.

 
Reply With Quote
 
t0
Guest
Posts: n/a
 
      06-23-2007
On Jun 9, 10:28 am, JF Mezei <(E-Mail Removed)> wrote:
> Lighning hit a 12kv line very near. (about 30-40m away). This was
> downstream from the transformer that feeds me electricity and there was
> a fuse between the point of impact and our transformer.
>
> A mac with long ethernet cable had its ethernet interface zapped. But
> otherwise the mac still works.
>
> It was plugged into port 10 of a Cisco 2924XL switch.
>
> The switch rebooted, but amber lights remained on ports 9, 10, 11 and 12.
>
> 9 and 11 were local devices (printer and one mac)
> 10 was the distant imac. All these 3 were turned off at the time of the
> event. These 4 ports are inoperative.
>
> The other ports seem to function. But the main status light remains amber.
>
> is it correct to assume that the hardware drives groups of 4 ports so if
> one port goes bezerk, it brings down the other 3 in that group ?
>
> Out of curiosity, what IOS command would tell me about the error in
> those ports ?
>
> show interface doesn't give anything of use:
> SWITCH1#show int fa0/10
> FastEthernet0/10 is down, line protocol is down
> Hardware is Fast Ethernet, address is 0004.4dfd.1a8a (bia 0004.4dfd.1a8a)
> Description: IMAC
> MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
> reliability 255/255, txload 1/255, rxload 1/255
> Encapsulation ARPA, loopback not set
> Keepalive not set
> Auto-duplex , Auto Speed , 100BaseTX/FX
> ARP type: ARPA, ARP Timeout 04:00:00
> Last input never, output 02:40:49, output hang never
> Last clearing of "show interface" counters never
> Queueing strategy: fifo
> Output queue 0/40, 0 drops; input queue 0/75, 0 drops
> 5 minute input rate 0 bits/sec, 0 packets/sec
> 5 minute output rate 0 bits/sec, 0 packets/sec
> 0 packets input, 0 bytes
> Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
> 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
> 0 watchdog, 0 multicast
> 0 input packets with dribble condition detected
> 0 packets output, 0 bytes, 0 underruns
> 0 output errors, 0 collisions, 1 interface resets
> 0 babbles, 0 late collision, 0 deferred
> 0 lost carrier, 0 no carrier
> 0 output buffer failures, 0 output buffers swapped out
> SWITCH1#
>
> What commands would show me the errors on the switch ? (I currently
> can't look at the serial port while the switch is booting because my
> cluster freezes during that time).
>
> Would there be a way to disable those 4 ports so that the unit would
> then report a healthy status ?
>
> Also, out of curiosity, would nearby lightning induce currents on long
> ethernet cables which would have explained this ? (devices near the
> switch in the basement were not damaged).


The SHOW HARD or SHOW TECH command will give you all the details, just
look throught it for the info you are looking for.

Regards
Tewfiq

 
Reply With Quote
 
JF Mezei
Guest
Posts: n/a
 
      06-23-2007
t0 wrote:

> The SHOW HARD or SHOW TECH command will give you all the details, just
> look throught it for the info you are looking for.


Thanks. Unfortunatly, they don't seem to show any errors. SHOW TECH
produces a list of the "SHOW INTERFACE fa0/xx" commands (which show no
errors/problems for those 4 ports), yet amber lights are on for those 4
ports, as well as one of the main status lights on the left of the unit
which is amber instead of green.

I am expecting a replacement unit in a couple of days and will then be
able to do more experiements in this damaged switch. (especially look
inside to see if there are traces of the smoke I smelled after lightning
struck).
 
Reply With Quote
 
w_tom
Guest
Posts: n/a
 
      06-23-2007
On Jun 23, 6:45 am, JF Mezei <(E-Mail Removed)> wrote:
> I am expecting a replacement unit in a couple of days and will then be
> able to do more experiements in this damaged switch. (especially look
> inside to see if there are traces of the smoke I smelled after lightning
> struck).


Most all surges leave no visible damage. Visible damage is usually
due to something called 'follow through current'. IOW a surge creates
this tiny little damage. A resulting short circuit permits massive
energy from AC mains to do further damage - create smoke.

Recording IC part numbers and manufacturer logo for ethernet
interface chips would also answer more questions such as how many
ports are controlled by one IC. Also part of a circuit (why ethernet
ports are so robust) would be a transformer between each ethernet wire
and that IC. Electrical characteristics of that transformer (does
galvanic isolation still exist?) may provide further insight.

If not obvious, autopsy is how experience educates from the
'experimental' side of knowledge. Useful experience comes from
repairing things at the component level. Then mate that with
'theoretical' concepts (the other side of knowledge) detailed in IEEE
papers, et al. Reason for doing so is to identify where lightning
entered the facility. Only then is defective earthing identified
(with confidence) and repaired.

 
Reply With Quote
 
Alain Fontaine
Guest
Posts: n/a
 
      06-27-2007
In article (Dans l'article)
<8951e$466a64bf$cef8887a$(E-Mail Removed)>,
JF Mezei <(E-Mail Removed)> wrote (ecrivait) :

> is it correct to assume that the hardware drives groups of 4 ports so if
> one port goes bezerk, it brings down the other 3 in that group ?


Quite typical for a 2924.

> Out of curiosity, what IOS command would tell me about the error in
> those ports ?


Did you try 'show post' after a reboot, or connect a console during the
reboot ?
 
Reply With Quote
 
JF Mezei
Guest
Posts: n/a
 
      06-27-2007
Alain Fontaine wrote:
> Did you try 'show post' after a reboot, or connect a console during the
> reboot ?



Ah ! Merci !

SWITCH1#show post
POST FAILED: FastEthernet0/9 failed front-end loopback test
POST FAILED: FastEthernet0/10 failed front-end loopback test
POST FAILED: FastEthernet0/11 failed front-end loopback test
POST FAILED: FastEthernet0/12 failed front-end loopback test
SWITCH1#


So that was the magic command to show the errors.

Replacement switch coming in today, so I should be able to then do a
good post mortem on the damaged switch !

(BTW, I also lost serial ports on 2 computers.)
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Cisco Switches vs HP ProCurve Switches Greg Cisco 5 07-01-2013 01:48 AM
switches, spanning tree question regarding designated ports and switches alefveld@versatel.nl Cisco 1 12-30-2008 01:24 AM
Cisco Aironet 1490 wireless bridge and lightning Rik Cisco 0 11-04-2005 03:55 PM
DVD Verdict reviews: LIGHTNING BUG, ELFEN LIED: VECTOR TWO (VOLUME 2), and more! DVD Verdict DVD Video 0 09-01-2005 08:12 AM
native vlan mismatch on 2 2924 switches w/ only 1 vlan defined (same on both switches) avraham shir-el Cisco 4 07-20-2004 08:08 AM



Advertisments