Catalysts trouble

Hi,
We have problem in our inside extensive network built by Cisco 3350-12 G,
3550 XL, 3508 , 6006
This network in base on 2 Laver with one central router Cat 6006.
From Sunday in our switches on Gigabit interfaces links are forced to down
on any switch.

Logs from switches:
1y11w: %SYS-2-MALLOCFAIL: Memory allocation of 1680 bytes failed from
0x13CC08,
pool I/O, alignment 0

1y11w: %ETHCNTR-3-LOOP_BACK_DETECTED: Loop-back detected on
GigabitEthernet0/xx.
The port is forced to linkdown.

1y11w: %SYS-3-CPUHOG: Task ran for 4476 msec (2481/52), process = Net Input,
PC
= 1529CC.

Any bug ?

I only enable udlp on the switches.

Thanks in advance.
grzybek

Have you upgraded into the last IOS/CatOS code? I noticed your uptime
as 1 Year, 11 weeks.. that's what made me think of this....

Others may disagree but that's a very long time for a switch to be
online without any code upgrades... we upgrade all our switches as
soon as upgrades come out and it hasn't come back to haunt us yet...

"grzybek" <> wrote in message news:<bvl00j$1j7b$>...
> Hi,
> We have problem in our inside extensive network built by Cisco 3350-12 G,
> 3550 XL, 3508 , 6006
> This network in base on 2 Laver with one central router Cat 6006.
> From Sunday in our switches on Gigabit interfaces links are forced to down
> on any switch.
>
> Logs from switches:
> 1y11w: %SYS-2-MALLOCFAIL: Memory allocation of 1680 bytes failed from
> 0x13CC08,
> pool I/O, alignment 0
>
> 1y11w: %ETHCNTR-3-LOOP_BACK_DETECTED: Loop-back detected on
> GigabitEthernet0/xx.
> The port is forced to linkdown.
>
> 1y11w: %SYS-3-CPUHOG: Task ran for 4476 msec (2481/52), process = Net Input,
> PC
> = 1529CC.
>
> Any bug ?
>
> I only enable udlp on the switches.
>
> Thanks in advance.
> grzybek

(Paul Stewart) writes:

>Have you upgraded into the last IOS/CatOS code? I noticed your uptime
>as 1 Year, 11 weeks.. that's what made me think of this....

>Others may disagree but that's a very long time for a switch to be
>online without any code upgrades... we upgrade all our switches as
>soon as upgrades come out and it hasn't come back to haunt us yet...

This is a somewhat philosophical debate. You're inducing a down time for
*every* upgrade - which adds up.

My main core switch (Cat 5500) has an uptime of...

Uptime is 737 days, 17 hours, 25 minutes

.... which means "*no* down time for over two years". I like that.

Of course we have the most recent software available, and we're reading
the release notes to watch out for things that could bite us. So if the
switch would start acting up, we're prepared.

gert
--
Yield to temptation ... it may not pass your way again! -- Lazarus Long
//www.muc.de/~gert
Gert Doering - Munich, Germany
fax: +49-89-3243328

Hi,

We had a similar problem in our datacenter yesterday.
We lost all layer-2 connectivity during about 3 minutes. ( looks like
spanning-tree was recalculated during three minutes , impact on all
trunks ).
Just before the event we had a similar error in one of our 3550-XL
catalysts :

Nov 23 11:48:38.711: %SYS-2-MALLOCFAIL: Memory allocation of 1680 bytes
failed from 0x177858, alignment 0

Pool: I/O Free: 2796 Cause: Memory fragmentation

Alternate Pool: None Free: 0 Cause: No Alternate pool



-Process= "Pool Manager", ipl= 0, pid= 5

-Traceback= 1BFE34 1C1488 17785C 1D064C 1D0894 1E0C44 1E423C


Nov 23 11:49:08.740: %SYS-2-MALLOCFAIL: Memory allocation of 1680 bytes
failed from 0x177858, alignment 0

Pool: I/O Free: 2796 Cause: Memory fragmentation

Alternate Pool: None Free: 0 Cause: No Alternate pool



-Process= "Pool Manager", ipl= 0, pid= 5

-Traceback= 1BFE34 1C1488 17785C 1D064C 1D07BC 1E0C44 1E423C

Nov 23 11:49:38.789: %SYS-2-MALLOCFAIL: Memory allocation of 1680 bytes
failed from 0x177858, alignment 0

Pool: I/O Free: 2796 Cause: Memory fragmentation

Alternate Pool: None Free: 0 Cause: No Alternate pool



-Process= "Pool Manager", ipl= 0, pid= 5

-Traceback= 1BFE34 1C1488 17785C 1D064C 1D07BC 1E0C44 1E423C





Can I get some more info about the problem you had , and how did you
solve it. I guess IOS upgrade.
Allthough we are not 100% sure that this caused our downtime. Just a
possible cause.

Bart

In article < .com>,
says...
> Hi,
>
> We had a similar problem in our datacenter yesterday.
> We lost all layer-2 connectivity during about 3 minutes. ( looks like
> spanning-tree was recalculated during three minutes , impact on all
> trunks ).
> Just before the event we had a similar error in one of our 3550-XL
> catalysts :
>
> Nov 23 11:48:38.711: %SYS-2-MALLOCFAIL: Memory allocation of 1680 bytes
> failed from 0x177858, alignment 0
>

A process wanted 1680 bytes of memory but there wasn't any free memory.

> Pool: I/O Free: 2796 Cause: Memory fragmentation

It was from I/O poll which was fragmented.
>
> -Process= "Pool Manager", ipl= 0, pid= 5
>

Pool Manager was doing allocation.


So, what can you do?

1) open Bug navigator on CCO and look for memory leak bugs. If you found
a bug and it is RESOLVED, then upgrade to image that has fix

2) open a TAC case and if they do not have image that fixes that problem
(unlikely) they willl create fix


--
-Ivan.

*** Use Rot13 to see my eMail address ***