[VERY LONG] Cisco 3620 and very low throghuput.

I'm geting crazy wiyh a Cisco 3620. It was the subject of few weeks ago on this NG.
I repost my question hoping to find a solutions.
It forward traffic only between 2 eth interfaces.
The throughput is close to 100 kbits/sec when it has a 100 Mbit/sec on both the interfaces.
If I replace it with a Linux box the throughput jupms to 3,5 Mbits very quietly.
All the outputs are made during a very big download (70 MBytes)
The total CPU load was at the average of 70 % but not any process shined for CPU load.
This all info I can give you:



--------- SH VER ------------------------------------------------------------

Cisco Internetwork Operating System Software
IOS (tm) 3600 Software (C3620-IK9O3S7-M), Version 12.3(13a), RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2005 by cisco Systems, Inc.
Compiled Tue 26-Apr-05 09:13 by ssearch
Image text-base: 0x60008B00, data-base: 0x61928000

ROM: System Bootstrap, Version 11.1(7)AX [kuong (7)AX], EARLY DEPLOYMENT RELEASE SOFTWARE (fc2)
ROM: 3600 Software (C3620-IK9O3S7-M), Version 12.3(13a), RELEASE SOFTWARE (fc2)

Borderline uptime is 1 week, 1 day, 13 hours, 33 minutes
System returned to ROM by power-on
System image file is "flash:c3620-ik9o3s7-mz.123-13a.bin"

[CUT]

cisco 3620 (R4700) processor (revision 0x81) with 61440K/4096K bytes of memory.
Processor board ID 056FT61
R4700 CPU at 80MHz, Implementation 33, Rev 1.0
Bridging software.
X.25 software, Version 3.0.0.
Basic Rate ISDN software, Version 1.1.
2 Ethernet/IEEE 802.3 interface(s)
1 Serial network interface(s)
4 ISDN Basic Rate interface(s)
DRAM configuration is 32 bits wide with parity disabled.
29K bytes of non-volatile configuration memory.
32768K bytes of processor board System flash (Read/Write)

----------------------- both SH INT E0/0 and E0/1 --------------------------------


Router#sh int e0/0
Ethernet0/0 is up, line protocol is up
Hardware is AmdP2, address is 00e0.1e56.7b61 (bia 00e0.1e56.7b61)
Internet address is 134.aaa.154.aaa/24
MTU 1500 bytes, BW 10000 Kbit, DLY 1000 usec,
reliability 254/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 00:20:57
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 68000 bits/sec, 13 packets/sec
5 minute output rate 12000 bits/sec, 11 packets/sec
12107 packets input, 9396669 bytes, 0 no buffer
Received 800 broadcasts, 0 runts, 0 giants, 0 throttles
121 input errors, 46 CRC, 46 frame, 0 overrun, 75 ignored
0 input packets with dribble condition detected
9613 packets output, 1358134 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out


Ethernet0/1 is up, line protocol is up
Hardware is AmdP2, address is 00e0.1e56.7b62 (bia 00e0.1e56.7b62)
Internet address is 192.168.32.142/29
MTU 1500 bytes, BW 10000 Kbit, DLY 1000 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:02, output 00:00:00, output hang never
Last clearing of "show interface" counters 00:21:38
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 5000 bits/sec, 4 packets/sec
5 minute output rate 64000 bits/sec, 6 packets/sec
6431 packets input, 1003588 bytes, 0 no buffer
Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
19 input errors, 19 CRC, 16 frame, 0 overrun, 0 ignored
0 input packets with dribble condition detected
8393 packets output, 9315674 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out

------------------ SH RUN -------------------------

!
version 12.3
service timestamps debug datetime msec
service timestamps log datetime msec
service password-encryption
!
hostname Borderline
!
boot-start-marker
boot-end-marker
!
enable secret fffffffffffffffffffffffffff
!
no aaa new-model
ip subnet-zero
!
!
ip cef
no ip domain lookup
ip domain name mine.com
!
ip audit po max-events 100
!
username xxxxxxxxxxxxxxxxxxx
!
!
interface Ethernet0/0
ip address xxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxx
ip nat outside
full-duplex
!
interface Serial0/0
no ip address
shutdown
!
interface Ethernet0/1
ip address 192.168.46.142 255.255.255.248
ip access-group 2 in
ip nat inside
full-duplex
!
interface BRI1/0
no ip address
shutdown
!
interface BRI1/1
no ip address
shutdown
!
interface BRI1/2
no ip address
shutdown
!
interface BRI1/3
no ip address
shutdown
!
ip nat translation max-entries 500
ip nat inside source list 112 interface Ethernet0/0 overload
ip nat inside source static tcp 192.168.46.137 22 interface Ethernet0/0 30022
ip nat inside source static tcp 192.168.46.137 443 interface Ethernet0/0 443
ip nat inside source static 192.168.46.193 CCCCCCCCCCCCCCCCC
ip nat inside source static 192.168.46.137 XXXXXXXXXXXXXXXXX
no ip http server
no ip http secure-server
ip classless
ip route 0.0.0.0 0.0.0.0 <NEXT HOP>
ip route 10.14.212.0 255.255.255.0 192.168.46.137
ip route 192.168.46.192 255.255.255.240 192.168.46.137
!
!
!
ip access-list extended vty-access
permit tcp 10.14.212.0 0.0.0.255 any eq 22
permit tcp 10.14.212.0 0.0.0.255 any eq telnet
access-list 1 permit 10.18.139.0 0.0.0.255
access-list 1 deny 10.0.0.0 0.255.255.255
access-list 1 deny 172.0.0.0 0.31.255.255
access-list 1 deny 192.168.0.0 0.0.255.255
access-list 1 permit any
access-list 1 deny any
access-list 2 deny 192.168.46.205
access-list 2 permit 192.168.46.136 0.0.0.7
access-list 2 permit 192.168.46.192 0.0.0.15
access-list 2 permit 10.14.212.0 0.0.0.255
access-list 2 deny any
access-list 100 permit udp host 192.168.46.137 eq isakmp host XXXXXXXXXXXXXXXXXXXXXXx eq isakmp
access-list 100 permit udp host 192.168.46.137 eq non500-isakmp host XXXXXXXXXXXXXXXXXXXX eq non500-isakmp
access-list 100 permit esp host 192.168.46.137 host XXXXXXXXXXXXXXXXXXXXXXX
access-list 100 deny ip any any
access-list 111 permit udp host 192.168.46.137 eq isakmp host XXXXXXXXXXXXXXXXXX eq isakmp
access-list 111 permit udp host 192.168.46.137 eq non500-isakmp host XXXXXXXXXXXXXXX eq non500-isakmp
access-list 111 permit esp host 192.168.46.137 host XXXXXXXXXXXXXXXXXX
access-list 112 permit ip 10.14.212.0 0.0.0.255 any
access-list 112 permit ip 192.168.46.136 0.0.0.7 any
access-list 112 permit ip 192.168.46.192 0.0.0.15 any
no cdp run
!
route-map NAT-VPN permit 10
match ip address 111
match interface Ethernet0/0
!
line con 0
line aux 0
line vty 0 4
access-class vty-access in
login local
!
!
end

-------------------- SH CEF ---------------------------------------

Ethernet0/0 is up (if_number 4)
Corresponding hwidb fast_if_number 4
Corresponding hwidb firstsw->if_number 4
Internet address is XXXXXXXXXXXXXXXXXXXxxx(24
ICMP redirects are always sent
Per packet load-sharing is disabled
IP unicast RPF check is disabled
Inbound access list is not set
Outbound access list is not set
IP policy routing is disabled
BGP based policy accounting is disabled
Hardware idb is Ethernet0/0
Fast switching type 1, interface type 61
IP CEF switching enabled
IP CEF Feature Fast switching turbo vector
Input fast flags 0x40, Output fast flags 0x100
ifindex 2(2)
Slot 0 Slot unit 0 Unit 0 VC -1
Transmit limit accumulator 0x0 (0x0)
IP MTU 1500

Ethernet0/1 is up (if_number 5)
Corresponding hwidb fast_if_number 5
Corresponding hwidb firstsw->if_number 5
Internet address is 192.168.46.142/29
ICMP redirects are always sent
Per packet load-sharing is disabled
IP unicast RPF check is disabled
Inbound access list is 2
Outbound access list is not set
IP policy routing is disabled
BGP based policy accounting is disabled
Hardware idb is Ethernet0/1
Fast switching type 1, interface type 61
IP CEF switching enabled
IP CEF Feature Fast switching turbo vector
Input fast flags 0x41, Output fast flags 0x100
ifindex 4(4)
Slot 0 Slot unit 1 Unit 1 VC -1
Transmit limit accumulator 0x0 (0x0)
IP MTU 1500

Router#sh cef not-cef-switched
CEF Packets passed on to next switching layer
Slot No_adj No_encap Unsupp'ted Redirect Receive Options Access Frag
RP 5 0 0 0 97397 0 0 0

---------------------------------------------------------------------------------------



That's problem it's weaking my mind, please...help me.
And sorry for the very long post

Alex.

 

I've worked for mutliple companies cleaning up problems and I can tell
you the number one problem is ignoring basics for sexy, cool,
impressive stuff.

Take care of the basics and the more complex stuff works out much
smoother and easier.

I'll point out what I can see here. I just corrected the same issue I
see here and througput on a server went from 50k to 4000k the moment I
corrected the setting.

You say above the interface is 100Mb, but this interface is only
running 10MB. Also since it reads ETHERNET and not FASTETHERNET, I
would assume it only support 10MB. If you are attaching this to a
10/100 Mb switch, you may have an auto-negotiation problem to boot.
This is a common problem. An error in speed/duplex negotiation can
produce a 99% decrease in throughput.

You set the interface for "full-duplex". On a fast ethernet(10/100)
interface if you hardcore negotiation you must do both SPEED and
DUPLEX and you MUST, MUST MUST do it on the device (router/pc/server)
*AND* the switchport you plug it into. One and not the other will
result in horrible throughput, physical errors on both or one side and
sometimes random effect with each power cycle of the device. On
ethernet you only need to worry about duplex. Make sure your switch is
not nogotiating. Set it for 10/FULL to match your interfaces.

Reliability should always be 255/255, otherwise you have
errors/physical problem.

All of these should always read 0. You'll get a couple when yanking
cables out of active interfaces, but that's the most you'll ever see.

After correcting checking speed duplex on the spwitch do a
CLEAR COUNTER E 0/0

If this interface attaches to a device you have no control over I
suggest putting it to half duplex. 10/full is more an exception, and
10/100 devices assumes old ethernets(which don't send negotiation
signals) operate at 10/half.

Again, errors here received here in the 21 minutes the interface has
been up.
Check speed/duplex on the attached switch.

The rest looks healthy enough.

I suggest

LOGGING BUFFERED INFO

to get a little more info on what's going on. SHO LOGGING may reveal
something else going on.


Get these two interface to be error free first. Don't bother
troubleshooting anything else. Once error-free reboot the router, the
errors should stay clear even past mutliple reboots.

DiGiTAL_ViNYL (no email)

 

Hello,

in addition to the suggestions in the other post, looking at your
configuration, a few things might not be needed, and I have a few
questions about others: first of all, which device has IP address
192.168.46.137 ? You have static routes in your configuration pointing
two of your to-be-NATted address spaces to that address. Since ip
routing occurs before NAT, these addresses will never get translated,
not sure if that is what you want.
Also, access lists 1, 100, 111, as well as the route-map appear to have
no purpose, so it might be a good idea to take those out.
So, putting it all together, I would suggest the following config:

version 12.3
service timestamps debug datetime msec
service timestamps log datetime msec
service password-encryption
!
hostname Borderline
!
boot-start-marker
boot-end-marker
!
enable secret fffffffffffffffffffffffffff
!
no aaa new-model
ip subnet-zero
!
!
ip cef
no ip domain lookup
ip domain name mine.com
!
ip audit po max-events 100
!
username xxxxxxxxxxxxxxxxxxx
!
!
interface Ethernet0/0
ip address xxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxx
ip nat outside
full-duplex
!
interface Serial0/0
no ip address
shutdown
!
interface Ethernet0/1
ip address 192.168.46.142 255.255.255.248
ip access-group 2 in
ip nat inside
full-duplex
!
interface BRI1/0
no ip address
shutdown
!
interface BRI1/1
no ip address
shutdown
!
interface BRI1/2
no ip address
shutdown
!
interface BRI1/3
no ip address
shutdown
!
ip nat translation max-entries 500
ip nat inside source list 112 interface Ethernet0/0 overload
ip nat inside source static tcp 192.168.46.137 22 interface Ethernet0/0
30022
ip nat inside source static tcp 192.168.46.137 443 interface
Ethernet0/0 443
ip nat inside source static 192.168.46.193 CCCCCCCCCCCCCCCCC
ip nat inside source static 192.168.46.137 XXXXXXXXXXXXXXXXX
no ip http server
no ip http secure-server
ip classless
ip route 0.0.0.0 0.0.0.0 Ethernet0/0
ip route 10.14.212.0 255.255.255.0 192.168.46.137
ip route 192.168.46.192 255.255.255.240 192.168.46.137
!
!
!
ip access-list extended vty-access
permit tcp 10.14.212.0 0.0.0.255 any eq 22
permit tcp 10.14.212.0 0.0.0.255 any eq telnet
access-list 2 deny 192.168.46.205
access-list 2 permit 192.168.46.136 0.0.0.7
access-list 2 permit 192.168.46.192 0.0.0.15
access-list 2 permit 10.14.212.0 0.0.0.255
access-list 2 deny any
access-list 112 permit ip 10.14.212.0 0.0.0.255 any
access-list 112 permit ip 192.168.46.136 0.0.0.7 any
access-list 112 permit ip 192.168.46.192 0.0.0.15 any
no cdp run
!
line con 0
line aux 0
line vty 0 4
access-class vty-access in
login local
!
!
end

That still might not help much, the important thing is to know which
addresses you want to have translated, and what the purpose is of the
static routes to 192.168.46.137 ...

Regards,

GP

 

interface Ethernet0/1

Hopefully, you have ALSO locked full duplex at the switch side? The input
errors suggest you might have a duplex mismatch. You should almost always
rely on autonegotiation unless you have a good reason not to.

 

AM:
Phillip Remaker:

Philip,

I agree with you however my agreement it theoretical,
I have not got the experience of 1000's of ports
to back it up.

Have you extensive experience that this works?

Thanks.

However:-
This is irrelevant I am certain to the issue at hand
which I have been following over the two threads.

We have:-

- High CPU (at Interrupt level)
- Low throughput
- Most packets using Speedy Switching (TM) (any kind of Fast sw)
- NAT present - Cisco say NAT overhead is low.

I am at the moment baffled.

How are the buffers? Could buffer misses be eating up the CPU?
Please post sh buff.

If you with I can try to assist with this.

Carry out a very controlled test that is repeatable.
Set "load-interval 30" on each interface.
service timestamps log datetime localtime

logging buffered 10000 debugging

Reboot router.
Carry out file transfer (one you can repeat exactly
tomorrow or next week) that will last for at lest 5 min.
issue:
term len 0

Arrange to capture data from terminal

sh mem (top bit only)

before transfer starts and subsequently every 2 minutes do:

Please paste these in as one block making sure that you select
an extra blank line at the bottom to get the
tail end timestamp correctly.

sh clock
sh proc cpu
sh int (relevant ones only)
sh buffers
sh interface switching
sh interface statistics
sh ip nat trans
sh ip traff
sh clock

After the transfer has finished:

repeat the above command list and add

sh log
sh run (sanitised as you wish)
sh mem (top lines only)

If you don't feel like posting that lot, post a message
requesting e-mail address with your e-mail address
(assuming it's not ) and I will send you
my address.

Reluctantly, the mind is drifting towards the idea of a random
software re-grade.
Maybe it's a Cisco plot to persuade you to get a 3725?