[VERY LONG] Cisco 3620 and very low throghuput.

Discussion in 'Cisco' started by AM, Oct 28, 2005.

  1. AM

    AM Guest

    I'm geting crazy wiyh a Cisco 3620. It was the subject of few weeks ago on this NG.
    I repost my question hoping to find a solutions.
    It forward traffic only between 2 eth interfaces.
    The throughput is close to 100 kbits/sec when it has a 100 Mbit/sec on both the interfaces.
    If I replace it with a Linux box the throughput jupms to 3,5 Mbits very quietly.
    All the outputs are made during a very big download (70 MBytes)
    The total CPU load was at the average of 70 % but not any process shined for CPU load.
    This all info I can give you:



    --------- SH VER ------------------------------------------------------------

    Cisco Internetwork Operating System Software
    IOS (tm) 3600 Software (C3620-IK9O3S7-M), Version 12.3(13a), RELEASE SOFTWARE (fc2)
    Technical Support: http://www.cisco.com/techsupport
    Copyright (c) 1986-2005 by cisco Systems, Inc.
    Compiled Tue 26-Apr-05 09:13 by ssearch
    Image text-base: 0x60008B00, data-base: 0x61928000

    ROM: System Bootstrap, Version 11.1(7)AX [kuong (7)AX], EARLY DEPLOYMENT RELEASE SOFTWARE (fc2)
    ROM: 3600 Software (C3620-IK9O3S7-M), Version 12.3(13a), RELEASE SOFTWARE (fc2)

    Borderline uptime is 1 week, 1 day, 13 hours, 33 minutes
    System returned to ROM by power-on
    System image file is "flash:c3620-ik9o3s7-mz.123-13a.bin"

    [CUT]

    cisco 3620 (R4700) processor (revision 0x81) with 61440K/4096K bytes of memory.
    Processor board ID 056FT61
    R4700 CPU at 80MHz, Implementation 33, Rev 1.0
    Bridging software.
    X.25 software, Version 3.0.0.
    Basic Rate ISDN software, Version 1.1.
    2 Ethernet/IEEE 802.3 interface(s)
    1 Serial network interface(s)
    4 ISDN Basic Rate interface(s)
    DRAM configuration is 32 bits wide with parity disabled.
    29K bytes of non-volatile configuration memory.
    32768K bytes of processor board System flash (Read/Write)

    ----------------------- both SH INT E0/0 and E0/1 --------------------------------


    Router#sh int e0/0
    Ethernet0/0 is up, line protocol is up
    Hardware is AmdP2, address is 00e0.1e56.7b61 (bia 00e0.1e56.7b61)
    Internet address is 134.aaa.154.aaa/24
    MTU 1500 bytes, BW 10000 Kbit, DLY 1000 usec,
    reliability 254/255, txload 1/255, rxload 1/255
    Encapsulation ARPA, loopback not set
    Keepalive set (10 sec)
    ARP type: ARPA, ARP Timeout 04:00:00
    Last input 00:00:00, output 00:00:00, output hang never
    Last clearing of "show interface" counters 00:20:57
    Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
    Queueing strategy: fifo
    Output queue: 0/40 (size/max)
    5 minute input rate 68000 bits/sec, 13 packets/sec
    5 minute output rate 12000 bits/sec, 11 packets/sec
    12107 packets input, 9396669 bytes, 0 no buffer
    Received 800 broadcasts, 0 runts, 0 giants, 0 throttles
    121 input errors, 46 CRC, 46 frame, 0 overrun, 75 ignored
    0 input packets with dribble condition detected
    9613 packets output, 1358134 bytes, 0 underruns
    0 output errors, 0 collisions, 0 interface resets
    0 babbles, 0 late collision, 0 deferred
    0 lost carrier, 0 no carrier
    0 output buffer failures, 0 output buffers swapped out


    Ethernet0/1 is up, line protocol is up
    Hardware is AmdP2, address is 00e0.1e56.7b62 (bia 00e0.1e56.7b62)
    Internet address is 192.168.32.142/29
    MTU 1500 bytes, BW 10000 Kbit, DLY 1000 usec,
    reliability 255/255, txload 1/255, rxload 1/255
    Encapsulation ARPA, loopback not set
    Keepalive set (10 sec)
    ARP type: ARPA, ARP Timeout 04:00:00
    Last input 00:00:02, output 00:00:00, output hang never
    Last clearing of "show interface" counters 00:21:38
    Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
    Queueing strategy: fifo
    Output queue: 0/40 (size/max)
    5 minute input rate 5000 bits/sec, 4 packets/sec
    5 minute output rate 64000 bits/sec, 6 packets/sec
    6431 packets input, 1003588 bytes, 0 no buffer
    Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
    19 input errors, 19 CRC, 16 frame, 0 overrun, 0 ignored
    0 input packets with dribble condition detected
    8393 packets output, 9315674 bytes, 0 underruns
    0 output errors, 0 collisions, 0 interface resets
    0 babbles, 0 late collision, 0 deferred
    0 lost carrier, 0 no carrier
    0 output buffer failures, 0 output buffers swapped out

    ------------------ SH RUN -------------------------

    !
    version 12.3
    service timestamps debug datetime msec
    service timestamps log datetime msec
    service password-encryption
    !
    hostname Borderline
    !
    boot-start-marker
    boot-end-marker
    !
    enable secret fffffffffffffffffffffffffff
    !
    no aaa new-model
    ip subnet-zero
    !
    !
    ip cef
    no ip domain lookup
    ip domain name mine.com
    !
    ip audit po max-events 100
    !
    username xxxxxxxxxxxxxxxxxxx
    !
    !
    interface Ethernet0/0
    ip address xxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxx
    ip nat outside
    full-duplex
    !
    interface Serial0/0
    no ip address
    shutdown
    !
    interface Ethernet0/1
    ip address 192.168.46.142 255.255.255.248
    ip access-group 2 in
    ip nat inside
    full-duplex
    !
    interface BRI1/0
    no ip address
    shutdown
    !
    interface BRI1/1
    no ip address
    shutdown
    !
    interface BRI1/2
    no ip address
    shutdown
    !
    interface BRI1/3
    no ip address
    shutdown
    !
    ip nat translation max-entries 500
    ip nat inside source list 112 interface Ethernet0/0 overload
    ip nat inside source static tcp 192.168.46.137 22 interface Ethernet0/0 30022
    ip nat inside source static tcp 192.168.46.137 443 interface Ethernet0/0 443
    ip nat inside source static 192.168.46.193 CCCCCCCCCCCCCCCCC
    ip nat inside source static 192.168.46.137 XXXXXXXXXXXXXXXXX
    no ip http server
    no ip http secure-server
    ip classless
    ip route 0.0.0.0 0.0.0.0 <NEXT HOP>
    ip route 10.14.212.0 255.255.255.0 192.168.46.137
    ip route 192.168.46.192 255.255.255.240 192.168.46.137
    !
    !
    !
    ip access-list extended vty-access
    permit tcp 10.14.212.0 0.0.0.255 any eq 22
    permit tcp 10.14.212.0 0.0.0.255 any eq telnet
    access-list 1 permit 10.18.139.0 0.0.0.255
    access-list 1 deny 10.0.0.0 0.255.255.255
    access-list 1 deny 172.0.0.0 0.31.255.255
    access-list 1 deny 192.168.0.0 0.0.255.255
    access-list 1 permit any
    access-list 1 deny any
    access-list 2 deny 192.168.46.205
    access-list 2 permit 192.168.46.136 0.0.0.7
    access-list 2 permit 192.168.46.192 0.0.0.15
    access-list 2 permit 10.14.212.0 0.0.0.255
    access-list 2 deny any
    access-list 100 permit udp host 192.168.46.137 eq isakmp host XXXXXXXXXXXXXXXXXXXXXXx eq isakmp
    access-list 100 permit udp host 192.168.46.137 eq non500-isakmp host XXXXXXXXXXXXXXXXXXXX eq non500-isakmp
    access-list 100 permit esp host 192.168.46.137 host XXXXXXXXXXXXXXXXXXXXXXX
    access-list 100 deny ip any any
    access-list 111 permit udp host 192.168.46.137 eq isakmp host XXXXXXXXXXXXXXXXXX eq isakmp
    access-list 111 permit udp host 192.168.46.137 eq non500-isakmp host XXXXXXXXXXXXXXX eq non500-isakmp
    access-list 111 permit esp host 192.168.46.137 host XXXXXXXXXXXXXXXXXX
    access-list 112 permit ip 10.14.212.0 0.0.0.255 any
    access-list 112 permit ip 192.168.46.136 0.0.0.7 any
    access-list 112 permit ip 192.168.46.192 0.0.0.15 any
    no cdp run
    !
    route-map NAT-VPN permit 10
    match ip address 111
    match interface Ethernet0/0
    !
    line con 0
    line aux 0
    line vty 0 4
    access-class vty-access in
    login local
    !
    !
    end

    -------------------- SH CEF ---------------------------------------

    Ethernet0/0 is up (if_number 4)
    Corresponding hwidb fast_if_number 4
    Corresponding hwidb firstsw->if_number 4
    Internet address is XXXXXXXXXXXXXXXXXXXxxx(24
    ICMP redirects are always sent
    Per packet load-sharing is disabled
    IP unicast RPF check is disabled
    Inbound access list is not set
    Outbound access list is not set
    IP policy routing is disabled
    BGP based policy accounting is disabled
    Hardware idb is Ethernet0/0
    Fast switching type 1, interface type 61
    IP CEF switching enabled
    IP CEF Feature Fast switching turbo vector
    Input fast flags 0x40, Output fast flags 0x100
    ifindex 2(2)
    Slot 0 Slot unit 0 Unit 0 VC -1
    Transmit limit accumulator 0x0 (0x0)
    IP MTU 1500

    Ethernet0/1 is up (if_number 5)
    Corresponding hwidb fast_if_number 5
    Corresponding hwidb firstsw->if_number 5
    Internet address is 192.168.46.142/29
    ICMP redirects are always sent
    Per packet load-sharing is disabled
    IP unicast RPF check is disabled
    Inbound access list is 2
    Outbound access list is not set
    IP policy routing is disabled
    BGP based policy accounting is disabled
    Hardware idb is Ethernet0/1
    Fast switching type 1, interface type 61
    IP CEF switching enabled
    IP CEF Feature Fast switching turbo vector
    Input fast flags 0x41, Output fast flags 0x100
    ifindex 4(4)
    Slot 0 Slot unit 1 Unit 1 VC -1
    Transmit limit accumulator 0x0 (0x0)
    IP MTU 1500

    Router#sh cef not-cef-switched
    CEF Packets passed on to next switching layer
    Slot No_adj No_encap Unsupp'ted Redirect Receive Options Access Frag
    RP 5 0 0 0 97397 0 0 0

    ---------------------------------------------------------------------------------------



    That's problem it's weaking my mind, please...help me.
    And sorry for the very long post

    Alex.
     
    AM, Oct 28, 2005
    #1
    1. Advertisements

  2. AM

    DigitalVinyl Guest

    I've worked for mutliple companies cleaning up problems and I can tell
    you the number one problem is ignoring basics for sexy, cool,
    impressive stuff.

    Take care of the basics and the more complex stuff works out much
    smoother and easier.

    I'll point out what I can see here. I just corrected the same issue I
    see here and througput on a server went from 50k to 4000k the moment I
    corrected the setting.


    You say above the interface is 100Mb, but this interface is only
    running 10MB. Also since it reads ETHERNET and not FASTETHERNET, I
    would assume it only support 10MB. If you are attaching this to a
    10/100 Mb switch, you may have an auto-negotiation problem to boot.
    This is a common problem. An error in speed/duplex negotiation can
    produce a 99% decrease in throughput.

    You set the interface for "full-duplex". On a fast ethernet(10/100)
    interface if you hardcore negotiation you must do both SPEED and
    DUPLEX and you MUST, MUST MUST do it on the device (router/pc/server)
    *AND* the switchport you plug it into. One and not the other will
    result in horrible throughput, physical errors on both or one side and
    sometimes random effect with each power cycle of the device. On
    ethernet you only need to worry about duplex. Make sure your switch is
    not nogotiating. Set it for 10/FULL to match your interfaces.
    Reliability should always be 255/255, otherwise you have
    errors/physical problem.
    All of these should always read 0. You'll get a couple when yanking
    cables out of active interfaces, but that's the most you'll ever see.
    After correcting checking speed duplex on the spwitch do a
    CLEAR COUNTER E 0/0
    If this interface attaches to a device you have no control over I
    suggest putting it to half duplex. 10/full is more an exception, and
    10/100 devices assumes old ethernets(which don't send negotiation
    signals) operate at 10/half.


    Again, errors here received here in the 21 minutes the interface has
    been up.
    Check speed/duplex on the attached switch.
    The rest looks healthy enough.

    I suggest

    LOGGING BUFFERED INFO

    to get a little more info on what's going on. SHO LOGGING may reveal
    something else going on.


    Get these two interface to be error free first. Don't bother
    troubleshooting anything else. Once error-free reboot the router, the
    errors should stay clear even past mutliple reboots.

    DiGiTAL_ViNYL (no email)
     
    DigitalVinyl, Oct 29, 2005
    #2
    1. Advertisements

  3. AM

    nazgulero Guest

    Hello,

    in addition to the suggestions in the other post, looking at your
    configuration, a few things might not be needed, and I have a few
    questions about others: first of all, which device has IP address
    192.168.46.137 ? You have static routes in your configuration pointing
    two of your to-be-NATted address spaces to that address. Since ip
    routing occurs before NAT, these addresses will never get translated,
    not sure if that is what you want.
    Also, access lists 1, 100, 111, as well as the route-map appear to have
    no purpose, so it might be a good idea to take those out.
    So, putting it all together, I would suggest the following config:

    version 12.3
    service timestamps debug datetime msec
    service timestamps log datetime msec
    service password-encryption
    !
    hostname Borderline
    !
    boot-start-marker
    boot-end-marker
    !
    enable secret fffffffffffffffffffffffffff
    !
    no aaa new-model
    ip subnet-zero
    !
    !
    ip cef
    no ip domain lookup
    ip domain name mine.com
    !
    ip audit po max-events 100
    !
    username xxxxxxxxxxxxxxxxxxx
    !
    !
    interface Ethernet0/0
    ip address xxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxx
    ip nat outside
    full-duplex
    !
    interface Serial0/0
    no ip address
    shutdown
    !
    interface Ethernet0/1
    ip address 192.168.46.142 255.255.255.248
    ip access-group 2 in
    ip nat inside
    full-duplex
    !
    interface BRI1/0
    no ip address
    shutdown
    !
    interface BRI1/1
    no ip address
    shutdown
    !
    interface BRI1/2
    no ip address
    shutdown
    !
    interface BRI1/3
    no ip address
    shutdown
    !
    ip nat translation max-entries 500
    ip nat inside source list 112 interface Ethernet0/0 overload
    ip nat inside source static tcp 192.168.46.137 22 interface Ethernet0/0
    30022
    ip nat inside source static tcp 192.168.46.137 443 interface
    Ethernet0/0 443
    ip nat inside source static 192.168.46.193 CCCCCCCCCCCCCCCCC
    ip nat inside source static 192.168.46.137 XXXXXXXXXXXXXXXXX
    no ip http server
    no ip http secure-server
    ip classless
    ip route 0.0.0.0 0.0.0.0 Ethernet0/0
    ip route 10.14.212.0 255.255.255.0 192.168.46.137
    ip route 192.168.46.192 255.255.255.240 192.168.46.137
    !
    !
    !
    ip access-list extended vty-access
    permit tcp 10.14.212.0 0.0.0.255 any eq 22
    permit tcp 10.14.212.0 0.0.0.255 any eq telnet
    access-list 2 deny 192.168.46.205
    access-list 2 permit 192.168.46.136 0.0.0.7
    access-list 2 permit 192.168.46.192 0.0.0.15
    access-list 2 permit 10.14.212.0 0.0.0.255
    access-list 2 deny any
    access-list 112 permit ip 10.14.212.0 0.0.0.255 any
    access-list 112 permit ip 192.168.46.136 0.0.0.7 any
    access-list 112 permit ip 192.168.46.192 0.0.0.15 any
    no cdp run
    !
    line con 0
    line aux 0
    line vty 0 4
    access-class vty-access in
    login local
    !
    !
    end

    That still might not help much, the important thing is to know which
    addresses you want to have translated, and what the purpose is of the
    static routes to 192.168.46.137 ...

    Regards,

    GP
     
    nazgulero, Oct 29, 2005
    #3
  4. interface Ethernet0/1
    Hopefully, you have ALSO locked full duplex at the switch side? The input
    errors suggest you might have a duplex mismatch. You should almost always
    rely on autonegotiation unless you have a good reason not to.
     
    Phillip Remaker, Nov 1, 2005
    #4
  5. AM

    anybody43 Guest

    AM:
    Phillip Remaker:
    Philip,

    I agree with you however my agreement it theoretical,
    I have not got the experience of 1000's of ports
    to back it up.

    Have you extensive experience that this works?

    Thanks.

    However:-
    This is irrelevant I am certain to the issue at hand
    which I have been following over the two threads.

    We have:-

    - High CPU (at Interrupt level)
    - Low throughput
    - Most packets using Speedy Switching (TM) (any kind of Fast sw)
    - NAT present - Cisco say NAT overhead is low.

    I am at the moment baffled.

    How are the buffers? Could buffer misses be eating up the CPU?
    Please post sh buff.

    If you with I can try to assist with this.

    Carry out a very controlled test that is repeatable.
    Set "load-interval 30" on each interface.
    service timestamps log datetime localtime

    logging buffered 10000 debugging

    Reboot router.
    Carry out file transfer (one you can repeat exactly
    tomorrow or next week) that will last for at lest 5 min.
    issue:
    term len 0

    Arrange to capture data from terminal

    sh mem (top bit only)

    before transfer starts and subsequently every 2 minutes do:

    Please paste these in as one block making sure that you select
    an extra blank line at the bottom to get the
    tail end timestamp correctly.

    sh clock
    sh proc cpu
    sh int (relevant ones only)
    sh buffers
    sh interface switching
    sh interface statistics
    sh ip nat trans
    sh ip traff
    sh clock

    After the transfer has finished:

    repeat the above command list and add

    sh log
    sh run (sanitised as you wish)
    sh mem (top lines only)

    If you don't feel like posting that lot, post a message
    requesting e-mail address with your e-mail address
    (assuming it's not ) and I will send you
    my address.

    Reluctantly, the mind is drifting towards the idea of a random
    software re-grade.
    Maybe it's a Cisco plot to persuade you to get a 3725?
     
    anybody43, Nov 1, 2005
    #5
    1. Advertisements

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.