Catalyst 3750G drops packets with IPv4 options

Discussion in 'Cisco' started by Spoon, May 7, 2007.

  1. Spoon

    Spoon Guest

    Hello,

    I have a problem with a Cisco Catalyst 3750G switch/router.

    In my tests, I could not get the 3750G to reliably route more than 3400
    IPv4 packets per second when these packets contained IP options.

    The packets were UDP/IPv4. I tried both small (payload = 100 bytes)
    and large (payload = 1450 bytes) packets.

    The IP options consisted of 4 or 8 NOPs (option type 1). Thus there
    were no alignment issues, and nothing extra for the router to process.

    3570 pps && payload= 100 : 1.1% packet loss
    3570 pps && payload=1450 : 2.3% packet loss
    5000 pps && payload=1450 : 30.2% packet loss

    The following page does not mention my situation:
    http://www.cisco.com/en/US/products/hw/switches/ps5023/products_tech_note09186a00807213f5.shtml

    I ran show process cpu:

    CPU utilization for five seconds: 87%/29%; one min: 89%; five min: 73%
    PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
    [...]
    150 7678123 5843620 1313 54.63% 53.74% 43.74% 0 IP Input

    A few entries show less than 0.15% CPU usage. All other show 0%.

    Where does the 87% figure come from?

    Is it 87% overall with 29% IRQ servicing => 58% other than IRQ.

    Why is the router dropping packets if the CPU is not maxed out?

    Is this a known problem?

    Has it perhaps been fixed in a more recent firmware revision?

    For reference:
    Cisco IOS Software, C3750 Software (C3750-IPSERVICES-M), Version
    12.2(25)SEC, RELEASE SOFTWARE (fc2)
    Copyright (c) 1986-2005 by Cisco Systems, Inc.
    Compiled Thu 14-Jul-05 21:33 by antonino

    ROM: Bootstrap program is C3750 boot loader
    BOOTLDR: C3750 Boot Loader (C3750-HBOOT-M) Version 12.2(25r)SE, RELEASE
    SOFTWARE (fc)

    cyrus uptime is 1 week, 2 days, 18 hours, 44 minutes
    System returned to ROM by power-on
    System image file is "flash:/c3750-ipservices-mz.122-25.SEC.bin"

    cisco WS-C3750G-24TS-1U (PowerPC405) processor (revision A0) with
    118784K/12280K bytes of memory.
    Processor board ID FOC0902U0GY
    Last reset from power-on
    17 Virtual Ethernet interfaces
    28 Gigabit Ethernet interfaces
    The password-recovery mechanism is enabled.

    512K bytes of flash-simulated non-volatile configuration memory.
    Base ethernet MAC Address : 00:13:19:86:CA:00
    Motherboard assembly number : 73-9637-07
    Power supply part number : 341-0098-01
    Motherboard serial number : FOC09020A2U
    Power supply serial number : DCA084701XH
    Model revision number : A0
    Motherboard revision number : A0
    Model number : WS-C3750G-24TS-S1U
    System serial number : FOC0902U0GY
    SFP Module assembly part number : 73-7757-02
    SFP Module revision Number : A0
    SFP Module serial number : CAT085207B5
    Top Assembly Part Number : 800-25730-01
    Top Assembly Revision Number : A0
    Version ID : 01
    CLEI Code Number : CNMWS00ARA
    Hardware Board Revision Number : 0x05


    Switch Ports Model SW Version SW Image
    ------ ----- ----- ---------- ----------
    * 1 28 WS-C3750G-24TS-1U 12.2(25)SEC
    C3750-IPSERVICES-M


    Configuration register is 0xF

    Regards.
     
    Spoon, May 7, 2007
    #1
    1. Advertisements

  2. Spoon

    Guest Guest

    check the speed and duplex of the interface

    Flamer.
     
    Guest, May 7, 2007
    #2
    1. Advertisements

  3. Spoon

    Bod43 Guest

    The 3750 is primarily a hardware basd router but
    can fall back to CPU routing under certain circumstances.
    You seem to have hit thse curcumstances.

    It will doo wire rate hardware routing but the software
    routing performance is poor. Further there are
    two kinds of software routing (fast switched and
    process switched) and you are getting a lot of
    process switching.

    CPU utilization for five seconds: 87%/29%; one min: 89%; five min:
    73%

    87% total CPU of which 29% is fast switching packets.

    The "IP Input" process is the one that does the process switching.

    If you look for documents that cover troubleshooting
    high cpu that ar relevant to you platform you may get
    something that explains all of the details.

    So whatever your options are doing is causing the
    switch to fall back to CPU based routing.
     
    Bod43, May 8, 2007
    #3
  4. Spoon

    Spoon Guest

    All interfaces involved are 100 Mbit/s and full duplex.

    On the sender:
    # ethtool eth2
    Settings for eth2:
    Supported ports: [ TP MII ]
    Supported link modes: 10baseT/Half 10baseT/Full
    100baseT/Half 100baseT/Full
    Supports auto-negotiation: Yes
    Advertised link modes: 10baseT/Half 10baseT/Full
    100baseT/Half 100baseT/Full
    Advertised auto-negotiation: Yes
    Speed: 100Mb/s
    Duplex: Full
    Port: MII
    PHYAD: 1
    Transceiver: internal
    Auto-negotiation: on
    Current message level: 0x000020c1 (8385)
    Link detected: yes

    That interface is connected to port 17 on the 3750:
    Name : Gi1/0/17
    Administrative Speed: auto
    Administrative Duplex: auto
    Administrative Auto-MDIX: on
    Administrative Power Inline: N/A
    Operational Speed: 100
    Operational Duplex: full
    Operational Auto-MDIX: on

    On the receiver:
    Settings for eth2:
    Supported ports: [ TP MII ]
    Supported link modes: 10baseT/Half 10baseT/Full
    100baseT/Half 100baseT/Full
    Supports auto-negotiation: Yes
    Advertised link modes: 10baseT/Half 10baseT/Full
    100baseT/Half 100baseT/Full
    Advertised auto-negotiation: Yes
    Speed: 100Mb/s
    Duplex: Full
    Port: MII
    PHYAD: 1
    Transceiver: internal
    Auto-negotiation: on
    Current message level: 0x000020c1 (8385)
    Link detected: yes

    That interface is connected to port 19 on the 3750:
    Name : Gi1/0/19
    Administrative Speed: auto
    Administrative Duplex: auto
    Administrative Auto-MDIX: on
    Administrative Power Inline: N/A
    Operational Speed: 100
    Operational Duplex: full
    Operational Auto-MDIX: on

    What problem do you suspect?

    I failed to mention that the router doesn't drop any packets when
    packets do not carry any IP option.

    Vlan17 is up, line protocol is up
    Hardware is EtherSVI, address is 0013.1986.ca48 (bia 0013.1986.ca48)
    Internet address is 10.1.17.254/24
    MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
    reliability 255/255, txload 1/255, rxload 10/255
    Encapsulation ARPA, loopback not set
    ARP type: ARPA, ARP Timeout 04:00:00
    Last input 00:00:00, output 00:00:01, output hang never
    Last clearing of "show interface" counters never
    Input queue: 21/75/35000/0 (size/max/drops/flushes); Total output
    drops: 0
    Queueing strategy: fifo
    Output queue: 0/40 (size/max)
    5 minute input rate 41465000 bits/sec, 3474 packets/sec
    5 minute output rate 0 bits/sec, 0 packets/sec
    43540876 packets input, 3580507536 bytes, 0 no buffer
    Received 0 broadcasts (364590 IP multicast)
    0 runts, 0 giants, 0 throttles
    0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
    52036 packets output, 3613284 bytes, 0 underruns
    0 output errors, 0 interface resets
    0 output buffer failures, 0 output buffers swapped out

    Vlan19 is up, line protocol is up
    Hardware is EtherSVI, address is 0013.1986.ca4a (bia 0013.1986.ca4a)
    Internet address is 10.1.19.254/24
    MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
    reliability 255/255, txload 10/255, rxload 1/255
    Encapsulation ARPA, loopback not set
    ARP type: ARPA, ARP Timeout 04:00:00
    Last input 00:27:22, output 00:00:00, output hang never
    Last clearing of "show interface" counters never
    Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
    Queueing strategy: fifo
    Output queue: 0/40 (size/max)
    5 minute input rate 0 bits/sec, 0 packets/sec
    5 minute output rate 41577000 bits/sec, 3469 packets/sec
    289 packets input, 17454 bytes, 0 no buffer
    Received 0 broadcasts (0 IP multicast)
    0 runts, 0 giants, 0 throttles
    0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
    33807273 packets output, 3212600286 bytes, 0 underruns
    0 output errors, 0 interface resets
    0 output buffer failures, 0 output buffers swapped out

    GigabitEthernet1/0/17 is up, line protocol is up (connected)
    Hardware is Gigabit Ethernet, address is 0013.1986.ca11 (bia
    0013.1986.ca11)
    MTU 9000 bytes, BW 100000 Kbit, DLY 100 usec,
    reliability 255/255, txload 1/255, rxload 122/255
    Encapsulation ARPA, loopback not set
    Keepalive set (10 sec)
    Full-duplex, 100Mb/s, media type is 10/100/1000BaseTX
    input flow-control is off, output flow-control is unsupported
    ARP type: ARPA, ARP Timeout 04:00:00
    Last input never, output 00:00:00, output hang never
    Last clearing of "show interface" counters never
    Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
    Queueing strategy: fifo
    Output queue: 0/40 (size/max)
    5 minute input rate 48059000 bits/sec, 4019 packets/sec
    5 minute output rate 0 bits/sec, 0 packets/sec
    56540589 packets input, 382292988 bytes, 0 no buffer
    Received 745135 broadcasts (0 multicast)
    0 runts, 0 giants, 0 throttles
    1 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
    0 watchdog, 745042 multicast, 0 pause input
    0 input packets with dribble condition detected
    434362 packets output, 35087682 bytes, 0 underruns
    0 output errors, 0 collisions, 1 interface resets
    0 babbles, 0 late collision, 0 deferred
    0 lost carrier, 0 no carrier, 0 PAUSE output
    0 output buffer failures, 0 output buffers swapped out

    GigabitEthernet1/0/19 is up, line protocol is up (connected)
    Hardware is Gigabit Ethernet, address is 0013.1986.ca13 (bia
    0013.1986.ca13)
    MTU 9000 bytes, BW 100000 Kbit, DLY 100 usec,
    reliability 255/255, txload 105/255, rxload 1/255
    Encapsulation ARPA, loopback not set
    Keepalive set (10 sec)
    Full-duplex, 100Mb/s, media type is 10/100/1000BaseTX
    input flow-control is off, output flow-control is unsupported
    ARP type: ARPA, ARP Timeout 04:00:00
    Last input never, output 00:00:00, output hang never
    Last clearing of "show interface" counters never
    Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
    Queueing strategy: fifo
    Output queue: 0/40 (size/max)
    5 minute input rate 0 bits/sec, 0 packets/sec
    5 minute output rate 41515000 bits/sec, 3471 packets/sec
    7177 packets input, 4052852 bytes, 0 no buffer
    Received 81 broadcasts (0 multicast)
    0 runts, 0 giants, 0 throttles
    0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
    0 watchdog, 18 multicast, 0 pause input
    0 input packets with dribble condition detected
    45385038 packets output, 1476776494 bytes, 0 underruns
    0 output errors, 0 collisions, 1 interface resets
    0 babbles, 0 late collision, 0 deferred
    0 lost carrier, 0 no carrier, 0 PAUSE output
    0 output buffer failures, 0 output buffers swapped out
     
    Spoon, May 9, 2007
    #4
  5. Spoon

    John Guest

    Hello,

    probably the switch goes to software forwarding as some people have
    already said. I suppose that you have not configured any of the "ip
    accounting, access-lists etc)"

    what you have sent shows that probably the packets are dropped in the
    input queue of SVI "Vlan17" (Input queue: 21/75/35000/0 (size/max/
    drops/flushes).

    You could try (as a workaround):
    a. raise the input queue (eg hold-queue 1000 in)
    b. try using l3 ports instead of SVI. (conf t; int fa0/17 ; no
    switchport; ip address 10.1.17.254 255.255.255.0)

    I guess that the best thing you can do is to upgrade to latest
    software and if the problems perists, open a case with TAC.

    --john
     
    John, May 9, 2007
    #5
  6. Spoon

    Thrill5 Guest

    You might need to issue the command "sdm prefer routing". The switch has
    only a certain amount of memory that it uses for switching and routing. By
    default most of the memory is allocated for MAC addresses, and if you have
    more than 2000 routes you will run out of memory for unicast routes at which
    point you are software routing packets. After issuing the command you will
    need to reload the switch to make it effective.

    Scott
     
    Thrill5, May 11, 2007
    #6
    1. Advertisements

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.