CEF slowing down FE to FE traffic

Discussion in 'Cisco' started by WAState, Feb 28, 2005.

  1. WAState

    WAState Guest

    I'm experimenting with CEF on a 2621. I have a fast PC on each FE port,
    each PC running a small tight program that opens a TCP socket and
    measures the throughput to/from the other PC. I wrote the code myself,
    it's very fast and is definitely not the limiting factor on speed (if
    the two PC's are connected through just a hub, throughput is ~80Mbps).

    When the 2621 sits between them, throughput is dramatically lower...
    ~10 Mbps in one direction and ~6600 Kbps in the other (with CEF
    disabled). I know the 2600 platform cannot sustain wire speeds, but at
    least this gives us a baseline.

    So I enabled CEF, expecting at least **some** increase in speed. Not
    so. In one direction (input F0/0 to output F0/1) the speed is
    unchanged; in the other (input F0/1 to output F0/0) the speed drops
    dramatically depending upon what interface settings I use.

    For example, with "ip route-cache flow" on both FE's the speed is
    roughly the same as the CEF-disabled rate (~6600 Kbps). However, add
    "ip route-cache cef" to both interfaces and the F0/1 to F0/0 speed
    drops to 700 Kbps (a drop of nearly 90%!). Removing all "ip
    route-cache" statements from both interfaces yields a middle ground of
    about 4500 Kbps.

    Meanwhile, in all cases the F0/0 to F0/1 traffic continues to be
    ~10Mbps, utterly unaffected by any CEF or route-cache settings.

    The only "extra" thing this router is doing is NAT. F0/0 is the outside
    interface, and F0/1 is the inside interface. NAT is using only static
    addresses. I don't know if NAT would be the limiting factor on speed,
    or if that burden would be unidirectional. Does NAT force all packets
    to be process switched, effectively disabling CEF? I would think that
    CEF's routing table could still be built even allowing for NAT, just as
    it allows for ACL's.

    Any ideas gratefully accepted. I'm just trying to understand what's
    happening here so I can get the most out of these routers.
     
    WAState, Feb 28, 2005
    #1
    1. Advertisements

  2. WAState

    WAState Guest

    I've just finished confirming (via "sho int stats") that CEF is indeed
    operational. The vast majority of packets are being handled by the
    "route cache" switching path and not the "processor" switching path, as
    reported by this command.

    Meanwhile, F0/1 -> F0/0 traffic continues to be roughly 1/2 the
    throughput of F0/0 -> F0/1....
     
    WAState, Feb 28, 2005
    #2
    1. Advertisements

  3. WAState

    Ivan Ostreš Guest

    It would be nice to see 'show proc cpu' when doing the tests with
    CEF....
     
    Ivan Ostreš, Mar 1, 2005
    #3
  4. WAState

    WAState Guest

    Yes, the CPU load is lower when running CEF. That's to be expected.
    Here are some approximate CPU load numbers...

    F0/0 -> F0/1 traffic: 87% w/o CEF, 30% w/ CEF
    F0/1 -> F0/0 traffic: 52% w/o CEF, 30% w/ CEF

    It's delightful that the CPU isn't as heavily loaded, but I don't like
    the 40-50% drop in throughput in the F0/1 -> F0/0 direction. Even more
    interesting, the throughput in the F0/0 -> F0/1 direction appears to be
    nearly constant (under 5% variation) regardless of all other factors.

    So we've proven that CEF saves CPU cycles. We still don't know why 1)
    one direction is slower than the other, and 2) why CEF is slower than
    process switching in that same direction.

    Any ideas, or suggested tests?
     
    WAState, Mar 1, 2005
    #4
  5. WAState

    Merv Guest

    Trying chnaging the IOS version in use on the box
     
    Merv, Mar 1, 2005
    #5
  6. WAState

    Hansang Bae Guest


    First things first. Have you tried SWAPPING your test programs? Move
    it from Fa0/0 to Fa0/1 and vice versa.

    You can always try a differnt IOS version if you have one handy (the
    release notes may have some blurb about a cef/switching bug.


    --

    hsb


    "Somehow I imagined this experience would be more rewarding" Calvin
    **************************ROT13 MY ADDRESS*************************
    Due to the volume of email that I receive, I may not not be able to
    reply to emails sent to my account. Please post a followup instead.
    ********************************************************************
     
    Hansang Bae, Mar 2, 2005
    #6
  7. WAState

    anybody43 Guest

    "I'm experimenting with CEF on a 2621"
    "F0/0 -> F0/1 traffic: 87% w/o CEF, 30% w/ CEF
    F0/1 -> F0/0 traffic: 52% w/o CEF, 30% w/ CEF"
    "It's delightful that the CPU isn't as heavily loaded"


    My experience is that the throughput of a router in
    a particular configuration is limited
    *either* by the CPU *or* by external factors.

    Since with CEF you are not hitting 100% CPU there is
    I suggest some external (to the router) factor affecting the tests.

    In summary the tests _must_ be broken! There is pretty much no
    other explanation.

    Just 2c worth.

    For example, you have a duplex missmatch and with
    process switching the rhythm of the conversation is
    such that you are not hitting it however when you switch to
    CEF the change in the rhythm causes lost frames.

    Finally, my understanding is that pretty much all of the fast
    switching methods, CEF, optimum, fast, flow have about
    the same performance (say +/- a few percent) however CEF
    does not have a cache that gets populated whenever the first
    matching packet is routed but has a FIB that gets populated
    from the routing table which only changes when the routing table
    changes. This difference could be critical for some applications.
     
    anybody43, Mar 2, 2005
    #7
  8. WAState

    Ivan Ostreš Guest

    What yours 'sh int fa0/0' and 'sh int fa0/1' says?
     
    Ivan Ostreš, Mar 2, 2005
    #8
  9. WAState

    WAState Guest

    Yes, I've flipped the "direction" of the test (so to speak) and the
    slowness continues to be in the same direction through the router.
     
    WAState, Mar 3, 2005
    #9
  10. WAState

    WAState Guest

    I have both FE ports forced to half duplex and 100Mbps (I've seen far
    too many problems with duplex and speed negotiation in the past).
     
    WAState, Mar 3, 2005
    #10
  11. WAState

    WAState Guest

    Snapshot taken today, last tests performed yesterday....

    #sho int f0/0
    FastEthernet0/0 is up, line protocol is up
    Hardware is AmdFE, address is 0004.c136.79c0 (bia 0004.c136.79c0)
    Internet address is www.xxx.yyy.zzz/27
    MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
    reliability 255/255, txload 1/255, rxload 1/255
    Encapsulation ARPA, loopback not set
    Keepalive set (10 sec)
    Half-duplex, 100Mb/s, 100BaseTX/FX
    ARP type: ARPA, ARP Timeout 04:00:00
    Last input 00:00:00, output 00:00:00, output hang never
    Last clearing of "show interface" counters 1w1d
    Queueing strategy: fifo
    Output queue 0/40, 0 drops; input queue 0/75, 0 drops
    5 minute input rate 1000 bits/sec, 1 packets/sec
    5 minute output rate 1000 bits/sec, 1 packets/sec
    386582 packets input, 127895373 bytes
    Received 235836 broadcasts, 0 runts, 0 giants, 0 throttles
    0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
    0 watchdog, 0 multicast
    0 input packets with dribble condition detected
    204303 packets output, 123685871 bytes, 0 underruns
    0 output errors, 94 collisions, 1 interface resets
    0 babbles, 0 late collision, 360 deferred
    0 lost carrier, 0 no carrier
    0 output buffer failures, 0 output buffers swapped out

    #sho int f0/1
    FastEthernet0/1 is up, line protocol is up
    Hardware is AmdFE, address is 0004.c136.79c1 (bia 0004.c136.79c1)
    Internet address is 192.168.254.201/24
    MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
    reliability 255/255, txload 1/255, rxload 1/255
    Encapsulation ARPA, loopback not set
    Keepalive set (10 sec)
    Half-duplex, 100Mb/s, 100BaseTX/FX
    ARP type: ARPA, ARP Timeout 04:00:00
    Last input 00:01:13, output 00:00:07, output hang never
    Last clearing of "show interface" counters 1w1d
    Queueing strategy: fifo
    Output queue 0/40, 0 drops; input queue 0/75, 0 drops
    5 minute input rate 0 bits/sec, 0 packets/sec
    5 minute output rate 0 bits/sec, 0 packets/sec
    111025 packets input, 110894489 bytes
    Received 3172 broadcasts, 0 runts, 0 giants, 0 throttles
    0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
    0 watchdog, 0 multicast
    0 input packets with dribble condition detected
    190936 packets output, 114023202 bytes, 0 underruns
    0 output errors, 1983 collisions, 1 interface resets
    0 babbles, 0 late collision, 4542 deferred
    0 lost carrier, 0 no carrier
    0 output buffer failures, 0 output buffers swapped out
     
    WAState, Mar 3, 2005
    #11
  12. WAState

    WAState Guest

    Also, as regards an "external" factor limiting the results: If I remove
    the router and connect the two machines through a hub, the results are
    ~80Mbps in both directions. That's 80% of the actual bit rate on Fast
    Ethernet... once you factor in overhead that's pretty close to the max
    real-world throughput.

    Add the 2621 back into the circuit and the throughput drops to ~10Mbps
    in one direction and ~6Mbps in the other. The ONLY thing changing is
    the presence or absence of the router.
     
    WAState, Mar 3, 2005
    #12
  13. WAState

    anybody43 Guest

    This does all seem pretty odd.

    It still worries me that the CPU is not 100%
    when the offered load is 10 times (80Mbps/10Mbps)
    the forwarding rate. How can the router CPU not be 100%?
    What is it doing if it is not working.? In something like a PC
    clearly it might be waiting on disk drives, whatever, but
    none of the routers that I have caried out performance tests
    on ever did anything other than forward more packets
    until the CPU ran out of steam.

    I do not understand how CPU metrics work, it seems a
    hard problem to me and I always have at the back
    of my mind that perhaps one day I will find some that
    lie. Maybe it's too busy to properly record what it is doing?

    OK back to reality.
    What ports are you using? NAT looks inside some packets
    eg DNS and H.323 (that I am aware of) (ftp?) and
    fudges up this and that. Maybe you are confusing
    it by sending random data on the DNS port (53 decimal)?

    If you want to pursue this I suggest that you go
    back to basics on the config. Take out NAT.

    Do the tests and gradually add more features until
    you replicate the issue.

    800 pps (10Mbps with 1500 byte packets) does not
    seem much for a 2621. With TCP I expect another 400pps
    in ACKs totalling 1200pps.

    Get a packet sniffer and check that there
    are no TCP retransmissions.

    OH!
    You do have enough memory? Rack up the logging
    level. Make sure that you are not getting memory
    alignment errors which are fixed up in software but hit
    performance.

    Finally!
    Back to square one.
    The tests must be broken and are not stressing the
    router otherwise the CPU would be 100%.

    How about bad memory (or bus) in the router
    corrupting packets and causing TCP
    re-transmissions?
    That would do it.

    Time to get the sniffer out.
     
    anybody43, Mar 3, 2005
    #13
    1. Advertisements

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.