CEF slowing down FE to FE traffic

I'm experimenting with CEF on a 2621. I have a fast PC on each FE port,
each PC running a small tight program that opens a TCP socket and
measures the throughput to/from the other PC. I wrote the code myself,
it's very fast and is definitely not the limiting factor on speed (if
the two PC's are connected through just a hub, throughput is ~80Mbps).

When the 2621 sits between them, throughput is dramatically lower...
~10 Mbps in one direction and ~6600 Kbps in the other (with CEF
disabled). I know the 2600 platform cannot sustain wire speeds, but at
least this gives us a baseline.

So I enabled CEF, expecting at least **some** increase in speed. Not
so. In one direction (input F0/0 to output F0/1) the speed is
unchanged; in the other (input F0/1 to output F0/0) the speed drops
dramatically depending upon what interface settings I use.

For example, with "ip route-cache flow" on both FE's the speed is
roughly the same as the CEF-disabled rate (~6600 Kbps). However, add
"ip route-cache cef" to both interfaces and the F0/1 to F0/0 speed
drops to 700 Kbps (a drop of nearly 90%!). Removing all "ip
route-cache" statements from both interfaces yields a middle ground of
about 4500 Kbps.

Meanwhile, in all cases the F0/0 to F0/1 traffic continues to be
~10Mbps, utterly unaffected by any CEF or route-cache settings.

The only "extra" thing this router is doing is NAT. F0/0 is the outside
interface, and F0/1 is the inside interface. NAT is using only static
addresses. I don't know if NAT would be the limiting factor on speed,
or if that burden would be unidirectional. Does NAT force all packets
to be process switched, effectively disabling CEF? I would think that
CEF's routing table could still be built even allowing for NAT, just as
it allows for ACL's.

Any ideas gratefully accepted. I'm just trying to understand what's
happening here so I can get the most out of these routers.

 

I've just finished confirming (via "sho int stats") that CEF is indeed
operational. The vast majority of packets are being handled by the
"route cache" switching path and not the "processor" switching path, as
reported by this command.

Meanwhile, F0/1 -> F0/0 traffic continues to be roughly 1/2 the
throughput of F0/0 -> F0/1....

 

It would be nice to see 'show proc cpu' when doing the tests with
CEF....

 

Yes, the CPU load is lower when running CEF. That's to be expected.
Here are some approximate CPU load numbers...

F0/0 -> F0/1 traffic: 87% w/o CEF, 30% w/ CEF
F0/1 -> F0/0 traffic: 52% w/o CEF, 30% w/ CEF

It's delightful that the CPU isn't as heavily loaded, but I don't like
the 40-50% drop in throughput in the F0/1 -> F0/0 direction. Even more
interesting, the throughput in the F0/0 -> F0/1 direction appears to be
nearly constant (under 5% variation) regardless of all other factors.

So we've proven that CEF saves CPU cycles. We still don't know why 1)
one direction is slower than the other, and 2) why CEF is slower than
process switching in that same direction.

Any ideas, or suggested tests?

 

Trying chnaging the IOS version in use on the box

 

First things first. Have you tried SWAPPING your test programs? Move
it from Fa0/0 to Fa0/1 and vice versa.

You can always try a differnt IOS version if you have one handy (the
release notes may have some blurb about a cef/switching bug.


--

hsb


"Somehow I imagined this experience would be more rewarding" Calvin
**************************ROT13 MY ADDRESS*************************
Due to the volume of email that I receive, I may not not be able to
reply to emails sent to my account. Please post a followup instead.
********************************************************************

 

"I'm experimenting with CEF on a 2621"
"F0/0 -> F0/1 traffic: 87% w/o CEF, 30% w/ CEF
F0/1 -> F0/0 traffic: 52% w/o CEF, 30% w/ CEF"
"It's delightful that the CPU isn't as heavily loaded"


My experience is that the throughput of a router in
a particular configuration is limited
*either* by the CPU *or* by external factors.

Since with CEF you are not hitting 100% CPU there is
I suggest some external (to the router) factor affecting the tests.

In summary the tests _must_ be broken! There is pretty much no
other explanation.

Just 2c worth.

For example, you have a duplex missmatch and with
process switching the rhythm of the conversation is
such that you are not hitting it however when you switch to
CEF the change in the rhythm causes lost frames.

Finally, my understanding is that pretty much all of the fast
switching methods, CEF, optimum, fast, flow have about
the same performance (say +/- a few percent) however CEF
does not have a cache that gets populated whenever the first
matching packet is routed but has a FIB that gets populated
from the routing table which only changes when the routing table
changes. This difference could be critical for some applications.

 

What yours 'sh int fa0/0' and 'sh int fa0/1' says?

 

Yes, I've flipped the "direction" of the test (so to speak) and the
slowness continues to be in the same direction through the router.

 

I have both FE ports forced to half duplex and 100Mbps (I've seen far
too many problems with duplex and speed negotiation in the past).

 

Snapshot taken today, last tests performed yesterday....

#sho int f0/0
FastEthernet0/0 is up, line protocol is up
Hardware is AmdFE, address is 0004.c136.79c0 (bia 0004.c136.79c0)
Internet address is www.xxx.yyy.zzz/27
MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Half-duplex, 100Mb/s, 100BaseTX/FX
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 1w1d
Queueing strategy: fifo
Output queue 0/40, 0 drops; input queue 0/75, 0 drops
5 minute input rate 1000 bits/sec, 1 packets/sec
5 minute output rate 1000 bits/sec, 1 packets/sec
386582 packets input, 127895373 bytes
Received 235836 broadcasts, 0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast
0 input packets with dribble condition detected
204303 packets output, 123685871 bytes, 0 underruns
0 output errors, 94 collisions, 1 interface resets
0 babbles, 0 late collision, 360 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out

#sho int f0/1
FastEthernet0/1 is up, line protocol is up
Hardware is AmdFE, address is 0004.c136.79c1 (bia 0004.c136.79c1)
Internet address is 192.168.254.201/24
MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Half-duplex, 100Mb/s, 100BaseTX/FX
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:01:13, output 00:00:07, output hang never
Last clearing of "show interface" counters 1w1d
Queueing strategy: fifo
Output queue 0/40, 0 drops; input queue 0/75, 0 drops
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
111025 packets input, 110894489 bytes
Received 3172 broadcasts, 0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast
0 input packets with dribble condition detected
190936 packets output, 114023202 bytes, 0 underruns
0 output errors, 1983 collisions, 1 interface resets
0 babbles, 0 late collision, 4542 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out

 

Also, as regards an "external" factor limiting the results: If I remove
the router and connect the two machines through a hub, the results are
~80Mbps in both directions. That's 80% of the actual bit rate on Fast
Ethernet... once you factor in overhead that's pretty close to the max
real-world throughput.

Add the 2621 back into the circuit and the throughput drops to ~10Mbps
in one direction and ~6Mbps in the other. The ONLY thing changing is
the presence or absence of the router.

 

This does all seem pretty odd.

It still worries me that the CPU is not 100%
when the offered load is 10 times (80Mbps/10Mbps)
the forwarding rate. How can the router CPU not be 100%?
What is it doing if it is not working.? In something like a PC
clearly it might be waiting on disk drives, whatever, but
none of the routers that I have caried out performance tests
on ever did anything other than forward more packets
until the CPU ran out of steam.

I do not understand how CPU metrics work, it seems a
hard problem to me and I always have at the back
of my mind that perhaps one day I will find some that
lie. Maybe it's too busy to properly record what it is doing?

OK back to reality.
What ports are you using? NAT looks inside some packets
eg DNS and H.323 (that I am aware of) (ftp?) and
fudges up this and that. Maybe you are confusing
it by sending random data on the DNS port (53 decimal)?

If you want to pursue this I suggest that you go
back to basics on the config. Take out NAT.

Do the tests and gradually add more features until
you replicate the issue.

800 pps (10Mbps with 1500 byte packets) does not
seem much for a 2621. With TCP I expect another 400pps
in ACKs totalling 1200pps.

Get a packet sniffer and check that there
are no TCP retransmissions.

OH!
You do have enough memory? Rack up the logging
level. Make sure that you are not getting memory
alignment errors which are fixed up in software but hit
performance.

Finally!
Back to square one.
The tests must be broken and are not stressing the
router otherwise the CPU would be 100%.

How about bad memory (or bus) in the router
corrupting packets and causing TCP
re-transmissions?
That would do it.

Time to get the sniffer out.