Optimizing perfomance on T3 line

Hi there,

I just posted the nearly same article on alt.certification.cisco, but I
think it is better placed in that group.

But now to the problem I'am facing:

My company recently connected eigth remote sites via T3 lines - each site
got a dedicated one.
The backbone routers are Cisco 7200. On the remote site are 3600 routers or
7200
(two in each site - connected via HSRP). One of each remote routers is
handling data traffic, one is handling VoIP traffic. Only in the case of the
failure of one of the two routers or lines, the one surviving the failure
must handle
both types of traffic. The problem that just occurs is, that the
WAN-performance is weak, because we can´t reach the 45MBit/s - normaly we
get 15-25.

The question by now is: What are your standard configurations of the
interface and do you include some special commands such as altered MTUs to
improve performace?

Thanx in advance

Christoph

 

Are you using CEF switching? If not, what switching are you using?
What do the interfaces look like? Rising errors? Clean? Do all sites
use the same ISP/Telco and experience the same problems? Can you post
your sanitised configs too? A few debugs perhaps? sh proc cpu, etc,
preferably when everything is at peak load.

Tim

 

Hi there,

CEF is enabled. We are connected via two different ISPs. The first provider
is the primary for each site, the second one acts as the backup line. The
performance only varies a little when switching the lines.
I can post some config lines when I return to my office next tuesday.
The interfaces show no problems or errors and the cpu load is at about 2%.
By analyzing the performance (normally done via extended ping to produce
heavy load on the line).
So, on the routers everything seems to be fine, since they don't show any
problems, but througput is bad.

Christoph

 

Is your ISP rate-limiting ICMP? Do you see ping loss across the ccts?
Do you have an internal machines you can ftp too/from?

What's the set up? Run BGP at all sites or do you tunnel through their
IP addresses to remote sites?

Tim

 

If you ping from router to router, you should not this traffic is
process switched, and pose a much higher load on the routers than
traffic forwarded through them.

So if you want to test performance, you should test between hosts
behind the routers.

 

OK, maybee my example of producing load on the line was not the best.
Of course we tested from hosts inside the main LAN to host in the remote LAN
and had the same bad perfomance (even worse). So we commited on using large
ICMP pakets for testing. The intention was, to exclude local problems, which
(we proved that) don't exist on neither the one or other site.
There is no limitation in traffic inside the ISPs network and the ACLs do no
harm on any type of traffic inside the internal network.
ACLs just limit acces to the routers themselfs, the router peers and
broadcasttraffic.

But I just received a configuration file from a collegue, so that you may
get an impression (OK, formatting is bad, the IPs, TACAS and ACLs are
removed, since they don't have any information):

Building configuration...

Current configuration : 5258 bytes

no service pad

service timestamps debug datetime localtime service timestamps log datetime
localtime service password-encryption !

hostname XYZ

!

card type t3 1

logging queue-limit 100

logging buffered 64000 debugging

enable secret 5 <removed>

!

clock timezone MET 1

clock summer-time MEST recurring last Sun Mar 2:00 last Sun Oct 2:00

ip cef

ip tftp source-interface Loopback0

no ip domain lookup

no voice hpi capture buffer

no voice hpi capture destination

!

!

mta receive maximum-recipients 0

!

!

controller T3 1/0

cablelength 10

!

!

!

interface Loopback0

ip address REMOVED

!

interface FastEthernet0/0

no ip address

speed 100

full-duplex

!

interface FastEthernet0/0.1

description VLAN Daten

encapsulation dot1Q 3

ip address REMOVED

ip helper-address REMOVED

no ip redirects

ip directed-broadcast 99

standby 1 ip REMOVED

standby 1 priority 110

standby 1 preempt

!

interface FastEthernet0/0.2

description VLAN Voice

encapsulation dot1Q 187

ip address REMOVED

ip helper-address REMOVED

no ip redirects

ip directed-broadcast 99

standby 2 ip 53.121.187.1

standby 2 priority 110

standby 2 preempt

!

interface FastEthernet0/1

description REMOVED

ip address REMOVED

speed 100
full-duplex

interface Serial1/0

description REMOVED

ip address REMOVED
ip route-cache flow dsu bandwidth 44210 !

router eigrp 121

passive-interface FastEthernet0/0.1

passive-interface FastEthernet0/0.2

network REMOVED

distribute-list 10 out Serial1/0

no auto-summary

no eigrp log-neighbor-changes

!

no ip http server

ip flow-export source Loopback0

ip flow-export version 5

ip flow-export destination REMOVED

ip classless ip tacacs source-interface Loopback0 !

!

!

logging REMOVED

!

mgcp profile default

!

!

!

dial-peer cor custom

!

!

!

rtr responder

!

line con 0

password 7 <removed>

line aux 0

access-class 1 in

password 7 <removed>

line vty 0 4

access-class 1 in

password 7 <removed>

!

ntp clock-period 17180490

ntp source Loopback0

ntp server REMOVED source Loopback0

end

 

The configuration look ok - you have to tell exactly what you did,
how did you measure the performance ?

File transfer via ftp, or ?

What does "show proc cpu" say on the 2 routers while transferring
traffic ?

What is the latency from site A to site B ?

Is it a p2p T3, or some L2 or L3 VPN service ?

 

Hello, Christoph!
You wrote on Sun, 23 Nov 2003 23:11:03 +0100:

Can you provide show interface from FastEthernet and Serial interfaces? Is there
any errors on FE? Any input/output drops on Serial? What's the round-trip time
looks like? Which systems were you using to conduct throughput test? Did you
test with single flow or with multiple ones? TCP or UDP?

Why I asking -

1. Duplex mismatch on FE kills performance. Usualy you will get 30% maximum from
FE throughput.
2. Output serial drops - you have FIFO queue on the interface which by default
keeps 40 packets. If you have bursts which can't be absorbed by the queue then
you are getting tail-drops and back-off in TCP flow. Depends on OS you are using
it might affect performance more or less.
3. Round-trip - receive window in TCP stack should be bigger than bandwidth * 8
* RTT. Windows NT, for example, by default has 8Kbyte windows.

With best regards,
Andrey.

 

Ok, since FTP is not provided in our network any more, we normaly use
robocopy from the old NT ressource kits.
Monitoring the cpu and memory load on the router by sending files of
different size won't stress the routers. The load increases at about 2% so
we get a load of about 4-5% in sum.
The latency is in better cases at about 9msec (not meaining a better
performance) and at about 15msec in the worst case. We can deal with that
times, cause that is the range, we expected on the lines and that doesn't
seem to be the problem since even on slower lines the throughput can be
better as on the faster ones.
The line isn't a VPN, just a p2p connection.
We assume that it is more a problem of the line then of the routers, so we
are looking for something to increase the stability on serial1/0 if.

 

Call your ISP, ask them to log onto their kit and take a good look at
your interface. Lines can appear clean at one end, but you can get
bundles of rising errors at the other. Ask them if they've got any
rate-limiting in place on the interface. It might be clear 45Mbit at
your end, but they may have made a mistake in their config. Does your
ISP have an FTP server with any big files you can download? Use it and
test for bottlenecks on their network, but make sure there are none of
your own first.

Tim

 

Please be a bit more specific. The 720x ranges from the old chassis to the
new VXR chassis and - even more relevant - can digest a wide range of NPEs
that can differ in throughput by an order of magnitude.

45Mbps is a somewhat decent speed to measure correctly, as it starts to
depend on certain aspects of the implementations used to do the testing.
There are, for instance, people who walk around in the belief that Fast
Ethernet cannot achieve more than approx. 7MiByte/s throughput, because
they had the strange idea to use Win9x/ME as a test platform. For a
decent throughput test, you should utilize a plain TCP connection that
is artificielly loaded to the max. I'm using NetIO for that purpose. The
c't magazine (which you might happen to know) provides an advanced version
of that (mainly a rewrite that builds on more platforms than the original)
which they use in their throughput tests.

I only know about E3 which is way slower than T3. But I didn't use any
special stuff besides CEF and route-cache flow. You might have a look
on the queuing settings, though.

 

Everything Andre said is relevant and should be looked at. But if you
have long haul DS3s, you really need to read up on RFC 1323.


--

hsb

"Somehow I imagined this experience would be more rewarding" Calvin
*************** USE ROT13 TO SEE MY EMAIL ADDRESS ****************
********************************************************************
Due to the volume of email that I receive, I may not not be able to
reply to emails sent to my account. Please post a followup instead.
********************************************************************

 

Lots of good stuff here in bits and pieces, I will have a go at
putting it together.

First:
Do a cisco extended ping and make sure that there is no packet loss
and that there are not excessive errors on any links.
These links shuold run pretty clean with less than 1/10,000 errors.
I use the figure that errors should be less that 1/1,000,000 but
I do not have much real experience of WAN links. Maybe I am
being unrealistic.

Fix any problems that you find.

Then:
The most likely "problem" is that the tests that you are doing
are not capable of loading the network to it's capacity.

The round trip latency causes many attemps to load
the network to be ineffective.

For example with a 9ms RTT using cisco 18000 byte pings on
a T3 the max theoretical throughput is computed below.
I have assumed (I don't know) the best case which is that the
far end router returns the packets one by one as they are received.
I think that this is probably wrong so the real result will be worse.


Send 18000 * 8 bits = 144000 bits = 144000/45000000 = 3ms
This is the time that the sending router takes to put the packets
on to the wire. Note that the wire (full round trip path)
will be 3/9ths full. i.e. 1/3rd full.

No more traffic will be sent until this bunch of traffic
gets to the far end, is reflected back and is received by
the originating router. This will have been accomplished when the
last bit sent has been to the far end and back.

This will take the RTT plus any delays in the far end router.

Once one packet is recieved router sends it back so delay
in far end router is vey small (3/12 ms = 0.25 ms) let's ignore it.

So total time then is Time to wire for traffic + RTT.

This will be 12ms.

The max theoretical traffic is therefore 3ms worth every 12ms or 25%.

This assumes that the router can actually generate this much traffic.

If we model the case where the whole 18k is first received by the
far end before it is sent back we have.

3ms to wire
9s for last bit to get there
3ms to wire
9ms for last bit to get bask.

24ms RTT per 18000 bytes.

A single extended ping sesson will therefore generate no more than

3/24 = 1/8 or 12.5% of the max bandwidth.

The key is that unless the tests that you do can FILL the pipe
with bits then you will not be able to saturate the network.

This is caled the bandwidth-delay product.

You need 45,000,000 * 0.009 bits in the pipe or you cannot saturate
the network. This is 405000 bits or 50k Bytes.

MS Windows is notoriously bad at filling the pipe.

For example I witnessed a case where a windows disk drive was
copied across the Atlantic by dragging and dropping using explorer
where the slowest part of the network path was a T3.

Resultant data rate was 4k Bytes per second. It took about 8 hours
(IIRC) to transfer the 2G bytes.

What can you do?

Use more than one cisco ping at the same time.
You can telnet on to the same router more than one.
Use the routers at both ends.
You MUST keep an eye on the CPU, if it gets over say 50% or 60%
you will not, I feel, be able to trust the results.
Use as many routers as you have, add telnet sessions with pings
until the CPU goes too high.

Use FTP instead of Windows file transfers.
YOu will still need several ftp sessions but at least you will not
be stressing the routers.

Get a network analyser (Free EtherReal is GREAT) and find
out what the window sizes are and make sure that the sum of the
window sizes of all of your test sessions is more than that magic
(in your case) 50k Bytes.

I think that all windows file transfers between a pair of
machines use a single TCP session. In some windows releases (w2k?)
the default window sizes were I think 4k and so a single
client-server session will only stress your link to 8%.

BEST of ALL.
Find out how to use some of the MANY free test tools that are available.

eg TTCP.


Good luck, I think that you rnetwork is probably just fine.

If you want to get computer operations to go faster then
that is a different matter.