Optimizing perfomance on T3 line

Discussion in 'Cisco' started by Christoph Schad, Nov 23, 2003.

  1. Hi there,

    I just posted the nearly same article on alt.certification.cisco, but I
    think it is better placed in that group.

    But now to the problem I'am facing:

    My company recently connected eigth remote sites via T3 lines - each site
    got a dedicated one.
    The backbone routers are Cisco 7200. On the remote site are 3600 routers or
    (two in each site - connected via HSRP). One of each remote routers is
    handling data traffic, one is handling VoIP traffic. Only in the case of the
    failure of one of the two routers or lines, the one surviving the failure
    must handle
    both types of traffic. The problem that just occurs is, that the
    WAN-performance is weak, because we can´t reach the 45MBit/s - normaly we
    get 15-25.

    The question by now is: What are your standard configurations of the
    interface and do you include some special commands such as altered MTUs to
    improve performace?

    Thanx in advance

    Christoph Schad, Nov 23, 2003
    1. Advertisements

  2. Christoph Schad

    Tim Thorne Guest

    Are you using CEF switching? If not, what switching are you using?
    What do the interfaces look like? Rising errors? Clean? Do all sites
    use the same ISP/Telco and experience the same problems? Can you post
    your sanitised configs too? A few debugs perhaps? sh proc cpu, etc,
    preferably when everything is at peak load.

    Tim Thorne, Nov 23, 2003
    1. Advertisements

  3. Hi there,

    CEF is enabled. We are connected via two different ISPs. The first provider
    is the primary for each site, the second one acts as the backup line. The
    performance only varies a little when switching the lines.
    I can post some config lines when I return to my office next tuesday.
    The interfaces show no problems or errors and the cpu load is at about 2%.
    By analyzing the performance (normally done via extended ping to produce
    heavy load on the line).
    So, on the routers everything seems to be fine, since they don't show any
    problems, but througput is bad.

    Christoph Schad, Nov 23, 2003
  4. Christoph Schad

    Tim Thorne Guest

    Is your ISP rate-limiting ICMP? Do you see ping loss across the ccts?
    Do you have an internal machines you can ftp too/from?

    What's the set up? Run BGP at all sites or do you tunnel through their
    IP addresses to remote sites?

    Tim Thorne, Nov 23, 2003
  5. If you ping from router to router, you should not this traffic is
    process switched, and pose a much higher load on the routers than
    traffic forwarded through them.

    So if you want to test performance, you should test between hosts
    behind the routers.
    Jesper Skriver, Nov 23, 2003
  6. OK, maybee my example of producing load on the line was not the best.
    Of course we tested from hosts inside the main LAN to host in the remote LAN
    and had the same bad perfomance (even worse). So we commited on using large
    ICMP pakets for testing. The intention was, to exclude local problems, which
    (we proved that) don't exist on neither the one or other site.
    There is no limitation in traffic inside the ISPs network and the ACLs do no
    harm on any type of traffic inside the internal network.
    ACLs just limit acces to the routers themselfs, the router peers and

    But I just received a configuration file from a collegue, so that you may
    get an impression (OK, formatting is bad, the IPs, TACAS and ACLs are
    removed, since they don't have any information):

    Building configuration...

    Current configuration : 5258 bytes

    no service pad

    service timestamps debug datetime localtime service timestamps log datetime
    localtime service password-encryption !

    hostname XYZ


    card type t3 1

    logging queue-limit 100

    logging buffered 64000 debugging

    enable secret 5 <removed>


    clock timezone MET 1

    clock summer-time MEST recurring last Sun Mar 2:00 last Sun Oct 2:00

    ip cef

    ip tftp source-interface Loopback0

    no ip domain lookup

    no voice hpi capture buffer

    no voice hpi capture destination



    mta receive maximum-recipients 0



    controller T3 1/0

    cablelength 10




    interface Loopback0

    ip address REMOVED


    interface FastEthernet0/0

    no ip address

    speed 100



    interface FastEthernet0/0.1

    description VLAN Daten

    encapsulation dot1Q 3

    ip address REMOVED

    ip helper-address REMOVED

    no ip redirects

    ip directed-broadcast 99

    standby 1 ip REMOVED

    standby 1 priority 110

    standby 1 preempt


    interface FastEthernet0/0.2

    description VLAN Voice

    encapsulation dot1Q 187

    ip address REMOVED

    ip helper-address REMOVED

    no ip redirects

    ip directed-broadcast 99

    standby 2 ip

    standby 2 priority 110

    standby 2 preempt


    interface FastEthernet0/1

    description REMOVED

    ip address REMOVED

    speed 100

    interface Serial1/0

    description REMOVED

    ip address REMOVED
    ip route-cache flow dsu bandwidth 44210 !

    router eigrp 121

    passive-interface FastEthernet0/0.1

    passive-interface FastEthernet0/0.2

    network REMOVED

    distribute-list 10 out Serial1/0

    no auto-summary

    no eigrp log-neighbor-changes


    no ip http server

    ip flow-export source Loopback0

    ip flow-export version 5

    ip flow-export destination REMOVED

    ip classless ip tacacs source-interface Loopback0 !



    logging REMOVED


    mgcp profile default




    dial-peer cor custom




    rtr responder


    line con 0

    password 7 <removed>

    line aux 0

    access-class 1 in

    password 7 <removed>

    line vty 0 4

    access-class 1 in

    password 7 <removed>


    ntp clock-period 17180490

    ntp source Loopback0

    ntp server REMOVED source Loopback0

    Christoph Schad, Nov 23, 2003
  7. The configuration look ok - you have to tell exactly what you did,
    how did you measure the performance ?

    File transfer via ftp, or ?

    What does "show proc cpu" say on the 2 routers while transferring
    traffic ?

    What is the latency from site A to site B ?

    Is it a p2p T3, or some L2 or L3 VPN service ?
    Jesper Skriver, Nov 23, 2003
  8. Hello, Christoph!
    You wrote on Sun, 23 Nov 2003 23:11:03 +0100:

    Can you provide show interface from FastEthernet and Serial interfaces? Is there
    any errors on FE? Any input/output drops on Serial? What's the round-trip time
    looks like? Which systems were you using to conduct throughput test? Did you
    test with single flow or with multiple ones? TCP or UDP?

    Why I asking -

    1. Duplex mismatch on FE kills performance. Usualy you will get 30% maximum from
    FE throughput.
    2. Output serial drops - you have FIFO queue on the interface which by default
    keeps 40 packets. If you have bursts which can't be absorbed by the queue then
    you are getting tail-drops and back-off in TCP flow. Depends on OS you are using
    it might affect performance more or less.
    3. Round-trip - receive window in TCP stack should be bigger than bandwidth * 8
    * RTT. Windows NT, for example, by default has 8Kbyte windows.

    With best regards,
    Andrey Tarasov, Nov 23, 2003
  9. Ok, since FTP is not provided in our network any more, we normaly use
    robocopy from the old NT ressource kits.
    Monitoring the cpu and memory load on the router by sending files of
    different size won't stress the routers. The load increases at about 2% so
    we get a load of about 4-5% in sum.
    The latency is in better cases at about 9msec (not meaining a better
    performance) and at about 15msec in the worst case. We can deal with that
    times, cause that is the range, we expected on the lines and that doesn't
    seem to be the problem since even on slower lines the throughput can be
    better as on the faster ones.
    The line isn't a VPN, just a p2p connection.
    We assume that it is more a problem of the line then of the routers, so we
    are looking for something to increase the stability on serial1/0 if.
    Christoph Schad, Nov 24, 2003
  10. Christoph Schad

    Tim Thorne Guest

    Call your ISP, ask them to log onto their kit and take a good look at
    your interface. Lines can appear clean at one end, but you can get
    bundles of rising errors at the other. Ask them if they've got any
    rate-limiting in place on the interface. It might be clear 45Mbit at
    your end, but they may have made a mistake in their config. Does your
    ISP have an FTP server with any big files you can download? Use it and
    test for bottlenecks on their network, but make sure there are none of
    your own first.

    Tim Thorne, Nov 24, 2003
  11. Christoph Schad

    Andre Beck Guest

    Please be a bit more specific. The 720x ranges from the old chassis to the
    new VXR chassis and - even more relevant - can digest a wide range of NPEs
    that can differ in throughput by an order of magnitude.
    45Mbps is a somewhat decent speed to measure correctly, as it starts to
    depend on certain aspects of the implementations used to do the testing.
    There are, for instance, people who walk around in the belief that Fast
    Ethernet cannot achieve more than approx. 7MiByte/s throughput, because
    they had the strange idea to use Win9x/ME as a test platform. For a
    decent throughput test, you should utilize a plain TCP connection that
    is artificielly loaded to the max. I'm using NetIO for that purpose. The
    c't magazine (which you might happen to know) provides an advanced version
    of that (mainly a rewrite that builds on more platforms than the original)
    which they use in their throughput tests.
    I only know about E3 which is way slower than T3. But I didn't use any
    special stuff besides CEF and route-cache flow. You might have a look
    on the queuing settings, though.
    Andre Beck, Dec 28, 2003
  12. Christoph Schad

    Hansang Bae Guest

    Everything Andre said is relevant and should be looked at. But if you
    have long haul DS3s, you really need to read up on RFC 1323.



    "Somehow I imagined this experience would be more rewarding" Calvin
    *************** USE ROT13 TO SEE MY EMAIL ADDRESS ****************
    Due to the volume of email that I receive, I may not not be able to
    reply to emails sent to my account. Please post a followup instead.
    Hansang Bae, Dec 28, 2003
  13. Christoph Schad

    AnyBody43 Guest

    Lots of good stuff here in bits and pieces, I will have a go at
    putting it together.

    Do a cisco extended ping and make sure that there is no packet loss
    and that there are not excessive errors on any links.
    These links shuold run pretty clean with less than 1/10,000 errors.
    I use the figure that errors should be less that 1/1,000,000 but
    I do not have much real experience of WAN links. Maybe I am
    being unrealistic.

    Fix any problems that you find.

    The most likely "problem" is that the tests that you are doing
    are not capable of loading the network to it's capacity.

    The round trip latency causes many attemps to load
    the network to be ineffective.

    For example with a 9ms RTT using cisco 18000 byte pings on
    a T3 the max theoretical throughput is computed below.
    I have assumed (I don't know) the best case which is that the
    far end router returns the packets one by one as they are received.
    I think that this is probably wrong so the real result will be worse.

    Send 18000 * 8 bits = 144000 bits = 144000/45000000 = 3ms
    This is the time that the sending router takes to put the packets
    on to the wire. Note that the wire (full round trip path)
    will be 3/9ths full. i.e. 1/3rd full.

    No more traffic will be sent until this bunch of traffic
    gets to the far end, is reflected back and is received by
    the originating router. This will have been accomplished when the
    last bit sent has been to the far end and back.

    This will take the RTT plus any delays in the far end router.

    Once one packet is recieved router sends it back so delay
    in far end router is vey small (3/12 ms = 0.25 ms) let's ignore it.

    So total time then is Time to wire for traffic + RTT.

    This will be 12ms.

    The max theoretical traffic is therefore 3ms worth every 12ms or 25%.

    This assumes that the router can actually generate this much traffic.

    If we model the case where the whole 18k is first received by the
    far end before it is sent back we have.

    3ms to wire
    9s for last bit to get there
    3ms to wire
    9ms for last bit to get bask.

    24ms RTT per 18000 bytes.

    A single extended ping sesson will therefore generate no more than

    3/24 = 1/8 or 12.5% of the max bandwidth.

    The key is that unless the tests that you do can FILL the pipe
    with bits then you will not be able to saturate the network.

    This is caled the bandwidth-delay product.

    You need 45,000,000 * 0.009 bits in the pipe or you cannot saturate
    the network. This is 405000 bits or 50k Bytes.

    MS Windows is notoriously bad at filling the pipe.

    For example I witnessed a case where a windows disk drive was
    copied across the Atlantic by dragging and dropping using explorer
    where the slowest part of the network path was a T3.

    Resultant data rate was 4k Bytes per second. It took about 8 hours
    (IIRC) to transfer the 2G bytes.

    What can you do?

    Use more than one cisco ping at the same time.
    You can telnet on to the same router more than one.
    Use the routers at both ends.
    You MUST keep an eye on the CPU, if it gets over say 50% or 60%
    you will not, I feel, be able to trust the results.
    Use as many routers as you have, add telnet sessions with pings
    until the CPU goes too high.

    Use FTP instead of Windows file transfers.
    YOu will still need several ftp sessions but at least you will not
    be stressing the routers.

    Get a network analyser (Free EtherReal is GREAT) and find
    out what the window sizes are and make sure that the sum of the
    window sizes of all of your test sessions is more than that magic
    (in your case) 50k Bytes.

    I think that all windows file transfers between a pair of
    machines use a single TCP session. In some windows releases (w2k?)
    the default window sizes were I think 4k and so a single
    client-server session will only stress your link to 8%.

    BEST of ALL.
    Find out how to use some of the MANY free test tools that are available.

    eg TTCP.

    Good luck, I think that you rnetwork is probably just fine.

    If you want to get computer operations to go faster then
    that is a different matter.
    AnyBody43, Jan 1, 2004
    1. Advertisements

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.