Need help understanding throughput

Discussion in 'Cisco' started by srp336, Mar 8, 2006.

  1. srp336

    srp336 Guest

    I need some helping understanding why 25% of the available bandwidth I
    think I have never gets used.

    I have 1 full T1 plus 14 channels of another T1, which should be a
    theoretical maximum bandwidth of 2432 kbit/s, correct? I'm using EIGRP
    unequal cost.

    According my router stats, there are occasional peaks at about 75% of
    1800 kbit/s, but for the most part a file copy operation will never get
    more than 1200 to 1400 kbit/s, even on a mostly quiet network.

    Can someone help me understand why this is happening so I can explain
    to management why we aren't getting what they expected?

    srp336, Mar 8, 2006
    1. Advertisements

  2. srp336

    Merv Guest

    Please provide more detail about the file copy operation.

    FTP or Windows copy ( read NETBIOS over TCP/IP)?

    Size of file being copied, how long it take, etc, etc
    Merv, Mar 8, 2006
    1. Advertisements

  3. Depending on the latency of your link you may be hitting window size
    problems (which will do this type of thing). If you have someone that is
    comfortable with Linux (to the point of building a custom built kernel)

    is a pointer to a tool that with a Linux server on one end of your link and
    a machine (preferably the machine doing the file transfer) on the other end
    with a Java enabled web browser will tell you all sorts of interesting things
    about the link and more importantly make suggestions about buffer changes on
    the end hosts to get better performance. Its a marvelous tool, it helped us
    take a highish latency (17 msec) gig link from 35 megabits per second to
    995 megabits per second (window scaling and increased tcp buffers in the
    kernel) on one of our research light paths.

    Peter Van Epp / Operations and Technical Support
    Simon Fraser University, Burnaby, B.C. Canada
    Peter Van Epp, Mar 9, 2006
  4. First question - how do you measure the throutput? Do you check the
    interface utilization on the router itself? Or 1200/1400/1800 Kbit/s is the
    speed, shown at your application? What application do you use? Is it FTP
    transfer? What is your end-point (where do you transfer to/from)? What
    Internet connection do they have? Do they limit transfer per-session?

    Another aspect - FTP (and any other application) measure throutput at the
    Application Layer of OSI model (file size divided by time). But real
    transfer has some additional information - FTP adds TCP header (application
    information), then IP protocol adds IP header (IP addresses), then Frame
    Relay or HDLC at your T1 adds it's own information. And only then the frame
    gets into the wire. Lets say the frame is 1500 bytes. Out of these 1500
    bytes "overhead" could be 60-70 bytes.

    Then, any TCP connection has some extra packets in addition to normal data
    transfer (confirmations, retransmissions, out-of-sequence frames, etc.).
    Thus could be minimal, but counting all this together could give you that
    25% of "missing bandwidth".

    Good luck,

, Mar 9, 2006
  5. srp336

    Charlie Root Guest

    Add 'load-interval 30', to start 10-50 sessions simultaneously and then
    periodically check 'sh int', then see what you get.

    Kind regards,
    Charlie Root, Mar 9, 2006
  6. srp336

    Sam Wilson Guest

    Also TCP works by tentatively increasing the amount of data it tries to
    send until a packet goes missing, whereupon it backs off and starts
    increasing again. The missing packet may be dropped by a router
    somewhere on the path or it may be because an output queue overflowed in
    the host itself. Thus for a single TCP stream throughput against time
    is a sawtooth pattern; it can never fill the whole capacity of the network.

    Sam Wilson, Mar 9, 2006
  7. srp336

    Adtran-ACES Guest

    YOu have to be configured into MLPPP between the two interfaces
    (interface 1 with 24 channels and the second interface with the 14
    channel) This creates a virual single interface that will then allow of
    dynamic use. Otherwise you only have 2 PtP

    Michael Stokes, CTO
    stokesTechnologies, Inc.
    Adtran-ACES, Mar 9, 2006
  8. srp336

    Merv Guest

    not true
    Merv, Mar 9, 2006
  9. srp336

    Merv Guest

    What is the busy-hour CPU utilization of the routers at each of of the
    two lingks

    show process cpu

    show process cpu history
    Merv, Mar 9, 2006
  10. srp336

    Sam Wilson Guest

    What's not true? Go on, give us a clue... :)

    Sam Wilson, Mar 9, 2006
  11. srp336

    rdymek Guest

    I'd have to agree with Merv. I believe what Merv was getting at is
    that MLPPP is not the only way to accomplish this (although is one way)
    - EIGRP does in fact do unequal cost load balancing; however, at the
    layer 3 perspective.

    EIGRP can be configured to use per-session or per-packet load balancing
    of up to 6 links. Of course, using MLPPP is layer two, and layer 3
    wouldn't have to bother with the load balancing. Personally I prefer
    to use Layer 3, EIGRP unequal cost per-packet load balancing, if the
    router has enough resources to handle per-packet. If not, I go for

    Regarding the load issue, its most likely a combination of overhead,
    TCP widowing, and flow control. If using regular windows file copy I
    have never seen a single file copy from a single host come close to
    maxing out a T1. Windows likes to throttle a lot preventing any single
    file copy to clobber the line...

    I've always said that general circumstances, you get about 25% loss
    with overhead (closer to 35% if using ATM) - which fits this scenario.

    rdymek, Mar 9, 2006
  12. srp336

    Adtran-ACES Guest

    My point in suggestion MLPPP was that it has a smaller over head as the
    bundling is done at layer 2 giving less tcp overhead need. I agree the
    combination of EIGRP and load balancing per-packet works well for
    load-balance/failover. I was addressing the question of monitoring the
    total bandwidth consumption which would be easier to determine and
    monitor (SNMP) vis-à-vis MLPPP
    Adtran-ACES, Mar 9, 2006
  13. srp336

    srp336 Guest

    Thanks to everyone for their responses... I've been getting a bit
    frustrated on this.

    I did want to mention a few further details on this. The router on this
    end is a 4700 and the remote side is a 2621. I did recently notice that
    fast switching is disabled.

    The test I am doing is a Windows file copy of a 100mb test file. I time
    the copy and figure out the average troughput (file size divided by
    time). After reading the responses here, I've tried one ftp transfer of
    the same file and can up with 816 kbit/s.

    The maximum CPU utilization I see in show proc cpu hist is about 50%.

    I was considering MLPPP vs. EIGRP. Maybe I'll give it a try.
    srp336, Mar 9, 2006
  14. srp336

    Merv Guest

    As a first priority you should enable CEF switching on both routers
    - to check if CEF is enabled "show ip cef"
    - to enable "ip cef"

    What IOS version is running on each router?

    BTW What is the business issue that you are trying to address?
    Merv, Mar 9, 2006
  15. srp336

    srp336 Guest

    I actually tried turning that on previously. It wasn't previously
    enable on either side. Once I enable that, it seems to turn ip
    route-cache back on (I've got that turned off now, too). After that,
    the copy tests seem to send all data through one serial port or the
    other and the resulting average throughput is even less than before.

    The use of these serial lines is mostly file transfers between the two
    locations. I just need to prove that we're getting the most out of them
    that we can (fastest transfer rate).
    srp336, Mar 10, 2006
  16. srp336

    Merv Guest

    If you have to disable CEF and fast switching, then every packet is
    being processed switched - that is bad news.

    Time to give MLPPP a whirl as you defintely want to be able to enable
    CEF or at least fast switching to get away from process switching.
    Merv, Mar 10, 2006
  17. srp336

    srp336 Guest

    OK... I'll move on to MLPPP. One question though...I'm still not
    certain why this didn't work. You can't have fast switching without
    route caching? And with route caching on, I didn't get any load
    balancing. Could I have done something wrong?
    srp336, Mar 10, 2006
  18. Route cache and fastswitcing are synonyms. Fast switching is what you
    get when you configure "ip route-cache". With fast switching you get
    per-destination load balancing, so any given connection will only use one
    of your many links.

    Process switching does per-packet load balancing.

    CEF is a later and greater fast switching that can do per-packet or
    per-destination load balancing. Per-destination is the default.
    Martin Gallagher, Mar 12, 2006
  19. srp336

    Merv Guest

    Based on Martin's post, also try CEF per-packet laod balancing and
    compare to MLPPP results

    see Cisco TAC article:

    for CEF per-packet load balancing you need to configure
    " ip load-sharing per-packet" on each T1 interface

    Make a note of the CPU utilization for each setup
    Merv, Mar 12, 2006
  20. srp336

    srp336 Guest

    OK... I'm trying cef per-packet now. So far, it doesn't really seem any
    faster for me, but I'll keep testing.

    Next, I'll try MLPPP. I'm trying to read up on it. I notice that I
    can't seem to create the "Interface Multilink1" on the local router. I
    can add "ppp multilink" without any options to the serial interfaces.
    Does the IOS on the local side need to be updated?
    srp336, Mar 13, 2006
    1. Advertisements

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.