Bandwidth issue

Discussion in 'Cisco' started by MC, Feb 2, 2007.

  1. MC

    MC Guest

    OK, Got a weird one.

    Broadwing MPLS between two sites each having full DS3 access into 3662
    router each site. was installed in November.

    Do not have any baselines to go by, but at first we did get a good copy
    of SANs data several times across the link and seems ran about 30-35
    mbps throughput, this was before Christmas.

    Sometime around the first of the year we had MTU issues on the broadwing
    MPLS net, they had replaced hardware and had an incorrect MTU setting.
    Since that was fixed seems that we get very low bandwidth, varies
    between 2mpbs and will jump sometimes to 10-13 mbps and back down to
    different rates in between. Not seeing any errors on the circuits.
    Broadwing has went over their network and routers at great lengths.

    Is their anything I can look at on the routers to see if anything on our
    routers configured wrong or other errors, etc ?

    I can post configs, just do not have access to the routers until morning.

    As mentioned, seemed like was working, and not after the MTU fix and we
    keep blaming broadwing but need to be 100 percent not on our end,
    Although I can not find anything wrong. Also router CPU is 3-5 percent
    total on each at the most. no interface ACL's, just a few for routing
    redistribution but just a few.

    I'll take any suggestions to look at anything.

    Thanks,

    MC
     
    MC, Feb 2, 2007
    #1
    1. Advertising

  2. MC

    Guest

    On 2 Feb, 00:04, MC <> wrote:
    > OK, Got a weird one.
    >
    > Broadwing MPLS between two sites each having full DS3 access into 3662
    > router each site. was installed in November.
    >
    > Do not have any baselines to go by, but at first we did get a good copy
    > of SANs data several times across the link and seems ran about 30-35
    > mbps throughput, this was before Christmas.
    >
    > Sometime around the first of the year we had MTU issues on the broadwing
    > MPLS net, they had replaced hardware and had an incorrect MTU setting.
    > Since that was fixed seems that we get very low bandwidth, varies
    > between 2mpbs and will jump sometimes to 10-13 mbps and back down to
    > different rates in between. Not seeing any errors on the circuits.
    > Broadwing has went over their network and routers at great lengths.
    >
    > Is their anything I can look at on the routers to see if anything on our
    > routers configured wrong or other errors, etc ?
    >
    > I can post configs, just do not have access to the routers until morning.
    >
    > As mentioned, seemed like was working, and not after the MTU fix and we
    > keep blaming broadwing but need to be 100 percent not on our end,
    > Although I can not find anything wrong. Also router CPU is 3-5 percent
    > total on each at the most. no interface ACL's, just a few for routing
    > redistribution but just a few.
    >
    > I'll take any suggestions to look at anything.
    >
    > Thanks,
    >
    > MC


    This is quite tricky. Measuring link bandwidth is not always
    straightforward
    and it is very easy to get the wrong answers.

    You want to check for fragmentation:-

    Get a packet capture tool (say Ethereal or Wireshark, BTH I find
    that anything later than 0.10.14 on Windows crashes frequently).
    Use it to find out how to send max size ping packets
    without fragmentation. Windows "ping -l xxxx", xxxx ~=1472.
    Send some across link and use capture at both ends to see
    if there is fragmentation.

    To do a pure bandwidth test use iperf in UDP mode
    OR more gently and with fewer problems use TCP
    but add enough iperf copies to saturate the link.
    You can tell when saturated since the rate of
    remaining copies of iperf reduces as new ones are
    added.
     
    , Feb 2, 2007
    #2
    1. Advertising

  3. MC

    MC Guest

    wrote:
    > On 2 Feb, 00:04, MC <> wrote:
    >> OK, Got a weird one.
    >>
    >> Broadwing MPLS between two sites each having full DS3 access into 3662
    >> router each site. was installed in November.
    >>
    >> Do not have any baselines to go by, but at first we did get a good copy
    >> of SANs data several times across the link and seems ran about 30-35
    >> mbps throughput, this was before Christmas.
    >>
    >> Sometime around the first of the year we had MTU issues on the broadwing
    >> MPLS net, they had replaced hardware and had an incorrect MTU setting.
    >> Since that was fixed seems that we get very low bandwidth, varies
    >> between 2mpbs and will jump sometimes to 10-13 mbps and back down to
    >> different rates in between. Not seeing any errors on the circuits.
    >> Broadwing has went over their network and routers at great lengths.
    >>
    >> Is their anything I can look at on the routers to see if anything on our
    >> routers configured wrong or other errors, etc ?
    >>
    >> I can post configs, just do not have access to the routers until morning.
    >>
    >> As mentioned, seemed like was working, and not after the MTU fix and we
    >> keep blaming broadwing but need to be 100 percent not on our end,
    >> Although I can not find anything wrong. Also router CPU is 3-5 percent
    >> total on each at the most. no interface ACL's, just a few for routing
    >> redistribution but just a few.
    >>
    >> I'll take any suggestions to look at anything.
    >>
    >> Thanks,
    >>
    >> MC

    >
    > This is quite tricky. Measuring link bandwidth is not always
    > straightforward
    > and it is very easy to get the wrong answers.
    >
    > You want to check for fragmentation:-
    >
    > Get a packet capture tool (say Ethereal or Wireshark, BTH I find
    > that anything later than 0.10.14 on Windows crashes frequently).
    > Use it to find out how to send max size ping packets
    > without fragmentation. Windows "ping -l xxxx", xxxx ~=1472.
    > Send some across link and use capture at both ends to see
    > if there is fragmentation.
    >
    > To do a pure bandwidth test use iperf in UDP mode
    > OR more gently and with fewer problems use TCP
    > but add enough iperf copies to saturate the link.
    > You can tell when saturated since the rate of
    > remaining copies of iperf reduces as new ones are
    > added.
    >
    >
    >
    >

    Thanks very much, I'll give that a try when get back to the office.
    MC
     
    MC, Feb 2, 2007
    #3
  4. MC

    Guest

    On 2 Feb, 10:30, MC <> wrote:
    > wrote:
    > > On 2 Feb, 00:04, MC <> wrote:
    > >> OK, Got a weird one.

    >
    > >> Broadwing MPLS between two sites each having full DS3 access into 3662
    > >> router each site. was installed in November.

    >
    > >> Do not have any baselines to go by, but at first we did get a good copy
    > >> of SANs data several times across the link and seems ran about 30-35
    > >> mbps throughput, this was before Christmas.

    >
    > >> Sometime around the first of the year we had MTU issues on the broadwing
    > >> MPLS net, they had replaced hardware and had an incorrect MTU setting.
    > >> Since that was fixed seems that we get very low bandwidth, varies
    > >> between 2mpbs and will jump sometimes to 10-13 mbps and back down to
    > >> different rates in between. Not seeing any errors on the circuits.
    > >> Broadwing has went over their network and routers at great lengths.

    >
    > >> Is their anything I can look at on the routers to see if anything on our
    > >> routers configured wrong or other errors, etc ?

    >
    > >> I can post configs, just do not have access to the routers until morning.

    >
    > >> As mentioned, seemed like was working, and not after the MTU fix and we
    > >> keep blaming broadwing but need to be 100 percent not on our end,
    > >> Although I can not find anything wrong. Also router CPU is 3-5 percent
    > >> total on each at the most. no interface ACL's, just a few for routing
    > >> redistribution but just a few.

    >
    > >> I'll take any suggestions to look at anything.

    >
    > >> Thanks,

    >
    > >> MC

    >
    > > This is quite tricky. Measuring link bandwidth is not always
    > > straightforward
    > > and it is very easy to get the wrong answers.

    >
    > > You want to check for fragmentation:-

    >
    > > Get a packet capture tool (say Ethereal or Wireshark, BTH I find
    > > that anything later than 0.10.14 on Windows crashes frequently).
    > > Use it to find out how to send max size ping packets
    > > without fragmentation. Windows "ping -l xxxx", xxxx ~=1472.
    > > Send some across link and use capture at both ends to see
    > > if there is fragmentation.

    >
    > > To do a pure bandwidth test use iperf in UDP mode
    > > OR more gently and with fewer problems use TCP
    > > but add enough iperf copies to saturate the link.
    > > You can tell when saturated since the rate of
    > > remaining copies of iperf reduces as new ones are
    > > added.

    >
    > Thanks very much, I'll give that a try when get back to the office.
    > MC- Hide quoted text -
    >
    > - Show quoted text -


    One more thing.
    You can use the sh int command to get an idea of the
    data rate but be aware that the 5min (x min) number
    takes data from more than 5 mins. The "5" refers to a
    constant in the mathematical technique
    used to compute the value. Search for [exponential weighted
    average time constant]. When offered a full scale step
    change of traffic level (say from 0 to 100%) the displayed
    number takes 20 minutes to stabilise completely.

    int xxx
    load-interval 30

    changes the "time constant" to 30 seconds.

    Also you can use snmp (getif.exe) to graph the values.
     
    , Feb 2, 2007
    #4
  5. MC

    MC Guest

    wrote:
    > On 2 Feb, 10:30, MC <> wrote:
    >> wrote:
    >>> On 2 Feb, 00:04, MC <> wrote:
    >>>> OK, Got a weird one.
    >>>> Broadwing MPLS between two sites each having full DS3 access into 3662
    >>>> router each site. was installed in November.
    >>>> Do not have any baselines to go by, but at first we did get a good copy
    >>>> of SANs data several times across the link and seems ran about 30-35
    >>>> mbps throughput, this was before Christmas.
    >>>> Sometime around the first of the year we had MTU issues on the broadwing
    >>>> MPLS net, they had replaced hardware and had an incorrect MTU setting.
    >>>> Since that was fixed seems that we get very low bandwidth, varies
    >>>> between 2mpbs and will jump sometimes to 10-13 mbps and back down to
    >>>> different rates in between. Not seeing any errors on the circuits.
    >>>> Broadwing has went over their network and routers at great lengths.
    >>>> Is their anything I can look at on the routers to see if anything on our
    >>>> routers configured wrong or other errors, etc ?
    >>>> I can post configs, just do not have access to the routers until morning.
    >>>> As mentioned, seemed like was working, and not after the MTU fix and we
    >>>> keep blaming broadwing but need to be 100 percent not on our end,
    >>>> Although I can not find anything wrong. Also router CPU is 3-5 percent
    >>>> total on each at the most. no interface ACL's, just a few for routing
    >>>> redistribution but just a few.
    >>>> I'll take any suggestions to look at anything.
    >>>> Thanks,
    >>>> MC
    >>> This is quite tricky. Measuring link bandwidth is not always
    >>> straightforward
    >>> and it is very easy to get the wrong answers.
    >>> You want to check for fragmentation:-
    >>> Get a packet capture tool (say Ethereal or Wireshark, BTH I find
    >>> that anything later than 0.10.14 on Windows crashes frequently).
    >>> Use it to find out how to send max size ping packets
    >>> without fragmentation. Windows "ping -l xxxx", xxxx ~=1472.
    >>> Send some across link and use capture at both ends to see
    >>> if there is fragmentation.
    >>> To do a pure bandwidth test use iperf in UDP mode
    >>> OR more gently and with fewer problems use TCP
    >>> but add enough iperf copies to saturate the link.
    >>> You can tell when saturated since the rate of
    >>> remaining copies of iperf reduces as new ones are
    >>> added.

    >> Thanks very much, I'll give that a try when get back to the office.
    >> MC- Hide quoted text -
    >>
    >> - Show quoted text -

    >
    > One more thing.
    > You can use the sh int command to get an idea of the
    > data rate but be aware that the 5min (x min) number
    > takes data from more than 5 mins. The "5" refers to a
    > constant in the mathematical technique
    > used to compute the value. Search for [exponential weighted
    > average time constant]. When offered a full scale step
    > change of traffic level (say from 0 to 100%) the displayed
    > number takes 20 minutes to stabilise completely.
    >
    > int xxx
    > load-interval 30
    >
    > changes the "time constant" to 30 seconds.
    >
    > Also you can use snmp (getif.exe) to graph the values.
    >
    >

    I do believe the testing we did was flawed, not sure why I did not think
    about it more. We were trying to compare FTP and other file transfer
    rates. Forgot about TCP default window sizing on windoz servers and
    effect of RTT delays across high speed WANs (45mb DS3 in this case).

    So we downloaded the iperf program and seems will work just as we need
    using the UDP protocol. Other program we used did have UDP capability
    but had it's own limitations and did not give accurate results.

    We were seeing server performance hang around 3mbps end-to-end across
    the WAN, 45mb DS3 on each side of the MPLS, running about 35 ms latency.
    This falls right into what should be expected. most of our WAN circuit
    have been no more than 3mb in the past. The only thing we had running
    across the DS3 at first was between two cisco 9216i SAN switches for SAN
    replication to the remote site which throughput was great. of course now
    realize they are optimized for WAN performance. The windows servers apps
    were originally all on the same LAN as all the clients, however now
    started having clients on the remote side use the servers at our local
    side. We had other remote sites on lower speed WANs and expected the
    slower performance. But when we brought up the new remote site on the
    45mb WAN we did not get any better performance as other remote sites and
    thought we started having an issue. We did have the MTU problem on the
    WAN provider network and thought was related.

    We now have decied anything we have done is meaningless and data can not
    be trusted so we are starting from scratch to se if we really do have
    any problems on the WAN.

    May be that we are just needing TCP tuning on the windoz applications or
    something.

    Thanks much for the info on the iperf tool.

    I'll post back next week after we run some more tests this weekend.
     
    MC, Feb 2, 2007
    #5
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. TJ

    WiFi Bandwidth issue

    TJ, Jul 19, 2004, in forum: Wireless Networking
    Replies:
    2
    Views:
    4,643
  2. Chris DeNardis

    Bandwidth issue in Milwaukee area(New Berlin)

    Chris DeNardis, Feb 19, 2004, in forum: Computer Support
    Replies:
    2
    Views:
    481
    Chris DeNardis
    Feb 20, 2004
  3. Replies:
    8
    Views:
    1,304
  4. Astaa_lavista

    Policy_map Bandwidth Issue

    Astaa_lavista, Oct 16, 2007, in forum: Cisco
    Replies:
    0
    Views:
    446
    Astaa_lavista
    Oct 16, 2007
  5. Replies:
    1
    Views:
    726
    BoBraxton
    Dec 20, 2007
Loading...

Share This Page