Cisco 3660 Performance

Discussion in 'Cisco' started by Arsen V., Feb 6, 2004.

  1. Arsen V.

    Arsen V. Guest

    Hello,

    We have a Cisco 3660 in the following configuration:

    - two 100Mbps interfaces (one is class C and one is 1.1.1.1)
    - two DS3 thru different providers (total 90Mpbs of bandwith)
    - ACL with about 10 lines per interface
    - CPU utilization at 22%
    - Bandwith utilization at around 14Mbps
    - 95% of traffic is HTTP

    We noticed that even though we have 90Mpbs of bandwith, the outbound
    connection (sending data from our web servers out to the internet) is
    always capped at around 14Mbps. This means that during our normal
    operation the remaining bandwidth of 90-14=76Mbps goes unused and the
    external performance is very slow.

    The inbound connection is capped at around 30Mpbs. In other words, we
    can download more stuff then we can send out.

    We thought that maybe we have a problem with our DS3.

    So we did the following test:

    - disconnect the router from our lan
    - put one computer on the class C interface (through a 100Mbps switch)
    - put 2nd computer on the 1.1.1.1 interface (through another 100Mpbs
    switch)
    - use iperf.exe tool to meause the bandwidth available between the
    1.1.1.1 and the C interface with router in between routing the packets
    from 1.1.1.1 to the class C.

    The results:

    - sending data from 1.1.1.1 to the class C interface gave about 30Mpbs
    throughput, even though there was no other data going over the router
    at this time. Why wasn't this 100Mpbs?

    - even more strange... sending data from the class C interface to
    1.1.1.1 resulted in at most 13 or 14Mbps. Why is this different from
    30Mpbs that we got for the other direction? Why are both of them so
    much lower than 100Mpbs that we would expect with 100Mpbs interfaces
    in the router?

    Are there any other tests we can perform to help us understand what is
    happening and how to fix this?

    Thank you in advance,
    Arsen
     
    Arsen V., Feb 6, 2004
    #1
    1. Advertising

  2. Arsen V.

    Guest Guest

    Look at using a firewall instead of all the ACL's.
    This will offload the CPU.
    I bet you are also Natting on the DS3 Internet links. although light this
    should also be offloaded to a Firewall.
    We have a 3660 with 8 T1's 8BRI and 2 FE segments and only average 5% (only
    one ACL) we also run Openview and Ciscoworks which hit the router every 5 to
    10 sec to collect SNMP Info.

    Also consider using CEF on all interfaces.


    As far as upload and download speed, its hard to say. If this is not BGP,
    you may want to look at some load balancing devices such as Radware
    linkproof or Cisco content switching.


    "Arsen V." <> wrote in message
    news:...
    > Hello,
    >
    > We have a Cisco 3660 in the following configuration:
    >
    > - two 100Mbps interfaces (one is class C and one is 1.1.1.1)
    > - two DS3 thru different providers (total 90Mpbs of bandwith)
    > - ACL with about 10 lines per interface
    > - CPU utilization at 22%
    > - Bandwith utilization at around 14Mbps
    > - 95% of traffic is HTTP
    >
    > We noticed that even though we have 90Mpbs of bandwith, the outbound
    > connection (sending data from our web servers out to the internet) is
    > always capped at around 14Mbps. This means that during our normal
    > operation the remaining bandwidth of 90-14=76Mbps goes unused and the
    > external performance is very slow.
    >
    > The inbound connection is capped at around 30Mpbs. In other words, we
    > can download more stuff then we can send out.
    >
    > We thought that maybe we have a problem with our DS3.
    >
    > So we did the following test:
    >
    > - disconnect the router from our lan
    > - put one computer on the class C interface (through a 100Mbps switch)
    > - put 2nd computer on the 1.1.1.1 interface (through another 100Mpbs
    > switch)
    > - use iperf.exe tool to meause the bandwidth available between the
    > 1.1.1.1 and the C interface with router in between routing the packets
    > from 1.1.1.1 to the class C.
    >
    > The results:
    >
    > - sending data from 1.1.1.1 to the class C interface gave about 30Mpbs
    > throughput, even though there was no other data going over the router
    > at this time. Why wasn't this 100Mpbs?
    >
    > - even more strange... sending data from the class C interface to
    > 1.1.1.1 resulted in at most 13 or 14Mbps. Why is this different from
    > 30Mpbs that we got for the other direction? Why are both of them so
    > much lower than 100Mpbs that we would expect with 100Mpbs interfaces
    > in the router?
    >
    > Are there any other tests we can perform to help us understand what is
    > happening and how to fix this?
    >
    > Thank you in advance,
    > Arsen
     
    Guest, Feb 6, 2004
    #2
    1. Advertising

  3. On 5 Feb 2004 22:44:14 -0800, (Arsen V.) wrote:

    >Hello,
    >
    >We have a Cisco 3660 in the following configuration:
    >
    >- two 100Mbps interfaces (one is class C and one is 1.1.1.1)
    >- two DS3 thru different providers (total 90Mpbs of bandwith)
    >- ACL with about 10 lines per interface
    >- CPU utilization at 22%
    >- Bandwith utilization at around 14Mbps
    >- 95% of traffic is HTTP
    >
    >We noticed that even though we have 90Mpbs of bandwith, the outbound
    >connection (sending data from our web servers out to the internet) is
    >always capped at around 14Mbps. This means that during our normal
    >operation the remaining bandwidth of 90-14=76Mbps goes unused and the
    >external performance is very slow.
    >
    >The inbound connection is capped at around 30Mpbs. In other words, we
    >can download more stuff then we can send out.


    Generally if the router's CPU isn't maxed out then the router isn't
    the bottleneck. There are a lot of variables when it comes to WAN
    link performance including the way that measurements are done.

    Inbound traffic: How did you measure it? Did you kick off one or more
    downloads from some workstations and add the throughput together, or
    are you just looking at aggregate throughput on the router (e.g., via
    SNMP polling) during business hours?

    Outbound traffic: Again, how did you measure it? Did you kick off
    some test downloads between Internet hosts and your web servers or are
    you looking at aggregate throughput? When the throughput hits 14Mbps,
    do the web servers then appear slow when accessed from the Internet?

    >We thought that maybe we have a problem with our DS3.
    >
    >So we did the following test:
    >
    >- disconnect the router from our lan
    >- put one computer on the class C interface (through a 100Mbps switch)
    >- put 2nd computer on the 1.1.1.1 interface (through another 100Mpbs
    >switch)
    >- use iperf.exe tool to meause the bandwidth available between the
    >1.1.1.1 and the C interface with router in between routing the packets
    >from 1.1.1.1 to the class C.
    >
    >The results:
    >
    >- sending data from 1.1.1.1 to the class C interface gave about 30Mpbs
    >throughput, even though there was no other data going over the router
    >at this time. Why wasn't this 100Mpbs?
    >
    >- even more strange... sending data from the class C interface to
    >1.1.1.1 resulted in at most 13 or 14Mbps. Why is this different from
    >30Mpbs that we got for the other direction? Why are both of them so
    >much lower than 100Mpbs that we would expect with 100Mpbs interfaces
    >in the router?


    Just because a router has 100Mbps interfaces doesn't mean it can route
    100Mbps worth of traffic. If a router maxes out at 40Mbps, for
    example, it's still worthwhile to put 100Mbps interfaces in it rather
    than 10Mbps interfaces.

    That being said, a 3660 should be able to do close to if not over
    100Mbps under ideal conditions (i.e., CEF, full-sized packets, no
    performance-reducing features such as ACLs/NAT/etc) based on tests
    that have been done with other models. You should definitely be able
    to clear 30Mbps under such conditions, and there are any number of
    possible reasons why you weren't able to:

    1) What other features besides ACLs are configured on the router that
    could potentially impact performance? Examples are NAT, policy
    routing, QoS, and so forth -- anything that falls outside of standard
    packet forwarding. Try disabling all such features and run the tests
    again, being careful to note the router's CPU usage during the tests.

    2) Some bandwidth-testing applications aren't very good at what they
    do. I've never used IPerf, but have had good success with TTCP
    (http://www.pcausa.com/Utilities/pcattcp.htm). So you may want to try
    at least one other testing application to see if you get different
    results.

    3) You said that there was a switch in between the router and each
    test host when you ran IPerf. To eliminate as many variables as
    possible I'd suggest running such tests with the hosts connected
    directly to the router via crossover cables.

    >Are there any other tests we can perform to help us understand what is
    >happening and how to fix this?


    I'd suggest starting with the basics: run TTCP between two hosts that
    are directly connected to each other via a crossover cable. You
    should get throughput that is reasonably close to 100Mbps. Then put
    the 3660 in between them (no switches) with CEF enabled and no ACLs or
    NAT or anything else non-standard, and run the test again. (Note the
    router's CPU usage while the test is running.) See if you get
    significantly less throughput when the router is in between the test
    hosts. If so then there's either something in the config that's
    slowing things down or possibly a physical layer issue such as a
    shotty cable or interface. Either way, let us know what the results
    are and we can go from there.

    -Terry
     
    Terry Baranski, Feb 7, 2004
    #3
  4. Arsen V.

    Ohad Dallal Guest

    Hey,

    I would say that the problem in the tests is not the router, but the
    workstations and the load generation method....

    The worksatations just can't receive\transmit in more than 14-30Mbps,
    because of CPU\MEM\stack limitations. Try using several workstations
    (at least 4) to create the traffic load, or use a traffic generator
    (like Smartbits, Ixea etc.) if you can get\rent one. I'm sure you'll
    see better router utilization....

    Ohad.

    Terry Baranski <0VE> wrote in message news:<>...
    > On 5 Feb 2004 22:44:14 -0800, (Arsen V.) wrote:
    >
    > >Hello,
    > >
    > >We have a Cisco 3660 in the following configuration:
    > >
    > >- two 100Mbps interfaces (one is class C and one is 1.1.1.1)
    > >- two DS3 thru different providers (total 90Mpbs of bandwith)
    > >- ACL with about 10 lines per interface
    > >- CPU utilization at 22%
    > >- Bandwith utilization at around 14Mbps
    > >- 95% of traffic is HTTP
    > >
    > >We noticed that even though we have 90Mpbs of bandwith, the outbound
    > >connection (sending data from our web servers out to the internet) is
    > >always capped at around 14Mbps. This means that during our normal
    > >operation the remaining bandwidth of 90-14=76Mbps goes unused and the
    > >external performance is very slow.
    > >
    > >The inbound connection is capped at around 30Mpbs. In other words, we
    > >can download more stuff then we can send out.

    >
    > Generally if the router's CPU isn't maxed out then the router isn't
    > the bottleneck. There are a lot of variables when it comes to WAN
    > link performance including the way that measurements are done.
    >
    > Inbound traffic: How did you measure it? Did you kick off one or more
    > downloads from some workstations and add the throughput together, or
    > are you just looking at aggregate throughput on the router (e.g., via
    > SNMP polling) during business hours?
    >
    > Outbound traffic: Again, how did you measure it? Did you kick off
    > some test downloads between Internet hosts and your web servers or are
    > you looking at aggregate throughput? When the throughput hits 14Mbps,
    > do the web servers then appear slow when accessed from the Internet?
    >
    > >We thought that maybe we have a problem with our DS3.
    > >
    > >So we did the following test:
    > >
    > >- disconnect the router from our lan
    > >- put one computer on the class C interface (through a 100Mbps switch)
    > >- put 2nd computer on the 1.1.1.1 interface (through another 100Mpbs
    > >switch)
    > >- use iperf.exe tool to meause the bandwidth available between the
    > >1.1.1.1 and the C interface with router in between routing the packets
    > >from 1.1.1.1 to the class C.
    > >
    > >The results:
    > >
    > >- sending data from 1.1.1.1 to the class C interface gave about 30Mpbs
    > >throughput, even though there was no other data going over the router
    > >at this time. Why wasn't this 100Mpbs?
    > >
    > >- even more strange... sending data from the class C interface to
    > >1.1.1.1 resulted in at most 13 or 14Mbps. Why is this different from
    > >30Mpbs that we got for the other direction? Why are both of them so
    > >much lower than 100Mpbs that we would expect with 100Mpbs interfaces
    > >in the router?

    >
    > Just because a router has 100Mbps interfaces doesn't mean it can route
    > 100Mbps worth of traffic. If a router maxes out at 40Mbps, for
    > example, it's still worthwhile to put 100Mbps interfaces in it rather
    > than 10Mbps interfaces.
    >
    > That being said, a 3660 should be able to do close to if not over
    > 100Mbps under ideal conditions (i.e., CEF, full-sized packets, no
    > performance-reducing features such as ACLs/NAT/etc) based on tests
    > that have been done with other models. You should definitely be able
    > to clear 30Mbps under such conditions, and there are any number of
    > possible reasons why you weren't able to:
    >
    > 1) What other features besides ACLs are configured on the router that
    > could potentially impact performance? Examples are NAT, policy
    > routing, QoS, and so forth -- anything that falls outside of standard
    > packet forwarding. Try disabling all such features and run the tests
    > again, being careful to note the router's CPU usage during the tests.
    >
    > 2) Some bandwidth-testing applications aren't very good at what they
    > do. I've never used IPerf, but have had good success with TTCP
    > (http://www.pcausa.com/Utilities/pcattcp.htm). So you may want to try
    > at least one other testing application to see if you get different
    > results.
    >
    > 3) You said that there was a switch in between the router and each
    > test host when you ran IPerf. To eliminate as many variables as
    > possible I'd suggest running such tests with the hosts connected
    > directly to the router via crossover cables.
    >
    > >Are there any other tests we can perform to help us understand what is
    > >happening and how to fix this?

    >
    > I'd suggest starting with the basics: run TTCP between two hosts that
    > are directly connected to each other via a crossover cable. You
    > should get throughput that is reasonably close to 100Mbps. Then put
    > the 3660 in between them (no switches) with CEF enabled and no ACLs or
    > NAT or anything else non-standard, and run the test again. (Note the
    > router's CPU usage while the test is running.) See if you get
    > significantly less throughput when the router is in between the test
    > hosts. If so then there's either something in the config that's
    > slowing things down or possibly a physical layer issue such as a
    > shotty cable or interface. Either way, let us know what the results
    > are and we can go from there.
    >
    > -Terry
     
    Ohad Dallal, Feb 7, 2004
    #4
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Adam
    Replies:
    0
    Views:
    3,843
  2. hari
    Replies:
    0
    Views:
    693
  3. hari

    cisco 3660 RAM problem,

    hari, Oct 27, 2003, in forum: Cisco
    Replies:
    10
    Views:
    10,249
  4. news.eircom.net

    Cisco 3660

    news.eircom.net, Nov 27, 2003, in forum: Cisco
    Replies:
    0
    Views:
    585
    news.eircom.net
    Nov 27, 2003
  5. Mike Sergeev

    "ALARM" port in cisco 3660

    Mike Sergeev, Jul 27, 2004, in forum: Cisco
    Replies:
    1
    Views:
    719
    Phillip Remaker
    Jul 28, 2004
Loading...

Share This Page