Network Performance 1GB ports

Discussion in 'Cisco' started by Personne, Nov 13, 2009.

  1. Personne

    Personne Guest

    Hi,

    I have a Cisco 2960 with 2 x 1 GB ports. I have plugged a computer on
    each port. and run a netperf (http://www.osnms.com/2008/07/network-
    performance-monitor-using-netperf/) between my 2 computers, and I'm
    very surprise and disappointed by the performance I get

    Info:
    Nothing else is connected to the switch (expect the power :) )
    The 2 x 1GB ports are configured to 1000 Full Duplex
    The 2 computers NIC are configured to 1000 Full Duplex
    The 2 cables I use are Cat5e

    The speed I get is only 60Mbytes/sec (or 480 Mbits/s) compare to the
    theoretical speed of a 1 GB port 125MBytes/s

    Why is this ?
     
    Personne, Nov 13, 2009
    #1
    1. Advertising

  2. Personne

    Mark Huizer Guest

    The wise Personne enlightened me with:
    > Hi,
    >
    > I have a Cisco 2960 with 2 x 1 GB ports. I have plugged a computer on
    > each port. and run a netperf (http://www.osnms.com/2008/07/network-
    > performance-monitor-using-netperf/) between my 2 computers, and I'm
    > very surprise and disappointed by the performance I get
    >
    > The speed I get is only 60Mbytes/sec (or 480 Mbits/s) compare to the
    > theoretical speed of a 1 GB port 125MBytes/s
    >
    > Why is this ?


    What performance do you get when you connect the two PC's directly with
    a cross cable? What network cards do you have? Did you try with jumbo
    frames?

    The performance limit doesn't have to be in the Cisco, and theoretical
    speeds are sometimes just that: theoretical.
    I would expect that you need decent PC's and decent network cards to
    fill 1Gb. And for real usage (where the PC has to use disk etc) I wonder
    if you can get anywhere near that.

    Mark
     
    Mark Huizer, Nov 13, 2009
    #2
    1. Advertising

  3. Personne

    Personne Guest

    Mark,

    I did already this test before your post, and you are correct it
    doesn't come from the CIsco Switch. With a crosscable (5e) between my
    2 computers (not servers) I get the same speed rate (around 50 and 60
    MBytes/s)
    I know TCP/IP has some overhead but 52% is really a lot.
    - Could be a driver issue
    - I haven't tried yet Jumbo Frame (I'm not even sure I have the option
    in my Windows drivers)
    - Netperf doesn't use disk, it is a RAM-RAM test, so no disk overhead.


    On Nov 13, 10:04 am, Mark Huizer <xaa
    > wrote:
    > The wise Personne enlightened me with:
    >
    > > Hi,

    >
    > > I have a Cisco 2960 with 2 x 1 GB ports. I have plugged a computer on
    > > each port. and run a netperf (http://www.osnms.com/2008/07/network-
    > > performance-monitor-using-netperf/) between my 2 computers, and I'm
    > > very surprise and disappointed by the performance I get

    >
    > > The speed I get is only 60Mbytes/sec (or 480 Mbits/s) compare to the
    > > theoretical speed of a 1 GB port 125MBytes/s

    >
    > > Why is this ?

    >
    > What performance do you get when you connect the two PC's directly with
    > a cross cable? What network cards do you have? Did you try with jumbo
    > frames?
    >
    > The performance limit doesn't have to be in the Cisco, and theoretical
    > speeds are sometimes just that: theoretical.
    > I would expect that you need decent PC's and decent network cards to
    > fill 1Gb. And for real usage (where the PC has to use disk etc) I wonder
    > if you can get anywhere near that.
    >
    > Mark
     
    Personne, Nov 13, 2009
    #3
  4. Personne

    Mark Huizer Guest

    The wise Personne enlightened me with:
    > Mark,
    >
    > I did already this test before your post, and you are correct it
    > doesn't come from the CIsco Switch. With a crosscable (5e) between my
    > 2 computers (not servers) I get the same speed rate (around 50 and 60
    > MBytes/s)
    > I know TCP/IP has some overhead but 52% is really a lot.
    > - Could be a driver issue
    > - I haven't tried yet Jumbo Frame (I'm not even sure I have the option
    > in my Windows drivers)
    > - Netperf doesn't use disk, it is a RAM-RAM test, so no disk overhead.
    >


    I know it's a RAM to RAM test, that's why I said that a 'real life case'
    probably would be even worse.
    TCP/IP doesn't have to be a problem, and you can account for that in
    your test. Not sure what you are testing precisely with netperf, but the
    parameters you are using can have enough impact on your measurements. I
    normally use iperf, I don't know netperf, but it might have the same
    options. You'll see different measurements for e.g. udp vs tcp, since
    udp is connectionless so no problems with window sizes. If you are using
    tcp, play around with the window size/buffer size, and see the
    difference.
    You might play with smaller/bigger sizes, you'll see a big difference.

    And I'm not entirely sure where PC hardware is nowadays. Can't tell you
    the max bandwidth of memory, PCI busses, etc. And you will see a big
    difference between using an el cheapo NIC and a topline NIC that does
    all kinds of offloading, because it will do stuff in hardware that your
    CPU doesn't have to worry about anymore.

    And to stress that again: don't expect to get anywhere near Gbit speed
    in real life usage with standard pc hardware. It's just not realistic.

    Mark
     
    Mark Huizer, Nov 13, 2009
    #4
  5. Personne

    Personne Guest

    Mark,
    "And to stress that again: don't expect to get anywhere near Gbit
    speed in real life usage with standard pc hardware. It's just not
    realistic."

    I'm not expecting to get a real 1GB connection, but I'm surprise to
    get only 50% of the theoretical number on a 1GB connection.
    When I use a 100M connection (with cross cable or with a switch) I get
    88% of the 100MB bandwidth.

    Thanks,
    Personne

    On Nov 13, 11:53 am, Mark Huizer <xaa
    > wrote:
    > The wise Personne enlightened me with:
    >
    > > Mark,

    >
    > > I did already this test before your post, and you are correct it
    > > doesn't come from the CIsco Switch. With a crosscable (5e) between my
    > > 2 computers (not servers) I get the same speed rate (around 50 and 60
    > > MBytes/s)
    > > I know TCP/IP has some overhead but 52% is really a lot.
    > > - Could be a driver issue
    > > - I haven't tried yet Jumbo Frame (I'm not even sure I have the option
    > > in my Windows drivers)
    > > - Netperf doesn't use disk, it is a RAM-RAM test, so no disk overhead.

    >
    > I know it's a RAM to RAM test, that's why I said that a 'real life case'
    > probably would be even worse.
    > TCP/IP doesn't have to be a problem, and you can account for that in
    > your test. Not sure what you are testing precisely with netperf, but the
    > parameters you are using can have enough impact on your measurements. I
    > normally use iperf, I don't know netperf, but it might have the same
    > options. You'll see different measurements for e.g. udp vs tcp, since
    > udp is connectionless so no problems with window sizes. If you are using
    > tcp, play around with the window size/buffer size, and see the
    > difference.
    > You might play with smaller/bigger sizes, you'll see a big difference.
    >
    > And I'm not entirely sure where PC hardware is nowadays. Can't tell you
    > the max bandwidth of memory, PCI busses, etc. And you will see a big
    > difference between using an el cheapo NIC and a topline NIC that does
    > all kinds of offloading, because it will do stuff in hardware that your
    > CPU doesn't have to worry about anymore.
    >
    > And to stress that again: don't expect to get anywhere near Gbit speed
    > in real life usage with standard pc hardware. It's just not realistic.
    >
    > Mark
     
    Personne, Nov 13, 2009
    #5
  6. Personne

    Rob Guest

    Personne <> wrote:
    > Mark,
    > "And to stress that again: don't expect to get anywhere near Gbit
    > speed in real life usage with standard pc hardware. It's just not
    > realistic."
    >
    > I'm not expecting to get a real 1GB connection, but I'm surprise to
    > get only 50% of the theoretical number on a 1GB connection.
    > When I use a 100M connection (with cross cable or with a switch) I get
    > 88% of the 100MB bandwidth.


    That is because the limit is not some percentage of the full bandwidth,
    but an absolute number. You are not hitting the limit of bus speed,
    cpu speed, memory speed etc. with your 100 Mbit network.

    I have done similar tests between Linux systems and got a little higher
    rates (600 Mbit if I remember correctly) but also not 1 Gbit.
    But I wasn't as surprised as you seem to be.

    (I was testing the troughput of a 3com Layer-3 switch when compared to a
    cisco router)
     
    Rob, Nov 13, 2009
    #6
  7. Personne

    Thrill5 Guest

    Getting 500Mb/s+ throughput on a standard PC is pretty good. Now you
    should know why servers cost so much, all parts of the architecture are
    optimized for throughput.

    The reason you only get 88% on a 100MB/s connection is because you can't
    actually use 100% of the bandwidth. The counters don't count the ethernet
    frame, plus there is a a required inter-frame gap between ethernet frames.


    "Personne" <> wrote in message
    news:...
    Mark,
    "And to stress that again: don't expect to get anywhere near Gbit
    speed in real life usage with standard pc hardware. It's just not
    realistic."

    I'm not expecting to get a real 1GB connection, but I'm surprise to
    get only 50% of the theoretical number on a 1GB connection.
    When I use a 100M connection (with cross cable or with a switch) I get
    88% of the 100MB bandwidth.

    Thanks,
    Personne

    On Nov 13, 11:53 am, Mark Huizer <xaa
    > wrote:
    > The wise Personne enlightened me with:
    >
    > > Mark,

    >
    > > I did already this test before your post, and you are correct it
    > > doesn't come from the CIsco Switch. With a crosscable (5e) between my
    > > 2 computers (not servers) I get the same speed rate (around 50 and 60
    > > MBytes/s)
    > > I know TCP/IP has some overhead but 52% is really a lot.
    > > - Could be a driver issue
    > > - I haven't tried yet Jumbo Frame (I'm not even sure I have the option
    > > in my Windows drivers)
    > > - Netperf doesn't use disk, it is a RAM-RAM test, so no disk overhead.

    >
    > I know it's a RAM to RAM test, that's why I said that a 'real life case'
    > probably would be even worse.
    > TCP/IP doesn't have to be a problem, and you can account for that in
    > your test. Not sure what you are testing precisely with netperf, but the
    > parameters you are using can have enough impact on your measurements. I
    > normally use iperf, I don't know netperf, but it might have the same
    > options. You'll see different measurements for e.g. udp vs tcp, since
    > udp is connectionless so no problems with window sizes. If you are using
    > tcp, play around with the window size/buffer size, and see the
    > difference.
    > You might play with smaller/bigger sizes, you'll see a big difference.
    >
    > And I'm not entirely sure where PC hardware is nowadays. Can't tell you
    > the max bandwidth of memory, PCI busses, etc. And you will see a big
    > difference between using an el cheapo NIC and a topline NIC that does
    > all kinds of offloading, because it will do stuff in hardware that your
    > CPU doesn't have to worry about anymore.
    >
    > And to stress that again: don't expect to get anywhere near Gbit speed
    > in real life usage with standard pc hardware. It's just not realistic.
    >
    > Mark
     
    Thrill5, Nov 14, 2009
    #7
  8. Personne

    6hopsaway Guest

    On Nov 14, 12:40 am, "Peter" <> wrote:
    > Hi Personne,
    >
    > > I'm not expecting to get a real 1GB connection, but I'm surprise to
    > > get only 50% of the theoretical number on a 1GB connection.
    > > When I use a 100M connection (with cross cable or with a switch) I get
    > > 88% of the 100MB bandwidth.

    >
    > If you have a 1Gbit interface and it negotiates a 1 Gbit link speed,
    > then you really do have 1 Gbit CAPABILITY on that connection.
    > HOWEVER... I work in Networking with lots of Cisco H/W and 100's of
    > top end Servers (Dell, IBM, HP, Sun) using many different OS's and ALL
    > machines have 1GB interfaces, and NONE of them are able to REGULARLY
    > push more than around 800Mbit through that interface without issues.
    > In fact most Servers rarely exceed around 700Mbit though a 1Gbit
    > interface. This is a limitation with the design of the H/W and S/W
    > being used in the Server, not with the Cisco H/W, as we really do see
    > aggregate Link speeds of 9.98Mbit throughput on EACH 1Gbit interface
    > of a MULTIPLE Gbit interfaces between Cisco Switches. The easiest way
    > to see this is to create Channel groups that consist of Multiple Gbit
    > interfaces.
    >
    > I hope this helps............pk.
    >
    > --
    > Peter from Auckland.


    For those of you looking to push the limits of your Ethernet
    connections you will need to turn on a few tcp options in windows.
    Check out RFC 1323 and Microsoft TcpOpts1323
    It is recommended to turn these features on with any link generating
    more than 16 mb/s operations. Cisco Routers enable this option by
    default. Routers only need tuned outside of the default range when
    using jumbo frames across them. Cisco gig links natively support up
    to 9k frames where fast Ethernet typically does not. Side note...
    Vista/ Windows 7 and 2008 Server all come with RFC1323 enabled. Linux
    and Unix have had these options set natively for years. Have fun
    pegging your links.
     
    6hopsaway, Nov 16, 2009
    #8
  9. Personne

    Personne Guest

    Thanks for all your replies
     
    Personne, Nov 16, 2009
    #9
  10. Personne

    alexd Guest

    Meanwhile, at the comp.dcom.sys.cisco Job Justification Hearings, Tosh chose the
    tried and tested strategy of:

    > All the times I did the same test I saw one of the server's cpu hitting 100%
    > usage, that was the limit.
    > Providing that you have multiple cpus on both servers, maybe if you run
    > iperf twice you can reach the adapter speed limit with the aggregate
    > throughput.


    Try 'iperf -c ... -P n', where 'n' is the number of threads to run. The man page
    does say "The threading implementation is rather heinous", so who knows. A quick
    loopback test [ie client and server] on this quad core shows -P 3 to give the
    highest 'throughput' [19.2Gbps], which I suppose makes sense.

    --
    <http://ale.cx/> (AIM:troffasky) ()
    19:09:13 up 31 days, 23:04, 4 users, load average: 0.13, 0.18, 0.22
    "Stupid is a condition. Ignorance is a choice" -- Wiley Miller
     
    alexd, Nov 18, 2009
    #10
  11. On Thu, 19 Nov 2009 19:03:05 +0100, "Tosh" <> wrote:
    >> Try 'iperf -c ... -P n', where 'n' is the number of threads to run. The
    >> man page
    >> does say "The threading implementation is rather heinous", so who knows. A
    >> quick
    >> loopback test [ie client and server] on this quad core shows -P 3 to give
    >> the
    >> highest 'throughput' [19.2Gbps], which I suppose makes sense.
    >>

    >
    >Did you try that in a real envoironment, I mean from host to host?
    >There is something that I don't understand in that, in all my previous tests
    >i've seen well clocked cpus jumping at 100% utilization with approx 400mbps
    >( megabit x sec) of unidirectional iperf traffic, 20 gig is too far from
    >those results to make sense to me, don't you agree?


    It says he's using client and server on the same PC, which means that
    probably the data isn't leaving the CPU, certainly not RAM, and should be
    limited by the throughput of memory bus or better.

    In host-to-host tests, it's usually the running of the actual network card
    that costs the cpu utilization, not the generating of the data or
    (apparently, according to this test) even the running of the IP stack.

    Jasper
     
    Jasper Janssen, Nov 19, 2009
    #11
  12. Personne

    alexd Guest

    Meanwhile, at the comp.dcom.sys.cisco Job Justification Hearings, Tosh chose the
    tried and tested strategy of:

    > There is something that I don't understand in that, in all my previous tests
    > i've seen well clocked cpus jumping at 100% utilization with approx 400mbps
    > ( megabit x sec) of unidirectional iperf traffic, 20 gig is too far from
    > those results to make sense to me, don't you agree?


    Depends what is using the CPU. Could be iperf, could be the NIC driver, could be
    the IP stack on the OS.

    --
    <http://ale.cx/> (AIM:troffasky) ()
    21:46:33 up 33 days, 2:05, 4 users, load average: 0.29, 0.23, 0.19
    Plant food is a made up drug
     
    alexd, Nov 19, 2009
    #12
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Mac
    Replies:
    4
    Views:
    643
  2. Homies
    Replies:
    8
    Views:
    14,590
    Pete Mainwaring
    Feb 26, 2004
  3. Jon Fanti
    Replies:
    4
    Views:
    8,457
  4. Graham Russell

    Capacity of 1GB SanDisk Ultra vs 1GB Microdrive

    Graham Russell, Nov 5, 2003, in forum: Digital Photography
    Replies:
    8
    Views:
    842
  5. Mike
    Replies:
    27
    Views:
    1,696
Loading...

Share This Page