Speed Mismatch?!?

Discussion in 'Cisco' started by mschunk, May 8, 2006.

  1. mschunk

    mschunk Guest

    I've been banging my head on this one for weeks.

    Can someone help me understand why network performance between Windows
    2k/xp/2k3 hosts with a gigabit interface, and those with a 100/full
    interface are slow so?

    Now, 100/full to 100/full transfers are plenty fast. Very. And 1000
    to 1000 transfers outright smoke...as gigabit should.

    But, for large TCP transfers from a 1gbit host to a 100mbit host it
    goes very Slow...

    1) No matter what NIC used. Intel, Broadcom, D-Link, 3COM....or driver
    version.
    2) No matter what version of windows to other version of windows...or
    different motherboards, etc...the slowness is very predictable no
    matter what hardware involved.
    3) No matter what type of gigabit switch....Cisco 3750, several
    Netgears, some managed others not.
    4) No matter how the switches involved are used...host and server in
    same switch, slow. Host and server in different switches...linked
    straight...routed between them...I even tried trunks between switches
    together w/ link ag....always slow. Always the same.
    5) No matter how many times I try different Ethernet cables...Cat5, 5e,
    6...some pre-made, others made by hand.
    6) No matter what TCP window sizes are set to anywhere, be it client or
    server side or both.

    I put gbit adapters in the servers, thinking they could use them to
    sever files to all the workstations that have 100mbit NIC's. I mean,
    it was logical...a gbit server "should" be able to handle 10x the
    throughput to 100mbit clients, right???? Nope.

    Note again that the performance problem is only one way. When I push
    files FROM the 100mbit hosts, to the 1gbit servers, speed is expected:
    fast, about 80mbit/sec. When I pull files form the 1bitg servers, to
    the 100mbit PC's...speed is absolutely terrible. I'm taking
    5mbit/sec, if that.

    Looking at the packets...I'm seeing gobs and gobs of TCP retransmit
    frames. It's like the switches (even the fancy Cisco one) are
    sending data to the PC's faster than the PC's can take it....but
    only when moving data from 1gbit to 100mbit. (So much for "store and
    forward" switching.)

    Also, in addition to the TCP re-Trans frames, the Cisco switch (when I
    put in the patch between these hosts) is telling me I've got "late
    collisions" on the gbit interface during the slow transfers. WTF?
    The PC's are all 100/full, in auto. Everything is full duplex, so
    why am I seeing ANY collisions of any kind at all???

    Duplex mismatch, sure, I've dealt with that nastiness before, and I
    think I understand how that messes things up....but I've NEVER heard
    of a SPEED mismatch on Ethernet. Have You?

    I'm coming to the conclusion that if I want to have gbit NICs around,
    AND use them at that speed...I have to dedicate a totally separate
    fabric for them. Great...so all my servers can talk to one another at
    1000...at least backups will still smoke...but they must all serve at
    the same speed as clients? Why did I even bother buying any gbit
    switches if I can't really use them?

    Help? What else can I try?
     
    mschunk, May 8, 2006
    #1
    1. Advertisements

  2. mschunk

    waterotter Guest

    Oh,it is very interesting.
    I will try it.
    Do you affirm that NIC was full duplex?
     
    waterotter, May 8, 2006
    #2
    1. Advertisements

  3. mschunk

    mschunk Guest

    Yes. In fact I played around with duplex settings quite a bit. I'm
    very certain that I've got 100/full on the 100mib side, and 1000/full
    (can't do 1/2 on gbit) on the gibt side.
     
    mschunk, May 8, 2006
    #3
  4. mschunk

    thrill5 Guest

    On the gig interfaces on the servers, do you have them set to auto or
    1000/full? You can set 1000/full, but it does NOT work. Gig ports MUST be
    set to auto ON BOTH SIDES on gigabit.

    Scott
     
    thrill5, May 8, 2006
    #4
  5. mschunk

    waterotter Guest

    Oh, really?
    i did't suppose it......
    it must AUTO.....
     
    waterotter, May 8, 2006
    #5
  6. mschunk

    mschunk Guest

    Thank you for the "duplex" warnings.

    I'm confidant that duplex issuess are not involved. I left out gobs of
    information about different duplex settings I have tried.

    And yes, we all know that the _only_ way to get 1000mbit/sec is in
    "Auto" as IEEE standards require "auto" as the _only_ way to get
    1000/full.

    For completness, I did test transfers between 1gbit and 100/half.
    Results: From 100/half to 1gbit = 60MBit/sec. From 1gbit to 100/half
    was so bad I canceled the test.

    I'm aware that all 100mbit devices on a fabric must operate at the same
    duplex...right down to every last devices, printers too, or nothing
    works well.

    My ehternet fabric concisits of either 1) 100/full, or 2) 1000/full
    (auto.)

    Recap: 100/full --> 1000/full at 80mbtis/sec. 1000/full --> 100/full
    = 5mbit/sec.

    Can anybody out there confirm, that just as there is a huge perforamce
    problem going between 100mbit and (older) 10mbit hosts...that the same
    really does apply to gbit and 100mbit?
     
    mschunk, May 8, 2006
    #6
  7. mschunk

    thrill5 Guest

    Most of our servers are connected via gig, and all of the workstations are
    100/full. I have not seen the problem you are describing except when there
    is a duplex mismatch. Have you made sure you don't have a duplex mismatch
    with other devices that are in the traffic path? You could have an uplink
    or router port with a duplex issue. I have run into problems with NIC
    drivers, but this was a few years ago. I know there were issues with
    drivers on Sun Gig boards, 3Com 3C590 embedded, and Broadcom 10/100 drivers
    on System V.

    The one sure way to diagnose you problem is to sniff the traffic as leaves
    the server and at the same time sniff where it enters the client. This
    will tell you if packets are disappearing and at which end. You can keep
    moving the sniffers to sniff the paths in-between to find out where the
    packets are being dropped.

    Scott
     
    thrill5, May 8, 2006
    #7
  8. mschunk

    rdymek Guest

    I'm 99% sure you have a duplex mismatch problem. I know you felt this
    wasn't the case but everything you have described points to this.

    Here are the top things that will happen with duplex mismatch:

    1. Collisions on a port shat should be full duplex (a properly
    operating port at full duplex should never have a collision).
    2. Problems (errors and performance issues) more in one direction than
    the other (duplex problems are asymmetrical).
    3. Retransmissions (these are due to the collisions and errors on the
    port

    Don't forget that you need the switch to be set the same as the host.
    You cannot have auto on one side and it set on the other.

    Turn on debugging on your switch - chances are you are getting a ton of
    duplex mismatch errors.

    If the duplex is fine then the only other way you can get collisions is
    due to a bad nic, bad switch port, bad cable or too long of a cable.

    Otherwise, all your TCP connections are throttled at the TCP layer from
    host to host. The switch mearly sends what its sent. If the Gig
    server is overrunning the 100mb port, it wouldn't be the switch that's
    the problem, but rather the Gig server itself. The switch does not do
    any throttling.

    Store and forward is simply a switching method - not a buffering
    mechanism. Its to provide 100% CRC reliability, but its also least
    efficient. It used to be in older days that you needed this
    reliability. With today's Ethernet standards, this is not necessary
    and cut-through tends to provide the same level of reliability, while
    providing much faster performance. But end result is that
    store-and-forward has absolutely nothing to do with "buffering" packets
    to help from higher speed links to lower speed ones.

    Ryan
     
    rdymek, May 9, 2006
    #8
  9. mschunk

    MC Guest

    I find that in some heavy traffic conditions auto negotiation can flap
    and flow control on gigabit links can throttle connections.

    Now for server a gigabit over cooper when using multiple speed devices,
    it is recommended to set at auto, however I set all switch and server to
    1000/full anyway and seem to get better reliability.

    I always set switch-switch links hard set at 1000/full and no auto
    negotiation and not flow control regardless.

    I have issues with flow control on switch-switch links causing traffic
    to halt intermittently, however never seem that issue with host
    connections at gigabit. Flow control may be needed on host based systems
    as may not be able to handle the entire bandwidth, most host system bus
    can not handle full throttle gigabit traffic.
     
    MC, May 9, 2006
    #9
  10. mschunk

    mschunk Guest

    Thanks Rdymek, I know it smells like a duplex problem.

    sh interfaces and eye-balling NIC config....no, it's not. I've got a
    good variety of switches, but have been using mostly just the cisco
    ones for working through this....many different nics of both kinds and
    PC's w/ different Windows versions (2k/xp/2k3) on them to play with.

    These Speed results are consistent...from Gi --> Fa, throughput really
    sucks.

    The collisions are not consistent. Today, I tried things out on a 2801
    that's not needed for the moment. No collisions detected today. Not
    one. No CRC errors, Just really bad performance between NIC's at same
    duplex but different speeds...What I find odd, is that the switch
    reports NO packet drops on either side. "sh interface <blah>"

    I can't get good debug events going. Packets have to switch by the
    processor for the debugger to catch them, right? Anyway....no duplex
    mismatch on interface state change.

    I have not tried sniffing both sides at the same time. Thanks for
    setting me straight on the "store and forward." I'll try
    "buffering" too...I have only one router, a 3825, that has both gi
    and fa interfaces on it. I'll stick it between two switches and see
    what happens. I won't be able to play with it until Sunday, during
    off-hours. If this works though....gbit overrunning the 100mbit
    host....how would I stop that even if true? It's my worst fear,
    here...if this works it confirms that I really do need to put 1000 and
    100 hosts on separate L2 fabrics.

    ....Till Sunday
     
    mschunk, May 9, 2006
    #10
  11. mschunk

    mschunk Guest

    What switch do you have that allows you to manually set "1000/full"
    It's aginst IEEE standards and I have no NIC or switch in the building
    that lets me.

    If you have one, please give us the model number?
     
    mschunk, May 9, 2006
    #11
  12. mschunk

    Merv Guest

    Please post show version from each Cisco switch in the traffic path
     
    Merv, May 9, 2006
    #12
  13. mschunk

    jay Guest

    Interesting issue, but it does make some sense..
    How about get one of thos DOS client/server Bandwidth tools that use
    UDP (not TCP). Check the ports utilisation on CLI, and you will see
    nothing wrong with the Switch/Duplex settings.
    Think about how TCP slow start works - one dropped packet resets the
    TCP window.. then ramps up again exponentially until another dropped
    packet. What is happening is that from 1000 -> 100 port packets gets
    dropped in the fabric - whilst I bet the 100 -> 1000 port, the packets
    gets buffered which introduces delay however the overall throughput is
    better.
    Simply, having a server on a gigport should not be servicing just one
    client. the idea is that you have at least 10 x 100mb ports/clients
    using the server on GIG. Even smarter if you have a linux server to cap
    to <100mb to any IP end point. Anyway, if you get a server to do a full
    gig speed useful data would need 64bit PCI cards and better than a dual
    Xeon : )
     
    jay, May 9, 2006
    #13
  14. mschunk

    anybody43 Guest

    Does a network infrastructure connecting devices
    1. TCP
    Answer no.

    I have here a PC on 100Mpbs and a couple of hops away
    a server on 1Gbps.

    The intermediate hops are all GBE or maybe a 4x
    GBE Etherchannel.

    ##################
    ## 1G bps Server end
    ##################
    C:\Documents and Settings\admin>iperf -s
    ------------------------------------------------------------
    Server listening on TCP port 5001
    TCP window size: 8.00 KByte (default)
    ------------------------------------------------------------
    [1896] local 192.168.5.33 port 5001 connected with 192.168.67.249 port
    2172
    [ ID] Interval Transfer Bandwidth
    [1896] 0.0-10.0 sec 112 MBytes 94.2 Mbits/sec

    C:\Documents and Settings\admin>iperf -c 192.168.67.249
    ------------------------------------------------------------
    Client connecting to 192.168.67.249, TCP port 5001
    TCP window size: 8.00 KByte (default)
    ------------------------------------------------------------
    [1908] local 192.168.5.33 port 2404 connected with 192.168.67.249 port
    5001
    [ ID] Interval Transfer Bandwidth
    [1908] 0.0-10.0 sec 110 MBytes 92.0 Mbits/sec



    #################
    ## 100Mbps PC end.
    #################
    M:\Documents and Settings\admin>iperf -c 192.168.5.33
    ------------------------------------------------------------
    Client connecting to 192.168.5.33, TCP port 5001
    TCP window size: 63.0 KByte (default)
    ------------------------------------------------------------
    [1704] local 192.168.67.249 port 2172 connected with 192.168.5.33 port
    5001
    [ ID] Interval Transfer Bandwidth
    [1704] 0.0-10.0 sec 112 MBytes 94.1 Mbits/sec


    M:\Documents and Settings\admin>iperf -s
    ------------------------------------------------------------
    Server listening on TCP port 5001
    TCP window size: 8.00 KByte (default)
    ------------------------------------------------------------
    [1668] local 192.168.67.249 port 5001 connected with 192.168.5.33 port
    2404
    [ ID] Interval Transfer Bandwidth
    [1668] 0.0-10.0 sec 110 MBytes 92.1 Mbits/sec

    This works in this case. No tuning, no optimisation
    just perfect performnce right out of the box.

    If it didn't work we would be in BIG trouble. All of our servers
    are on GBE and a lot of the clients are on FE.

    Millions and millions of people are using just such a
    setup as you describe.

    OK, yours isn't working.

    Maybe this is the clue?
    "Late collisions" are your problem, or at least one of them.

    They are caused by two things in modern switched networks.
    Firstly collisions can only occur on Half Duplex ports.

    Late collisions are a consequence of a duplex missmatch.
    They occur on the HD end since the FD end just transmitts
    when it feels like it and sometimes there is an out of window
    collision.

    I have found (a while back) that WIndows drivers can report
    duplex settings incorrectly. In dicfficult cases I trust
    ONLY the port counters on Cisco networking kit.

    Use the counters to determine if you have a duplex missmatch.

    If there is a missmatch AND there is enough traffic
    (let's say) there WILL BE late collisions on the HD end.

    The FD end will be seeing CRC and Align errors.

    So, very carefully determine what to do to emiminate
    the duplex missmatch and then roll it out onto the
    various kit.

    Pick a couple of ports on one switch set up 1000 on
    one and 100 on another.

    Clear the counters

    run some traffic if the performance is poor send

    sh run int fa xx
    sh run int gi xx

    sh int fa xx
    sh int gi xx

    If you download iperf and use it to create traffic in
    both directions simultaneously you will for sure
    show up any problems.

    For some reason when I do both directions simultaneously I
    get only 90Mbps one way and 25Mbps he other way.
    I don't know why but I think that if it was a duplex missmatch
    it would be worse.
     
    anybody43, May 9, 2006
    #14
  15. mschunk

    anybody43 Guest

    Does a network infrastructure connecting devices
    Changing the iperf "buffer size" from the default 8k to 32k fixed it.
    85Mbps each way now.

    I expect that this is the amount of data sent before waiting on
    an acknowledgement.

    With 500k I get 90Mbps each way. In this acse we will be
    running into the TCP Rx window which seems to be 64k.
    No scaling.
     
    anybody43, May 9, 2006
    #15
  16. mschunk

    mschunk Guest

    Thank you everybody for your support, how about I show some numbers?
    You will soon see that there is no duplex issue being reported by the
    switches.


    Both client and server in this test running Win2k3, SP1...I don't have
    a single linux box in the building :(

    Client = Nivida nForce2 on-board 10/100
    Server = Nivida nForce4 on-board 10/100/1000

    I'll run tests aginst server twice:
    1) both NIC and switchport manually set at 100/full
    2) both NIC and switchport set at "auto"

    Although I'ved gotten the same speed results on many different
    switches, I'll run this today

    on the production stack of 3750's.

    I'll use iperf, and run a set of tests in TCP, and another in UDP.

    -------------------------
    Stack Info
    -------------------------

    3750Stack# sh ver

    Cisco Internetwork Operating System Software
    IOS (tm) C3750 Software (C3750-I5-M), Version 12.1(19)EA1d, RELEASE
    SOFTWARE (fc1)
    Copyright (c) 1986-2004 by cisco Systems, Inc.
    Compiled Mon 05-Apr-04 22:06 by antonino
    Image text-base: 0x00003000, data-base: 0x009206D8

    ROM: Bootstrap program is C3750 boot loader
    BOOTLDR: C3750 Boot Loader (C3750-HBOOT-M) Version 12.1(14r)EA1a,
    RELEASE SOFTWARE (fc1)

    (...)

    Switch Ports Model SW Version SW Image
    ------ ----- ----- ---------- ----------
    * 1 28 WS-C3750G-24TS 12.1(19)EA1d C3750-I5-M
    2 52 WS-C3750-48P 12.1(19)EA1d C3750-I9-M
    3 52 WS-C3750-48P 12.1(19)EA1d C3750-I9-M
    4 52 WS-C3750-48P 12.1(19)EA1d C3750-I9-M
    5 52 WS-C3750-48P 12.1(19)EA1d C3750-I9-M
    6 52 WS-C3750-48P 12.1(19)EA1d C3750-I5-M

    (...)

    3750Stack# sh run

    (...)
    !
    ip subnet-zero
    ip routing
    !
    ip host vg 192.168.8.3
    ip host rld-1760 192.168.8.2
    ip host rld-3725 192.168.8.1
    ip name-server 192.168.1.5
    mls qos map cos-dscp 0 8 16 26 34 46 48 56
    mls qos map ip-prec-dscp 0 8 16 26 34 46 48 56
    mls qos
    !
    !
    spanning-tree mode pvst
    spanning-tree loopguard default
    no spanning-tree optimize bpdu transmission
    spanning-tree extend system-id
    !
    (...)
    !
    interface GigabitEthernet1/0/24
    description ------------ SERVER PORT -----
    switchport access vlan 105
    switchport mode access
    no ip address
    no mdix auto
    storm-control broadcast level 2.00
    storm-control multicast level 2.00
    !
    (...)
    !
    interface FastEthernet5/0/24
    description ------------ CLIENT PORT -----
    switchport access vlan 100
    switchport mode access
    switchport voice vlan 200
    no ip address
    no mdix auto
    storm-control broadcast level 2.00
    storm-control multicast level 2.00
    spanning-tree portfast
    spanning-tree bpduguard enable
    !
    (...)
    !
    interface Vlan1
    no ip address
    shutdown
    !
    interface Vlan100
    ip address 192.168.0.1 255.255.248.0
    ip helper-address 192.168.1.5
    !
    interface Vlan105
    ip address 10.30.5.1 255.255.255.0
    ip helper-address 192.168.1.5
    !
    interface Vlan200
    ip address 192.168.8.4 255.255.248.0
    ip helper-address 192.168.1.5
    !
    (...)

    -----------------
    Stack Notes
    -----------------

    There are VoIP phones in use.
    The stack does inter-vlan rounting:

    3750Stack# sh ip route

    (...)
    C 10.30.5.0 is directly connected, Vlan105
    S* 0.0.0.0/0 [1/0] via 192.168.0.2
    C 192.168.8.0/21 is directly connected, Vlan200
    C 192.168.0.0/21 is directly connected, Vlan100
    (...)

    Client IP address = 192.168.1.6
    Server IP Address = 10.30.5.5

    ------------------
    iperf, 100/full <--> 100/full
    ------------------

    The results of this test will surprize no one:



    C:\>iperf -f m -B 192.168.1.6 -c 10.30.5.5 -r
    ------------------------------------------------------------
    Server listening on TCP port 5001
    Binding to local address 192.168.1.6
    TCP window size: 0.01 MByte (default)
    ------------------------------------------------------------
    ------------------------------------------------------------
    Client connecting to 10.30.5.5, TCP port 5001
    Binding to local address 192.168.1.6
    TCP window size: 0.01 MByte (default)
    ------------------------------------------------------------
    [1192] local 192.168.1.6 port 3523 connected with 10.30.5.5 port 5001
    [ ID] Interval Transfer Bandwidth
    [1192] 0.0-10.0 sec 96.8 MBytes 81.1 Mbits/sec
    [1948] local 192.168.1.6 port 5001 connected with 10.30.5.5 port 6025
    [ ID] Interval Transfer Bandwidth
    [1948] 0.0-10.0 sec 98.2 MBytes 82.4 Mbits/sec





    See, TCP is about 80Mbit/sec, and symetrical Looks good. Let's try
    UDP:





    C:>iperf -f m -B 192.168.1.6 -c 10.30.5.5 -r -u
    ------------------------------------------------------------
    Server listening on UDP port 5001
    Binding to local address 192.168.1.6
    Receiving 1470 byte datagrams
    UDP buffer size: 0.01 MByte (default)
    ------------------------------------------------------------
    ------------------------------------------------------------
    Client connecting to 10.30.5.5, UDP port 5001
    Binding to local address 192.168.1.6
    Sending 1470 byte datagrams
    UDP buffer size: 0.01 MByte (default)
    ------------------------------------------------------------
    [1192] local 192.168.1.6 port 3601 connected with 10.30.5.5 port 5001
    [ ID] Interval Transfer Bandwidth
    [1192] 0.0-10.0 sec 1.25 MBytes 1.05 Mbits/sec
    [1192] Server Report:
    [1192] 0.0-10.0 sec 1.25 MBytes 1.05 Mbits/sec 0.000 ms 0/ 893
    (0%)
    [1192] Sent 893 datagrams
    [1244] local 192.168.1.6 port 5001 connected with 10.30.5.5 port 6258
    [ ID] Interval Transfer Bandwidth Jitter Lost/Total
    Datagrams
    [1244] 0.0-10.0 sec 1.25 MBytes 1.05 Mbits/sec 0.977 ms 0/ 894
    (0%)





    Whoa! Only 1.05Mbit/sec for UDP? (This is still 100/full <-->
    100/full) Maybe I just don't know enough about UDP and/or iperf?
    (First time I'ved used iperf was today, up till now I had been using
    robocopy.)


    -----------------------------
    Let's now do the (server) GigabitEthernet <--> (client) FastEthernet
    tests.
    -------------------------

    But first, clear switch counter.

    3750Stack# clear counters gi1/0/24
    3750Stack# clear counters fa5/0/24

    ....uh-oh. running iperf as client from "client" results in connection
    timeout. (not consection refused, as happens when I forget to start
    iperf as server)

    ....ok, I'll do iperf -s on the client instead...we are only goint to
    see the "slow" path.

    C:\>iperf -f m -c 192.168.1.6
    ------------------------------------------------------------
    Client connecting to 192.168.1.6, TCP port 5001
    TCP window size: 0.01 MByte (default)
    ------------------------------------------------------------
    [1908] local 10.30.5.5 port 4608 connected with 192.168.1.6 port 5001
    [ ID] Interval Transfer Bandwidth
    [1908] 0.0-10.1 sec 1.74 MBytes 1.45 Mbits/sec

    C:\>


    Yup, slow as ethernet over carrier pigeon. Now, lets see some
    interface counters:


    3750Stack#sh interfaces gi1/0/24
    GigabitEthernet1/0/24 is up, line protocol is up (connected)
    Hardware is Gigabit Ethernet, address is 0011.bb99.2598 (bia
    0011.bb99.2598)
    MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
    reliability 255/255, txload 1/255, rxload 1/255
    Encapsulation ARPA, loopback not set
    Keepalive set (10 sec)
    Full-duplex, 1000Mb/s, media type is RJ45
    output flow-control is off, input flow-control is off
    ARP type: ARPA, ARP Timeout 04:00:00
    Last input never, output 00:00:01, output hang never
    Last clearing of "show interface" counters 00:05:31
    Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
    Queueing strategy: fifo
    Output queue: 0/40 (size/max)
    5 minute input rate 0 bits/sec, 0 packets/sec
    5 minute output rate 0 bits/sec, 0 packets/sec
    2126 packets input, 2261304 bytes, 0 no buffer
    Received 11 broadcasts (0 multicast)
    0 runts, 0 giants, 0 throttles
    0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
    0 watchdog, 0 multicast, 0 pause input
    0 input packets with dribble condition detected
    1690 packets output, 300868 bytes, 0 underruns
    0 output errors, 0 collisions, 0 interface resets
    0 babbles, 0 late collision, 0 deferred
    0 lost carrier, 0 no carrier, 0 PAUSE output
    0 output buffer failures, 0 output buffers swapped out
    3750Stack#
    3750Stack#sh interfaces fa5/0/24
    FastEthernet5/0/24 is up, line protocol is up (connected)
    Hardware is Fast Ethernet, address is 0011.20a7.6b1a (bia
    0011.20a7.6b1a)
    MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
    reliability 255/255, txload 1/255, rxload 1/255
    Encapsulation ARPA, loopback not set
    Keepalive set (10 sec)
    Full-duplex, 100Mb/s
    input flow-control is off, output flow-control is off
    ARP type: ARPA, ARP Timeout 04:00:00
    Last input 00:00:21, output 00:00:57, output hang never
    Last clearing of "show interface" counters 00:17:29
    Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
    Queueing strategy: fifo
    Output queue: 0/40 (size/max)
    5 minute input rate 1000 bits/sec, 1 packets/sec
    5 minute output rate 48000 bits/sec, 14 packets/sec
    3232 packets input, 437440 bytes, 0 no buffer
    Received 31 broadcasts (0 multicast)
    0 runts, 0 giants, 0 throttles
    0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
    0 watchdog, 18 multicast, 0 pause input
    0 input packets with dribble condition detected
    13545 packets output, 7019546 bytes, 0 underruns
    0 output errors, 0 collisions, 0 interface resets
    0 babbles, 0 late collision, 0 deferred
    0 lost carrier, 0 no carrier, 0 PAUSE output
    0 output buffer failures, 0 output buffers swapped out



    I'm no CCNA....but the above tells me there is no duplex issue. Or
    droped packets?!? In fact, I'm starting to think the collisions I had
    been seeing were generated by some other device. (I found out
    yesterday there are some old printers around that operate at 10/half.)

    I don't understand WHY I can't get iperf to run in this, the test I
    wanted you all to see.

    I'll skip the UDP test this time, you can already see how bad the
    situation is already with the server transmitting tcp to the client at
    a rate of 1.45Mbits/sec

    I want to demo asymitry...shame iperf cound not connect both ways...Let
    me flip back to robocopy:


    -------------------------------------------------------------------------------
    ROBOCOPY :: Robust File Copy for Windows :: Version
    XP010
    -------------------------------------------------------------------------------

    Started : Tue May 09 13:11:59 2006

    Source : <Client>\Foo
    Dest : <Server>\Foo

    Files : *.*

    Options : *.* /S /E /COPY:DAT /MOVE /R:1000000 /W:30

    ------------------------------------------------------------------------------

    New Dir 1 \Foo
    100% New File 32.6 m BigFile.mp3

    ------------------------------------------------------------------------------

    Total Copied Skipped Mismatch FAILED Extras
    Dirs : 1 1 0 0 0 0
    Files : 1 1 0 0 0 0
    Bytes : 32.68 m 32.68 m 0 0 0 0
    Times : 0:00:04 0:00:04 0:00:00 0:00:00

    Speed : 6920528 Bytes/sec.
    Speed : 395.995 MegaBytes/min.

    Ended : Tue May 09 13:12:04 2006

    -------------------------------------------------------------------------------
    ROBOCOPY :: Robust File Copy for Windows :: Version
    XP010
    -------------------------------------------------------------------------------

    Started : Tue May 09 13:12:04 2006

    Source : <Client>\Foo
    Dest : <Client>\Foo

    Files : *.*

    Options : *.* /S /E /COPY:DAT /MOVE /R:1000000 /W:30

    ------------------------------------------------------------------------------

    New Dir 1 \Foo
    100% New File 32.6 m BigFile.mp3

    ------------------------------------------------------------------------------

    Total Copied Skipped Mismatch FAILED Extras
    Dirs : 1 1 0 0 0 0
    Files : 1 1 0 0 0 0
    Bytes : 32.68 m 32.68 m 0 0 0 0
    Times : 0:01:19 0:01:19 0:00:00 0:00:00

    Speed : 433561 Bytes/sec.
    Speed : 24.808 MegaBytes/min.

    Ended : Tue May 09 13:13:23 2006



    So while iperf could not transfer both ways, windows file system could.


    Client --> server = 395.995 MegaBytes/min.
    Server --> Client = 24.808 MegaBytes/min.

    Note that when I do robocopy between 100/fll and 100/full...robocoy
    gets 591 MBytes/min both directions. So, even though the FastE -->
    GigE doen't suck as much as GigE -> FastE.....FastE -> FastE is giving
    me the best performace.




    Comments?


    Merv, if you are a "couple hope away" than you are further away then I.
    Remember, in the above there are NO ROUTED HOPS between client and
    server, it's all switched at L2. and there is no "packet buffering" as
    was pointed out to me above. I'll still be placing a L3 router between
    the two on Sunday and see what happens. My goal here is aviod a
    seperate L2 fabric dedicated to gigabit....that's seams to be what you
    already have, no?
     
    mschunk, May 9, 2006
    #16
  17. mschunk

    anybody43 Guest

    Poor network performance 100Mbps <--> 1Gbps ports

    I think that the following description of iperf is OK but I am
    not 100% sure.

    When using iperf TCP tests are very different from UDP tests.

    TCP uses TCP (?) but does modify the behaviour
    slightly since by default iperf has a buffer size of 8k.

    It sends 8k and then waits until all of the data has
    been acknowledged before sending another 8k.
    In your case on one switch (or stack) and at 100Mbps
    this is not significant and you saw ~100Mbps.

    In the case of UDP there are no acknowledgements.
    It just blasts the data into the network at the configured rate.
    There is a default rate that is quite low and this acn be changed
    with commnd line parameters.

    The observed behaviour is anomalous. Your network should not be
    behaving in this way.

    You appear to have eliminated the duplex issue (if the counters are to
    be trusted).

    As a next step I would like to see packet captures from both ends
    of the link.

    You can download Ethereal (and of course winpcap) and
    install it on both ends. If you want to mount the resultant
    dump files on a server on the internet I could have look.

    I would also suggest eliminating the "stack" from the equation
    by using ports on the same switch.

    Can you get any stacking-link statistics?

    On the 4500 when the backplane is oversubscribed
    (to keep it simple) you seem to get.

    sh int Gi3/22
    GigabitEthernet3/22 is up, line protocol is up (connected)
    ......
    109 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored


    sh int Gi3/22 cou det
    ......
    Port Rx-No-Pkt-Buff RxPauseFrames TxPauseFrames
    PauseFramesDrop
    Gi3/22 109 0 0
    0


    I don't supose you have a spare switch? Ideally this should
    be reproduced off line.

    Do you have Cisco support? If I was seeing this I would
    raise a TAC case.

    You could get this problem if there were no
    buffers in the path between the 1000Mbps port and the 100Mbps port.

    TCP sends (small-ish:) bursts of packets at wire rate back to back and
    the switch is REQUIRED to buffer them.

    Maybe for some reason yours is not doing this.
    This could be due to a bug, due to the buffer
    memory being use by something else,
    perhaps due to miss-configuration.

    I see that you have storm control enabled,
    how about turning that off?


    I notice that you software is a bit aged?
    You could check the release notes for likely bugs.
    You have - 12.1(19)EA1d --- 12.1.19-EA1 is "Deferred"
    which I think means that a serious problem was found with it.

    I would consider changing the software.
    Note that in a substantial network that this
    is not something to be undertaken lightly.

    Implementation plan
    Test plan
    Backout plan.

    Good luck.
     
    anybody43, May 10, 2006
    #17
  18. mschunk

    mschunk Guest

    Thanks "anybod"

    I feel somewhat vindicated in that you are able to look at this and say
    to yourself, "WTF"

    I really appreciate your time; I won't waste it w/ packet dumps quite
    yet.

    I do have TAC support...I prefer Usenet over these idiots but I
    degrees. I keep smartnet mostly for RMA's and Software updates. I
    knew this IOS was older, I'll look into this releases "caveats" and
    then upgrading it before going further.

    As you say, you don't just do that on a dime. I don't think I'll have
    the switch up to date in less than three Sundays.

    I'll post back in time, let you all know what happens.

    Thanks again.
     
    mschunk, May 10, 2006
    #18
  19. mschunk

    anybody43 Guest

    Hmmmm.

    You do need to understand how to drive TAC but
    I find them pretty good. You, of course,
    have to jump through a few hoops and send
    a bit of info but if you have a clearly defined issue like this
    it should be quite easy to get the results that you want
    in quite a short period.

    My impression is that Cisco really, really do want TAC to
    provide a good service.

    A while back I was a network consultant with a gold partner
    and I got a lot of experience with TAC. The stuff we were
    sending then had already been picked over
    pretty thoroughly (there were 10 CCIEs in the office - not me)
    and it was very rare not to get at least a decent
    explanation of why we were being stupid:)

    The TAC is a key part of why I still recommend and use
    Cisco kit pretty much exclusively.

    Since the software that you are using is deferred it will
    be difficult to resist an upgrade recommendation though.
    It may of course be deferred for a security reason or something
    and so they may let you keep it.

    Back to the issue.

    Maybe the switch really is full up with traffic?

    Take off the storm control.

    Check the CPU when the problem is ocurring just in case
    for some reason the packets are being software switched.
    This would be indicated by high CPU .

    See if the behaviour is the same with both ports on one module.
    (Just as a simplification of the problem.)

    Good luck again.
     
    anybody43, May 11, 2006
    #19
  20. mschunk

    mschunk Guest

    Thanks Anybod.

    Bad TAC experience...CallManager and IPCC support is how I developed a
    bad taste for them.

    Storm-control. I did try removing it. It had no impact...so I put it
    back before I forgot. That's there for a very good reason...a coworker
    ran "ghost" in broadcast mode during peak hours...not only did it flood
    the switch, it literally destroyed two T1 ports on an attached 3725.
    (that's right! a packet storm that caused hardware failure. The
    gateway was RMA'd a few months later too; never was the same again.)

    For a while I played with mls - I didn't set this switch up - least I
    know more about mls than I did before but toying with that didn't do
    anything either.

    Both on same Module.... yes. Did that a few weeks ago. I get the same
    results going from gi1/0/24 to gi1/0/23 as I do in the results I
    posted.

    The switches are linked through a stack ring cable...sh switch neigh,
    detail...ummm...are there stats for the ring?

    I mentioned that 1gig <--> 1gig performance smokes, right?
    732mbit/sec. (same server I've been testing w/ one that's in
    production.) (which reminds me...a few servers I still have at gig as
    they only talk to each other, the file and database servers are set to
    manual 100/full for the moment.)

    CPU utilization....I've never seen it above 10% on the "5 min"
    interval....hey, my routers do a "sh proc cpu hist" for a dandy
    graph...catalyst IOS can't do that???

    There is another, very odd, side to this story I have not mentioned.

    99% of users access servers like this:

    PC-->IP Phone-->Stack-->Server

    I'm set up a bit different for various reasons:

    PC-->Switch-->Switch-->Stack-->Server

    Apparently, the performance issue has been with us since this network
    was put together...about 1.5 years. I never really noticed. Poeple
    said "this is slow" and - the in-house apps are big so I brushed it
    off, then one day I saw what they had been talking about and paid
    attention.

    I said that 100/full to 100/full (both ports on stack) got about
    80mbits/sec....and I get aobut that too with the extra switch in my
    patch. I was getting about 30mbit/sec to gbit the server in my
    special, extra-switch-than-most path. Still very asymmetrical, though.
    I discovered that, to a point, the more switches I placed between the
    100/full and gbit hosts...things get noticeably better, but still not
    what I think it should be. (Plus such a "fix" is impractical to
    implement for all.)

    I'll open a TAC case.
     
    mschunk, May 11, 2006
    #20
    1. Advertisements

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.