Speed Mismatch?!?

Discussion in 'Cisco' started by mschunk, May 8, 2006.

  1. mschunk

    anybody43 Guest

    TAC

    Here is how I would put the case.

    Choose the simplest setup that can reproduce the issue
    i.e. no stack link involved.

    use 3 ports (or ideally 4)
    2 x 1G
    2 x 100M

    Do the performance tests and
    send them the results.

    Send full sh run and relevant sh int to show that the
    interfaces are reporting as running clean.

    If you do it like this and make NO changes to anything
    betwen runs then they cannot get diverted
    down the blind alleys that you have already investigated
    thoroughly. Can they???

    Let's call the computers Gi1, Gi2 and Fa1, Fa2.

    If you show Gi1 <-> Gi2 works OK AND Fa1 <-> Fa2 is OK
    then ask them to fix Gi1 <-> Fa1 which is broken.

    One advantage of using iperf is that they can
    reproduce it for themselves.

    Send
    H:\>iperf -v
    iperf version 1.7.0 (13 Mar 2003) win32 threads
    H:\>
    If that is the path you choose.

    It would be interesting to know if it was indeed packets
    towards the end of a burst that were disappearing.

    Remember that iperf by default will only send
    8k (or about 5 or 6 packtes) in one burst.
    The TCP then send as it sees fit:) which
    can only reduce the burst size further.

    IDEA:-
    Try a test with an iperf buffer of less than 1 packet.
    If that works, you MAY be able to improve the
    local performance by setting the TCP Receive Window to
    just over 1 packet (less than two). Or one
    packet exactly? 1460 bytes usually.
    Why not try it? Set the MS Windows
    TCP Receive Window to 1460 and see what happens.

    You will also have to set the other end to
    send TCP acks for each segment.
    http://support.microsoft.com/kb/328890/
    tcpackfrequency.
    Only available on XP and 2003 Server.
    Set to ONE.


    iperf -c 1.1.1.1 -l1460 :: I have had a look and it is not
    :: clear that this does what I
    thought.

    This will though destroy WAN performance.

    # # # #

    If my current model is correct and the issue is indeed that
    the buffers between Gi ports and Fa ports are not working
    then adding a "buffering" switch to the path would help.


    source--Gi-->-buffer-sw--Fa-->-Stack--Fa-->-sink

    The other thing that adding switches to the path does is
    that it adds latency. To a good approximation all modern
    switches are store and forward. Each switch in the path
    adds 1 Packets worth of Transmssion Delay. The
    value is dependent on the length of each packet
    and the speed of each interface.

    IPERF
    I have had a look and I don't think that IPERF does wait
    for all of the acks after sending a block. It is not clear
    to me at present.
     
    anybody43, May 11, 2006
    #21
    1. Advertisements

  2. mschunk

    mschunk Guest

    TAC has been "...[looking] at the captures..." for the last day+.

    No word yet.
    This _is_ a clue. It's buried above, but the reason it took me a year
    and a half to realise that somthing was horribly wrong, not just slow
    servers etc - was because I have an extra switch in my path compared to
    the rest of the users. I can get 30mbit/sec to Gi. Still crapy and
    asym....80 going to Gi, 30 coming from.

    I have not tried exactly what you have above, I did this:

    source-gi-->buffer-sw-Gi-->Stack-->Fa-sink.

    IOW, I did not try buffering w/ a switch that had an Fa link to the
    stack...I stayed on Gi.

    I also took this a little further and setup an 4x etherchannel, still
    all gi...I _WAS_ trying to buffer somthing somwhere, not really knowing
    how but wondering if packet A overran packet B to the point that the
    stack could not see the etherframe signature of A anymore...reports no
    drop becuase it saw no packet in the first place. I know there is a
    drop somwhere, where is this drop occuring? Packet drop counters
    increment if and only if IOS is aware that a drop has occured...I doubt
    interface counters care what goes on between the guts that contect
    other interfaces. Stack ring counters? Still don't know how to get
    any.

    Anyway, I have time right now to try what you suggest.

    < time passes >

    It works at 65 mbit/sec.

    Gi-source-->buffer-sw-Fa-->stack-->Fa-sink

    iperf...

    Recall that I'm unable to conect from fa-client to gi-server to run the
    test. I get "connection failed" If I fail to lanuch "ipsef -s" on Gi,
    I get "Connection refused" I have the same problem with pcattcp.
    Thing is client-fa-->server-Gi is not the slow path. It's slower that
    it should be using robocoy.

    Gi-server-->stack-->Gi-sink

    ....kicks so much ass. Very fast.

    I got no clue about the inerds of a 3750...is there supposed to be a
    buffer/queue dedicated for moving data between fa and gi?!?!?!?

    I'm been thinkg about the odds of a gi-host overruning an fa-host. I
    don't see it. TCP does send some segments out...but do not most TCP
    stacks "wait" at some point for some ACKs beofre sending more data? In
    UDP, I could see Gi sending so much that packets just drop in the
    switch once egrees queues filled up. Or does it not work that way?
    I'm just making assumption based on prior data; not knowing how much
    value said prior data/experiance has in this case. </end bable>
     
    mschunk, May 12, 2006
    #22
    1. Advertisements

  3. mschunk

    anybody43 Guest

    Poor GI <--> Fa end node performance across a switch,

    I have just remembered something.
    GI Ethernet has a feature called "Pause Frames"

    ############
    # Begin guesswork
    ############
    These are designed to prevent or reduce
    the number of 'overrun frames' (my
    terminology) sent. Some network
    protocols can send wire rate bursts of frames
    that can overflow the buffers in intermediate
    or in end stations. Such overflow packets
    are lost.

    Pause frames are sent by the station that is experiencing
    the overflowing buffers towards the source and in
    some way signal to the source to slow down a bit.

    ############
    # End guesswork
    ############

    I have no real idea if you network _needs_ pause
    frames to be operational to function (I would
    strongly doubt it) however what I was wondering was
    if you did in fact have partially operational
    pause frames and maybe there might be a bug in
    the way that the pause frames were operating.

    If this model is correct you will NOT be seeing lost
    packets on the network.

    I know nothing about pause frames. They could
    for example not show up on a sniffer 'cos
    they could quite reasonably be implemented
    at Layer 1 and a cheap and cheerful sniffer
    could quite reasonably not be able to see them.

    You could quite reasonably post this issue
    on comp.dcom.lans.ethernet. Leave in the
    cisco references. Someone there who is
    familiar with the pause frame implementation
    for example may just see the issue right away.


    Out of interest, you aren't using huge TCP
    windows are you? i.e. you haven't got
    TCP Window scaling turned on? Consider
    all of your hosts.

    If you were getting buffer overflows you should
    I would imaging be seeing output drops on the Fa
    interfaces.
    sh int | inc drops
    If there are none then maybe the counters are broken
    or maybe the drops are internal to the switch or
    maybe there is a 'pause frame bug'.
     
    anybody43, May 14, 2006
    #23
  4. mschunk

    mschunk Guest

    I thought about Pause frames too.

    I'm certain I now even less about them than yourself.

    I do know that I have all flow-control turned off on the stack, as is
    the default. Useing braodcoms diag tools for thier NIC's...During link
    negotiation, the stack is saying "I don't do flow control" and the NIC,
    who has flow control set to auto, noted this.

    I'm watiing for TAC to say _somthing_ but at this point they have not
    responded to what I set or asked any more questions. I will bring up
    flow control with them.

    I looked into flow control because, befor I realized that _ALL_ my GigE
    NIC's are affected, I was thinking "bad driver" and the RTM driver for
    the broadcom NIC's in question did, arrounding to broadcom, have a bu
    in them with reguards to flow control...the older driver would revice a
    pause frame and them pause for too long.
     
    mschunk, May 14, 2006
    #24
  5. mschunk

    rdymek Guest

    What brand/models do you have this experience with? I've currently got
    two 6509's curreently deployed with Gig hard coded on both ends of the
    link (auto disabled). I've also got 2 2948G-48 switches deployed with
    gig hard coded on both ends of the link. We get a full 1gb throughput
    without a single error on the port.

    Ryan
     
    rdymek, May 15, 2006
    #25
  6. mschunk

    rdymek Guest

    Personally, my experiences have been just the opposite. Honestly, I
    can't quote the IEEE standard on this one as I've never looked it up
    and just assumed that you could set it or put it on auto just like
    100mb.

    I've currently got 6500's using the WS-X6148-GE-TX copper 48 port
    blades, as well as 2948G's that are hard coded to gig. Both on the
    server and the switch.

    I'm not saying its not against IEEE, but If it were against IEEE
    standards, then why would the server as well as the switch allow this?

    Ryan
     
    rdymek, May 15, 2006
    #26
    1. Advertisements

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.