unexplained bad performance of multipoint gre tunnel ...

Discussion in 'Cisco' started by Garry Glendown, Dec 9, 2003.

  1. Yet again, working on some backup links ...

    I have a three router setup of two multipoint gre tunnels (w/nhrp) which
    as such work quite well - at least while they are not used (apart for
    OSPF routing updates or direct pings/transfer tests). All routers are
    3640, running the same 12.2(13)T9 IOS. I have a /28 defined for the two
    tunnels, on of them being a 1500/200 ADSL, the second a 2300 SDSL
    connection through another provider. Both connections are utilizing
    PPPoE on a FastEthernet port (though running at 10mbit half duplex)

    Configuration has been done using a CISCO example minus the (3)DES
    encryption.

    Both spoke routers are configured identically, except for the local
    tunnel IP address.

    The first router (the one with the ADSL connection) sets up the backup
    just fine - if the primary link (2Meg digital) goes down, there is a
    short transition of about 1-2seconds, then the backup link is used
    without noticeable delays or performance problems (the site is mainly
    using downlink, the used bandwidth is usually in about the same range or
    slightly higher than the ADSL bandwidth available). So far, so good.

    The problem is the second link - the site has higher bandwidth use
    (usually running on a 155meg line), but at the time of tests, an active
    usage of less than 2mbit in both directions. Before shutting down the
    main line, ping values through the tunnel are in the range of 8-12ms.

    When shutting down the line, a parallel continous ping to a remote site
    loses some 5-10 packets (considerably more than on the other line), then
    quickly jumps to 4sec delays:

    64 bytes from heise.de (193.99.144.71): icmp_seq=81 ttl=247 time=11.5 ms
    64 bytes from heise.de (193.99.144.71): icmp_seq=82 ttl=247 time=10.8 ms
    64 bytes from heise.de (193.99.144.71): icmp_seq=83 ttl=247 time=11.0 ms
    <"shutdown" on POS line of second router>
    64 bytes from heise.de (193.99.144.71): icmp_seq=89 ttl=246 time=44.5 ms
    64 bytes from heise.de (193.99.144.71): icmp_seq=90 ttl=247 time=160 ms
    64 bytes from heise.de (193.99.144.71): icmp_seq=92 ttl=247 time=922 ms
    64 bytes from heise.de (193.99.144.71): icmp_seq=93 ttl=247 time=2568 ms
    64 bytes from heise.de (193.99.144.71): icmp_seq=94 ttl=247 time=3402 ms
    64 bytes from heise.de (193.99.144.71): icmp_seq=95 ttl=247 time=4367 ms
    64 bytes from heise.de (193.99.144.71): icmp_seq=96 ttl=247 time=4300 ms

    (this delay is kept up - with occasional packet loss of 25% or so,
    latency will stay in the range of 3900-4400ms)

    At the same time, a reset counter on the tunnel interface only shows a
    usage of only ~200kbit in both directions, cricket graphs on the dialer
    and the tunnel confirm this bandwidth usage.

    I have confirmed the available bandwidth on the vanilla SDSL line to be
    over 2mbit, so I keep getting less than 10% of the actual line - I had
    suspected a possible problem with the line, but the audit program
    confirmed the correct operation on line level.

    I'm at a total loss right now of where else to look anymore - does
    anybody have an idea what the reason could be, or what other things I
    should check into???

    Thanks in advance,

    -garry
    Garry Glendown, Dec 9, 2003
    #1
    1. Advertising

  2. Garry Glendown <> wrote in message news:<br5hca$v37$>...
    > Yet again, working on some backup links ...
    >
    > I have a three router setup of two multipoint gre tunnels (w/nhrp) which
    > as such work quite well - at least while they are not used (apart for
    > OSPF routing updates or direct pings/transfer tests). All routers are
    > 3640, running the same 12.2(13)T9 IOS. I have a /28 defined for the two
    > tunnels, on of them being a 1500/200 ADSL, the second a 2300 SDSL
    > connection through another provider. Both connections are utilizing
    > PPPoE on a FastEthernet port (though running at 10mbit half duplex)
    > ...
    > I'm at a total loss right now of where else to look anymore - does
    > anybody have an idea what the reason could be, or what other things I
    > should check into???
    >
    > Thanks in advance,
    >
    > -garry


    Garry,
    I am having similar problems on a 256k pppoe sdsl line.
    I have set up 2 spokes, one cisco 831 via pppoe on 256k sdsl and one
    test cisco 831 on 100 Mbit lan connecting to a cisco 831 multipoint
    gre hub also on the lan. All IOS Version c831-k9o3sy6-mz.123-4.T1.bin.
    The lan tunnel is always working ok even on heavy load (hard to tell
    on 100 Mbit).
    But the sdsl tunnel is not ok, except if there is not much traffic.
    If you put heavy load (like ping -f from another linux box) on the
    sdsl line, regular packets (tested with ping) to the tunnel ip (and
    also wan ip!) get corrupted!
    It seems that if the buffers fill up, the packets get damaged before
    going on the wire. Since I enabled tunnel checksum, packets damaged on
    the wire should be dropped by the router and not reach any host. My
    conclusions, there is either a bug in the cisco ios which damages
    packets in memory, or the memory itself is broken or in deed a wire
    problem, which passes through tunnel checksum. I think this is not a
    tunnel issue itself.

    I will set up another SDSL tunnel with a differen cisco 831 for
    testing (if it really is a ios or memory fault).

    Anyone any ideas?
    Thanks
    Oliver Kaufmann

    hub:
    interface Tunnel0
    ip address 10.10.10.1 255.255.255.0
    no ip redirects
    ip nhrp authentication xxx
    ip nhrp network-id xxx
    ip nhrp holdtime 300
    tunnel source Ethernet1
    tunnel mode gre multipoint
    tunnel key xxx
    tunnel sequence-datagrams
    tunnel checksum

    SDSL spoke:
    interface Tunnel0
    ip address 10.10.10.3 255.255.255.0
    no ip redirects
    ip nhrp authentication xxx
    ip nhrp map 10.10.10.1 xxx
    ip nhrp network-id xxx
    ip nhrp holdtime 300
    ip nhrp nhs 10.10.10.1
    tunnel source Dialer1
    tunnel destination xxx
    tunnel key xxx
    tunnel sequence-datagrams
    tunnel checksum

    corrupted ping reply:

    wrong data byte #0 should be 0xaf but was 0xaeae f ec 3f 80 c4 2 0
    8 9 a b c d e f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e
    1f 20 21 22 23 24 25 26 27
    28 29 2a 2b 2c 2d 2e 2f 30 31 32 33 34 35 36 37 38 39 3a 3b 3c
    3d 3e 3f 40 41 42 43 44 45 46 47
    48 49 4a 4b 4c 4d 4e 4f 50 51 52 53 54 55 56 57 58 59 5a 5b 5c
    5d 5e 5f 60 61 62 63 64 65 66 67
    68 69 6a 6b 6c 6d 6e 6f 70 71 72 73 74 75 76 77 78 79 7a 7b 7c
    7d 7e 7f 80 81 82 83 84 85 86 87
    88 89 8a 8b 8c 8d 8e 8f 90 91 92 93 94 95 96 97 98 99 9a 9b 9c
    9d 9e 9f a0 a1 a2 a3 a4 a5 a6 a7
    a8 a9 aa ab ac ad ae af b0 b1 b2 b3 b4 b5 b6 b7 b8 b9 ba bb bc
    bd be bf c0 c1 c2 c3 c4 c5 c6 c7
    c8 c9 ca cb cc cd ce cf d0 d1 d2 d3 d4 d5 d6 d7 d8 d9 da db dc
    dd de df e0 e1 e2 e3 e4 e5 e6 e7
    e8 e9 ea eb ec ed ee ef f0 f1 f2 f3 f4 f5 f6 f7 f8 f9 fa fb fc
    fd fe ff 0 1 2 3 4 5 6 7
    8 9 a b c d e f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e
    1f 20 21 22 23 24 25 26 27
    28 29 2a 2b 2c 2d 2e 2f 30 31 32 33 34 35 36 37 38 39 3a 3b 3c
    3d 3e 3f 40 41 42 43 44 45 46 47
    48 49 4a 4b 4c 4d 4e 4f 50 51 52 53 54 55 56 57 58 59 5a 5b 5c
    5d 5e 5f 60 61 62 63 64 65 66 67
    68 69 6a 6b 6c 6d 6e 6f 70 71 72 73 74 75 76 77 78 79 7a 7b 7c
    7d 7e 7f 80 81 82 83 84 85 86 87
    88 89 8a 8b 8c 8d 8e 8f 90 91 92 93 94 95 96 97 98 99 9a 9b 9c
    9d 9e 9f a0 a1 a2 a3 a4 a5 a6 a7
    a8 a9 aa ab ac ad ae af b0 b1 b2 b3 b4 b5 b6 b7 b8 b9 ba bb bc
    bd be bf c0 c1 c2 c3 c4 c5 c6 c7
    c8 c9 ca cb cc cd ce cf d0 d1 d2 d3 d4 d5 d6 d7 d8 d9 da db dc
    dd de df e0 e1 e2 e3 e4 e5 e6 e7
    e8 e9 ea eb ec ed ee ef f0 f1 f2 f3 f4 f5 f6 f7 f8 f9 fa fb fc
    fd fe ff 0 1 2 3 4 5 6 7
    8 9 a b c d e f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e
    1f 20 21 22 23 24 25 26 27
    28 29 2a 2b 2c 2d 2e 2f 30 31 32 33 34 35 36 37 38 39 3a 3b 3c
    3d 3e 3f 40 41 42 43 44 45 46 47
    48 49 4a 4b 4c 4d 4e 4f 50 51 52 53 54 55 56 57 58 59 5a 5b 5c
    5d 5e 5f 60 61 62 63 64 65 66 67
    68 69 6a 6b 6c 6d 6e 6f 70 71 72 73 74 75 76 77 78 79 7a 7b 7c
    7d 7e 7f 80 81 82 83 84 85 86 87
    88 89 8a 8b 8c 8d 8e 8f 90 91 92 93 94 95 96 97 98 99 9a 9b 9c
    9d 9e 9f a0 a1 a2 a3 a4 a5 a6 a7
    a8 a9 aa ab ac ad ae af b0 b1 b2 b3 b4 b5 b6 b7 b8 b9 ba bb bc
    bd be bf c0 c1 c2 c3 c4 c5 c6 c7
    c8 c9 ca cb cc cd ce cf d0 d1 d2 d3 d4 d5 d6 d7 d8 d9 da db dc
    dd de df e0 e1 e2 e3 e4 e5 e6 e7
    e8 e9 ea eb ec ed ee ef f0 f1 f2 f3 f4 f5 f6 f7 f8 f9 fa fb fc
    fd fe ff 0 1 2 3 4 5 6 7
    8 9 a b c d e f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e
    1f 20 21 22 23 24 25 26 27
    28 29 2a 2b 2c 2d 2e 2f 30 31 32 33 34 35 36 37 38 39 3a 3b 3c
    3d 3e 3f 40 41 42 43 44 45 46 47
    48 49 4a 4b 4c 4d 4e 4f 50 51 52 53 54 55 56 57 58 59 5a 5b 5c
    5d 5e 5f 60 61 62 63 64 65 66 67
    68 69 6a 6b 6c 6d 6e 6f 70 71 72 73 74 75 76 77 78 79 7a 7b 7c
    7d 7e 7f 80 81 82 83 84 85 86 87
    88 89 8a 8b 8c 8d 8e 8f 90 91 92 93 94 95 96 97 98 99 9a 9b 9c
    9d 9e 9f a0 a1 a2 a3 a4 a5 a6 a7
    a8 a9 aa ab ac ad ae af b0 b1 b2 b3 b4 b5 b6 b7 b8 b9 ba bb bc
    bd be bf c0 c1 c2 c3 c4 c5 c6 c7
    c8 c9 ca cb cc cd ce cf d0 d1 d2 d3 d4 d5 d6 d7 d8 d9 da db dc
    dd de df e0 e1 e2 e3 e4 e5 e6 e7
    e8 e9 ea eb ec ed ee ef f0 f1 f2 f3 f4 f5 f6 f7 f8 f9 fa fb fc
    fd fe ff 0 1 2 3 4 5 6 7
    8 9 a b c d e f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e
    1f 20 21 22 23 24 25 26 27
    28 29 2a 2b 2c 2d 2e 2f 30 31 32 33 34 35 36 37 38 39 3a 3b 3c
    3d 3e 3f 40 41 42 43 44 45 46 47
    48 49 4a 4b 4c 4d 4e 4f 50 51 52 53 54 55 56 57 58 59 5a 5b 5c
    5d 5e 5f 60 61 62 63 64 65 66 67
    68 69 6a 6b 6c 6d 6e 6f 70 71 72 73 74 75 76 77 78 79 7a 7b 7c
    7d 7e 7f 80 81 82 83 84 85 86 87
    88 89 8a 8b 8c 8d 8e 8f 90 91 92 93 94 95 96 97 98 99 9a 9b 9c
    9d 9e 9f a0 a1 a2 a3 a4 a5 a6 a7
    a8 a9 aa ab ac ad ae af b0 b1 b2 b3 b4 b5 b6 b7 b8 b9 ba bb bc
    bd be bf c0 c1 c2 c3 c4 c5 c6 c7
    c8 c9 ca cb cc cd ce cf d0 d1 d2 d3 d4 d5 d6 d7 d8 d9 da db dc
    dd de df e0 e1 e2 e3 e4 e5 e6 e7
    e8 e9 ea eb ec ed ee ef f0 f1 f2 f3 f4 f5 f6 f7 f8 f9 fa fb fc
    fd fe ff 0 1 2 3 4 5 6 7
    8 9 a b c d e f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e
    1f 20 21 22 23 24 25 26 27
    28 29 2a 2b 2c 2d 2e 2f 30 31 32 33 34 35 36 37 38 39 3a 3b 3c
    3d 3e 3f 40 41 42 43 44 45 46 47
    48 49 4a 4b 4c 4d 4e 4f 50 51 52 53 54 55 56 57 58 59 5a 5b 5c
    5d 5e 5f 60 61 62 63 64 65 66 67
    68 69 6a 6b 6c 6d 6e 6f 70 71 72 73 74 75 76 77 78 79 7a 7b 7c
    7d 7e 7f 80 81 82 83 84 85 86 87
    88 89 8a 8b 8c 8d 8e 8f 90 91 92 93 94 95 96 97 98 99 9a 9b 9c
    9d 9e 9f a0 a1 a2 a3 a4 a5 a6 a7
    a8 a9 aa ab ac ad ae af b0 b1 b2 b3 b4 b5 b6 b7 b8 b9 ba bb bc
    bd be bf c0 c1 c2 c3 c4 c5 c6 c7
    c8 c9 ca cb cc cd ce cf d0 d1 d2 d3
    Oliver Kaufmann, Dec 26, 2003
    #2
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. John Ireland
    Replies:
    1
    Views:
    1,075
    Claude LeFort
    Nov 11, 2003
  2. JS
    Replies:
    13
    Views:
    22,269
  3. a.nonny mouse
    Replies:
    2
    Views:
    1,095
  4. Replies:
    6
    Views:
    29,431
  5. Replies:
    12
    Views:
    2,942
    Michael Alan Chary
    Feb 23, 2005
Loading...

Share This Page