Adjusting BGP advertisements

Discussion in 'Cisco' started by Michael Munford, Dec 14, 2003.

  1. I've inherited the maintenance and upkeep duties of my companies small ISP
    platform as part of my normal job duties... (As if I don't have enough to do
    already!!) We supply Internet connectivity for a subset of our normal
    customer base. We have about 150 "Downstream" customers connected via T1's,
    Frac T's, resold DSL's, ISDN's and Dial-ups. Our current platform is
    currently a Cisco 3660 connected to a Tier 1 provider via four individual T1
    lines. These T1's were added to the platform as the customer base expanded.
    They were basically provisioned as two sets of geographically 'Diverse'
    T's. At some point in the recent past, we actually had two interconnected
    Routers, each with one of the 'Diverse' pairs. Due to some recent customer
    shrinkage, we've conolidated back down to the single router with all four
    upstream lines.We've got two /23 blocks for use on the network. We've
    reserved the 1st /25 for our Publicly hosted services such as DNS, Web and
    mail services. The rest of the blocks are subnetted out for the various
    customer circuits. For the sake of discussion, the two /23 blocks will be
    referred to as 10.10.122.0/23 and 10.11.34.0/23.

    I've been noticing that one of our upstream lines is usually close to
    saturation while the others have fairly low utilization and have been trying
    to figure out how to re-distribute the traffic by adjusting the BGP
    route-maps that are in place. These Route-maps for adjusting BGP
    advertisements were originally constructed with the aid of the Tier1
    providers support department. The Route-maps were originally configured to
    divide each /23 into its individual /24 subnets and then advertise them with
    varying metrics to each upstream peer. After the recent consolidation, I've
    adjusted these to some extent, to try and get the /23 blocks advertised with
    a 0 metric to two of the upstream peers.

    I'm not sure if I should be pursuing this a different way, because I'm still
    seeing similar unbalanced traffic on the upstream links.

    Anybody have any suggestions???

    Below is a scrubbed configuration example:
    ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

    version 12.2
    !
    hostname CoreRouter
    !
    enable secret 5 XXXXXXXXXXXXXXX
    !
    ip subnet-zero
    no ip source-route
    !
    !
    interface FastEthernet1/0
    description Public Ethernet Segment
    ip address 10.10.122.1 255.255.255.128
    duplex auto
    speed auto
    no cdp enable
    !
    !
    interface Serial3/2
    description Upstream Link #1
    bandwidth 1536
    no ip address
    encapsulation frame-relay IETF
    ip route-cache flow
    no fair-queue
    serial restart-delay 0
    frame-relay lmi-type ansi
    !
    interface Serial3/2.1 point-to-point
    description Upstream Link #1-SubInterface
    bandwidth 1536
    ip address 192.168.45.82 255.255.255.252
    no ip redirects
    no ip proxy-arp
    no cdp enable
    frame-relay interface-dlci 500 IETF
    !
    !
    interface Serial4/0
    description Upstream Link #2
    bandwidth 1536
    no ip address
    encapsulation frame-relay IETF
    ip route-cache flow
    no fair-queue
    serial restart-delay 0
    frame-relay lmi-type ansi
    !
    interface Serial4/0.1 point-to-point
    description Upstream Link #2-SubInterface
    bandwidth 1536
    ip address 192.168.59.150 255.255.255.252
    no ip redirects
    no ip proxy-arp
    no cdp enable
    frame-relay interface-dlci 500 IETF
    !
    !
    interface Serial6/0
    description Upstream Link #3
    bandwidth 1536
    no ip address
    encapsulation frame-relay IETF
    ip route-cache flow
    no fair-queue
    frame-relay lmi-type ansi
    !
    interface Serial6/0.1 point-to-point
    description Upstream Link #3
    bandwidth 1536
    ip address 172.16.89.238 255.255.255.252
    no ip redirects
    no ip proxy-arp
    no cdp enable
    frame-relay interface-dlci 500 IETF
    !
    !
    interface Serial6/1
    description Upstream Link #4
    bandwidth 1536
    no ip address
    encapsulation frame-relay IETF
    ip route-cache flow
    no fair-queue
    frame-relay lmi-type ansi
    !
    interface Serial6/1.1 point-to-point
    description Upstream Link #4
    bandwidth 1536
    ip address 172.16.89.242 255.255.255.252
    no ip redirects
    no ip proxy-arp
    no cdp enable
    frame-relay interface-dlci 500 IETF
    !
    router bgp 1111
    bgp log-neighbor-changes
    network 10.10.122.0 mask 255.255.255.0
    network 10.10.123.0 mask 255.255.255.0
    network 10.10.122.0 mask 255.255.254.0
    network 10.11.34.0 mask 255.255.255.0
    network 10.11.35.0 mask 255.255.255.0
    network 10.11.34.0 mask 255.255.254.0
    neighbor 172.16.89.237 remote-as 222
    neighbor 172.16.89.237 version 4
    neighbor 172.16.89.237 send-community
    neighbor 172.16.89.237 route-map to-UpstreamLink3 out
    neighbor 172.16.89.241 remote-as 222
    neighbor 172.16.89.241 version 4
    neighbor 172.16.89.241 send-community
    neighbor 172.16.89.241 route-map to-UpstreamLink4 out
    neighbor 192.168.45.81 remote-as 222
    neighbor 192.168.45.81 version 4
    neighbor 192.168.45.81 send-community
    neighbor 192.168.45.81 route-map to-UpstreamLink1 out
    neighbor 192.168.59.149 remote-as 222
    neighbor 192.168.59.149 version 4
    neighbor 192.168.59.149 send-community
    neighbor 192.168.59.149 route-map to-UpstreamLink2 out
    !
    ip classless
    ip route 0.0.0.0 0.0.0.0 172.16.89.237
    ip route 0.0.0.0 0.0.0.0 172.16.89.241
    ip route 0.0.0.0 0.0.0.0 192.168.45.81 2
    ip route 0.0.0.0 0.0.0.0 192.168.59.149 2
    ip route 10.10.122.0 255.255.254.0 Null0
    ip route 10.10.122.0 255.255.255.0 Null0
    ip route 10.10.123.0 255.255.255.0 Null0
    ip route 172.16.0.0 255.255.0.0 172.16.89.237
    ip route 172.16.0.0 255.255.0.0 172.16.89.241
    ip route 10.11.34.0 255.255.254.0 Null0
    ip route 10.11.34.0 255.255.255.0 Null0
    ip route 10.11.35.0 255.255.255.0 10.10.122.202
    ip route 10.11.35.0 255.255.255.0 Null0
    ip route 192.168.0.0 255.255.0.0 192.168.59.149
    ip route 192.168.0.0 255.255.0.0 192.168.45.81
    no ip http server
    !
    !
    !
    access-list 10 permit 10.10.122.0 0.0.0.255
    access-list 10 deny any
    access-list 11 permit 10.11.34.0 0.0.0.255
    access-list 11 deny any
    access-list 20 permit 10.10.123.0 0.0.0.255
    access-list 20 deny any
    access-list 21 permit 10.11.35.0 0.0.0.255
    access-list 21 deny any
    access-list 22 permit 10.11.34.0 0.0.1.255
    access-list 22 deny any
    access-list 23 permit 10.10.122.0 0.0.1.255
    access-list 23 deny any
    !
    !
    !
    route-map to-UpstreamLink3 permit 10
    match ip address 22
    set metric 0
    set community no-export
    !
    route-map to-UpstreamLink3 permit 20
    match ip address 23
    set metric 1
    set community no-export
    !
    route-map to-UpstreamLink4 permit 10
    match ip address 22
    set metric 0
    set community no-export
    !
    route-map to-UpstreamLink4 permit 20
    match ip address 23
    set metric 1
    set community no-export
    !
    route-map to-UpstreamLink1 permit 10
    match ip address 23
    set metric 0
    set community no-export
    !
    route-map to-UpstreamLink1 permit 20
    match ip address 22
    set metric 1
    set community no-export
    !
    route-map to-UpstreamLink2 permit 10
    match ip address 23
    set metric 0
    set community no-export
    !
    route-map to-UpstreamLink2 permit 20
    match ip address 22
    set metric 1
    set community no-export
    !
    !
    line con 0
    exec-timeout 0 0
    line aux 0
    line vty 0 4
    exec-timeout 15 0
    password 7 XXXXXXXXXXXXXXXX
    session-limit 2
    login
    !
    end
    Michael Munford, Dec 14, 2003
    #1
    1. Advertising

  2. Michael Munford

    joe Guest

    this is about this worst way to do this.. sorry :)

    your running 4 peering sessions to the same ISP from the same
    bgp border router ???

    For bgp sake, you could tweak local pref with ip prefix lists/route maps
    to prefer a different part of the internet out each T-1, and send
    a different metric out on each advertisement to influence inbound). Still
    would probally lead to bad routing.

    Your fix here (being it's the same isp upstream) is to run MLPPP.
    You said two sets of T-1's... does this mean 2 x 2 T-1's to different
    isp edge routers ? or are are the t-1's just carried over diverse facilities
    at layer 1/2 and then still drop on the same edge router at the isp ?

    (this is why good communication with an ISP is a must).

    So you looking at 1 or 2 MLPPP interfaces (T-1 bundles).

    Now, this will give you 1 or 2 bigger pipes to play with, and will remove
    bgp from the traffic sharing process. You really don't bgp having a say
    in load sharing, if you can avoid it. It's much better to do this at lower
    layers of the OSI model.

    Get your ISP on the phone, find out what LAYER 3 router(s) your T-1's all end
    up on. If one router, you now have one 6Mbps pipe !!!, Two routers,
    two 3Mbps pipes. Don't worry about all the MLPPP overhead horseshit often
    thrown around here. I have been running it for years on 3600+'s with no
    issues. Then once you get your T-1's aggregated as much as possible with your
    isp, then redesign your bgp config. Say you get it down to two "logical" links,
    assuming two of your t-1's each go to different isp edge routers.

    I would then use as path prepending on one /23 on each connection, but
    advertise both. So one 2 x T-1 MLPPP bundle will always be "primary" for
    each /23, with the other being "backup". Also, I would monitor my internet
    traffic (ntop, or other flow export utility) and divide my 100 top traffic
    destination as's amougnst both bgp sessions). This will be done with
    as regular expressions, raising local preference on each session.

    Please get back to us, I'm curious how this all works out.

    "Michael Munford" <> wrote in message news:<yG6Db.7196$WQ3.6246@lakeread05>...
    > I've inherited the maintenance and upkeep duties of my companies small ISP
    > platform as part of my normal job duties... (As if I don't have enough to do
    > already!!) We supply Internet connectivity for a subset of our normal
    > customer base. We have about 150 "Downstream" customers connected via T1's,
    > Frac T's, resold DSL's, ISDN's and Dial-ups. Our current platform is
    > currently a Cisco 3660 connected to a Tier 1 provider via four individual T1
    > lines. These T1's were added to the platform as the customer base expanded.
    > They were basically provisioned as two sets of geographically 'Diverse'
    > T's. At some point in the recent past, we actually had two interconnected
    > Routers, each with one of the 'Diverse' pairs. Due to some recent customer
    > shrinkage, we've conolidated back down to the single router with all four
    > upstream lines.We've got two /23 blocks for use on the network. We've
    > reserved the 1st /25 for our Publicly hosted services such as DNS, Web and
    > mail services. The rest of the blocks are subnetted out for the various
    > customer circuits. For the sake of discussion, the two /23 blocks will be
    > referred to as 10.10.122.0/23 and 10.11.34.0/23.
    >
    > I've been noticing that one of our upstream lines is usually close to
    > saturation while the others have fairly low utilization and have been trying
    > to figure out how to re-distribute the traffic by adjusting the BGP
    > route-maps that are in place. These Route-maps for adjusting BGP
    > advertisements were originally constructed with the aid of the Tier1
    > providers support department. The Route-maps were originally configured to
    > divide each /23 into its individual /24 subnets and then advertise them with
    > varying metrics to each upstream peer. After the recent consolidation, I've
    > adjusted these to some extent, to try and get the /23 blocks advertised with
    > a 0 metric to two of the upstream peers.
    >
    > I'm not sure if I should be pursuing this a different way, because I'm still
    > seeing similar unbalanced traffic on the upstream links.
    >
    > Anybody have any suggestions???
    >
    > Below is a scrubbed configuration example:
    > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
    >
    > version 12.2
    > !
    > hostname CoreRouter
    > !
    > enable secret 5 XXXXXXXXXXXXXXX
    > !
    > ip subnet-zero
    > no ip source-route
    > !
    > !
    > interface FastEthernet1/0
    > description Public Ethernet Segment
    > ip address 10.10.122.1 255.255.255.128
    > duplex auto
    > speed auto
    > no cdp enable
    > !
    > !
    > interface Serial3/2
    > description Upstream Link #1
    > bandwidth 1536
    > no ip address
    > encapsulation frame-relay IETF
    > ip route-cache flow
    > no fair-queue
    > serial restart-delay 0
    > frame-relay lmi-type ansi
    > !
    > interface Serial3/2.1 point-to-point
    > description Upstream Link #1-SubInterface
    > bandwidth 1536
    > ip address 192.168.45.82 255.255.255.252
    > no ip redirects
    > no ip proxy-arp
    > no cdp enable
    > frame-relay interface-dlci 500 IETF
    > !
    > !
    > interface Serial4/0
    > description Upstream Link #2
    > bandwidth 1536
    > no ip address
    > encapsulation frame-relay IETF
    > ip route-cache flow
    > no fair-queue
    > serial restart-delay 0
    > frame-relay lmi-type ansi
    > !
    > interface Serial4/0.1 point-to-point
    > description Upstream Link #2-SubInterface
    > bandwidth 1536
    > ip address 192.168.59.150 255.255.255.252
    > no ip redirects
    > no ip proxy-arp
    > no cdp enable
    > frame-relay interface-dlci 500 IETF
    > !
    > !
    > interface Serial6/0
    > description Upstream Link #3
    > bandwidth 1536
    > no ip address
    > encapsulation frame-relay IETF
    > ip route-cache flow
    > no fair-queue
    > frame-relay lmi-type ansi
    > !
    > interface Serial6/0.1 point-to-point
    > description Upstream Link #3
    > bandwidth 1536
    > ip address 172.16.89.238 255.255.255.252
    > no ip redirects
    > no ip proxy-arp
    > no cdp enable
    > frame-relay interface-dlci 500 IETF
    > !
    > !
    > interface Serial6/1
    > description Upstream Link #4
    > bandwidth 1536
    > no ip address
    > encapsulation frame-relay IETF
    > ip route-cache flow
    > no fair-queue
    > frame-relay lmi-type ansi
    > !
    > interface Serial6/1.1 point-to-point
    > description Upstream Link #4
    > bandwidth 1536
    > ip address 172.16.89.242 255.255.255.252
    > no ip redirects
    > no ip proxy-arp
    > no cdp enable
    > frame-relay interface-dlci 500 IETF
    > !
    > router bgp 1111
    > bgp log-neighbor-changes
    > network 10.10.122.0 mask 255.255.255.0
    > network 10.10.123.0 mask 255.255.255.0
    > network 10.10.122.0 mask 255.255.254.0
    > network 10.11.34.0 mask 255.255.255.0
    > network 10.11.35.0 mask 255.255.255.0
    > network 10.11.34.0 mask 255.255.254.0
    > neighbor 172.16.89.237 remote-as 222
    > neighbor 172.16.89.237 version 4
    > neighbor 172.16.89.237 send-community
    > neighbor 172.16.89.237 route-map to-UpstreamLink3 out
    > neighbor 172.16.89.241 remote-as 222
    > neighbor 172.16.89.241 version 4
    > neighbor 172.16.89.241 send-community
    > neighbor 172.16.89.241 route-map to-UpstreamLink4 out
    > neighbor 192.168.45.81 remote-as 222
    > neighbor 192.168.45.81 version 4
    > neighbor 192.168.45.81 send-community
    > neighbor 192.168.45.81 route-map to-UpstreamLink1 out
    > neighbor 192.168.59.149 remote-as 222
    > neighbor 192.168.59.149 version 4
    > neighbor 192.168.59.149 send-community
    > neighbor 192.168.59.149 route-map to-UpstreamLink2 out
    > !
    > ip classless
    > ip route 0.0.0.0 0.0.0.0 172.16.89.237
    > ip route 0.0.0.0 0.0.0.0 172.16.89.241
    > ip route 0.0.0.0 0.0.0.0 192.168.45.81 2
    > ip route 0.0.0.0 0.0.0.0 192.168.59.149 2
    > ip route 10.10.122.0 255.255.254.0 Null0
    > ip route 10.10.122.0 255.255.255.0 Null0
    > ip route 10.10.123.0 255.255.255.0 Null0
    > ip route 172.16.0.0 255.255.0.0 172.16.89.237
    > ip route 172.16.0.0 255.255.0.0 172.16.89.241
    > ip route 10.11.34.0 255.255.254.0 Null0
    > ip route 10.11.34.0 255.255.255.0 Null0
    > ip route 10.11.35.0 255.255.255.0 10.10.122.202
    > ip route 10.11.35.0 255.255.255.0 Null0
    > ip route 192.168.0.0 255.255.0.0 192.168.59.149
    > ip route 192.168.0.0 255.255.0.0 192.168.45.81
    > no ip http server
    > !
    > !
    > !
    > access-list 10 permit 10.10.122.0 0.0.0.255
    > access-list 10 deny any
    > access-list 11 permit 10.11.34.0 0.0.0.255
    > access-list 11 deny any
    > access-list 20 permit 10.10.123.0 0.0.0.255
    > access-list 20 deny any
    > access-list 21 permit 10.11.35.0 0.0.0.255
    > access-list 21 deny any
    > access-list 22 permit 10.11.34.0 0.0.1.255
    > access-list 22 deny any
    > access-list 23 permit 10.10.122.0 0.0.1.255
    > access-list 23 deny any
    > !
    > !
    > !
    > route-map to-UpstreamLink3 permit 10
    > match ip address 22
    > set metric 0
    > set community no-export
    > !
    > route-map to-UpstreamLink3 permit 20
    > match ip address 23
    > set metric 1
    > set community no-export
    > !
    > route-map to-UpstreamLink4 permit 10
    > match ip address 22
    > set metric 0
    > set community no-export
    > !
    > route-map to-UpstreamLink4 permit 20
    > match ip address 23
    > set metric 1
    > set community no-export
    > !
    > route-map to-UpstreamLink1 permit 10
    > match ip address 23
    > set metric 0
    > set community no-export
    > !
    > route-map to-UpstreamLink1 permit 20
    > match ip address 22
    > set metric 1
    > set community no-export
    > !
    > route-map to-UpstreamLink2 permit 10
    > match ip address 23
    > set metric 0
    > set community no-export
    > !
    > route-map to-UpstreamLink2 permit 20
    > match ip address 22
    > set metric 1
    > set community no-export
    > !
    > !
    > line con 0
    > exec-timeout 0 0
    > line aux 0
    > line vty 0 4
    > exec-timeout 15 0
    > password 7 XXXXXXXXXXXXXXXX
    > session-limit 2
    > login
    > !
    > end
    joe, Dec 16, 2003
    #2
    1. Advertising

  3. Michael Munford

    MC Guest

    We have been running several bundled T1 lines to our ISP for increased
    bandwidth for several different applications, have four T1 bundles and some
    that are two T1 bundles, are using CEF to do the load balancing on the T1
    bundles. With CEF you can do per destination or per packet load balancing
    with the per packet being more cpu intensive. We are running a two T1 bundle
    on a 2610, CPU about 25% both T1s at 90% traffic, Works OK. The other T1
    bundles are on a 7500 platform, installing DS3 and will use the T1s as
    backup for now until we get more DS3's (have to upgrade our OC-3 ring access
    to OC-12 first.

    CEF is prefered because it can switch traffic better than normal process
    switching or even fast switching on the router which can handle some forms
    of Internet attackes without maxing the cpu on the router.

    Sorry, Not much on details at the moment, fighting the flu, can't tink width
    taken alll dis madisin.

    MC

    "joe" <> wrote in message
    news:...
    > this is about this worst way to do this.. sorry :)
    >
    > your running 4 peering sessions to the same ISP from the same
    > bgp border router ???
    >
    > For bgp sake, you could tweak local pref with ip prefix lists/route maps
    > to prefer a different part of the internet out each T-1, and send
    > a different metric out on each advertisement to influence inbound). Still
    > would probally lead to bad routing.
    >
    > Your fix here (being it's the same isp upstream) is to run MLPPP.
    > You said two sets of T-1's... does this mean 2 x 2 T-1's to different
    > isp edge routers ? or are are the t-1's just carried over diverse

    facilities
    > at layer 1/2 and then still drop on the same edge router at the isp ?
    >
    > (this is why good communication with an ISP is a must).
    >
    > So you looking at 1 or 2 MLPPP interfaces (T-1 bundles).
    >
    > Now, this will give you 1 or 2 bigger pipes to play with, and will remove
    > bgp from the traffic sharing process. You really don't bgp having a say
    > in load sharing, if you can avoid it. It's much better to do this at lower
    > layers of the OSI model.
    >
    > Get your ISP on the phone, find out what LAYER 3 router(s) your T-1's all

    end
    > up on. If one router, you now have one 6Mbps pipe !!!, Two routers,
    > two 3Mbps pipes. Don't worry about all the MLPPP overhead horseshit often
    > thrown around here. I have been running it for years on 3600+'s with no
    > issues. Then once you get your T-1's aggregated as much as possible with

    your
    > isp, then redesign your bgp config. Say you get it down to two "logical"

    links,
    > assuming two of your t-1's each go to different isp edge routers.
    >
    > I would then use as path prepending on one /23 on each connection, but
    > advertise both. So one 2 x T-1 MLPPP bundle will always be "primary" for
    > each /23, with the other being "backup". Also, I would monitor my internet
    > traffic (ntop, or other flow export utility) and divide my 100 top traffic
    > destination as's amougnst both bgp sessions). This will be done with
    > as regular expressions, raising local preference on each session.
    >
    > Please get back to us, I'm curious how this all works out.
    >
    > "Michael Munford" <> wrote in message

    news:<yG6Db.7196$WQ3.6246@lakeread05>...
    > > I've inherited the maintenance and upkeep duties of my companies small

    ISP
    > > platform as part of my normal job duties... (As if I don't have enough

    to do
    > > already!!) We supply Internet connectivity for a subset of our normal
    > > customer base. We have about 150 "Downstream" customers connected via

    T1's,
    > > Frac T's, resold DSL's, ISDN's and Dial-ups. Our current platform is
    > > currently a Cisco 3660 connected to a Tier 1 provider via four

    individual T1
    > > lines. These T1's were added to the platform as the customer base

    expanded.
    > > They were basically provisioned as two sets of geographically 'Diverse'
    > > T's. At some point in the recent past, we actually had two

    interconnected
    > > Routers, each with one of the 'Diverse' pairs. Due to some recent

    customer
    > > shrinkage, we've conolidated back down to the single router with all

    four
    > > upstream lines.We've got two /23 blocks for use on the network. We've
    > > reserved the 1st /25 for our Publicly hosted services such as DNS, Web

    and
    > > mail services. The rest of the blocks are subnetted out for the various
    > > customer circuits. For the sake of discussion, the two /23 blocks will

    be
    > > referred to as 10.10.122.0/23 and 10.11.34.0/23.
    > >
    > > I've been noticing that one of our upstream lines is usually close to
    > > saturation while the others have fairly low utilization and have been

    trying
    > > to figure out how to re-distribute the traffic by adjusting the BGP
    > > route-maps that are in place. These Route-maps for adjusting BGP
    > > advertisements were originally constructed with the aid of the Tier1
    > > providers support department. The Route-maps were originally configured

    to
    > > divide each /23 into its individual /24 subnets and then advertise them

    with
    > > varying metrics to each upstream peer. After the recent consolidation,

    I've
    > > adjusted these to some extent, to try and get the /23 blocks advertised

    with
    > > a 0 metric to two of the upstream peers.
    > >
    > > I'm not sure if I should be pursuing this a different way, because I'm

    still
    > > seeing similar unbalanced traffic on the upstream links.
    > >
    > > Anybody have any suggestions???
    > >
    > > Below is a scrubbed configuration example:
    > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
    > >
    > > version 12.2
    > > !
    > > hostname CoreRouter
    > > !
    > > enable secret 5 XXXXXXXXXXXXXXX
    > > !
    > > ip subnet-zero
    > > no ip source-route
    > > !
    > > !
    > > interface FastEthernet1/0
    > > description Public Ethernet Segment
    > > ip address 10.10.122.1 255.255.255.128
    > > duplex auto
    > > speed auto
    > > no cdp enable
    > > !
    > > !
    > > interface Serial3/2
    > > description Upstream Link #1
    > > bandwidth 1536
    > > no ip address
    > > encapsulation frame-relay IETF
    > > ip route-cache flow
    > > no fair-queue
    > > serial restart-delay 0
    > > frame-relay lmi-type ansi
    > > !
    > > interface Serial3/2.1 point-to-point
    > > description Upstream Link #1-SubInterface
    > > bandwidth 1536
    > > ip address 192.168.45.82 255.255.255.252
    > > no ip redirects
    > > no ip proxy-arp
    > > no cdp enable
    > > frame-relay interface-dlci 500 IETF
    > > !
    > > !
    > > interface Serial4/0
    > > description Upstream Link #2
    > > bandwidth 1536
    > > no ip address
    > > encapsulation frame-relay IETF
    > > ip route-cache flow
    > > no fair-queue
    > > serial restart-delay 0
    > > frame-relay lmi-type ansi
    > > !
    > > interface Serial4/0.1 point-to-point
    > > description Upstream Link #2-SubInterface
    > > bandwidth 1536
    > > ip address 192.168.59.150 255.255.255.252
    > > no ip redirects
    > > no ip proxy-arp
    > > no cdp enable
    > > frame-relay interface-dlci 500 IETF
    > > !
    > > !
    > > interface Serial6/0
    > > description Upstream Link #3
    > > bandwidth 1536
    > > no ip address
    > > encapsulation frame-relay IETF
    > > ip route-cache flow
    > > no fair-queue
    > > frame-relay lmi-type ansi
    > > !
    > > interface Serial6/0.1 point-to-point
    > > description Upstream Link #3
    > > bandwidth 1536
    > > ip address 172.16.89.238 255.255.255.252
    > > no ip redirects
    > > no ip proxy-arp
    > > no cdp enable
    > > frame-relay interface-dlci 500 IETF
    > > !
    > > !
    > > interface Serial6/1
    > > description Upstream Link #4
    > > bandwidth 1536
    > > no ip address
    > > encapsulation frame-relay IETF
    > > ip route-cache flow
    > > no fair-queue
    > > frame-relay lmi-type ansi
    > > !
    > > interface Serial6/1.1 point-to-point
    > > description Upstream Link #4
    > > bandwidth 1536
    > > ip address 172.16.89.242 255.255.255.252
    > > no ip redirects
    > > no ip proxy-arp
    > > no cdp enable
    > > frame-relay interface-dlci 500 IETF
    > > !
    > > router bgp 1111
    > > bgp log-neighbor-changes
    > > network 10.10.122.0 mask 255.255.255.0
    > > network 10.10.123.0 mask 255.255.255.0
    > > network 10.10.122.0 mask 255.255.254.0
    > > network 10.11.34.0 mask 255.255.255.0
    > > network 10.11.35.0 mask 255.255.255.0
    > > network 10.11.34.0 mask 255.255.254.0
    > > neighbor 172.16.89.237 remote-as 222
    > > neighbor 172.16.89.237 version 4
    > > neighbor 172.16.89.237 send-community
    > > neighbor 172.16.89.237 route-map to-UpstreamLink3 out
    > > neighbor 172.16.89.241 remote-as 222
    > > neighbor 172.16.89.241 version 4
    > > neighbor 172.16.89.241 send-community
    > > neighbor 172.16.89.241 route-map to-UpstreamLink4 out
    > > neighbor 192.168.45.81 remote-as 222
    > > neighbor 192.168.45.81 version 4
    > > neighbor 192.168.45.81 send-community
    > > neighbor 192.168.45.81 route-map to-UpstreamLink1 out
    > > neighbor 192.168.59.149 remote-as 222
    > > neighbor 192.168.59.149 version 4
    > > neighbor 192.168.59.149 send-community
    > > neighbor 192.168.59.149 route-map to-UpstreamLink2 out
    > > !
    > > ip classless
    > > ip route 0.0.0.0 0.0.0.0 172.16.89.237
    > > ip route 0.0.0.0 0.0.0.0 172.16.89.241
    > > ip route 0.0.0.0 0.0.0.0 192.168.45.81 2
    > > ip route 0.0.0.0 0.0.0.0 192.168.59.149 2
    > > ip route 10.10.122.0 255.255.254.0 Null0
    > > ip route 10.10.122.0 255.255.255.0 Null0
    > > ip route 10.10.123.0 255.255.255.0 Null0
    > > ip route 172.16.0.0 255.255.0.0 172.16.89.237
    > > ip route 172.16.0.0 255.255.0.0 172.16.89.241
    > > ip route 10.11.34.0 255.255.254.0 Null0
    > > ip route 10.11.34.0 255.255.255.0 Null0
    > > ip route 10.11.35.0 255.255.255.0 10.10.122.202
    > > ip route 10.11.35.0 255.255.255.0 Null0
    > > ip route 192.168.0.0 255.255.0.0 192.168.59.149
    > > ip route 192.168.0.0 255.255.0.0 192.168.45.81
    > > no ip http server
    > > !
    > > !
    > > !
    > > access-list 10 permit 10.10.122.0 0.0.0.255
    > > access-list 10 deny any
    > > access-list 11 permit 10.11.34.0 0.0.0.255
    > > access-list 11 deny any
    > > access-list 20 permit 10.10.123.0 0.0.0.255
    > > access-list 20 deny any
    > > access-list 21 permit 10.11.35.0 0.0.0.255
    > > access-list 21 deny any
    > > access-list 22 permit 10.11.34.0 0.0.1.255
    > > access-list 22 deny any
    > > access-list 23 permit 10.10.122.0 0.0.1.255
    > > access-list 23 deny any
    > > !
    > > !
    > > !
    > > route-map to-UpstreamLink3 permit 10
    > > match ip address 22
    > > set metric 0
    > > set community no-export
    > > !
    > > route-map to-UpstreamLink3 permit 20
    > > match ip address 23
    > > set metric 1
    > > set community no-export
    > > !
    > > route-map to-UpstreamLink4 permit 10
    > > match ip address 22
    > > set metric 0
    > > set community no-export
    > > !
    > > route-map to-UpstreamLink4 permit 20
    > > match ip address 23
    > > set metric 1
    > > set community no-export
    > > !
    > > route-map to-UpstreamLink1 permit 10
    > > match ip address 23
    > > set metric 0
    > > set community no-export
    > > !
    > > route-map to-UpstreamLink1 permit 20
    > > match ip address 22
    > > set metric 1
    > > set community no-export
    > > !
    > > route-map to-UpstreamLink2 permit 10
    > > match ip address 23
    > > set metric 0
    > > set community no-export
    > > !
    > > route-map to-UpstreamLink2 permit 20
    > > match ip address 22
    > > set metric 1
    > > set community no-export
    > > !
    > > !
    > > line con 0
    > > exec-timeout 0 0
    > > line aux 0
    > > line vty 0 4
    > > exec-timeout 15 0
    > > password 7 XXXXXXXXXXXXXXXX
    > > session-limit 2
    > > login
    > > !
    > > end
    MC, Dec 17, 2003
    #3
  4. There are four T's total... Two 'original' circuits that are
    switch/geographically diverse, and the two 'newer' circuits that are also
    switch/geographically diverse... the 'newer' circuits were added with
    similar far-end provisioning to the 'original' pair. So, for example,
    circuit 1 terminates in Richmond, circuit 2 terminates in DC, circuit 3
    terminates in Richmond, circuit 4 terminates in DC. It would appear that
    both Richmond circuits are on the same router, probably because it is a
    smaller POP than DC. The DC circuits appear to terminate on different
    routers.


    Actually, I was on the phone with a support rep from the upstream ISP on an
    unrelated issue today and posed the question of some sort of reprovisioning.
    He mentioned the possibility of "n x T1" bundling, which I would need to
    discuss with teh provisioning/design departments later (unrelated to my case
    at hand...). Is this the same or similar to the MLPPP you mentioned??

    I am hoping to get a NetFlow/RRDTool box running soon (as if I don't have
    enough to do already...), so I can make better sense of which downstream
    customers are pulling the heavy traffic....




    "joe" <> wrote in message
    news:...
    > this is about this worst way to do this.. sorry :)
    >
    > your running 4 peering sessions to the same ISP from the same
    > bgp border router ???
    >
    > For bgp sake, you could tweak local pref with ip prefix lists/route maps
    > to prefer a different part of the internet out each T-1, and send
    > a different metric out on each advertisement to influence inbound). Still
    > would probally lead to bad routing.
    >
    > Your fix here (being it's the same isp upstream) is to run MLPPP.
    > You said two sets of T-1's... does this mean 2 x 2 T-1's to different
    > isp edge routers ? or are are the t-1's just carried over diverse

    facilities
    > at layer 1/2 and then still drop on the same edge router at the isp ?
    >
    > (this is why good communication with an ISP is a must).
    >
    > So you looking at 1 or 2 MLPPP interfaces (T-1 bundles).
    >
    > Now, this will give you 1 or 2 bigger pipes to play with, and will remove
    > bgp from the traffic sharing process. You really don't bgp having a say
    > in load sharing, if you can avoid it. It's much better to do this at lower
    > layers of the OSI model.
    >
    > Get your ISP on the phone, find out what LAYER 3 router(s) your T-1's all

    end
    > up on. If one router, you now have one 6Mbps pipe !!!, Two routers,
    > two 3Mbps pipes. Don't worry about all the MLPPP overhead horseshit often
    > thrown around here. I have been running it for years on 3600+'s with no
    > issues. Then once you get your T-1's aggregated as much as possible with

    your
    > isp, then redesign your bgp config. Say you get it down to two "logical"

    links,
    > assuming two of your t-1's each go to different isp edge routers.
    >
    > I would then use as path prepending on one /23 on each connection, but
    > advertise both. So one 2 x T-1 MLPPP bundle will always be "primary" for
    > each /23, with the other being "backup". Also, I would monitor my internet
    > traffic (ntop, or other flow export utility) and divide my 100 top traffic
    > destination as's amougnst both bgp sessions). This will be done with
    > as regular expressions, raising local preference on each session.
    >
    > Please get back to us, I'm curious how this all works out.
    >
    > "Michael Munford" <> wrote in message

    news:<yG6Db.7196$WQ3.6246@lakeread05>...
    > > I've inherited the maintenance and upkeep duties of my companies small

    ISP
    > > platform as part of my normal job duties... (As if I don't have enough

    to do
    > > already!!) We supply Internet connectivity for a subset of our normal
    > > customer base. We have about 150 "Downstream" customers connected via

    T1's,
    > > Frac T's, resold DSL's, ISDN's and Dial-ups. Our current platform is
    > > currently a Cisco 3660 connected to a Tier 1 provider via four

    individual T1
    > > lines. These T1's were added to the platform as the customer base

    expanded.
    > > They were basically provisioned as two sets of geographically 'Diverse'
    > > T's. At some point in the recent past, we actually had two

    interconnected
    > > Routers, each with one of the 'Diverse' pairs. Due to some recent

    customer
    > > shrinkage, we've conolidated back down to the single router with all

    four
    > > upstream lines.We've got two /23 blocks for use on the network. We've
    > > reserved the 1st /25 for our Publicly hosted services such as DNS, Web

    and
    > > mail services. The rest of the blocks are subnetted out for the various
    > > customer circuits. For the sake of discussion, the two /23 blocks will

    be
    > > referred to as 10.10.122.0/23 and 10.11.34.0/23.
    > >
    > > I've been noticing that one of our upstream lines is usually close to
    > > saturation while the others have fairly low utilization and have been

    trying
    > > to figure out how to re-distribute the traffic by adjusting the BGP
    > > route-maps that are in place. These Route-maps for adjusting BGP
    > > advertisements were originally constructed with the aid of the Tier1
    > > providers support department. The Route-maps were originally configured

    to
    > > divide each /23 into its individual /24 subnets and then advertise them

    with
    > > varying metrics to each upstream peer. After the recent consolidation,

    I've
    > > adjusted these to some extent, to try and get the /23 blocks advertised

    with
    > > a 0 metric to two of the upstream peers.
    > >
    > > I'm not sure if I should be pursuing this a different way, because I'm

    still
    > > seeing similar unbalanced traffic on the upstream links.
    > >
    > > Anybody have any suggestions???
    > >
    > > Below is a scrubbed configuration example:
    > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
    > >
    > > version 12.2
    > > !
    > > hostname CoreRouter
    > > !
    > > enable secret 5 XXXXXXXXXXXXXXX
    > > !
    > > ip subnet-zero
    > > no ip source-route
    > > !
    > > !
    > > interface FastEthernet1/0
    > > description Public Ethernet Segment
    > > ip address 10.10.122.1 255.255.255.128
    > > duplex auto
    > > speed auto
    > > no cdp enable
    > > !
    > > !
    > > interface Serial3/2
    > > description Upstream Link #1
    > > bandwidth 1536
    > > no ip address
    > > encapsulation frame-relay IETF
    > > ip route-cache flow
    > > no fair-queue
    > > serial restart-delay 0
    > > frame-relay lmi-type ansi
    > > !
    > > interface Serial3/2.1 point-to-point
    > > description Upstream Link #1-SubInterface
    > > bandwidth 1536
    > > ip address 192.168.45.82 255.255.255.252
    > > no ip redirects
    > > no ip proxy-arp
    > > no cdp enable
    > > frame-relay interface-dlci 500 IETF
    > > !
    > > !
    > > interface Serial4/0
    > > description Upstream Link #2
    > > bandwidth 1536
    > > no ip address
    > > encapsulation frame-relay IETF
    > > ip route-cache flow
    > > no fair-queue
    > > serial restart-delay 0
    > > frame-relay lmi-type ansi
    > > !
    > > interface Serial4/0.1 point-to-point
    > > description Upstream Link #2-SubInterface
    > > bandwidth 1536
    > > ip address 192.168.59.150 255.255.255.252
    > > no ip redirects
    > > no ip proxy-arp
    > > no cdp enable
    > > frame-relay interface-dlci 500 IETF
    > > !
    > > !
    > > interface Serial6/0
    > > description Upstream Link #3
    > > bandwidth 1536
    > > no ip address
    > > encapsulation frame-relay IETF
    > > ip route-cache flow
    > > no fair-queue
    > > frame-relay lmi-type ansi
    > > !
    > > interface Serial6/0.1 point-to-point
    > > description Upstream Link #3
    > > bandwidth 1536
    > > ip address 172.16.89.238 255.255.255.252
    > > no ip redirects
    > > no ip proxy-arp
    > > no cdp enable
    > > frame-relay interface-dlci 500 IETF
    > > !
    > > !
    > > interface Serial6/1
    > > description Upstream Link #4
    > > bandwidth 1536
    > > no ip address
    > > encapsulation frame-relay IETF
    > > ip route-cache flow
    > > no fair-queue
    > > frame-relay lmi-type ansi
    > > !
    > > interface Serial6/1.1 point-to-point
    > > description Upstream Link #4
    > > bandwidth 1536
    > > ip address 172.16.89.242 255.255.255.252
    > > no ip redirects
    > > no ip proxy-arp
    > > no cdp enable
    > > frame-relay interface-dlci 500 IETF
    > > !
    > > router bgp 1111
    > > bgp log-neighbor-changes
    > > network 10.10.122.0 mask 255.255.255.0
    > > network 10.10.123.0 mask 255.255.255.0
    > > network 10.10.122.0 mask 255.255.254.0
    > > network 10.11.34.0 mask 255.255.255.0
    > > network 10.11.35.0 mask 255.255.255.0
    > > network 10.11.34.0 mask 255.255.254.0
    > > neighbor 172.16.89.237 remote-as 222
    > > neighbor 172.16.89.237 version 4
    > > neighbor 172.16.89.237 send-community
    > > neighbor 172.16.89.237 route-map to-UpstreamLink3 out
    > > neighbor 172.16.89.241 remote-as 222
    > > neighbor 172.16.89.241 version 4
    > > neighbor 172.16.89.241 send-community
    > > neighbor 172.16.89.241 route-map to-UpstreamLink4 out
    > > neighbor 192.168.45.81 remote-as 222
    > > neighbor 192.168.45.81 version 4
    > > neighbor 192.168.45.81 send-community
    > > neighbor 192.168.45.81 route-map to-UpstreamLink1 out
    > > neighbor 192.168.59.149 remote-as 222
    > > neighbor 192.168.59.149 version 4
    > > neighbor 192.168.59.149 send-community
    > > neighbor 192.168.59.149 route-map to-UpstreamLink2 out
    > > !
    > > ip classless
    > > ip route 0.0.0.0 0.0.0.0 172.16.89.237
    > > ip route 0.0.0.0 0.0.0.0 172.16.89.241
    > > ip route 0.0.0.0 0.0.0.0 192.168.45.81 2
    > > ip route 0.0.0.0 0.0.0.0 192.168.59.149 2
    > > ip route 10.10.122.0 255.255.254.0 Null0
    > > ip route 10.10.122.0 255.255.255.0 Null0
    > > ip route 10.10.123.0 255.255.255.0 Null0
    > > ip route 172.16.0.0 255.255.0.0 172.16.89.237
    > > ip route 172.16.0.0 255.255.0.0 172.16.89.241
    > > ip route 10.11.34.0 255.255.254.0 Null0
    > > ip route 10.11.34.0 255.255.255.0 Null0
    > > ip route 10.11.35.0 255.255.255.0 10.10.122.202
    > > ip route 10.11.35.0 255.255.255.0 Null0
    > > ip route 192.168.0.0 255.255.0.0 192.168.59.149
    > > ip route 192.168.0.0 255.255.0.0 192.168.45.81
    > > no ip http server
    > > !
    > > !
    > > !
    > > access-list 10 permit 10.10.122.0 0.0.0.255
    > > access-list 10 deny any
    > > access-list 11 permit 10.11.34.0 0.0.0.255
    > > access-list 11 deny any
    > > access-list 20 permit 10.10.123.0 0.0.0.255
    > > access-list 20 deny any
    > > access-list 21 permit 10.11.35.0 0.0.0.255
    > > access-list 21 deny any
    > > access-list 22 permit 10.11.34.0 0.0.1.255
    > > access-list 22 deny any
    > > access-list 23 permit 10.10.122.0 0.0.1.255
    > > access-list 23 deny any
    > > !
    > > !
    > > !
    > > route-map to-UpstreamLink3 permit 10
    > > match ip address 22
    > > set metric 0
    > > set community no-export
    > > !
    > > route-map to-UpstreamLink3 permit 20
    > > match ip address 23
    > > set metric 1
    > > set community no-export
    > > !
    > > route-map to-UpstreamLink4 permit 10
    > > match ip address 22
    > > set metric 0
    > > set community no-export
    > > !
    > > route-map to-UpstreamLink4 permit 20
    > > match ip address 23
    > > set metric 1
    > > set community no-export
    > > !
    > > route-map to-UpstreamLink1 permit 10
    > > match ip address 23
    > > set metric 0
    > > set community no-export
    > > !
    > > route-map to-UpstreamLink1 permit 20
    > > match ip address 22
    > > set metric 1
    > > set community no-export
    > > !
    > > route-map to-UpstreamLink2 permit 10
    > > match ip address 23
    > > set metric 0
    > > set community no-export
    > > !
    > > route-map to-UpstreamLink2 permit 20
    > > match ip address 22
    > > set metric 1
    > > set community no-export
    > > !
    > > !
    > > line con 0
    > > exec-timeout 0 0
    > > line aux 0
    > > line vty 0 4
    > > exec-timeout 15 0
    > > password 7 XXXXXXXXXXXXXXXX
    > > session-limit 2
    > > login
    > > !
    > > end
    Michael Munford, Dec 17, 2003
    #4
  5. I have enabled CEF to allow NetFlow reporting, but haven't looked into using
    CEF to affect traffic flow... How is this accomplished??


    "MC" <> wrote in message
    news:_GNDb.6871$...
    > We have been running several bundled T1 lines to our ISP for increased
    > bandwidth for several different applications, have four T1 bundles and

    some
    > that are two T1 bundles, are using CEF to do the load balancing on the T1
    > bundles. With CEF you can do per destination or per packet load balancing
    > with the per packet being more cpu intensive. We are running a two T1

    bundle
    > on a 2610, CPU about 25% both T1s at 90% traffic, Works OK. The other T1
    > bundles are on a 7500 platform, installing DS3 and will use the T1s as
    > backup for now until we get more DS3's (have to upgrade our OC-3 ring

    access
    > to OC-12 first.
    >
    > CEF is prefered because it can switch traffic better than normal process
    > switching or even fast switching on the router which can handle some forms
    > of Internet attackes without maxing the cpu on the router.
    >
    > Sorry, Not much on details at the moment, fighting the flu, can't tink

    width
    > taken alll dis madisin.
    >
    > MC
    >
    > "joe" <> wrote in message
    > news:...
    > > this is about this worst way to do this.. sorry :)
    > >
    > > your running 4 peering sessions to the same ISP from the same
    > > bgp border router ???
    > >
    > > For bgp sake, you could tweak local pref with ip prefix lists/route maps
    > > to prefer a different part of the internet out each T-1, and send
    > > a different metric out on each advertisement to influence inbound).

    Still
    > > would probally lead to bad routing.
    > >
    > > Your fix here (being it's the same isp upstream) is to run MLPPP.
    > > You said two sets of T-1's... does this mean 2 x 2 T-1's to different
    > > isp edge routers ? or are are the t-1's just carried over diverse

    > facilities
    > > at layer 1/2 and then still drop on the same edge router at the isp ?
    > >
    > > (this is why good communication with an ISP is a must).
    > >
    > > So you looking at 1 or 2 MLPPP interfaces (T-1 bundles).
    > >
    > > Now, this will give you 1 or 2 bigger pipes to play with, and will

    remove
    > > bgp from the traffic sharing process. You really don't bgp having a say
    > > in load sharing, if you can avoid it. It's much better to do this at

    lower
    > > layers of the OSI model.
    > >
    > > Get your ISP on the phone, find out what LAYER 3 router(s) your T-1's

    all
    > end
    > > up on. If one router, you now have one 6Mbps pipe !!!, Two routers,
    > > two 3Mbps pipes. Don't worry about all the MLPPP overhead horseshit

    often
    > > thrown around here. I have been running it for years on 3600+'s with no
    > > issues. Then once you get your T-1's aggregated as much as possible with

    > your
    > > isp, then redesign your bgp config. Say you get it down to two "logical"

    > links,
    > > assuming two of your t-1's each go to different isp edge routers.
    > >
    > > I would then use as path prepending on one /23 on each connection, but
    > > advertise both. So one 2 x T-1 MLPPP bundle will always be "primary" for
    > > each /23, with the other being "backup". Also, I would monitor my

    internet
    > > traffic (ntop, or other flow export utility) and divide my 100 top

    traffic
    > > destination as's amougnst both bgp sessions). This will be done with
    > > as regular expressions, raising local preference on each session.
    > >
    > > Please get back to us, I'm curious how this all works out.
    > >
    > > "Michael Munford" <> wrote in message

    > news:<yG6Db.7196$WQ3.6246@lakeread05>...
    > > > I've inherited the maintenance and upkeep duties of my companies small

    > ISP
    > > > platform as part of my normal job duties... (As if I don't have enough

    > to do
    > > > already!!) We supply Internet connectivity for a subset of our normal
    > > > customer base. We have about 150 "Downstream" customers connected via

    > T1's,
    > > > Frac T's, resold DSL's, ISDN's and Dial-ups. Our current platform is
    > > > currently a Cisco 3660 connected to a Tier 1 provider via four

    > individual T1
    > > > lines. These T1's were added to the platform as the customer base

    > expanded.
    > > > They were basically provisioned as two sets of geographically

    'Diverse'
    > > > T's. At some point in the recent past, we actually had two

    > interconnected
    > > > Routers, each with one of the 'Diverse' pairs. Due to some recent

    > customer
    > > > shrinkage, we've conolidated back down to the single router with all

    > four
    > > > upstream lines.We've got two /23 blocks for use on the network. We've
    > > > reserved the 1st /25 for our Publicly hosted services such as DNS, Web

    > and
    > > > mail services. The rest of the blocks are subnetted out for the

    various
    > > > customer circuits. For the sake of discussion, the two /23 blocks will

    > be
    > > > referred to as 10.10.122.0/23 and 10.11.34.0/23.
    > > >
    > > > I've been noticing that one of our upstream lines is usually close to
    > > > saturation while the others have fairly low utilization and have been

    > trying
    > > > to figure out how to re-distribute the traffic by adjusting the BGP
    > > > route-maps that are in place. These Route-maps for adjusting BGP
    > > > advertisements were originally constructed with the aid of the Tier1
    > > > providers support department. The Route-maps were originally

    configured
    > to
    > > > divide each /23 into its individual /24 subnets and then advertise

    them
    > with
    > > > varying metrics to each upstream peer. After the recent consolidation,

    > I've
    > > > adjusted these to some extent, to try and get the /23 blocks

    advertised
    > with
    > > > a 0 metric to two of the upstream peers.
    > > >
    > > > I'm not sure if I should be pursuing this a different way, because I'm

    > still
    > > > seeing similar unbalanced traffic on the upstream links.
    > > >
    > > > Anybody have any suggestions???
    > > >
    > > > Below is a scrubbed configuration example:
    > > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
    > > >
    > > > version 12.2
    > > > !
    > > > hostname CoreRouter
    > > > !
    > > > enable secret 5 XXXXXXXXXXXXXXX
    > > > !
    > > > ip subnet-zero
    > > > no ip source-route
    > > > !
    > > > !
    > > > interface FastEthernet1/0
    > > > description Public Ethernet Segment
    > > > ip address 10.10.122.1 255.255.255.128
    > > > duplex auto
    > > > speed auto
    > > > no cdp enable
    > > > !
    > > > !
    > > > interface Serial3/2
    > > > description Upstream Link #1
    > > > bandwidth 1536
    > > > no ip address
    > > > encapsulation frame-relay IETF
    > > > ip route-cache flow
    > > > no fair-queue
    > > > serial restart-delay 0
    > > > frame-relay lmi-type ansi
    > > > !
    > > > interface Serial3/2.1 point-to-point
    > > > description Upstream Link #1-SubInterface
    > > > bandwidth 1536
    > > > ip address 192.168.45.82 255.255.255.252
    > > > no ip redirects
    > > > no ip proxy-arp
    > > > no cdp enable
    > > > frame-relay interface-dlci 500 IETF
    > > > !
    > > > !
    > > > interface Serial4/0
    > > > description Upstream Link #2
    > > > bandwidth 1536
    > > > no ip address
    > > > encapsulation frame-relay IETF
    > > > ip route-cache flow
    > > > no fair-queue
    > > > serial restart-delay 0
    > > > frame-relay lmi-type ansi
    > > > !
    > > > interface Serial4/0.1 point-to-point
    > > > description Upstream Link #2-SubInterface
    > > > bandwidth 1536
    > > > ip address 192.168.59.150 255.255.255.252
    > > > no ip redirects
    > > > no ip proxy-arp
    > > > no cdp enable
    > > > frame-relay interface-dlci 500 IETF
    > > > !
    > > > !
    > > > interface Serial6/0
    > > > description Upstream Link #3
    > > > bandwidth 1536
    > > > no ip address
    > > > encapsulation frame-relay IETF
    > > > ip route-cache flow
    > > > no fair-queue
    > > > frame-relay lmi-type ansi
    > > > !
    > > > interface Serial6/0.1 point-to-point
    > > > description Upstream Link #3
    > > > bandwidth 1536
    > > > ip address 172.16.89.238 255.255.255.252
    > > > no ip redirects
    > > > no ip proxy-arp
    > > > no cdp enable
    > > > frame-relay interface-dlci 500 IETF
    > > > !
    > > > !
    > > > interface Serial6/1
    > > > description Upstream Link #4
    > > > bandwidth 1536
    > > > no ip address
    > > > encapsulation frame-relay IETF
    > > > ip route-cache flow
    > > > no fair-queue
    > > > frame-relay lmi-type ansi
    > > > !
    > > > interface Serial6/1.1 point-to-point
    > > > description Upstream Link #4
    > > > bandwidth 1536
    > > > ip address 172.16.89.242 255.255.255.252
    > > > no ip redirects
    > > > no ip proxy-arp
    > > > no cdp enable
    > > > frame-relay interface-dlci 500 IETF
    > > > !
    > > > router bgp 1111
    > > > bgp log-neighbor-changes
    > > > network 10.10.122.0 mask 255.255.255.0
    > > > network 10.10.123.0 mask 255.255.255.0
    > > > network 10.10.122.0 mask 255.255.254.0
    > > > network 10.11.34.0 mask 255.255.255.0
    > > > network 10.11.35.0 mask 255.255.255.0
    > > > network 10.11.34.0 mask 255.255.254.0
    > > > neighbor 172.16.89.237 remote-as 222
    > > > neighbor 172.16.89.237 version 4
    > > > neighbor 172.16.89.237 send-community
    > > > neighbor 172.16.89.237 route-map to-UpstreamLink3 out
    > > > neighbor 172.16.89.241 remote-as 222
    > > > neighbor 172.16.89.241 version 4
    > > > neighbor 172.16.89.241 send-community
    > > > neighbor 172.16.89.241 route-map to-UpstreamLink4 out
    > > > neighbor 192.168.45.81 remote-as 222
    > > > neighbor 192.168.45.81 version 4
    > > > neighbor 192.168.45.81 send-community
    > > > neighbor 192.168.45.81 route-map to-UpstreamLink1 out
    > > > neighbor 192.168.59.149 remote-as 222
    > > > neighbor 192.168.59.149 version 4
    > > > neighbor 192.168.59.149 send-community
    > > > neighbor 192.168.59.149 route-map to-UpstreamLink2 out
    > > > !
    > > > ip classless
    > > > ip route 0.0.0.0 0.0.0.0 172.16.89.237
    > > > ip route 0.0.0.0 0.0.0.0 172.16.89.241
    > > > ip route 0.0.0.0 0.0.0.0 192.168.45.81 2
    > > > ip route 0.0.0.0 0.0.0.0 192.168.59.149 2
    > > > ip route 10.10.122.0 255.255.254.0 Null0
    > > > ip route 10.10.122.0 255.255.255.0 Null0
    > > > ip route 10.10.123.0 255.255.255.0 Null0
    > > > ip route 172.16.0.0 255.255.0.0 172.16.89.237
    > > > ip route 172.16.0.0 255.255.0.0 172.16.89.241
    > > > ip route 10.11.34.0 255.255.254.0 Null0
    > > > ip route 10.11.34.0 255.255.255.0 Null0
    > > > ip route 10.11.35.0 255.255.255.0 10.10.122.202
    > > > ip route 10.11.35.0 255.255.255.0 Null0
    > > > ip route 192.168.0.0 255.255.0.0 192.168.59.149
    > > > ip route 192.168.0.0 255.255.0.0 192.168.45.81
    > > > no ip http server
    > > > !
    > > > !
    > > > !
    > > > access-list 10 permit 10.10.122.0 0.0.0.255
    > > > access-list 10 deny any
    > > > access-list 11 permit 10.11.34.0 0.0.0.255
    > > > access-list 11 deny any
    > > > access-list 20 permit 10.10.123.0 0.0.0.255
    > > > access-list 20 deny any
    > > > access-list 21 permit 10.11.35.0 0.0.0.255
    > > > access-list 21 deny any
    > > > access-list 22 permit 10.11.34.0 0.0.1.255
    > > > access-list 22 deny any
    > > > access-list 23 permit 10.10.122.0 0.0.1.255
    > > > access-list 23 deny any
    > > > !
    > > > !
    > > > !
    > > > route-map to-UpstreamLink3 permit 10
    > > > match ip address 22
    > > > set metric 0
    > > > set community no-export
    > > > !
    > > > route-map to-UpstreamLink3 permit 20
    > > > match ip address 23
    > > > set metric 1
    > > > set community no-export
    > > > !
    > > > route-map to-UpstreamLink4 permit 10
    > > > match ip address 22
    > > > set metric 0
    > > > set community no-export
    > > > !
    > > > route-map to-UpstreamLink4 permit 20
    > > > match ip address 23
    > > > set metric 1
    > > > set community no-export
    > > > !
    > > > route-map to-UpstreamLink1 permit 10
    > > > match ip address 23
    > > > set metric 0
    > > > set community no-export
    > > > !
    > > > route-map to-UpstreamLink1 permit 20
    > > > match ip address 22
    > > > set metric 1
    > > > set community no-export
    > > > !
    > > > route-map to-UpstreamLink2 permit 10
    > > > match ip address 23
    > > > set metric 0
    > > > set community no-export
    > > > !
    > > > route-map to-UpstreamLink2 permit 20
    > > > match ip address 22
    > > > set metric 1
    > > > set community no-export
    > > > !
    > > > !
    > > > line con 0
    > > > exec-timeout 0 0
    > > > line aux 0
    > > > line vty 0 4
    > > > exec-timeout 15 0
    > > > password 7 XXXXXXXXXXXXXXXX
    > > > session-limit 2
    > > > login
    > > > !
    > > > end

    >
    >
    Michael Munford, Dec 17, 2003
    #5
  6. Michael Munford

    joe Guest

    "n x T1" is a common name for mlppp service. much better than cef.
    cef can do nothing for your inbound, if the isp messes up. CEF in my
    experience (per-packet) is lame. MLPPP is nearly perfect load sharing,
    at layer 2. Where bgp, routing, clearing the cef adj. table, etc, cant
    hurt it..

    MLPPP is the closest thing to a bigger pipe you can do. You should
    have 2 mlppp bundles, each of 2 t-1's, one to richmond, one to dc.

    then run 2 (instead of 4) bgp peering sessions, and monitor your bandwidth
    carefully. make sure you know what your using, etc. I would then tweak
    my bgp policy carefully, based on my 100 top outbound asn's (in bytes),
    with flow export, to send 50 out one bundle, and 50 out another.
    for inbound load balancing, dont use med (metric). Advertise one block
    AS prepended on each bundle. then each block will be preferred in one bundle,
    unless one is down (Then of course they both use the same bundle !)

    "Michael Munford" <> wrote in message news:<ErQDb.9134$WQ3.369@lakeread05>...
    > There are four T's total... Two 'original' circuits that are
    > switch/geographically diverse, and the two 'newer' circuits that are also
    > switch/geographically diverse... the 'newer' circuits were added with
    > similar far-end provisioning to the 'original' pair. So, for example,
    > circuit 1 terminates in Richmond, circuit 2 terminates in DC, circuit 3
    > terminates in Richmond, circuit 4 terminates in DC. It would appear that
    > both Richmond circuits are on the same router, probably because it is a
    > smaller POP than DC. The DC circuits appear to terminate on different
    > routers.
    >
    >
    > Actually, I was on the phone with a support rep from the upstream ISP on an
    > unrelated issue today and posed the question of some sort of reprovisioning.
    > He mentioned the possibility of "n x T1" bundling, which I would need to
    > discuss with teh provisioning/design departments later (unrelated to my case
    > at hand...). Is this the same or similar to the MLPPP you mentioned??
    >
    > I am hoping to get a NetFlow/RRDTool box running soon (as if I don't have
    > enough to do already...), so I can make better sense of which downstream
    > customers are pulling the heavy traffic....
    >
    >
    >
    >
    > "joe" <> wrote in message
    > news:...
    > > this is about this worst way to do this.. sorry :)
    > >
    > > your running 4 peering sessions to the same ISP from the same
    > > bgp border router ???
    > >
    > > For bgp sake, you could tweak local pref with ip prefix lists/route maps
    > > to prefer a different part of the internet out each T-1, and send
    > > a different metric out on each advertisement to influence inbound). Still
    > > would probally lead to bad routing.
    > >
    > > Your fix here (being it's the same isp upstream) is to run MLPPP.
    > > You said two sets of T-1's... does this mean 2 x 2 T-1's to different
    > > isp edge routers ? or are are the t-1's just carried over diverse

    > facilities
    > > at layer 1/2 and then still drop on the same edge router at the isp ?
    > >
    > > (this is why good communication with an ISP is a must).
    > >
    > > So you looking at 1 or 2 MLPPP interfaces (T-1 bundles).
    > >
    > > Now, this will give you 1 or 2 bigger pipes to play with, and will remove
    > > bgp from the traffic sharing process. You really don't bgp having a say
    > > in load sharing, if you can avoid it. It's much better to do this at lower
    > > layers of the OSI model.
    > >
    > > Get your ISP on the phone, find out what LAYER 3 router(s) your T-1's all

    > end
    > > up on. If one router, you now have one 6Mbps pipe !!!, Two routers,
    > > two 3Mbps pipes. Don't worry about all the MLPPP overhead horseshit often
    > > thrown around here. I have been running it for years on 3600+'s with no
    > > issues. Then once you get your T-1's aggregated as much as possible with

    > your
    > > isp, then redesign your bgp config. Say you get it down to two "logical"

    > links,
    > > assuming two of your t-1's each go to different isp edge routers.
    > >
    > > I would then use as path prepending on one /23 on each connection, but
    > > advertise both. So one 2 x T-1 MLPPP bundle will always be "primary" for
    > > each /23, with the other being "backup". Also, I would monitor my internet
    > > traffic (ntop, or other flow export utility) and divide my 100 top traffic
    > > destination as's amougnst both bgp sessions). This will be done with
    > > as regular expressions, raising local preference on each session.
    > >
    > > Please get back to us, I'm curious how this all works out.
    > >
    > > "Michael Munford" <> wrote in message

    > news:<yG6Db.7196$WQ3.6246@lakeread05>...
    > > > I've inherited the maintenance and upkeep duties of my companies small

    > ISP
    > > > platform as part of my normal job duties... (As if I don't have enough

    > to do
    > > > already!!) We supply Internet connectivity for a subset of our normal
    > > > customer base. We have about 150 "Downstream" customers connected via

    > T1's,
    > > > Frac T's, resold DSL's, ISDN's and Dial-ups. Our current platform is
    > > > currently a Cisco 3660 connected to a Tier 1 provider via four

    > individual T1
    > > > lines. These T1's were added to the platform as the customer base

    > expanded.
    > > > They were basically provisioned as two sets of geographically 'Diverse'
    > > > T's. At some point in the recent past, we actually had two

    > interconnected
    > > > Routers, each with one of the 'Diverse' pairs. Due to some recent

    > customer
    > > > shrinkage, we've conolidated back down to the single router with all

    > four
    > > > upstream lines.We've got two /23 blocks for use on the network. We've
    > > > reserved the 1st /25 for our Publicly hosted services such as DNS, Web

    > and
    > > > mail services. The rest of the blocks are subnetted out for the various
    > > > customer circuits. For the sake of discussion, the two /23 blocks will

    > be
    > > > referred to as 10.10.122.0/23 and 10.11.34.0/23.
    > > >
    > > > I've been noticing that one of our upstream lines is usually close to
    > > > saturation while the others have fairly low utilization and have been

    > trying
    > > > to figure out how to re-distribute the traffic by adjusting the BGP
    > > > route-maps that are in place. These Route-maps for adjusting BGP
    > > > advertisements were originally constructed with the aid of the Tier1
    > > > providers support department. The Route-maps were originally configured

    > to
    > > > divide each /23 into its individual /24 subnets and then advertise them

    > with
    > > > varying metrics to each upstream peer. After the recent consolidation,

    > I've
    > > > adjusted these to some extent, to try and get the /23 blocks advertised

    > with
    > > > a 0 metric to two of the upstream peers.
    > > >
    > > > I'm not sure if I should be pursuing this a different way, because I'm

    > still
    > > > seeing similar unbalanced traffic on the upstream links.
    > > >
    > > > Anybody have any suggestions???
    > > >
    > > > Below is a scrubbed configuration example:
    > > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
    > > >
    > > > version 12.2
    > > > !
    > > > hostname CoreRouter
    > > > !
    > > > enable secret 5 XXXXXXXXXXXXXXX
    > > > !
    > > > ip subnet-zero
    > > > no ip source-route
    > > > !
    > > > !
    > > > interface FastEthernet1/0
    > > > description Public Ethernet Segment
    > > > ip address 10.10.122.1 255.255.255.128
    > > > duplex auto
    > > > speed auto
    > > > no cdp enable
    > > > !
    > > > !
    > > > interface Serial3/2
    > > > description Upstream Link #1
    > > > bandwidth 1536
    > > > no ip address
    > > > encapsulation frame-relay IETF
    > > > ip route-cache flow
    > > > no fair-queue
    > > > serial restart-delay 0
    > > > frame-relay lmi-type ansi
    > > > !
    > > > interface Serial3/2.1 point-to-point
    > > > description Upstream Link #1-SubInterface
    > > > bandwidth 1536
    > > > ip address 192.168.45.82 255.255.255.252
    > > > no ip redirects
    > > > no ip proxy-arp
    > > > no cdp enable
    > > > frame-relay interface-dlci 500 IETF
    > > > !
    > > > !
    > > > interface Serial4/0
    > > > description Upstream Link #2
    > > > bandwidth 1536
    > > > no ip address
    > > > encapsulation frame-relay IETF
    > > > ip route-cache flow
    > > > no fair-queue
    > > > serial restart-delay 0
    > > > frame-relay lmi-type ansi
    > > > !
    > > > interface Serial4/0.1 point-to-point
    > > > description Upstream Link #2-SubInterface
    > > > bandwidth 1536
    > > > ip address 192.168.59.150 255.255.255.252
    > > > no ip redirects
    > > > no ip proxy-arp
    > > > no cdp enable
    > > > frame-relay interface-dlci 500 IETF
    > > > !
    > > > !
    > > > interface Serial6/0
    > > > description Upstream Link #3
    > > > bandwidth 1536
    > > > no ip address
    > > > encapsulation frame-relay IETF
    > > > ip route-cache flow
    > > > no fair-queue
    > > > frame-relay lmi-type ansi
    > > > !
    > > > interface Serial6/0.1 point-to-point
    > > > description Upstream Link #3
    > > > bandwidth 1536
    > > > ip address 172.16.89.238 255.255.255.252
    > > > no ip redirects
    > > > no ip proxy-arp
    > > > no cdp enable
    > > > frame-relay interface-dlci 500 IETF
    > > > !
    > > > !
    > > > interface Serial6/1
    > > > description Upstream Link #4
    > > > bandwidth 1536
    > > > no ip address
    > > > encapsulation frame-relay IETF
    > > > ip route-cache flow
    > > > no fair-queue
    > > > frame-relay lmi-type ansi
    > > > !
    > > > interface Serial6/1.1 point-to-point
    > > > description Upstream Link #4
    > > > bandwidth 1536
    > > > ip address 172.16.89.242 255.255.255.252
    > > > no ip redirects
    > > > no ip proxy-arp
    > > > no cdp enable
    > > > frame-relay interface-dlci 500 IETF
    > > > !
    > > > router bgp 1111
    > > > bgp log-neighbor-changes
    > > > network 10.10.122.0 mask 255.255.255.0
    > > > network 10.10.123.0 mask 255.255.255.0
    > > > network 10.10.122.0 mask 255.255.254.0
    > > > network 10.11.34.0 mask 255.255.255.0
    > > > network 10.11.35.0 mask 255.255.255.0
    > > > network 10.11.34.0 mask 255.255.254.0
    > > > neighbor 172.16.89.237 remote-as 222
    > > > neighbor 172.16.89.237 version 4
    > > > neighbor 172.16.89.237 send-community
    > > > neighbor 172.16.89.237 route-map to-UpstreamLink3 out
    > > > neighbor 172.16.89.241 remote-as 222
    > > > neighbor 172.16.89.241 version 4
    > > > neighbor 172.16.89.241 send-community
    > > > neighbor 172.16.89.241 route-map to-UpstreamLink4 out
    > > > neighbor 192.168.45.81 remote-as 222
    > > > neighbor 192.168.45.81 version 4
    > > > neighbor 192.168.45.81 send-community
    > > > neighbor 192.168.45.81 route-map to-UpstreamLink1 out
    > > > neighbor 192.168.59.149 remote-as 222
    > > > neighbor 192.168.59.149 version 4
    > > > neighbor 192.168.59.149 send-community
    > > > neighbor 192.168.59.149 route-map to-UpstreamLink2 out
    > > > !
    > > > ip classless
    > > > ip route 0.0.0.0 0.0.0.0 172.16.89.237
    > > > ip route 0.0.0.0 0.0.0.0 172.16.89.241
    > > > ip route 0.0.0.0 0.0.0.0 192.168.45.81 2
    > > > ip route 0.0.0.0 0.0.0.0 192.168.59.149 2
    > > > ip route 10.10.122.0 255.255.254.0 Null0
    > > > ip route 10.10.122.0 255.255.255.0 Null0
    > > > ip route 10.10.123.0 255.255.255.0 Null0
    > > > ip route 172.16.0.0 255.255.0.0 172.16.89.237
    > > > ip route 172.16.0.0 255.255.0.0 172.16.89.241
    > > > ip route 10.11.34.0 255.255.254.0 Null0
    > > > ip route 10.11.34.0 255.255.255.0 Null0
    > > > ip route 10.11.35.0 255.255.255.0 10.10.122.202
    > > > ip route 10.11.35.0 255.255.255.0 Null0
    > > > ip route 192.168.0.0 255.255.0.0 192.168.59.149
    > > > ip route 192.168.0.0 255.255.0.0 192.168.45.81
    > > > no ip http server
    > > > !
    > > > !
    > > > !
    > > > access-list 10 permit 10.10.122.0 0.0.0.255
    > > > access-list 10 deny any
    > > > access-list 11 permit 10.11.34.0 0.0.0.255
    > > > access-list 11 deny any
    > > > access-list 20 permit 10.10.123.0 0.0.0.255
    > > > access-list 20 deny any
    > > > access-list 21 permit 10.11.35.0 0.0.0.255
    > > > access-list 21 deny any
    > > > access-list 22 permit 10.11.34.0 0.0.1.255
    > > > access-list 22 deny any
    > > > access-list 23 permit 10.10.122.0 0.0.1.255
    > > > access-list 23 deny any
    > > > !
    > > > !
    > > > !
    > > > route-map to-UpstreamLink3 permit 10
    > > > match ip address 22
    > > > set metric 0
    > > > set community no-export
    > > > !
    > > > route-map to-UpstreamLink3 permit 20
    > > > match ip address 23
    > > > set metric 1
    > > > set community no-export
    > > > !
    > > > route-map to-UpstreamLink4 permit 10
    > > > match ip address 22
    > > > set metric 0
    > > > set community no-export
    > > > !
    > > > route-map to-UpstreamLink4 permit 20
    > > > match ip address 23
    > > > set metric 1
    > > > set community no-export
    > > > !
    > > > route-map to-UpstreamLink1 permit 10
    > > > match ip address 23
    > > > set metric 0
    > > > set community no-export
    > > > !
    > > > route-map to-UpstreamLink1 permit 20
    > > > match ip address 22
    > > > set metric 1
    > > > set community no-export
    > > > !
    > > > route-map to-UpstreamLink2 permit 10
    > > > match ip address 23
    > > > set metric 0
    > > > set community no-export
    > > > !
    > > > route-map to-UpstreamLink2 permit 20
    > > > match ip address 22
    > > > set metric 1
    > > > set community no-export
    > > > !
    > > > !
    > > > line con 0
    > > > exec-timeout 0 0
    > > > line aux 0
    > > > line vty 0 4
    > > > exec-timeout 15 0
    > > > password 7 XXXXXXXXXXXXXXXX
    > > > session-limit 2
    > > > login
    > > > !
    > > > end
    joe, Dec 17, 2003
    #6
  7. I'm guessing that the following statement assumes that my router is
    receiving full BGP routes from the ISP:
    "I would then tweak my bgp policy carefully, based on my 100 top outbound
    asn's (in bytes),"

    Unfortunately, we're currently just doing static default routes for outbound
    traffic (remember, I've inherited this rig...), which is also contributing
    to our traffic flow control issues...
    I guess I'm a 'newbie' to "advanced BGP", so do you mind giving me a brief
    rundown of what is required to switch-over to utilizing "full BGP routes"
    for outbound traffic???


    One more question: Do these 'bundles' continue to function if a bundle
    element goes down??

    Thanks for your help so far!!!


    "joe" <> wrote in message
    news:...
    > "n x T1" is a common name for mlppp service. much better than cef.
    > cef can do nothing for your inbound, if the isp messes up. CEF in my
    > experience (per-packet) is lame. MLPPP is nearly perfect load sharing,
    > at layer 2. Where bgp, routing, clearing the cef adj. table, etc, cant
    > hurt it..
    >
    > MLPPP is the closest thing to a bigger pipe you can do. You should
    > have 2 mlppp bundles, each of 2 t-1's, one to richmond, one to dc.
    >
    > then run 2 (instead of 4) bgp peering sessions, and monitor your bandwidth
    > carefully. make sure you know what your using, etc. I would then tweak
    > my bgp policy carefully, based on my 100 top outbound asn's (in bytes),
    > with flow export, to send 50 out one bundle, and 50 out another.
    > for inbound load balancing, dont use med (metric). Advertise one block
    > AS prepended on each bundle. then each block will be preferred in one

    bundle,
    > unless one is down (Then of course they both use the same bundle !)
    >
    > "Michael Munford" <> wrote in message

    news:<ErQDb.9134$WQ3.369@lakeread05>...
    > > There are four T's total... Two 'original' circuits that are
    > > switch/geographically diverse, and the two 'newer' circuits that are

    also
    > > switch/geographically diverse... the 'newer' circuits were added with
    > > similar far-end provisioning to the 'original' pair. So, for example,
    > > circuit 1 terminates in Richmond, circuit 2 terminates in DC, circuit 3
    > > terminates in Richmond, circuit 4 terminates in DC. It would appear that
    > > both Richmond circuits are on the same router, probably because it is a
    > > smaller POP than DC. The DC circuits appear to terminate on different
    > > routers.
    > >
    > >
    > > Actually, I was on the phone with a support rep from the upstream ISP on

    an
    > > unrelated issue today and posed the question of some sort of

    reprovisioning.
    > > He mentioned the possibility of "n x T1" bundling, which I would need to
    > > discuss with teh provisioning/design departments later (unrelated to my

    case
    > > at hand...). Is this the same or similar to the MLPPP you mentioned??
    > >
    > > I am hoping to get a NetFlow/RRDTool box running soon (as if I don't

    have
    > > enough to do already...), so I can make better sense of which downstream
    > > customers are pulling the heavy traffic....
    > >
    > >
    > >
    > >
    > > "joe" <> wrote in message
    > > news:...
    > > > this is about this worst way to do this.. sorry :)
    > > >
    > > > your running 4 peering sessions to the same ISP from the same
    > > > bgp border router ???
    > > >
    > > > For bgp sake, you could tweak local pref with ip prefix lists/route

    maps
    > > > to prefer a different part of the internet out each T-1, and send
    > > > a different metric out on each advertisement to influence inbound).

    Still
    > > > would probally lead to bad routing.
    > > >
    > > > Your fix here (being it's the same isp upstream) is to run MLPPP.
    > > > You said two sets of T-1's... does this mean 2 x 2 T-1's to different
    > > > isp edge routers ? or are are the t-1's just carried over diverse

    > > facilities
    > > > at layer 1/2 and then still drop on the same edge router at the isp ?
    > > >
    > > > (this is why good communication with an ISP is a must).
    > > >
    > > > So you looking at 1 or 2 MLPPP interfaces (T-1 bundles).
    > > >
    > > > Now, this will give you 1 or 2 bigger pipes to play with, and will

    remove
    > > > bgp from the traffic sharing process. You really don't bgp having a

    say
    > > > in load sharing, if you can avoid it. It's much better to do this at

    lower
    > > > layers of the OSI model.
    > > >
    > > > Get your ISP on the phone, find out what LAYER 3 router(s) your T-1's

    all
    > > end
    > > > up on. If one router, you now have one 6Mbps pipe !!!, Two routers,
    > > > two 3Mbps pipes. Don't worry about all the MLPPP overhead horseshit

    often
    > > > thrown around here. I have been running it for years on 3600+'s with

    no
    > > > issues. Then once you get your T-1's aggregated as much as possible

    with
    > > your
    > > > isp, then redesign your bgp config. Say you get it down to two

    "logical"
    > > links,
    > > > assuming two of your t-1's each go to different isp edge routers.
    > > >
    > > > I would then use as path prepending on one /23 on each connection, but
    > > > advertise both. So one 2 x T-1 MLPPP bundle will always be "primary"

    for
    > > > each /23, with the other being "backup". Also, I would monitor my

    internet
    > > > traffic (ntop, or other flow export utility) and divide my 100 top

    traffic
    > > > destination as's amougnst both bgp sessions). This will be done with
    > > > as regular expressions, raising local preference on each session.
    > > >
    > > > Please get back to us, I'm curious how this all works out.
    > > >
    > > > "Michael Munford" <> wrote in message

    > > news:<yG6Db.7196$WQ3.6246@lakeread05>...
    > > > > I've inherited the maintenance and upkeep duties of my companies

    small
    > > ISP
    > > > > platform as part of my normal job duties... (As if I don't have

    enough
    > > to do
    > > > > already!!) We supply Internet connectivity for a subset of our

    normal
    > > > > customer base. We have about 150 "Downstream" customers connected

    via
    > > T1's,
    > > > > Frac T's, resold DSL's, ISDN's and Dial-ups. Our current platform is
    > > > > currently a Cisco 3660 connected to a Tier 1 provider via four

    > > individual T1
    > > > > lines. These T1's were added to the platform as the customer base

    > > expanded.
    > > > > They were basically provisioned as two sets of geographically

    'Diverse'
    > > > > T's. At some point in the recent past, we actually had two

    > > interconnected
    > > > > Routers, each with one of the 'Diverse' pairs. Due to some recent

    > > customer
    > > > > shrinkage, we've conolidated back down to the single router with all

    > > four
    > > > > upstream lines.We've got two /23 blocks for use on the network.

    We've
    > > > > reserved the 1st /25 for our Publicly hosted services such as DNS,

    Web
    > > and
    > > > > mail services. The rest of the blocks are subnetted out for the

    various
    > > > > customer circuits. For the sake of discussion, the two /23 blocks

    will
    > > be
    > > > > referred to as 10.10.122.0/23 and 10.11.34.0/23.
    > > > >
    > > > > I've been noticing that one of our upstream lines is usually close

    to
    > > > > saturation while the others have fairly low utilization and have

    been
    > > trying
    > > > > to figure out how to re-distribute the traffic by adjusting the BGP
    > > > > route-maps that are in place. These Route-maps for adjusting BGP
    > > > > advertisements were originally constructed with the aid of the Tier1
    > > > > providers support department. The Route-maps were originally

    configured
    > > to
    > > > > divide each /23 into its individual /24 subnets and then advertise

    them
    > > with
    > > > > varying metrics to each upstream peer. After the recent

    consolidation,
    > > I've
    > > > > adjusted these to some extent, to try and get the /23 blocks

    advertised
    > > with
    > > > > a 0 metric to two of the upstream peers.
    > > > >
    > > > > I'm not sure if I should be pursuing this a different way, because

    I'm
    > > still
    > > > > seeing similar unbalanced traffic on the upstream links.
    > > > >
    > > > > Anybody have any suggestions???
    > > > >
    > > > > Below is a scrubbed configuration example:
    > > > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
    > > > >
    > > > > version 12.2
    > > > > !
    > > > > hostname CoreRouter
    > > > > !
    > > > > enable secret 5 XXXXXXXXXXXXXXX
    > > > > !
    > > > > ip subnet-zero
    > > > > no ip source-route
    > > > > !
    > > > > !
    > > > > interface FastEthernet1/0
    > > > > description Public Ethernet Segment
    > > > > ip address 10.10.122.1 255.255.255.128
    > > > > duplex auto
    > > > > speed auto
    > > > > no cdp enable
    > > > > !
    > > > > !
    > > > > interface Serial3/2
    > > > > description Upstream Link #1
    > > > > bandwidth 1536
    > > > > no ip address
    > > > > encapsulation frame-relay IETF
    > > > > ip route-cache flow
    > > > > no fair-queue
    > > > > serial restart-delay 0
    > > > > frame-relay lmi-type ansi
    > > > > !
    > > > > interface Serial3/2.1 point-to-point
    > > > > description Upstream Link #1-SubInterface
    > > > > bandwidth 1536
    > > > > ip address 192.168.45.82 255.255.255.252
    > > > > no ip redirects
    > > > > no ip proxy-arp
    > > > > no cdp enable
    > > > > frame-relay interface-dlci 500 IETF
    > > > > !
    > > > > !
    > > > > interface Serial4/0
    > > > > description Upstream Link #2
    > > > > bandwidth 1536
    > > > > no ip address
    > > > > encapsulation frame-relay IETF
    > > > > ip route-cache flow
    > > > > no fair-queue
    > > > > serial restart-delay 0
    > > > > frame-relay lmi-type ansi
    > > > > !
    > > > > interface Serial4/0.1 point-to-point
    > > > > description Upstream Link #2-SubInterface
    > > > > bandwidth 1536
    > > > > ip address 192.168.59.150 255.255.255.252
    > > > > no ip redirects
    > > > > no ip proxy-arp
    > > > > no cdp enable
    > > > > frame-relay interface-dlci 500 IETF
    > > > > !
    > > > > !
    > > > > interface Serial6/0
    > > > > description Upstream Link #3
    > > > > bandwidth 1536
    > > > > no ip address
    > > > > encapsulation frame-relay IETF
    > > > > ip route-cache flow
    > > > > no fair-queue
    > > > > frame-relay lmi-type ansi
    > > > > !
    > > > > interface Serial6/0.1 point-to-point
    > > > > description Upstream Link #3
    > > > > bandwidth 1536
    > > > > ip address 172.16.89.238 255.255.255.252
    > > > > no ip redirects
    > > > > no ip proxy-arp
    > > > > no cdp enable
    > > > > frame-relay interface-dlci 500 IETF
    > > > > !
    > > > > !
    > > > > interface Serial6/1
    > > > > description Upstream Link #4
    > > > > bandwidth 1536
    > > > > no ip address
    > > > > encapsulation frame-relay IETF
    > > > > ip route-cache flow
    > > > > no fair-queue
    > > > > frame-relay lmi-type ansi
    > > > > !
    > > > > interface Serial6/1.1 point-to-point
    > > > > description Upstream Link #4
    > > > > bandwidth 1536
    > > > > ip address 172.16.89.242 255.255.255.252
    > > > > no ip redirects
    > > > > no ip proxy-arp
    > > > > no cdp enable
    > > > > frame-relay interface-dlci 500 IETF
    > > > > !
    > > > > router bgp 1111
    > > > > bgp log-neighbor-changes
    > > > > network 10.10.122.0 mask 255.255.255.0
    > > > > network 10.10.123.0 mask 255.255.255.0
    > > > > network 10.10.122.0 mask 255.255.254.0
    > > > > network 10.11.34.0 mask 255.255.255.0
    > > > > network 10.11.35.0 mask 255.255.255.0
    > > > > network 10.11.34.0 mask 255.255.254.0
    > > > > neighbor 172.16.89.237 remote-as 222
    > > > > neighbor 172.16.89.237 version 4
    > > > > neighbor 172.16.89.237 send-community
    > > > > neighbor 172.16.89.237 route-map to-UpstreamLink3 out
    > > > > neighbor 172.16.89.241 remote-as 222
    > > > > neighbor 172.16.89.241 version 4
    > > > > neighbor 172.16.89.241 send-community
    > > > > neighbor 172.16.89.241 route-map to-UpstreamLink4 out
    > > > > neighbor 192.168.45.81 remote-as 222
    > > > > neighbor 192.168.45.81 version 4
    > > > > neighbor 192.168.45.81 send-community
    > > > > neighbor 192.168.45.81 route-map to-UpstreamLink1 out
    > > > > neighbor 192.168.59.149 remote-as 222
    > > > > neighbor 192.168.59.149 version 4
    > > > > neighbor 192.168.59.149 send-community
    > > > > neighbor 192.168.59.149 route-map to-UpstreamLink2 out
    > > > > !
    > > > > ip classless
    > > > > ip route 0.0.0.0 0.0.0.0 172.16.89.237
    > > > > ip route 0.0.0.0 0.0.0.0 172.16.89.241
    > > > > ip route 0.0.0.0 0.0.0.0 192.168.45.81 2
    > > > > ip route 0.0.0.0 0.0.0.0 192.168.59.149 2
    > > > > ip route 10.10.122.0 255.255.254.0 Null0
    > > > > ip route 10.10.122.0 255.255.255.0 Null0
    > > > > ip route 10.10.123.0 255.255.255.0 Null0
    > > > > ip route 172.16.0.0 255.255.0.0 172.16.89.237
    > > > > ip route 172.16.0.0 255.255.0.0 172.16.89.241
    > > > > ip route 10.11.34.0 255.255.254.0 Null0
    > > > > ip route 10.11.34.0 255.255.255.0 Null0
    > > > > ip route 10.11.35.0 255.255.255.0 10.10.122.202
    > > > > ip route 10.11.35.0 255.255.255.0 Null0
    > > > > ip route 192.168.0.0 255.255.0.0 192.168.59.149
    > > > > ip route 192.168.0.0 255.255.0.0 192.168.45.81
    > > > > no ip http server
    > > > > !
    > > > > !
    > > > > !
    > > > > access-list 10 permit 10.10.122.0 0.0.0.255
    > > > > access-list 10 deny any
    > > > > access-list 11 permit 10.11.34.0 0.0.0.255
    > > > > access-list 11 deny any
    > > > > access-list 20 permit 10.10.123.0 0.0.0.255
    > > > > access-list 20 deny any
    > > > > access-list 21 permit 10.11.35.0 0.0.0.255
    > > > > access-list 21 deny any
    > > > > access-list 22 permit 10.11.34.0 0.0.1.255
    > > > > access-list 22 deny any
    > > > > access-list 23 permit 10.10.122.0 0.0.1.255
    > > > > access-list 23 deny any
    > > > > !
    > > > > !
    > > > > !
    > > > > route-map to-UpstreamLink3 permit 10
    > > > > match ip address 22
    > > > > set metric 0
    > > > > set community no-export
    > > > > !
    > > > > route-map to-UpstreamLink3 permit 20
    > > > > match ip address 23
    > > > > set metric 1
    > > > > set community no-export
    > > > > !
    > > > > route-map to-UpstreamLink4 permit 10
    > > > > match ip address 22
    > > > > set metric 0
    > > > > set community no-export
    > > > > !
    > > > > route-map to-UpstreamLink4 permit 20
    > > > > match ip address 23
    > > > > set metric 1
    > > > > set community no-export
    > > > > !
    > > > > route-map to-UpstreamLink1 permit 10
    > > > > match ip address 23
    > > > > set metric 0
    > > > > set community no-export
    > > > > !
    > > > > route-map to-UpstreamLink1 permit 20
    > > > > match ip address 22
    > > > > set metric 1
    > > > > set community no-export
    > > > > !
    > > > > route-map to-UpstreamLink2 permit 10
    > > > > match ip address 23
    > > > > set metric 0
    > > > > set community no-export
    > > > > !
    > > > > route-map to-UpstreamLink2 permit 20
    > > > > match ip address 22
    > > > > set metric 1
    > > > > set community no-export
    > > > > !
    > > > > !
    > > > > line con 0
    > > > > exec-timeout 0 0
    > > > > line aux 0
    > > > > line vty 0 4
    > > > > exec-timeout 15 0
    > > > > password 7 XXXXXXXXXXXXXXXX
    > > > > session-limit 2
    > > > > login
    > > > > !
    > > > > end
    Michael Munford, Dec 19, 2003
    #7
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. AA
    Replies:
    5
    Views:
    1,170
    Barry Margolin
    Apr 15, 2004
  2. Replies:
    1
    Views:
    4,138
    Charlie Root
    Mar 3, 2006
  3. Mike
    Replies:
    4
    Views:
    458
  4. Heidi Manway

    Quality Reliable Software to block Advertisements?

    Heidi Manway, Nov 20, 2006, in forum: Computer Support
    Replies:
    9
    Views:
    475
    Beauregard T. Shagnasty
    Nov 21, 2006
  5. Ryan

    iBGP Advertisements

    Ryan, Nov 9, 2007, in forum: Cisco
    Replies:
    1
    Views:
    518
    Barry Margolin
    Nov 9, 2007
Loading...

Share This Page