«

I am debating whether to use a Cisco 3725 router or Cisco 3745 router for
this scenario. I will have a building with about 4 VLAN's and approx. 900
connections - users, printers, wireless. The switches are L2 and gigabit
ethernet connected to each other. The Cisco 3725/3745 will be providing L3
via two Fast Ethernet cards in a trunk. The routers will also have an ATM
OC-3 card in it to route back to the Data Center site. At highest
utilization, we have seen the OC-3 link from the building to the Data Center
at 40Mb/s. The router will have a default of 128MB of RAM. Will a 3725
router suffice or would it be better to go with a 3745. Cisco says that the
3725 has a max. process switch capacity of 125,000 pps whereas the 3745 has
250,000 pps when using CEF switching.

Thanks

On Fri, 23 Jan 2004 16:21:12 -0600, <> wrote:

>I am debating whether to use a Cisco 3725 router or Cisco 3745 router for
>this scenario. I will have a building with about 4 VLAN's and approx. 900
>connections - users, printers, wireless. The switches are L2 and gigabit
>ethernet connected to each other. The Cisco 3725/3745 will be providing L3
>via two Fast Ethernet cards in a trunk. The routers will also have an ATM
>OC-3 card in it to route back to the Data Center site. At highest
>utilization, we have seen the OC-3 link from the building to the Data Center
>at 40Mb/s. The router will have a default of 128MB of RAM. Will a 3725
>router suffice or would it be better to go with a 3745. Cisco says that the
>3725 has a max. process switch capacity of 125,000 pps whereas the 3745 has
>250,000 pps when using CEF switching.

I was able to do 200Mbps through a 3725 (100Mbps bidirectional between
two FE interfaces via CEF, 1500 byte packets) with a SmartBits. The
router didn't seem to be maxed out, but it only had the two FE
interfaces so I couldn't try pushing more bandwidth through it.

Take it for what it's worth. Note two things about router throughput:

1) Cisco's pps ratings reflect performance with 64 byte packets, the
smallest Ethernet packets possible. This is because, in general
smaller packets = higher pps. Such pps ratings don't tell you much
about maximum throughput potential, since pps usually decreases as
packet size increases.

2) Throughput testing via SmartBits/TTCP/etc generally gives "best
case" results. Things usually aren't so optimal in the real world,
due to varying packet sizes on most networks and performance-reducing
features such as ACLs, NAT, and so forth that may be configured.

This has been discussed here before so you can Google prior threads
for more information.

-Terry

In article <>,
Terry Baranski <0VE> wrote:
:1) Cisco's pps ratings reflect performance with 64 byte packets, the
:smallest Ethernet packets possible. This is because, in general
:smaller packets = higher pps. Such pps ratings don't tell you much
:about maximum throughput potential, since pps usually decreases as
acket size increases.

I think you might have cause and effect reversed. Cisco uses 64 byte
packets in their ratings because 64 byte packets are the hardest on
the equipment, so the 64-byte packet rating tells you how well the
device can keep up to the maximum load.

Historically there have numerous routers that could handle the
maximum possible throughput (by using the longest possible packets),
but could not keep up when shorter packets were used.

Also, real traffic tends to be bimodal, involving lots of short
packets, then a noticable drop-off, then a [lesser] peak at the maximum
size packets. Real-life packet sizes are not evenly distributed.
--
Warhol's Law: every Usenet user is entitled to his or her very own
fifteen minutes of flame -- The Squoire

On 24 Jan 2004 02:19:07 GMT, (Walter
Roberson) wrote:

>In article <>,
>Terry Baranski <0VE> wrote:
>:1) Cisco's pps ratings reflect performance with 64 byte packets, the
>:smallest Ethernet packets possible. This is because, in general
>:smaller packets = higher pps. Such pps ratings don't tell you much
>:about maximum throughput potential, since pps usually decreases as
>acket size increases.
>
>I think you might have cause and effect reversed. Cisco uses 64 byte
>packets in their ratings because 64 byte packets are the hardest on
>the equipment, so the 64-byte packet rating tells you how well the
>device can keep up to the maximum load.

Why would small packets stress routers more than large packets?
You're not the first to propose this theory but I just don't see it.
I've been assuming that people say this because small packets are so
inefficient from a bandwidth standpoint that you tend to run into a
pps/forwarding limit before you hit a bandwidth limit. So the fact
that (in general) less aggregate throughput is possible with smaller
packet sizes creates the impression that such packets stress equipment
more. But this isn't the case -- you just have to route more of them
to acheive a given throughput level, so as you decrease packet size
you'll hit a pps limit (before you hit a bandwidth limit) eventually.

>Historically there have numerous routers that could handle the
>maximum possible throughput (by using the longest possible packets),
>but could not keep up when shorter packets were used.

Again, this is because as packet size decreases, more of them need to
be forwarded to sustain maximum interface bandwidth. It makes perfect
sense that with small enough packets you'll eventually max out the
forwarding engine before you can saturate an interface's bandwidth.
That this is the case doesn't mean smaller packets stress routers more
-- it's just an issue of efficiency.

One of the examples I gave a couple weeks ago when this came up is a
2621, rated at 25k pps. With 64 byte packets this is around 12Mbps.
A throughput test that I did with TTCP between two FE interfaces with
full-sized (1500 byte) packets maxed out at 40Mbps, at which point the
router's CPU was at 100% utilization. This is around 3.5k pps and
not even half of interface bandwidth capacity.

So, forwarding performance (pps) was substantially lower with
full-sized packets even though interface capacity wasn't reached,
which contradicts the notion that smaller packets stress equipment
more. In fact, on a per-packet basis it is larger packets that tend
to be more stressful due to internal bandwidth/buffering limitations
and so forth (i.e., the router has to move more bytes around for each
packet forwarded). Even ignoring cases of interface saturation, pps
tends to decrease as packet size increases based on a handful of
throughput tests that either I myself have done or that I've seen
reported by others.

-Terry

P.S. When a bunch of Marketing people sit down in a room and try to
figure out how to attach a performance metric to the routers that they
sell, they're not thinking "How can we best stress our equipment to
give the most complete & accurate picture of its performance under
maximum load?". What they're thinking is "How can we make these
performance numbers as high as possible?" Minimum-sized packets are
the answer when it comes to packet forwarding rates.

In article <>,
Terry Baranski <0VE> wrote:
:Why would small packets stress routers more than large packets?
:You're not the first to propose this theory but I just don't see it.
:I've been assuming that people say this because small packets are so
:inefficient from a bandwidth standpoint that you tend to run into a
ps/forwarding limit before you hit a bandwidth limit.

It depends in part on the router architecture. There are usually
per-port packet buffers, and logic at that level to do basic CRC and
other sanity checks. After that, a routing decision has to be made.

Cisco architectures that use line cards and supervisors try to avoid
sending header information around the bus -- they traditionally started
by sending only the source and destination IP addresses onward and only
transferred the port information if it turned out to be needed (i.e.,
for an acl.) As this cross-bus is shared by all the ports, time on it
is expensive -- when you are sending data about one packet over it, you
can't be sending data about a different packet on a different port.

Anyhow, once a forwarding decision is made for a port, a dma engine
kicks in to copy the rewritten packet to the appropriate destination
port buffers -- and until that dma engine finishes, you aren't
going to be getting a service request from that port. Thus, the
longer the packet, the longer the "breather" that the supervisory
logic gets from paying attention the port, during which the
supervisory logic can do other work. Conversely, when packets are
short, that port is going to end up wanting to interrupt "soon".
The shorter the packets in use, the less time the supervisory logic
has to make the routing decision before the next packet is ready.


:It makes perfect
:sense that with small enough packets you'll eventually max out the
:forwarding engine before you can saturate an interface's bandwidth.
:That this is the case doesn't mean smaller packets stress routers more
:-- it's just an issue of efficiency.

What do you mean by 'stress' then? Less time to react implies requires
faster busses and faster processing to maintain line rates. Faster
busses and faster processing usually implies greater heat and
electrical requirements, which physically stresses the electronics.
Certainly from time to time new substrates and transitor types are
found that are inherently faster and less energy hungry, but a lot
of electronics speed up comes from reducing feature sizes and pathways --
in effect packing more energy into a smaller area.

Have you ever looked at the energy consumption of an Itanium^2 chip?
130 Watts! And if you could run your processor 20 times slower because
you are using 1380 byte packets instead of 64 byte packets, would that
not stress the router less?
--
Reviewers should be required to produce a certain number of
negative reviews - like police given quotas for handing out
speeding tickets. -- The Audio Anarchist

"Terry Baranski" <0VE> wrote in message
news:...
> On 24 Jan 2004 02:19:07 GMT, (Walter
> Roberson) wrote:
>
> >In article <>,
> >Terry Baranski <0VE> wrote:
> >:1) Cisco's pps ratings reflect performance with 64 byte packets, the
> >:smallest Ethernet packets possible. This is because, in general
> >:smaller packets = higher pps. Such pps ratings don't tell you much
> >:about maximum throughput potential, since pps usually decreases as
> >acket size increases.
> >
> >I think you might have cause and effect reversed. Cisco uses 64 byte
> >packets in their ratings because 64 byte packets are the hardest on
> >the equipment, so the 64-byte packet rating tells you how well the
> >device can keep up to the maximum load.
>
> Why would small packets stress routers more than large packets?

this is a characteristic of software driven routers - not just cisco.
Because this is the way router have behaved for many years, tests are based
on small packets to give the highest potential load.

the way these things work is that a lot of the load on the processing is per
packet header - deciding where to send it and so on.

there is some software load dependent on packet size, such as checksum
tests, but it is a 2nd order effect. Also, many routers are designed to use
DMA and other hardware acceleration to reduce the data scan and move "costs"

in fact the same things seem to happen on hardware driven layer 2 or layer 3
switches - many switches are "wire speed" when passing parge packets, but
cant keep up as the packet size reduces, and the numbers of packets
increase.

> You're not the first to propose this theory but I just don't see it.
> I've been assuming that people say this because small packets are so
> inefficient from a bandwidth standpoint that you tend to run into a
> pps/forwarding limit before you hit a bandwidth limit. So the fact
> that (in general) less aggregate throughput is possible with smaller
> packet sizes creates the impression that such packets stress equipment
> more. But this isn't the case -- you just have to route more of them
> to acheive a given throughput level, so as you decrease packet size
> you'll hit a pps limit (before you hit a bandwidth limit) eventually.

Not true - throughput is generally the bps on the wire, not application
level - so all those headers still get counted.
>
> >Historically there have numerous routers that could handle the
> >maximum possible throughput (by using the longest possible packets),
> >but could not keep up when shorter packets were used.
>
> Again, this is because as packet size decreases, more of them need to
> be forwarded to sustain maximum interface bandwidth. It makes perfect
> sense that with small enough packets you'll eventually max out the
> forwarding engine before you can saturate an interface's bandwidth.
> That this is the case doesn't mean smaller packets stress routers more
> -- it's just an issue of efficiency.

yes - but isnt that what performance numbers are designed to measure - what
happens under conditions of max stress?
>
> One of the examples I gave a couple weeks ago when this came up is a
> 2621, rated at 25k pps. With 64 byte packets this is around 12Mbps.
> A throughput test that I did with TTCP between two FE interfaces with
> full-sized (1500 byte) packets maxed out at 40Mbps, at which point the
> router's CPU was at 100% utilization. This is around 3.5k pps and
> not even half of interface bandwidth capacity.
>
> So, forwarding performance (pps) was substantially lower with
> full-sized packets even though interface capacity wasn't reached,
> which contradicts the notion that smaller packets stress equipment
> more. In fact, on a per-packet basis it is larger packets that tend
> to be more stressful due to internal bandwidth/buffering limitations
> and so forth (i.e., the router has to move more bytes around for each
> packet forwarded). Even ignoring cases of interface saturation, pps
> tends to decrease as packet size increases based on a handful of
> throughput tests that either I myself have done or that I've seen
> reported by others.
>
> -Terry
>
> P.S. When a bunch of Marketing people sit down in a room and try to
> figure out how to attach a performance metric to the routers that they
> sell, they're not thinking "How can we best stress our equipment to
> give the most complete & accurate picture of its performance under
> maximum load?". What they're thinking is "How can we make these
> performance numbers as high as possible?" Minimum-sized packets are
> the answer when it comes to packet forwarding rates.

traditionally hardware manufacturers always try to avoid giving numbers
unless they think they have an "edge".

but 64 byte packets are more or less the standard to use - there is even an
RFC (1944) that defines what and how testing should happen - amoung others
things that specifies max and min frame sizes for the media.
http://www.ietf.org/rfc/rfc1944.txt?number=1944
--
Regards

Stephen Hope - remove xx from email to reply

On Sat, 24 Jan 2004 17:02:34 -0000, "shope"
<> wrote:

>"Terry Baranski" <0VE> wrote in message
>news:.. .
>>
>> Why would small packets stress routers more than large packets?
>
>this is a characteristic of software driven routers - not just cisco.
>Because this is the way router have behaved for many years, tests are based
>on small packets to give the highest potential load.
>
>the way these things work is that a lot of the load on the processing is per
>packet header - deciding where to send it and so on.
>
>there is some software load dependent on packet size, such as checksum
>tests, but it is a 2nd order effect. Also, many routers are designed to use
>DMA and other hardware acceleration to reduce the data scan and move "costs"
>
>in fact the same things seem to happen on hardware driven layer 2 or layer 3
>switches - many switches are "wire speed" when passing parge packets, but
>cant keep up as the packet size reduces, and the numbers of packets
>increase.

I think we just have a semantics issue here. I've already said a few
times (you even quoted me saying it below) that smaller packets are
more likely to cause a given device to hit a forwarding limit before
hitting wire speed for a given interface. However, to me this doesn't
make it accurate to simply say that smaller packets stress routers
more, and this is what I'm taking issue with. This makes it sound
like you're talking about stress on a per-packet basis; i.e., one 64
byte packet requires more effort to route than one 1500 byte packet.
This clearly isn't the case, so I think the statement is misleading.

>> You're not the first to propose this theory but I just don't see it.
>> I've been assuming that people say this because small packets are so
>> inefficient from a bandwidth standpoint that you tend to run into a
>> pps/forwarding limit before you hit a bandwidth limit. So the fact
>> that (in general) less aggregate throughput is possible with smaller
>> packet sizes creates the impression that such packets stress equipment
>> more. But this isn't the case -- you just have to route more of them
>> to acheive a given throughput level, so as you decrease packet size
>> you'll hit a pps limit (before you hit a bandwidth limit) eventually.
>
>Not true - throughput is generally the bps on the wire, not application
>level - so all those headers still get counted.

No -- Cisco's 64 byte pps ratings are nowhere close to wire speed for
modern interfaces (e.g., FE), whether you include header bits in the
calculation or not. Two quick examples: 1) 2621, 25k pps rating X 64
bytes = 12.2Mbps; 2) 3640, 50-70k pps rating X 64 bytes = 24.4 -
34.1Mbps. The lowest-end router that can saturate an FE interface
(unidirectional) with 64 byte packets per Cisco's ratings is the 3745,
rated at 225k pps X 64 bytes = 109Mbps.

So again, aggregate throughput (including headers) tends to drop as
packet size decreases because you eventually hit a forwarding engine
limit (packets forwarded per second) which is completely unrelated to
the maximum potential bandwidth of whatever interface you happen to be
trying to saturate.

>> Again, this is because as packet size decreases, more of them need to
>> be forwarded to sustain maximum interface bandwidth. It makes perfect
>> sense that with small enough packets you'll eventually max out the
>> forwarding engine before you can saturate an interface's bandwidth.
>> That this is the case doesn't mean smaller packets stress routers more
>> -- it's just an issue of efficiency.
>
>yes - but isnt that what performance numbers are designed to measure - what
>happens under conditions of max stress?

I don't have an issue with using 64 byte packets for pps ratings. I
think it makes perfect sense to do so because you're less likely to
hit some type of bandwidth limit (which isn't what you're trying to
achieve in a pps test). What's at issue here was what I think was a
misleading statement that smaller packets stress routers more. Since
we all seem to be in agreement that you need to send more of them
relative to larger packets to acheive a given "level of stress" (a
high-level & generic term, but it works for our purposes), the way I
would put it is something along the lines of -

'Smaller packets allow one to better test purely the packet forwarding
performance (i.e., route lookups) potential of a given network device,
as they reduce the chances of other performance limitations (such as
interface bandwidth, internal bus bandwidth, or internal buffering)
coming into play.'

-Terry

See my response to Stephen -- I think we're essentially on the same
page here, it's just semantics causing confusion. I'm talking about
"stress" on a per-packet basis since, unless specified otherwise,
that's the way I interpret the term when it's used in statements such
as "these types of packets stress routers more than these other types
of packets." So with this in mind I don't think we're in major
disagreement on anything -- we're just using terms differently.

-Terry

On 24 Jan 2004 04:14:36 GMT, (Walter
Roberson) wrote:

>In article <>,
>Terry Baranski <0VE> wrote:
>:Why would small packets stress routers more than large packets?
>:You're not the first to propose this theory but I just don't see it.
>:I've been assuming that people say this because small packets are so
>:inefficient from a bandwidth standpoint that you tend to run into a
>ps/forwarding limit before you hit a bandwidth limit.
>
>It depends in part on the router architecture. There are usually
>per-port packet buffers, and logic at that level to do basic CRC and
>other sanity checks. After that, a routing decision has to be made.
>
>Cisco architectures that use line cards and supervisors try to avoid
>sending header information around the bus -- they traditionally started
>by sending only the source and destination IP addresses onward and only
>transferred the port information if it turned out to be needed (i.e.,
>for an acl.) As this cross-bus is shared by all the ports, time on it
>is expensive -- when you are sending data about one packet over it, you
>can't be sending data about a different packet on a different port.
>
>Anyhow, once a forwarding decision is made for a port, a dma engine
>kicks in to copy the rewritten packet to the appropriate destination
>port buffers -- and until that dma engine finishes, you aren't
>going to be getting a service request from that port. Thus, the
>longer the packet, the longer the "breather" that the supervisory
>logic gets from paying attention the port, during which the
>supervisory logic can do other work. Conversely, when packets are
>short, that port is going to end up wanting to interrupt "soon".
>The shorter the packets in use, the less time the supervisory logic
>has to make the routing decision before the next packet is ready.
>
>
>:It makes perfect
>:sense that with small enough packets you'll eventually max out the
>:forwarding engine before you can saturate an interface's bandwidth.
>:That this is the case doesn't mean smaller packets stress routers more
>:-- it's just an issue of efficiency.
>
>What do you mean by 'stress' then? Less time to react implies requires
>faster busses and faster processing to maintain line rates. Faster
>busses and faster processing usually implies greater heat and
>electrical requirements, which physically stresses the electronics.
>Certainly from time to time new substrates and transitor types are
>found that are inherently faster and less energy hungry, but a lot
>of electronics speed up comes from reducing feature sizes and pathways --
>in effect packing more energy into a smaller area.
>
>Have you ever looked at the energy consumption of an Itanium^2 chip?
>130 Watts! And if you could run your processor 20 times slower because
>you are using 1380 byte packets instead of 64 byte packets, would that
>not stress the router less?

In article <>,
Terry Baranski <0VE> wrote:
:See my response to Stephen -- I think we're essentially on the same
age here, it's just semantics causing confusion. I'm talking about
:"stress" on a per-packet basis since, unless specified otherwise,
:that's the way I interpret the term when it's used in statements such
:as "these types of packets stress routers more than these other types
f packets." So with this in mind I don't think we're in major
:disagreement on anything -- we're just using terms differently.

But in order to achieve wire-rate, the router has to process
the short packet in a short time -- so the *per-packet* processing
is stressed by short packets (especially in non-distributed
architectures.)
--
"Meme" is self-referential; memes exist if and only if the "meme" meme
exists. "Meme" is thus logically a meta-meme; but until the existance
of meta-memes is more widely recognized, "meta-meme" is not a meme.
-- A Child's Garden Of Memes

wrote:
> I am debating whether to use a Cisco 3725 router or Cisco 3745 router for
> this scenario. I will have a building with about 4 VLAN's and approx. 900
> connections - users, printers, wireless. The switches are L2 and gigabit
> ethernet connected to each other. The Cisco 3725/3745 will be providing L3
> via two Fast Ethernet cards in a trunk. The routers will also have an ATM
> OC-3 card in it to route back to the Data Center site. At highest
> utilization, we have seen the OC-3 link from the building to the Data Center
> at 40Mb/s. The router will have a default of 128MB of RAM. Will a 3725
> router suffice or would it be better to go with a 3745. Cisco says that the
> 3725 has a max. process switch capacity of 125,000 pps whereas the 3745 has
> 250,000 pps when using CEF switching.
>
> Thanks
>
>

At 40Mbps it will not matter. At 100Mbps it might. Although another
poster indicated he got 200Mbps from a 3725, our experience with OC3
blades in these routers indicates that the OC3's use more processor than
FE interfaces. Not sure why. We found FE->OC3 performance less than FE->FE.

--Mike