Benchmarking C compilers for embedded systems
 

I'd like to compare C compilers targetting embedded systems - the kind
that only have a few KB of RAM, etc.

However there seems to be a lack of software I could use. Waht I'm
looking for should be free, written (mostly) in standard C, be not
totally different from typical embedded applications.

So far I see:
- dhrystone
- The Contiki OS
- the files I've been using to track sdcc over revisions at
https://sourceforge.net/apps/trac/sd...%20TODO%20list

Are there any other suitable benchmarks I could look at? They don't have
to be designed as benchmarks, other software, like the Contiki OS
mentioned above is OK, too. THey don't even have to "run", so libraries
would be OK, too.

Philipp

Re: Benchmarking C compilers for embedded systems
 

On Sep 19, 10:43*pm, Philipp Klaus Krause <p...@spth.de> wrote:
> I'd like to compare C compilers targetting embedded systems - the kind
> that only have a few KB of RAM, etc.
>
> However there seems to be a lack of software I could use. Waht I'm
> looking for should be free, written (mostly) in standard C, be not
> totally different from typical embedded applications.
>
> So far I see:
> - dhrystone
> - The Contiki OS
> - the files I've been using to track sdcc over revisions athttps://sourceforge.net/apps/trac/sdcc/wiki/Philipp%27s%20TODO%20list
>
> Are there any other suitable benchmarks I could look at? They don't have
> to be designed as benchmarks, other software, like the Contiki OS
> mentioned above is OK, too. THey don't even have to "run", so libraries
> would be OK, too.
>
> Philipp

Hi Philipp,

Have you tried Coremark from EEMBC (www.coremark.org)?
It is available for free.
The other benchmarking suites from EEMBC require you to pay small
membership fee i guess.

Regards,
Abhishek

Re: Benchmarking C compilers for embedded systems
 

Am 20.09.2011 09:50, schrieb Abhishek:

>
> Have you tried Coremark from EEMBC (www.coremark.org)?
> It is available for free.
> The other benchmarking suites from EEMBC require you to pay small
> membership fee i guess.

I now did, however the license is a problem:
1) The license. I can understand when benchmarking organizations don't
distribute older versions of their benchmarks and no longer accept
reults generated using an old version, like SPEC does. However the
Coremark license explicitly forbids you from just running or even
keeping a copy of older versions: You loose all rights to them each time
a new version is released.

I'll probably use it anyway and just hope they don't release a new
version anytime soon. Still, I'm looking for further benchmarks. The
Contiki OS seems perfect, but I'd like to find more of that kind.

Philipp

Re: Benchmarking C compilers for embedded systems
 

On 09/21/11 07:40 PM, Philipp Klaus Krause wrote:
> Am 20.09.2011 09:50, schrieb Abhishek:
>
>>
>> Have you tried Coremark from EEMBC (www.coremark.org)?
>> It is available for free.
>> The other benchmarking suites from EEMBC require you to pay small
>> membership fee i guess.
>
> I now did, however the license is a problem:
> 1) The license. I can understand when benchmarking organizations don't
> distribute older versions of their benchmarks and no longer accept
> reults generated using an old version, like SPEC does. However the
> Coremark license explicitly forbids you from just running or even
> keeping a copy of older versions: You loose all rights to them each time
> a new version is released.
>
> I'll probably use it anyway and just hope they don't release a new
> version anytime soon. Still, I'm looking for further benchmarks. The
> Contiki OS seems perfect, but I'd like to find more of that kind.

What about your own code?

--
Ian Collins

Re: Benchmarking C compilers for embedded systems
 

Am 21.09.2011 11:06, schrieb Ian Collins:
> On 09/21/11 07:40 PM, Philipp Klaus Krause wrote:
>> Am 20.09.2011 09:50, schrieb Abhishek:
>>
>>>
>>> Have you tried Coremark from EEMBC (www.coremark.org)?
>>> It is available for free.
>>> The other benchmarking suites from EEMBC require you to pay small
>>> membership fee i guess.
>>
>> I now did, however the license is a problem:
>> 1) The license. I can understand when benchmarking organizations don't
>> distribute older versions of their benchmarks and no longer accept
>> reults generated using an old version, like SPEC does. However the
>> Coremark license explicitly forbids you from just running or even
>> keeping a copy of older versions: You loose all rights to them each time
>> a new version is released.
>>
>> I'll probably use it anyway and just hope they don't release a new
>> version anytime soon. Still, I'm looking for further benchmarks. The
>> Contiki OS seems perfect, but I'd like to find more of that kind.
>
> What about your own code?
>

I'd like to get results that are useful to other people as well. I
intend to use some of my code as one of the benchmarks benchmark, but in
general I'd prefer to use more generally accepted stuff.

Philipp

Re: Benchmarking C compilers for embedded systems
 

Am 19.09.2011 22:10, schrieb David Brown:
> On 19/09/11 19:43, Philipp Klaus Krause wrote:
>> I'd like to compare C compilers targetting embedded systems - the kind
>> that only have a few KB of RAM, etc.
>>
>> However there seems to be a lack of software I could use. Waht I'm
>> looking for should be free, written (mostly) in standard C, be not
>> totally different from typical embedded applications.
>>
>> So far I see:
>> - dhrystone
>> - The Contiki OS
>> - the files I've been using to track sdcc over revisions at
>> https://sourceforge.net/apps/trac/sd...%20TODO%20list
>>
>> Are there any other suitable benchmarks I could look at? They don't have
>> to be designed as benchmarks, other software, like the Contiki OS
>> mentioned above is OK, too. THey don't even have to "run", so libraries
>> would be OK, too.
>>
>> Philipp
>
> There are no good general benchmarks for small embedded systems - partly
> because it would be a great deal of work (standard C is not standard for
> many compilers - you need to use compiler-specific features to get the
> best out of them), and partly because it would be of very little use.
>
> What you want to know is which tools are the best for /your/ code, not
> some general code written for completely different uses. Most tool
> vendors provide free evaluation software - test them out on your own code.
>

I don't just want to know which tool is best for my code. I want to
know, which tool is best for typical code.
I am one of the developers of the free sdcc compiler and currently
working on register and stack allcoation. I see that what I implemented
results in better code when compiling my own programs, Contiki,
dhrystone, Coremark. But I'd like to know if there is any code for which
the new approach performs worse than the on epreviously used in sdcc.
And I'd like to know it know, instead of having to wait for bug reports
about code size regressions coming in after the next release.

Philipp

Re: Benchmarking C compilers for embedded systems
 

On 09/22/11 07:12 PM, Philipp Klaus Krause wrote:
> Am 21.09.2011 11:06, schrieb Ian Collins:
>> On 09/21/11 07:40 PM, Philipp Klaus Krause wrote:
>>> Am 20.09.2011 09:50, schrieb Abhishek:
>>>
>>>>
>>>> Have you tried Coremark from EEMBC (www.coremark.org)?
>>>> It is available for free.
>>>> The other benchmarking suites from EEMBC require you to pay small
>>>> membership fee i guess.
>>>
>>> I now did, however the license is a problem:
>>> 1) The license. I can understand when benchmarking organizations don't
>>> distribute older versions of their benchmarks and no longer accept
>>> reults generated using an old version, like SPEC does. However the
>>> Coremark license explicitly forbids you from just running or even
>>> keeping a copy of older versions: You loose all rights to them each time
>>> a new version is released.
>>>
>>> I'll probably use it anyway and just hope they don't release a new
>>> version anytime soon. Still, I'm looking for further benchmarks. The
>>> Contiki OS seems perfect, but I'd like to find more of that kind.
>>
>> What about your own code?
>>
>
> I'd like to get results that are useful to other people as well. I
> intend to use some of my code as one of the benchmarks benchmark, but in
> general I'd prefer to use more generally accepted stuff.

Every target and code base combination will behave differently, even on
different members of the same processor family. This makes general
benchmarks useless for embedded systems.

--
Ian Collins

Re: Benchmarking C compilers for embedded systems
 

On 09/19/2011 07:43 PM, Philipp Klaus Krause wrote:
> I'd like to compare C compilers targetting embedded systems - the kind
> that only have a few KB of RAM, etc.
>
> However there seems to be a lack of software I could use. Waht I'm
> looking for should be free, written (mostly) in standard C, be not
> totally different from typical embedded applications.
>
> So far I see:
> - dhrystone
> - The Contiki OS
> - the files I've been using to track sdcc over revisions at
> https://sourceforge.net/apps/trac/sd...%20TODO%20list
>
> Are there any other suitable benchmarks I could look at? They don't have
> to be designed as benchmarks, other software, like the Contiki OS
> mentioned above is OK, too. THey don't even have to "run", so libraries
> would be OK, too.

Elm's FAT filesystem implementation:

http://elm-chan.org/fsw/ff/00index_e.html

Re: Benchmarking C compilers for embedded systems
 

Philipp Klaus Krause wrote:

> I'd like to compare C compilers targetting embedded systems - the kind
> that only have a few KB of RAM, etc.
>
> However there seems to be a lack of software I could use. Waht I'm
> looking for should be free, written (mostly) in standard C, be not
> totally different from typical embedded applications.
>
> So far I see:
> - dhrystone
> - The Contiki OS
> - the files I've been using to track sdcc over revisions at
> https://sourceforge.net/apps/trac/sd...%20TODO%20list
>
> Are there any other suitable benchmarks I could look at? They don't have
> to be designed as benchmarks, other software, like the Contiki OS
> mentioned above is OK, too. THey don't even have to "run", so libraries
> would be OK, too.
>

Philipp,

There are tons of embedded systems benchmarks on the internet. In our
compiler work we have found the most informative benchmarks are application
type benchmarks close to the actual intended application.

The application fragment type benchmarks (fft, filters, math packs) are
useful generally is seeing how focused a compiler is in optimizing a
specific application area.

Generic benchmarks would likely need to also have some definition of the
intended application area.

There isn't much meaning in comparing the performance of processors used in
mice and processors used in home appliance controllers. Both are non hosted
embedded controllers with a few K of code but very different requirements.
They may even be members of the same family.

Walter..