30°C Power On 12 to 36 watts - for fanless sealed system design ...

ESE Magazine Jan/Feb 06
20
Benchmarks: more
than a marketing tool
Markus Levy, EEMBC
The latest of the regular columns from EEMBC explains the thinking
behind the wider licensing of processor benchmarks
CERTIFIED EEMBC scores for many
processors have always been available for
free on the EEMBC Web site, but since
the consortium’s founding only members
had access to the benchmark source code
itself and the ability to run the benchmarks.
Under a new program, OEMs, design consultants,
and other qualified users can now license
any or all of EEMBC’s benchmark suites. While
the consortium will continue to allow the publication
only of scores that have been verified by
its certification lab, the new licensing program is
set to make EEMBC benchmark software more
widely available as a relevant, focused, objective
tool for engineers choosing between
processors for their applications.
Typically, benchmarks are viewed as a marketing
tool to allow vendors to compete on performance.
However, the EEMBC benchmark software
is also regularly used as a tool to analyze,
tune, and validate processor architectures and
the systems that encompass these processors.
Although it will always be true that the best
benchmark is the system designer’s application
itself, the beauty of using the EEMBC software
is that it’s easy to port to different architectures
and platforms and user-obtained scores can be
matched to the large database of scores on the
EEMBC website. Furthermore, because it is such
a widely-used code base, system designers can
get direct support from processor vendors
(because the majority of them are EEMBC members
already familiar with the code).
University
The licensing model has quite a successful
precedent in EEMBC U, under which faculty
members at universities and college are entitled
to license the benchmark source code for teaching
and experimental purposes. EEMBC U continues
to grow apace thanks to recent public
exposure at several industry and academic venues,
and its members now include faculty at
some of the most highly regarded universities in
the United States, Europe, and Asia. The SPEC
consortium provides another example of how
membership and licensing can successfully
coexist within a single organization.
The licensing program is also a result of
what we’ve learned, after nearly 10 years of
existence, about measuring EEMBC’s progress
as a standards-setting enterprise. At various
times, we’ve tended to gauge success according
to the number of our member companies (which
has averaged about 50 over the past five years),
the number of certified processor benchmark
scores being published, and the rate at which
we bring new benchmark suites to completion.
But none of these is as significant as another
metric which is somewhat more difficult to
know: the demand by customers of processor
vendors for disclosure of EEMBC benchmark
scores as a condition of considering any new
chip for one of their system designs. The anecdotal
evidence is that this demand has gone up
significantly in the past few years, and more
than one member company has told me frankly
that its interest in joining EEMBC was the direct
result of interest by its customers in seeing
EEMBC benchmark scores.
Benchmark suite
EEMBC’s application orientation to benchmarks
and its certification rules are two obvious
respects in which it differs from Dhrystone mips
and other synthetic benchmarks. Less apparent
but just as important is the EEMBC process of
benchmark development, in which members
companies both large and small have an equal
vote in the many decisions that ultimately lead to
a completed benchmark suite. The process is not
always fast, but it does lead to credible results.
This is one reason why EEMBC is now being
approached by OEMs who are not only interested
in the consortium’s existing suites but would
like us to develop new ones that address the par-
Figure 1: Markus Levy, president EEMBC.
Benchmark software is also regularly used as a tool to analyze, tune, and validate
processor architectures and the systems that encompass these processors
ticular needs of a given embedded application.
This is how EEMBC came recently to form a new
subcommittee that is working on network storage
benchmarks and being led by Adaptec.
Until now, most users of the EEMBC benchmarks
have been the processor vendors that
developed them, so part of our task in the coming
weeks and months will be making the benchmarking
process even more user-friendly for a
broader audience of embedded systems designers
and others for whom the processor is just
part of the bigger picture. We’ll be adding to the
information about the benchmarks on the
EEMBC Web site, improving benchmark documentation,
and providing more specific information
on the workloads that EEMBC benchmarks
address. We welcome contributions to this
effort from everyone in the embedded systems
community.
www.eembc.org