<?xml version="1.0"?>
<document>

    <properties>
        <author email="burton at apache.org">Kevin A. Burton</author>
        <title>Overview - Commons Benchmark</title>
    </properties>

    <body>

        <section name="Goals">
            
            <p>
                Java (and UNIX) has for a long time lacked the equivalent (for
                the most part) of Windows perfmon.  While perfmon was by no
                means perfect the goals were admirable.  Developers could export
                simple statistics from their application and perfmon could
                introspect the values and graph them.
            </p>

            <p>
                With commons benchmark we provide similar functionality by
                allowing developers to export statistics and then tool
                developers can write frontends to monitor application
                performance.
            </p>

            <p>
                Of course one approach is to use a Java profiler (and many
                exist) but profilers suffer a number of fatal problems:
            </p>

            <dl>
                
                <dt>Slow</dt>

                <dd>
                    Most java profilers are slow due to profiling of ALL method
                    calls or allocation points.  This is a fatal flaw of their
                    design not their implementation.  Since they don't know what
                    they're looking for they essentially have to look for
                    everything..

                </dd>

                <dt>Require a LOT of memory</dt>

                <dt>Crash</dt>

                <dt>Not always enabled</dt>

                <dd>
                    Benchmarks are designed to run in <q>always on</q> mode
                    where collection of performance data has a negligible impact
                    on the VM.  Most profilers on the other hand are not
                    designed to do this and often cause significant resource
                    restraints.
                </dd>
                
                <dt>Proprietary</dt>

            </dl>

            <p>
                Infrasture code such as rrdtool, ganglia, and jrobin, provide
                the foundation to view the data and commons benchmarks provides
                a way to get at the data.
            </p>

            <p>
                Therefore we provide two core components for runtime performance
                analysis of Java applications:

                <ol>

                    <li>A simple benchmark API for Java</li>

                    <li>An XMLRPC export mechanism</li>
                    
                    <li>A graphing package based on JRobin</li>

                </ol>

            </p>

        </section>

        <section name="Benchmark">

            <p>
                Benchmark that allows cheap and lightweight "benchmarking" (go
                figure) of arbitrary code.  All you have to do is call start()
                every time a method starts which will then increment the
                benchmark and perform any operations necessary to maintain the
                benchmark.  Just call complete() when your method is done.
            </p>
  
            <p>
                This class is lightweight and only requires 500 bytes per
                benchmark of memory with no external storage requirements.  This
                class is also threadsafe so if you need to call this from
                multithreaded code to benchmark then you'll be ok.
            </p>
 
            <p>
                The benchmark is maintained as number of starts and completes
                per minute.  This can be any type of operation you want.
                Technically the interval can be longer than a minute but we will
                end up with stale data.  That's the tradeoff with this type of
                benchmark.  Its cheap and easy to maintain but anything more
                than 60 seconds worth of data and you'll end up with a stale
                benchmark.
            </p>
 
            <p>
                Internally we use an incremented value which is accumulated and
                reset ever 60 seconds.  When we reset the benchmark we reset the
                current value so that we can start accumulating again.
            </p>
            
            <source>
                
                //get a benchmark
                Benchmark benchmark = Benchmark.getBenchmark( "my_benchmark_name" );
                
                try {
                
                    benchmark.start();
                
                    //do something expensive
                
                } finally {
                    benchmark.complete();
                }
                
            </source>
            
            <p>
                The method overhead is very light. One a modern machine you can
                perform about 1M benchmarks per second.  For code thats only
                called a few thousand times you won't notice any performance
                overhead.
            </p>

            <p>
                We provide shortcuts for determining the benchmark name from the
                caller.  For example if youre class
                is <code>mypackage.MyClass</code> and you call
                <code>Benchmark.getBenchmark()</code> or simply <code>new
                Benchmark()</code> we automatically determine the class name and
                use this as the name of the benchmark.
            </p>

        </section>

        <section name="Proxy Benchmarks">

            <p>
                We also support <q>proxy</q> benchmarks which can benchmark
                every method of a given class.  The only requirements are that
                the class be backed by an interface.
            </p>

            <p>
                Here we're going to benchmark java.util.LinkedList:
            </p>

            <source>

                List list = new LinkedList();
                list = (List)Benchmark.getBenchmarkAsProxy( list, List.class );
                
                list.add( "hello" );
                list.add( "world" );
                list.iterator();
                
                Benchmark benchmark = Benchmark.getBenchmark( "java.util.List.iterator" );

                System.out.println( benchmark.getName() + ": " + benchmark );

            </source>

        </section>

        <section name="Graphing">
            
            <p>
                We provide integration automated graph output for benchmark
                stats.  Initially this is provided via integration with JRobin
                which used Round Robin Database (RRD) files to maintain
                historical data.
            </p>

            <p>
                You can define your datasources with a benchmark.xml
                configuration file and a daemon will periodically (usually once
                per minute) connect to a running VM via XMLRPC (or in-process)
                and log a benchmark stat to an RRD file.
            </p>

            <source>

                ----------------------------------------------------
                | Target VM using benchmark.start/complete() calls |
                ----------------------------------------------------
                                        ^
                                        |
                
                         --------------------------------
                         | Source VM running RRD logger |
                         --------------------------------
                                        |
                                        V
                
                         --------------------------------
                         |  WWW server exporting graphs |
                         --------------------------------

            </source>

            <p>
                The XMLRPC daemon can also be integrated with
                <a href="http://ganglia.info">ganglia</a> with the ganglia
                gmetric script.  Slight configuration is required in order to
                specify the multicast channel, port, cluster name, ec.
            </p>

        </section>

        <section name="Requirements">

            <p>
                The benchmark API is designed to be very lightweight but
                depending on the role deployed can require more than a few
                components
            </p>

            <p>
                The following modes are supported:
            </p>

            <dl>
                <dt>base</dt>

                <dd>
                    All benchmarks enabled.  Fast with very little memory
                    overhead.  No remote operation.
                </dd>

                <dt>
                    xmlrpc
                </dt>

                <dd>
                    All benchmarks enabled/disabled, configuration support, and
                    remote benchmark probing.
                </dd>

                <dt>
                    ganglia
                </dt>

                <dd>
                    Benchmarks are broadcast on the local submnet via unicast.
                </dd>

            </dl>

            <table>

                <tr>
                    <th></th>
                    <th>commons-beanutils</th>
                    <th>xmlrpc</th>
                </tr>

                <tr>
                    <td>
                        base
                    </td>

                    <td>
                        no
                    </td>

                    <td>
                        no
                    </td>
                </tr>

                <tr>
                    <td>
                        xmlrpc
                    </td>

                    <td>
                        no
                    </td>

                    <td>
                        yes
                    </td>
                </tr>

            </table>

        </section>

    </body>

</document>
