<?xml version="1.0" encoding="UTF-8"?>
<chapter version="5.0" xml:id="performance"
         xmlns="http://docbook.org/ns/docbook"
         xmlns:xlink="http://www.w3.org/1999/xlink"
         xmlns:xi="http://www.w3.org/2001/XInclude"
         xmlns:svg="http://www.w3.org/2000/svg"
         xmlns:m="http://www.w3.org/1998/Math/MathML"
         xmlns:html="http://www.w3.org/1999/xhtml"
         xmlns:db="http://docbook.org/ns/docbook">
<!--
/**
 * Licensed to the Apache Software Foundation (ASF) under one
 * or more contributor license agreements.  See the NOTICE file
 * distributed with this work for additional information
 * regarding copyright ownership.  The ASF licenses this file
 * to you under the Apache License, Version 2.0 (the
 * "License"); you may not use this file except in compliance
 * with the License.  You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
-->
  <title>Performance Tuning</title>

  <section xml:id="perf.os">
    <title>Operating System</title>
        <section xml:id="perf.os.ram">
          <title>Memory</title>
          <para>RAM, RAM, RAM.  Don't starve HBase.</para>
        </section>
        <section xml:id="perf.os.64">
          <title>64-bit</title>
          <para>Use a 64-bit platform (and 64-bit JVM).</para>
        </section>
        <section xml:id="perf.os.swap">
          <title>Swapping</title>
          <para>Watch out for swapping.  Set swappiness to 0.</para>
        </section>
  </section>
  <section xml:id="perf.network">
    <title>Network</title>
    <para>
    Perhaps the most important factor in avoiding network issues degrading Hadoop and HBbase performance is the switching hardware
    that is used, decisions made early in the scope of the project can cause major problems when you double or triple the size of your cluster (or more). 
    </para>
    <para>
    Important items to consider:
        <itemizedlist>
          <listitem>Switching capacity of the device</listitem>
          <listitem>Number of systems connected</listitem>
          <listitem>Uplink capacity</listitem>
        </itemizedlist>
    </para>
    <section xml:id="perf.network.1switch">
      <title>Single Switch</title>
      <para>The single most important factor in this configuration is that the switching capacity of the hardware is capable of 
      handling the traffic which can be generated by all systems connected to the switch. Some lower priced commodity hardware
      can have a slower switching capacity than could be utilized by a full switch. 
      </para>
    </section>
    <section xml:id="perf.network.2switch">
      <title>Multiple Switches</title>
      <para>Multiple switches are a potential pitfall in the architecture.   The most common configuration of lower priced hardware is a
      simple 1Gbps uplink from one switch to another. This often overlooked pinch point can easily become a bottleneck for cluster communication. 
      Especially with MapReduce jobs that are both reading and writing a lot of data the communication across this uplink could be saturated.
      </para>
      <para>Mitigation of this issue is fairly simple and can be accomplished in multiple ways:
      <itemizedlist>
        <listitem>Use appropriate hardware for the scale of the cluster which you're attempting to build.</listitem>
        <listitem>Use larger single switch configurations i.e. single 48 port as opposed to 2x 24 port</listitem>
        <listitem>Configure port trunking for uplinks to utilize multiple interfaces to increase cross switch bandwidth.</listitem>
      </itemizedlist>
      </para>
    </section>
    <section xml:id="perf.network.multirack">
      <title>Multiple Racks</title>
      <para>Multiple rack configurations carry the same potential issues as multiple switches, and can suffer performance degradation from two main areas:
         <itemizedlist>
           <listitem>Poor switch capacity performance</listitem>
           <listitem>Insufficient uplink to another rack</listitem>
         </itemizedlist>
      If the the switches in your rack have appropriate switching capacity to handle all the hosts at full speed, the next most likely issue will be caused by homing 
      more of your cluster across racks.  The easiest way to avoid issues when spanning multiple racks is to use port trunking to create a bonded uplink to other racks.
      The downside of this method however, is in the overhead of ports that could potentially be used. An example of this is, creating an 8Gbps port channel from rack
      A to rack B, using 8 of your 24 ports to communicate between racks gives you a poor ROI, using too few however can mean you're not getting the most out of your cluster. 
      </para>
      <para>Using 10Gbe links between racks will greatly increase performance, and assuming your switches support a 10Gbe uplink or allow for an expansion card will allow you to
      save your ports for machines as opposed to uplinks.
      </para>
      
    </section>
  </section>  <!-- network -->
  <section xml:id="jvm">
    <title>Java</title>

    <section xml:id="gc">
      <title>The Garbage Collector and HBase</title>

      <section xml:id="gcpause">
        <title>Long GC pauses</title>

        <para>In his presentation, <link
        xlink:href="http://www.slideshare.net/cloudera/hbase-hug-presentation">Avoiding
        Full GCs with MemStore-Local Allocation Buffers</link>, Todd Lipcon
        describes two cases of stop-the-world garbage collections common in
        HBase, especially during loading; CMS failure modes and old generation
        heap fragmentation brought. To address the first, start the CMS
        earlier than default by adding
        <code>-XX:CMSInitiatingOccupancyFraction</code> and setting it down
        from defaults. Start at 60 or 70 percent (The lower you bring down the
        threshold, the more GCing is done, the more CPU used). To address the
        second fragmentation issue, Todd added an experimental facility that
        must be explicitly enabled in HBase 0.90.x (Its defaulted to be on in
        0.92.x HBase). See <code>hbase.hregion.memstore.mslab.enabled</code>
        to true in your <classname>Configuration</classname>. See the cited
        slides for background and detail<footnote><para>The latest jvms do better
        regards fragmentation so make sure you are running a recent release.
        Read down in the message,
        <link xlink:href="http://osdir.com/ml/hotspot-gc-use/2011-11/msg00002.html">Identifying concurrent mode failures caused by fragmentation</link>.</para></footnote>.</para>
        <para>For more information about GC logs, see <xref linkend="trouble.log.gc" />.
        </para>
      </section>
    </section>
  </section>

  <section xml:id="perf.configurations">
    <title>HBase Configurations</title>

    <para>See <xref linkend="recommended_configurations" />.</para>

    <section xml:id="perf.number.of.regions">
      <title>Number of Regions</title>

      <para>The number of regions for an HBase table is driven by the <xref
              linkend="bigger.regions" />. Also, see the architecture
          section on <xref linkend="arch.regions.size" /></para>
       <para>A lower number of regions is preferred, generally in the range of 20 to low-hundreds
       per RegionServer.  Adjust the regionsize as appropriate to achieve this number. 
       </para>
       <para>For the 0.90.x codebase, the upper-bound of regionsize is about 4Gb.
       For 0.92.x codebase, due to the HFile v2 change much larger regionsizes can be supported (e.g., 20Gb).
       </para>
       <para>You may need to experiment with this setting based on your hardware configuration and application needs.
       </para>
    </section>

    <section xml:id="perf.compactions.and.splits">
      <title>Managing Compactions</title>

      <para>For larger systems, managing <link
      linkend="disable.splitting">compactions and splits</link> may be
      something you want to consider.</para>
    </section>

    <section xml:id="perf.handlers">
        <title><varname>hbase.regionserver.handler.count</varname></title>
        <para>See <xref linkend="hbase.regionserver.handler.count"/>. 
            This setting in essence sets how many requests are
            concurrently being processed inside the RegionServer at any
            one time.  If set too high, then throughput may suffer as
            the concurrent requests contend; if set too low, requests will
            be stuck waiting to get into the machine.  You can get a
            sense of whether you have too little or too many handlers by
            <xref linkend="rpc.logging" />
            on an individual RegionServer then tailing its logs (Queued requests
            consume memory).</para>
    </section>
    <section xml:id="perf.hfile.block.cache.size">
        <title><varname>hfile.block.cache.size</varname></title>
        <para>See <xref linkend="hfile.block.cache.size"/>. 
        A memory setting for the RegionServer process.
        </para>
    </section>    
    <section xml:id="perf.rs.memstore.upperlimit">
        <title><varname>hbase.regionserver.global.memstore.upperLimit</varname></title>
        <para>See <xref linkend="hbase.regionserver.global.memstore.upperLimit"/>.  
        This memory setting is often adjusted for the RegionServer process depending on needs.
        </para>
    </section>    
    <section xml:id="perf.rs.memstore.lowerlimit">
        <title><varname>hbase.regionserver.global.memstore.lowerLimit</varname></title>
        <para>See <xref linkend="hbase.regionserver.global.memstore.lowerLimit"/>.  
        This memory setting is often adjusted for the RegionServer process depending on needs.
        </para>
    </section>
    <section xml:id="perf.hstore.blockingstorefiles">
        <title><varname>hbase.hstore.blockingStoreFiles</varname></title>
        <para>See <xref linkend="hbase.hstore.blockingStoreFiles"/>.  
        If there is blocking in the RegionServer logs, increasing this can help.
        </para>
    </section>
    <section xml:id="perf.hregion.memstore.block.multiplier">
        <title><varname>hbase.hregion.memstore.block.multiplier</varname></title>
        <para>See <xref linkend="hbase.hregion.memstore.block.multiplier"/>.  
        If there is enough RAM, increasing this can help.  
        </para>
    </section>

  </section>

  <section xml:id="perf.schema">
      <title>Schema Design</title>
  
    <section xml:id="perf.number.of.cfs">
      <title>Number of Column Families</title>
      <para>See <xref linkend="number.of.cfs" />.</para>
    </section>
    <section xml:id="perf.schema.keys">
      <title>Key and Attribute Lengths</title>
      <para>See <xref linkend="keysize" />.</para>
    </section>
    <section xml:id="schema.regionsize"><title>Table RegionSize</title>
    <para>The regionsize can be set on a per-table basis via <code>setFileSize</code> on
    <link xlink:href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/HTableDescriptor.html">HTableDescriptor</link> in the 
    event where certain tables require different regionsizes than the configured default regionsize.
    </para>
    <para>See <xref linkend="perf.number.of.regions"/> for more information.
    </para>
    </section>
    <section xml:id="schema.bloom">
    <title>Bloom Filters</title>
    <para>Bloom Filters can be enabled per-ColumnFamily.
        Use <code>HColumnDescriptor.setBloomFilterType(NONE | ROW |
        ROWCOL)</code> to enable blooms per Column Family. Default =
        <varname>NONE</varname> for no bloom filters. If
        <varname>ROW</varname>, the hash of the row will be added to the bloom
        on each insert. If <varname>ROWCOL</varname>, the hash of the row +
        column family + column family qualifier will be added to the bloom on
        each key insert.</para>
    <para>See <link xlink:href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/HColumnDescriptor.html">HColumnDescriptor</link> and 
    <xref linkend="blooms"/> for more information.
    </para>
    </section>
    <section xml:id="schema.cf.blocksize"><title>ColumnFamily BlockSize</title>
    <para>The blocksize can be configured for each ColumnFamily in a table, and this defaults to 64k.  Larger cell values require larger blocksizes. 
    There is an inverse relationship between blocksize and the resulting StoreFile indexes (i.e., if the blocksize is doubled then the resulting
    indexes should be roughly halved).
    </para>
    <para>See <link xlink:href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/HColumnDescriptor.html">HColumnDescriptor</link> 
    and <xref linkend="store"/>for more information.
    </para>
    </section>
    <section xml:id="cf.in.memory">
    <title>In-Memory ColumnFamilies</title>
    <para>ColumnFamilies can optionally be defined as in-memory.  Data is still persisted to disk, just like any other ColumnFamily.  
    In-memory blocks have the highest priority in the <xref linkend="block.cache" />, but it is not a guarantee that the entire table
    will be in memory.
    </para>
    <para>See <link xlink:href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/HColumnDescriptor.html">HColumnDescriptor</link> for more information.
    </para>
    </section>
    <section xml:id="perf.compression">
      <title>Compression</title>
      <para>Production systems should use compression with their ColumnFamily definitions.  See <xref linkend="compression" /> for more information.
      </para>
    </section>
  </section>  <!--  perf schema -->
  
  <section xml:id="perf.writing">
    <title>Writing to HBase</title>

    <section xml:id="perf.batch.loading">
      <title>Batch Loading</title>
      <para>Use the bulk load tool if you can.  See
        <link xlink:href="http://hbase.apache.org/bulk-loads.html">Bulk Loads</link>.
        Otherwise, pay attention to the below.
      </para>
    </section>  <!-- batch loading -->

    <section xml:id="precreate.regions">
    <title>
    Table Creation: Pre-Creating Regions
    </title>
<para>
Tables in HBase are initially created with one region by default.  For bulk imports, this means that all clients will write to the same region until it is large enough to split and become distributed across the cluster.  A useful pattern to speed up the bulk import process is to pre-create empty regions.  Be somewhat conservative in this, because too-many regions can actually degrade performance.  An example of pre-creation using hex-keys is as follows (note:  this example may need to be tweaked to the individual applications keys):
</para>
<para>
<programlisting>public static boolean createTable(HBaseAdmin admin, HTableDescriptor table, byte[][] splits)
throws IOException {
  try {
    admin.createTable( table, splits );
    return true;
  } catch (TableExistsException e) {
    logger.info("table " + table.getNameAsString() + " already exists");
    // the table already exists...
    return false;  
  }
}

public static byte[][] getHexSplits(String startKey, String endKey, int numRegions) {
  byte[][] splits = new byte[numRegions-1][];
  BigInteger lowestKey = new BigInteger(startKey, 16);
  BigInteger highestKey = new BigInteger(endKey, 16);
  BigInteger range = highestKey.subtract(lowestKey);
  BigInteger regionIncrement = range.divide(BigInteger.valueOf(numRegions));
  lowestKey = lowestKey.add(regionIncrement);
  for(int i=0; i &lt; numRegions-1;i++) {
    BigInteger key = lowestKey.add(regionIncrement.multiply(BigInteger.valueOf(i)));
    byte[] b = String.format("%016x", key).getBytes();
    splits[i] = b;
  }
  return splits;
}</programlisting>
  </para>
  </section>
    <section xml:id="def.log.flush">
    <title>
    Table Creation: Deferred Log Flush
    </title>
<para>
The default behavior for Puts using the Write Ahead Log (WAL) is that <classname>HLog</classname> edits will be written immediately.  If deferred log flush is used, 
WAL edits are kept in memory until the flush period.  The benefit is aggregated and asynchronous <classname>HLog</classname>- writes, but the potential downside is that if
 the RegionServer goes down the yet-to-be-flushed edits are lost.  This is safer, however, than not using WAL at all with Puts.
</para>
<para>
Deferred log flush can be configured on tables via <link
      xlink:href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/HTableDescriptor.html">HTableDescriptor</link>.  The default value of <varname>hbase.regionserver.optionallogflushinterval</varname> is 1000ms.
</para>
    </section>  

    <section xml:id="perf.hbase.client.autoflush">
      <title>HBase Client:  AutoFlush</title>

      <para>When performing a lot of Puts, make sure that setAutoFlush is set
      to false on your <link
      xlink:href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HTable.html">HTable</link>
      instance. Otherwise, the Puts will be sent one at a time to the
      RegionServer. Puts added via <code> htable.add(Put)</code> and <code> htable.add( &lt;List&gt; Put)</code>
      wind up in the same write buffer. If <code>autoFlush = false</code>,
      these messages are not sent until the write-buffer is filled. To
      explicitly flush the messages, call <methodname>flushCommits</methodname>.
      Calling <methodname>close</methodname> on the <classname>HTable</classname>
      instance will invoke <methodname>flushCommits</methodname>.</para>
    </section>
    <section xml:id="perf.hbase.client.putwal">
      <title>HBase Client:  Turn off WAL on Puts</title>
      <para>A frequently discussed option for increasing throughput on <classname>Put</classname>s is to call <code>writeToWAL(false)</code>.  Turning this off means
          that the RegionServer will <emphasis>not</emphasis> write the <classname>Put</classname> to the Write Ahead Log,
          only into the memstore, HOWEVER the consequence is that if there
          is a RegionServer failure <emphasis>there will be data loss</emphasis>.
          If <code>writeToWAL(false)</code> is used, do so with extreme caution.  You may find in actuality that
          it makes little difference if your load is well distributed across the cluster.
      </para>
      <para>In general, it is best to use WAL for Puts, and where loading throughput
          is a concern to use <link linkend="perf.batch.loading">bulk loading</link> techniques instead.  
      </para>
    </section>
    <section xml:id="perf.hbase.client.regiongroup">
      <title>HBase Client: Group Puts by RegionServer</title>
      <para>In addition to using the writeBuffer, grouping <classname>Put</classname>s by RegionServer can reduce the number of client RPC calls per writeBuffer flush. 
      There is a utility <classname>HTableUtil</classname> currently on TRUNK that does this, but you can either copy that or implement your own verison for
      those still on 0.90.x or earlier.
      </para>
    </section>    
    <section xml:id="perf.hbase.write.mr.reducer">
      <title>MapReduce:  Skip The Reducer</title>
      <para>When writing a lot of data to an HBase table from a MR job (e.g., with <link
      xlink:href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/mapreduce/TableOutputFormat.html">TableOutputFormat</link>), and specifically where Puts are being emitted
      from the Mapper, skip the Reducer step.  When a Reducer step is used, all of the output (Puts) from the Mapper will get spooled to disk, then sorted/shuffled to other 
      Reducers that will most likely be off-node.  It's far more efficient to just write directly to HBase.   
      </para>
      <para>For summary jobs where HBase is used as a source and a sink, then writes will be coming from the Reducer step (e.g., summarize values then write out result). 
      This is a different processing problem than from the the above case. 
      </para>
    </section>

  <section xml:id="perf.one.region">
    <title>Anti-Pattern:  One Hot Region</title>
    <para>If all your data is being written to one region at a time, then re-read the
    section on processing <link linkend="timeseries">timeseries</link> data.</para>
    <para>Also, see <xref linkend="precreate.regions"/>, as well as <xref linkend="perf.configurations"/> </para>   
  </section>

  </section>  <!--  writing -->
  
  <section xml:id="perf.reading">
    <title>Reading from HBase</title>

    <section xml:id="perf.hbase.client.caching">
      <title>Scan Caching</title>

      <para>If HBase is used as an input source for a MapReduce job, for
      example, make sure that the input <link
      xlink:href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Scan.html">Scan</link>
      instance to the MapReduce job has <methodname>setCaching</methodname> set to something greater
      than the default (which is 1). Using the default value means that the
      map-task will make call back to the region-server for every record
      processed. Setting this value to 500, for example, will transfer 500
      rows at a time to the client to be processed. There is a cost/benefit to
      have the cache value be large because it costs more in memory for both
      client and RegionServer, so bigger isn't always better.</para>
      <section xml:id="perf.hbase.client.caching.mr">
        <title>Scan Caching in MapReduce Jobs</title>
        <para>Scan settings in MapReduce jobs deserve special attention.  Timeouts can result (e.g., UnknownScannerException)
        in Map tasks if it takes longer to process a batch of records before the client goes back to the RegionServer for the
        next set of data.  This problem can occur because there is non-trivial processing occuring per row.  If you process
        rows quickly, set caching higher.  If you process rows more slowly (e.g., lots of transformations per row, writes), 
        then set caching lower.
        </para>
        <para>Timeouts can also happen in a non-MapReduce use case (i.e., single threaded HBase client doing a Scan), but the
        processing that is often performed in MapReduce jobs tends to exacerbate this issue.
        </para>
      </section>
    </section>
    <section xml:id="perf.hbase.client.selection">
      <title>Scan Attribute Selection</title>

      <para>Whenever a Scan is used to process large numbers of rows (and especially when used
      as a MapReduce source), be aware of which attributes are selected.   If <code>scan.addFamily</code> is called
      then <emphasis>all</emphasis> of the attributes in the specified ColumnFamily will be returned to the client.
      If only a small number of the available attributes are to be processed, then only those attributes should be specified
      in the input scan because attribute over-selection is a non-trivial performance penalty over large datasets.
      </para>
    </section>

    <section xml:id="perf.hbase.client.scannerclose">
      <title>Close ResultScanners</title>

      <para>This isn't so much about improving performance but rather
      <emphasis>avoiding</emphasis> performance problems. If you forget to
      close <link
      xlink:href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/ResultScanner.html">ResultScanners</link>
      you can cause problems on the RegionServers. Always have ResultScanner
      processing enclosed in try/catch blocks... <programlisting>
Scan scan = new Scan();
// set attrs...
ResultScanner rs = htable.getScanner(scan);
try {
  for (Result r = rs.next(); r != null; r = rs.next()) {
  // process result...
} finally {
  rs.close();  // always close the ResultScanner!
}
htable.close();</programlisting></para>
    </section>

    <section xml:id="perf.hbase.client.blockcache">
      <title>Block Cache</title>

      <para><link
      xlink:href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Scan.html">Scan</link>
      instances can be set to use the block cache in the RegionServer via the
      <methodname>setCacheBlocks</methodname> method. For input Scans to MapReduce jobs, this should be
      <varname>false</varname>. For frequently accessed rows, it is advisable to use the block
      cache.</para>
    </section>
    <section xml:id="perf.hbase.client.rowkeyonly">
      <title>Optimal Loading of Row Keys</title>
      <para>When performing a table <link xlink:href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Scan.html">scan</link>
            where only the row keys are needed (no families, qualifiers, values or timestamps), add a FilterList with a
            <varname>MUST_PASS_ALL</varname> operator to the scanner using <methodname>setFilter</methodname>. The filter list
            should include both a <link xlink:href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/filter/FirstKeyOnlyFilter.html">FirstKeyOnlyFilter</link>
            and a <link xlink:href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/filter/KeyOnlyFilter.html">KeyOnlyFilter</link>.
            Using this filter combination will result in a worst case scenario of a RegionServer reading a single value from disk
            and minimal network traffic to the client for a single row.
      </para>
    </section>
   <section xml:id="perf.hbase.read.dist">
      <title>Concurrency:  Monitor Data Spread</title>
      <para>When performing a high number of concurrent reads, monitor the data spread of the target tables.  If the target table(s) have 
      too few regions then the reads could likely be served from too few nodes.  </para>
      <para>See <xref linkend="precreate.regions"/>, as well as <xref linkend="perf.configurations"/> </para>   
   </section>
    
  </section>  <!--  reading -->
  
  <section xml:id="perf.deleting">
    <title>Deleting from HBase</title>
     <section xml:id="perf.deleting.queue">
       <title>Using HBase Tables as Queues</title>
       <para>HBase tables are sometimes used as queues.  In this case, special care must be taken to regularly perform major compactions on tables used in
       this manner.  As is documented in <xref linkend="datamodel" />, marking rows as deleted creates additional StoreFiles which then need to be processed
       on reads.  Tombstones only get cleaned up with major compactions.
       </para>
       <para>See also <xref linkend="compaction" /> and <link xlink:href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HBaseAdmin.html#majorCompact%28java.lang.String%29">HBaseAdmin.majorCompact</link>.
       </para>
     </section>
     <section xml:id="perf.deleting.rpc">
       <title>Delete RPC Behavior</title>
       <para>Be aware that <code>htable.delete(Delete)</code> doesn't use the writeBuffer.  It will execute an RegionServer RPC with each invocation.
       For a large number of deletes, consider <code>htable.delete(List)</code>.
       </para>
       <para>See <link xlink:href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HTable.html#delete%28org.apache.hadoop.hbase.client.Delete%29"></link>
       </para>
     </section>
  </section>  <!--  deleting -->

  <section xml:id="perf.hdfs"><title>HDFS</title>
   <para>Because HBase runs on <xref linkend="arch.hdfs" /> it is important to understand how it works and how it affects
   HBase.
   </para>
    <section xml:id="perf.hdfs.curr"><title>Current Issues With Low-Latency Reads</title>
      <para>The original use-case for HDFS was batch processing.  As such, there low-latency reads were historically not a priority.
      With the increased adoption of HBase this is changing, and several improvements are already in development.
      See the 
      <link xlink:href="https://issues.apache.org/jira/browse/HDFS-1599">Umbrella Jira Ticket for HDFS Improvements for HBase</link>.
      </para>
    </section>
    <section xml:id="perf.hdfs.comp"><title>Performance Comparisons of HBase vs. HDFS</title>
     <para>A fairly common question on the dist-list is why HBase isn't as performant as HDFS files in a batch context (e.g., as 
     a MapReduce source or sink).  The short answer is that HBase is doing a lot more than HDFS (e.g., reading the KeyValues, 
     returning the most current row or specified timestamps, etc.), and as such HBase is 4-5 times slower than HDFS in this 
     processing context.  Not that there isn't room for improvement (and this gap will, over time, be reduced), but HDFS
      will always be faster in this use-case.
     </para>
    </section>
  </section>
  
  <section xml:id="perf.ec2"><title>Amazon EC2</title>
   <para>Performance questions are common on Amazon EC2 environments because it is a shared environment.  You will
   not see the same throughput as a dedicated server.  In terms of running tests on EC2, run them several times for the same
   reason (i.e., it's a shared environment and you don't know what else is happening on the server).
   </para>
   <para>If you are running on EC2 and post performance questions on the dist-list, please state this fact up-front that
    because EC2 issues are practically a separate class of performance issues.
   </para>
  </section>
</chapter>
