exp-bbv/docs/bbv-manual.xml

<?xml version="1.0"?> <!-- -*- sgml -*- -->
<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
  "http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd">

<chapter id="bbv-manual" xreflabel="BBV">
  <title>BBV: an experimental basic block vector generation tool</title>

<para>To use this tool, you must specify
<option>--tool=exp-bbv</option> on the Valgrind
command line.</para>

<sect1 id="bbv-manual.overview" xreflabel="Overview">
<title>Overview</title>

<para>
   A basic block is a linear section of code with one entry point and one exit
   point.  A <emphasis>basic block vector</emphasis> (BBV) is a list of all
   basic blocks entered during program execution, and a count of how many
   times each basic block was run.
</para>

<para>
   BBV is a tool that generates basic block vectors for use with the 
   <ulink url="http://www.cse.ucsd.edu/~calder/simpoint/">SimPoint</ulink>
   analysis tool. 
   The SimPoint methodology enables speeding up architectural 
   simulations by only running a small portion of a program
   and then extrapolating total behavior from this
   small portion.  Most programs exhibit phase-based behavior, which
   means that at various times during execution a program will encounter 
   intervals of time where the code behaves similarly to a previous
   interval.  If you can detect these intervals and group them together, 
   an approximation of the total program behavior can be obtained
   by only simulating a bare minimum number of intervals, and then scaling 
   the results.
</para>

<para>
  In computer architecture research, running a 
  benchmark on a cycle-accurate simulator can cause slowdowns on the order
  of 1000 times, making it take days, weeks, or even longer to run full
  benchmarks.  By utilizing SimPoint this can be reduced significantly,
  usually by 90-95%, while still retaining reasonable accuracy.
</para>

<para>
   A more complete introduction to how SimPoint works can be 
   found in the paper "Automatically Characterizing Large Scale 
   Program Behavior" by T. Sherwood, E. Perelman, G. Hamerly, and 
   B. Calder.  
</para>

</sect1>

<sect1 id="bbv-manual.quickstart" xreflabel="Quick Start">
<title>Using Basic Block Vectors to create SimPoints</title>

<para>
   To quickly create a basic block vector file, you will call Valgrind
   like this:

   <programlisting>valgrind --tool=exp-bbv /bin/ls</programlisting>

   In this case we are running on <filename>/bin/ls</filename>,
   but this can be any program.  By default a file called
   <computeroutput>bb.out.PID</computeroutput> will be created,
   where PID is replaced by the process ID of the running process.
   This file contains the basic block vector.  For long-running programs
   this file can be quite large, so it might be wise to compress
   it with gzip or some other compression program.
</para>   

<para>
   To create actual SimPoint results, you will need the SimPoint utility,
   available from the 
   <ulink url="http://www.cse.ucsd.edu/~calder/simpoint/">SimPoint webpage</ulink>.
   Assuming you have downloaded SimPoint 3.2 and compiled it,
   create SimPoint results with a command like the following:
      
   <programlisting><![CDATA[
./SimPoint.3.2/bin/simpoint -inputVectorsGzipped \
    -loadFVFile bb.out.1234.gz \
    -k 5 -saveSimpoints results.simpts \
    -saveSimpointWeights results.weights]]></programlisting>

   where bb.out.1234.gz is your compressed basic block vector file
   generated by BBV.
</para>

<para>   
   The SimPoint utility does random linear projection using 15-dimensions,
   then does k-mean clustering to calculate which intervals are 
   of interest.  In this example we specify 5 intervals with the 
   -k 5 option.   
</para>   
   
<para>   
   The outputs from the SimPoint run are the 
   <computeroutput>results.simpts</computeroutput>
   and <computeroutput>results.weights</computeroutput> files.
   The first holds the 5 most relevant intervals of the program.
   The seconds holds the weight to scale each interval by when
   extrapolating full-program behavior.  The intervals and the weights
   can be used in conjunction with a simulator that supports
   fast-forwarding; you fast-forward to the interval of interest,
   collect stats for the desired interval length, then use
   statistics gathered in conjunction with the weights to 
   calculate your results.
</para> 
   
</sect1>

<sect1 id="bbv-manual.usage" xreflabel="BBV Command-line Options">
<title>BBV Command-line Options</title>

<para> BBV-specific command-line options are:</para>

<!-- start of xi:include in the manpage -->
<variablelist id="bbv.opts.list">

  <varlistentry id="opt.bb-out-file" xreflabel="--bb-out-file">
     <term>
        <option><![CDATA[--bb-out-file=<name> [default: bb.out.%p] ]]></option>
     </term>
     <listitem>
        <para>
           This option selects the name of the basic block vector file.  The
           <option>%p</option> and <option>%q</option> format specifiers can be
           used to embed the process ID and/or the contents of an environment
           variable in the name, as is the case for the core option
           <option><xref linkend="opt.log-file"/></option>.
        </para>
     </listitem>
  </varlistentry>

  <varlistentry id="opt.pc-out-file" xreflabel="--pc-out-file">
     <term>
        <option><![CDATA[--pc-out-file=<name> [default: pc.out.%p] ]]></option>
     </term>
     <listitem>
        <para>
           This option selects the name of the PC file.  
           This file holds program counter addresses
           and function name info for the various basic blocks.
           This can be used in conjunction
           with the basic block vector file to fast-forward via function names
           instead of just instruction counts.  The 
           <option>%p</option> and <option>%q</option> format specifiers can be
           used to embed the process ID and/or the contents of an environment
           variable in the name, as is the case for the core option
           <option><xref linkend="opt.log-file"/></option>.
        </para>
     </listitem>
   </varlistentry>

   <varlistentry id="opt.interval-size" xreflabel="--interval-size">
      <term>
        <option><![CDATA[--interval-size=<number> [default: 100000000] ]]></option>
      </term>
      <listitem>
      <para>
         This option selects the size of the interval to use.  
         The default is 100 
         million instructions, which is a commonly used value.  
         Other sizes can be used; smaller intervals can help programs
         with finer-grained phases.  However smaller interval size
         can lead to accuracy issues due to warm-up effects 
         (When fast-forwarding the various architectural features
         will be un-initialized, and it will take some number
         of instructions before they "warm up" to the state a 
         full simulation would be at without the fast-forwarding.
         Large interval sizes tend to mitigate this.)
      </para>
      </listitem>
  </varlistentry>

  <varlistentry id="opt.instr-count-only" xreflabel="--instr-count-only">
     <term>
        <option><![CDATA[--instr-count-only [default: no] ]]></option>
     </term>
     <listitem>
        <para>
           This option tells the tool to only display instruction count
           totals, and to not generate the actual basic block vector file.
           This is useful for debugging, and for gathering instruction count
           info without generating the large basic block vector files.
        </para>
     </listitem>
   </varlistentry>
  

</variablelist>
<!-- end of xi:include in the manpage -->

</sect1>

<sect1 id="bbv-manual.fileformat" xreflabel="BBV File Format">
<title>Basic Block Vector File Format</title>

<para>  
  The Basic Block Vector is dumped at fixed intervals.  This
  is commonly done every 100 million instructions; the 
  <option>--interval-size</option> option can be 
  used to change this.
</para>

<para>
  The output file looks like this:
</para>

<programlisting><![CDATA[
T:45:1024 :189:99343
T:11:78573 :15:1353  :56:1
T:18:45 :12:135353 :56:78 314:4324263]]></programlisting>

<para>
  Each new interval starts with a T.   This is followed on the same line
  by a series of basic block and frequency pairs, one for each
  basic block that was entered during the interval.  The format for
  each block/frequency pair is a colon, followed by a number that
  uniquely identifies the basic block, another colon, and then
  the frequency (which is the number of times the block was entered,
  multiplied by the number of instructions in the block).  The
  pairs are separated from each other by a space.
</para>

<para>
  The frequency count is multiplied by the number of instructions that are 
  in the basic block, in order to weigh the count so that instructions in 
  small basic blocks aren't counted as more important than instructions 
  in large basic blocks.
</para>

<para>
  The SimPoint program only processes lines that start with a "T".  All
  other lines are ignored.  Traditionally comments are indicated by
  starting a line with a "#" character.  Some other BBV generation tools,
  such as PinPoints, generate lines beginning with letters other than "T"
  to indicate more information about the program being run.  We do
  not generate these, as the SimPoint utility ignores them.
</para>

</sect1>

<sect1 id="bbv-manual.implementation" xreflabel="Implementation">
<title>Implementation</title>

<para>
   Valgrind provides all of the information necessary to create
   BBV files.  In the current implementation, all instructions
   are instrumented.  This is slower (by approximately a factor
   of two) than a method that instruments at the basic block level, 
   but there are some complications (especially with rep prefix
   detection) that make that method more difficult.
</para>
  
<para>
   Valgrind actually provides instrumentation at a superblock level.
   A superblock has one entry point but unlike basic blocks can
   have multiple exit points.  Once a branch occurs into the middle
   of a block, it is split into a new basic block.  Because
   Valgrind cannot produce "true" basic blocks, the generated
   BBV vectors will be different than those generated by other tools.
   In practice this does not seem to affect the accuracy of the
   SimPoint results.  We do internally force the
   <option>--vex-guest-chase-thresh=0</option>
   option to Valgrind which forces a more basic-block-like
   behavior.
</para>

<para>
   When a superblock is run for the first time, it is instrumented
   with our BBV routine.  A block info (bbInfo) structure is allocated
   which holds the various information and statistics for the block.
   A unique block ID is assigned to the block, and then the
   structure is placed into an ordered set.
   Then each native instruction in the block is instrumented to
   call an instruction counting routine with a pointer to the block
   info structure as an argument.
</para>

<para>
   At run-time, our instruction counting routines are called once
   per native instruction.  The relevant block info structure is accessed
   and the block count and total instruction count is updated.   
   If the total instruction count overflows the interval size 
   then we walk the ordered set, writing out the statistics for
   any block that was accessed in the interval, then resetting the
   block counters to zero.
</para>

<para>
   On the x86 and amd64 architectures the counting code has extra
   code to handle rep-prefixed string instructions.  This is because 
   actual hardware counts a rep-prefixed instruction 
   as one instruction, while a naive Valgrind implementation
   would count it as many (possibly hundreds, thousands or even millions)
   of instructions.  We handle rep-prefixed instructions specially,
   in order to make the results match those obtained with hardware performance
   counters.
</para>   
   
<para>
   BBV also counts the fldcw instruction.  This instruction is used on 
   x86 machines in various ways; it is most commonly found when converting 
   floating point values into integers.
   On Pentium 4 systems the retired instruction performance
   counter counts this instruction as two instructions (all other 
   known processors only count it as one).
   This can affect results when using SimPoint on Pentium 4 systems.
   We provide the fldcw count so that users can evaluate whether it
   will impact their results enough to avoid using Pentium 4 machines
   for their experiments.  It would be possible to add an option to 
   this tool that mimics the double-counting so that the generated BBV
   files would be usable for experiments using hardware performance
   counters on Pentium 4 systems.
</para>

</sect1>

<sect1 id="bbv-manual.threadsupport" xreflabel="BBV Threaded Support">
<title>Threaded Executable Support</title>

<para>
   BBV supports threaded programs.  When a program has multiple threads,
   an additional basic block vector file is created for each thread (each
   additional file is the specified filename with the thread number
   appended at the end).
</para>

<para>
   There is no official method of using SimPoint with
   threaded workloads.  The most common method is to run
   SimPoint on each thread's results independently, and use 
   some method of deterministic execution to try to match the
   original workload.  This should be possible with the current
   BBV.
</para>

</sect1>

<sect1 id="bbv-manual.validation" xreflabel="BBV Validation">
<title>Validation</title>

<para>
   BBV has been tested on x86, amd64, and ppc32 platforms.
   An earlier version of BBV was tested in detail using
   hardware performance counters, this work is described in a paper 
   from the HiPEAC'08 conference, "Using Dynamic Binary Instrumentation 
   to Generate Multi-Platform SimPoints: Methodology and Accuracy" by
   V.M. Weaver and S.A. McKee.
</para>
 
</sect1>
 
<sect1 id="bbv-manual.performance" xreflabel="BBV Performance">
<title>Performance</title>

<para>
  Using this program slows down execution by roughly a factor of 40
  over native execution.  This varies depending on the machine
  used and the benchmark being run.
  On the SPEC CPU 2000 benchmarks running on a 3.4GHz Pentium D 
  processor, the slowdown ranges from 24x (mcf) to 340x (vortex.2).
</para>

</sect1>

</chapter>