Getting Started with HLVM

Here's the quick start for getting up and running quickly with HLVM. Note that this is for experts and the impatient only.

1. Read the documentation.
2. Read the documentation.
3. Remember that you were warned twice about reading the documentation.
4. Obtain, build, and install the dependent packages. You need llvm, llvm-gcc4, apr, apr-util, libxml2, scons, gperf, gcc, dejagnu, python, tcl, and expect.
5. Obtain the HLVM Source Code (http://llvm.org/svn/llvm-project/hlvm/trunk).
6. Build hlvm with: scons -Q mode=debug confpath=/paths/to/pkgs
7. The configuration utility might ask you where your 3rd party software is located. Please enter the paths appropriately.
8. Run the test suite with: scons -Q check
9. Install hlvm with: scons -Q mode=debug install prefix=/path/to/put/hlvm

Welcome to HLVM! This document shows you how to get started with the High Level Virtual Machine. Before you do that, lets get some questions answered up front to make sure you actually want to get started with HLVM.

What is HLVM?

HLVM is a toolkit for building virtual machines for dynamic languages. Its not something everyone is going to want to mess with. Generally, it is highly technical and doesn't really do much for an end user. On the other hand, if you're developing your own dynamic programming language, HLVM is just the ticket.

Who uses HLVM?

Generally, programming language researchers and software tool vendors use HLVM.

What does HLVM do?

HLVM doesn't really do anything. While it provides a few executables, these are ancillary in nature. Most of HLVM's capabilities are provided through a set of code libraries that can be incorporated into other programs. HLVM does provide a virtual machine executable but without a program to run (one that has been compiled by HLVM based tools), the virtual machine won't do anything. So, if you're an end user of software, HLVM probably isn't for you. If you're a developer, however, we think you'll find HLVM quite spiffy.

How hard is HLVM to build?

Although we've tried to make building HLVM dead simple, it is a complicated system and you must understand some of those complexities to get it right. One of its major complexities is that it incorporates many other packages that are not distributed with HLVM. You must build and install those packages yourself.

Okay, if you're still interested in HLVM, then we have a few more introductory things to cover:

1. HLVM provides a suite of libraries and a few tools. These are statically linked (archive files and object files) so that you can incorporate the appropriate pieces into your project. This was done purposefully since we don't believe that creating a monolithic shared library (DLL) is particularly useful. This architecture gives you the freedom to construct your own libraries by picking the appropriate things from the palette that HLVM offers.
2. Although HLVM provides several tools, these are ancillary in nature. They are provided to help either with testing HLVM or to help you use HLVM. The hlvm program (the actual virtual machine) is a reference implementation. You may choose to implement it differently (with the help of HLVM's runtime libraries, of course).

Before you attempt to build HLVM, you need to understand its requirements. HLVM is non-trivial and depends on a number of other software packages. These packages will be required before you can build HLVM. Taking a moment to familiarize yourself with HLVM's requirements may save you some time in the long run.

The table below shows the software that HLVM depends on. You must obtain and build these packages yourself if they are not already present on your system. HLVM won't compile or run without them.

Hardware

HLVM supports whatever hardware LLVM supports. For details, please see LLVM's documentation

Software

Compiling HLVM requires that you have several software packages installed. The table below lists those required packages. The Package column is the usual name for the software package that HLVM depends on. The Version column provides "known to work" versions of the package. The Notes column describes how HLVM uses the package and provides other details.

Package	Version	Notes
gcc	≥3.4.6 (4.1 recommended)	C/C++ compiler for compiling HLVM.
llvm 1	1.8cvs	Low Level Virtual Machine. This compler infrastructure provides the low level code generation and optimization, linking, bytecode, JIT, and other facilities HLVM needs. LLVM is HLVM's sister project. It handles all the low level details for
llvm-gcc4 1	4	LLVM C/C++ Frontend, Version 4. This is a full GCC (Gnu Compiler Collection) compiler but with extensions for generating code via LLVM instead of GCC's normal RTL-based backend. This tool can compile C, C++, Objective-C and Objective-C++ languages into either native binary or LLVM bytecode. In either case, it utilizes LLVM's suite of passes for code optimization.
apr	1.2.7	Apache Portable Runtime. APR is used as the portability layer to abstract away differences between the various operating systems that HLVM runs on. This simplifies the HLVM Runtime library implementation and also endows it with some robustness. It handles various OS abstraction such as file I/O, threading, locking, loading, etc.
apr-util	1.2.7	Apache Portable Runtime Utilities. Additional runtime utilities for DBM database access, code set conversion (iconv) and miscellaneous other utilities.
libxml2	2.6.24	XML parsing and validation toolkit. This is used for reading and writing XML forms of the Abstract Syntax Tree.
gperf	2.7.2 (3.0.1 won't work)	GNU Perfect Hash Function Generator. This is used for recognizing element and attribute names in XML documents.
subversion	≥1.1	Subversion source Subversion access to HLVM source repository.
scons	0.96.92	Software Construction system used for HLVM's build system.
python	2.3	Python programming language, required by scons
dejagnu	1.4.2	Test suite automation package. You only need this if you want to run the automated test suite.
expect	5.38.0	Test scripting language. DejaGnu is based on this. You only need this if you want to run the automated test suite.
tcl	8.3, 8.4	Test Control Language. Expect is based on this. You only need this if you want to run the automated test suite.
doxygen	≥1.4.43	C/C++ automated documentation generator for building the XHTML documentation from the comments in the HLVM header files. You only need this if you intend to build the documentation.
xsltproc	≥2.6.243	XSLT Processor (comes with libxml2). This is used to transform the Relax NG schema for the HLVM Abstract Syntax Tree (AST) into documentation that describes the AST. You only need this if you intend to build the documentation.

Notes:

Additionally, your compilation host is expected to have the usual Unix utilities, specifically:

• ar - archive library builder
• gzip* - gzip command for distribution generation
• gunzip* - gunzip command for distribution checking
• ranlib - symbol table builder for archive libraries
• tar - tape archive for distribution generation

Single Separate Install Area

It is suggested that you keep a separate installation area for building the things upon which HLVM depends. This is the location in your file system where you will install the built packages. It is the parameter to the --prefix options to the configure programs of those packages. By using a separate install area, you ensure that the HLVM version of required packages doesn't interfere with any of your system installed packages.

libxml2

• If your system doesn't already have a modern version of libxml2, you can generally build this quite painlessly as it doesn't depend on much else.
• You need a modern version of this package, in the 2.6 series, in order to successfully build HLVM. HLVM uses the Relax/NG validator that was added and improved in the 2.6 series of releases. Reid used 2.6.24 to develop HLVM and you are advised to use that version or a later one.
• See the README file in the libxml2 root directory.
• Build with:

      ./configure --prefix=/where/to/install 
      make 
      make install

• Use ./configure --help to get a full list of libxml2's configuration options; it has many. Not everything libxml2 provides is needed. We need the parser, the textwriter and the relaxng validator.

apr

• This package builds quite easily as well and doesn't depend on anything other than your operating system.
• You need version 1.2.7 or later.
• See the README.dev for build instructions.
• Build with:

      ./buildconf    #generates the configure script
      ./configure --prefix=/where/to/install --enable-threads --enable-other-child
      make
      make install

• Optionally use the --enable-debug configure flag for making debugging easier (and your HLVM programs run slower).

apr-util

• This package builds quite easily and doesn't depend on much.
• Build with:

      ./buildconf    #generates the configure script
      ./configure --prefix=/where/to/install --with-apr=/path/to/apr
      make
      make install

• apr-util provides ldap, dbm, and iconv services all of which HLVM will eventually use. apr-util's configure script will generally be able to figure out what your system has and use it. However, if it can't you might want to use the various --with-* options to tell apr-util where your packages are located.

llvm

• You must use the latest CVS version of LLVM. LLVM is HLVM's sister project and the two are intricately connected. Consequently, patches are made to LLVM in order to satisfy HLVM's requirements. At some point, this restriction will be lifted as a released version of LLVM will contain all of HLVM's requirements. However, while in development you should checkout the latest version of LLVM and keep it up to date.
• Build with:

      cd /path/to/llvm/
      mkdir ../build
      cd ../build
      ../llvm/configure --prefix=/install/dir --with-llvmgccdir=/path/to/llvmgcc
      make ENABLE_OPTIMIZED=1 tools-only
      make tools-only
      make install

• Note: that you are actually building two copies of LLVM here. The first one is built with ENABLE_OPTIMIZED=1. This is for llvm-gcc4 to use and is optimized for quick compiles. The llvm-gcc4 configuration will know how to find the build targets of this optimized release version. The second build is a debug build for linking with HLVM. This ensures that you can debug into LLVM, should you need to.
• Note: that we only build the tools-only target. This saves us from building the llvm-gcc3 runtime libraries and examples that we won't use. It also gets around a chicken-and-egg problem where the LLVM runtime libraries depend on llvm-gcc3 but llvm-gcc3 depends on the LLVM tools being built. Use of llvm-gcc4 completely gets around this problem and so it is only necessary to build the LLVM tools and not the runtime library.

llvm-gcc4

• You must use version 4 of llvm-gcc, not version 3. Version 4 is much better integrated with LLVM, supports more languages, builds native object files by default, and otherwise looks and acts more like a full gcc compiler than did llvm-gcc3.
• Obtain llvm-gcc4 source from http://nondot.org/sabre/2006-06-01-llvm-gcc-4.tar.gz
• Alternatively, you can obtain a binary version of llvm-gcc4 for x86 Linux (Fedora Core 3) from http://llvm.org/2006-06-01-llvm-gcc4-linux-x86-binary.tar.gz
• You must install llvm-gcc4 into an installation area that is not used by any other software, particularly gcc. This is required because you might clobber your compiler's runtime libraries and tools otherwise. Just pick an installation directory where no other software is installed.
• See the README.LLVM in the top of the llvm-gcc4 source tree if you're inclined to build llvm-gcc4 on your own. It isn't hard but there are a few things you need to do correctly.
• Basic build steps:

      mkdir llvm-gcc
      cd llvm-gcc
      tar zxf llvm-gcc4-x.y.source.tar.gz
      mkdir obj
      mkdir install
      cd obj
      ../llvm-gcc4-x.y.source/configure --prefix=`pwd`/../install/ \
        -enable-llvm=$LLVMOBJDIR --enable-languages=c,c++,objc,obj-c++
      make
      make install

• Note: that the -enable-llvm option should specify the root of the build directory in which LLVM was built. If you built LLVM according to the instructions above, this directory will contain directories named Release and Debug. llvm-gcc4 will look for its tools in the Release directory.
• Note: that the --enable-languages option shown above provides the complete list of languages that could be built with llvm-gcc4. However, HLVM only needs C and C++ so you can shorten the build for llvm-gcc4 a little by dropping ,objc,obj-c++ from this option.

gperf

• Pretty simple tool to build.
• You need version 2.7.2. HLVM hasn't been tried with any other version.
• Build with:

      ./configure --prefix=/install/path
      make
      make install

deja-gnu/

• Simple tool to build
• You need version 1.4.4 or later
• Build with:

      ./configure --prefix=/install/path
      make
      make install

Quick Start

If you took our advice and built all dependent packages (except llvm-gcc4) into a separate install area, then you can build HLVM quickly by using this command:

  scons -Q mode=debug confpath=/path/to/llvm-gcc4/install:/path/to/install

The confpath option indicates to the HLVM build system some additional places to search for software. The first path should be the location in which the llmv-gcc4 software was installed. The second path should be the Single Separate Install Area where you installed all the other packages such as llvm, apr, apr-util, etc.

About SCons And HLVM

HLVM uses the SCons tool for software construction. While make and the various auto* tools were originally used, the full programming language support of SCons (Python) was attractive because of the platform support and the need to do intricate things to build HLVM. HLVM's use of SCons departs a bit from the norm because SCons doesn't provide everything that HLVM needs. If you are not familiar with SCons, we recommend you take a quick peek at the SCons User Manual. While an in depth knowledge is not needed to build HLVM, having some understanding of scons will definitely help.

The details on using scons for HLVM can be found in the Developer's Guide, but here's a precis to help you get started:

• You can still use make as the Makefile in the top level has the customary targets that will invoke the corresponding scons command.
• HLVM uses a number of customizations to the facilities provided in order to keep the SConstruct and SConscript files minimized. These facilities are provided in a Python module named build at the root directory. This departure was necessary in order to hide the implementation details of many of the build rules and configuration details that HLVM requires.
• An important thing to know is that configuration is done on every build. However, if the dependencies haven't changed, the result is cached and this runs quite quickly. This is a huge help to the project because it automatically detects environment changes and reconfigures HLVM accordingly which saves time and also catches problems very early.
• There are numerous configuration options available. You should use the scons -Q --help option to get a description of all of them.
• Configuration options are saved in cache files so you don't have to keep repeating them on each invocation of scons

SCons Options Supported

The important options that may be given on the scons command line are:

• -Q. Normally scons is quite verbose about all the configuration checks it is doing. This option just tells scons to be quiet about such things.
• --prefx=/path/to/install/dir. This option tells the build system where HLVM should be installed. The default is /usr/local.
• confpath=<path>. This option provides a colon (:) separated list of paths to search when configuring HLVM. This is handy if you normally install your software in an unusual place. For example, some of our developers install software to /proj/install or /more/install. By using this option, the configuration code will search the paths for the packages and tools that it is looking for. Furthermore, it will give priority to the confpath paths over the regular places that it looks.
• mode=<mode_name>. This gives a name to your configuration. You can replace <mode_name> with any name you like. For example you might think of your configuration as "debug" or "optimized". When you name a configuration this way, your configuration parameters will be saved in a file named .<mode_name>_options. To repeat the options specified the first time, simply issue the command scons mode=<mode_name> and all your configuration options will be read from the cache file.
• debug=0|1. This turns debug mode on and off. Debug mode causes the compiler to emit debug symbols and turns on the HLVM_DEBUG pre-processor symbol so that debug code can be included.
• assertions=0|1. This turns assertions on or off. This is separate from debug because we might want to have a release or optimized version that includes assertions but not debug code.
• inline=0|1. This controls whether the compiler should emit inline functions as real functions or attempt to inline them. When building a version you intend to debug with gdb, it is often handy to turn this off. You pay a performance penalty but it is also much easier to debug the code.
• optimize. This option tells the compiler to optimize the generated code. This may make it difficult to debug the resulting program. However, it will also make the program run much faster.
• with_*. There are a variety of options that are prefixed with with_, for example with_llvm or with_apr. Thise options allow you to tell scons where a specific package is located on your system. See the scons -Q --help output for a full list of these options.

Build Targets

The default build target will simply build all the libraries and tools. There are additional pseudo-targets (aliases in SCons lingo) that you can build:

• check. This will run the Deja-Gnu test suite located in the test directory. If you're making changes to HLVM, it is handy to run this frequently to ensure that you haven't caused any regressions. Patches that do not pass this test suite will not be accepted.
• install. This alias will install the HLVM header files, libraries, tools and documentation to the install directory (which was specified with the --prefix= configuration option).
• docs. By default, documentation is not built because it is time consuming and not necessary for each build. When this alias is used, the doxygen and XSLT generated documentation will be built.

If you don't want to learn how to use SCons, HLVM provides a Makefile that will invoke it for you. All the usual targets are provided. To use this facility, simply do a local edit of the Makefile and adjust the three variables:

MYMODE

Indicates the name of the build mode you want to build by default. This can be any string, but it will be more useful if it is one of these:

• Debug - specifies a debug build with assertions turned on and inline functions turned off. Initialize this build mode with make Debug.
• Optimized - specifies an optimized build with debug turned off, inline turned on, and assertions turned on. Initialize this build mode with make Optimized.
• Release - specifies an optimized build with assertions turned off and symbols stripped. Initialize this build with make Release.
• Profile - specifies an optimized profiling build with assertions turned off, inline turned on and with profiling options suitable for profiling the tools with gmon. Initialize this build with make Profile.

MYPATH

This provides a path for the 3rd party software that HLVM depends on. You should change this path to match your local environment. If you don't get this right then the first time you run scons, it will ask you to manually enter paths for things like the APR library and LLVM.

MYPREFIX

This provides the directory into which HLVM will be installed. If it is not specifed, the default will be /usr/local.

Once you've made these adjustments you can use the various targets in the Makefile to build HLVM. Note that these targets just convert the target into the appropriate invocation of the scons command.

Targets Supported

You can use the following targets to build HLVM in various ways:

all

Builds HLVM in the default mode, as specified by MYMODE. Typically this is a Debug build.

debug Debug

This builds HLVM in a mode named "Debug" and sets the various options appropriately for a debug build. You should use this target the first time you want a Debug build.

optimized Optimized

This builds HLVM in a mode named "Optimized" and sets the various options appropriately for an optimized build. You should use this target the first time you want an Optimized build.

release Release

This builds HLVM in a mode named "Release" and sets the various options appropriately for a release build. You should use this target the first time you want a Release build.

profile Profile

This builds HLVM in a mode named "Profile" and sets the various options appropriately for an profiled build. You should use this target the first time you want an Profile build.

check

This doesn't build HLVM but instead invokes the test suite via dejagnu. Use this to make sure that your local modifications haven't caused any regressions.

clean

This removes all the build targets that were previously built. The next time you build the all target, everything will be rebuilt.

doc

This builds the documentation for HLVM. It will run doxygen and xsltproc to build the generated portion of the documentation.

hlvm

This provides a partial build of HLVM. It descends into the hlvm directory and builds only the libraries located there.

install

This will installed HLVM libraries, headers and executable tools into the directory named by the MYPREFIX variable.

tools

This provides a partial build of HLVM. It skips building the libraries in the hlvm directory and instead descends into the tools and builds the executable tools there.

Tips And Tricks

• Override MYMODE. On the make command line you can override the value of MYMODE makefile variable. This allows you to use any of the usual targets with a different build mode. For example, suppose your Makefile specifies a value of "Debug" for MYMODE. This situation indicates that the "Debug" configuration will be used for all regular builds. However, if you want to install the Release version, say, you could issue this command:

      make install MYMODE=Release