I've been wanting to learn a new language for a while. Ruby is fun and all, but sometimes it's good to get a different perspective. There are definitely some interesting ones that have shown up on my radar: Lua, Haskell, Erlang, and Scala, to name a few. I've always been intrigued by Lisp, though. I don't think anyone would deny that John McCarthy (R.I.P.) was a genius for pioneering a language that's so simple, yet so powerful.

While there have been a few implementations of Lisp (Common Lisp, Scheme, Arc) throughout the years, and while it is used for certain applications (AI, music typesetting, text editors), it really hasn't reached mainstream appeal in the programming world the way that Java, Perl, or Lua has (three languages that were developed for one specific purpose and has since spread to other subfields). I think one of the reasons for this -- besides the fact that Lisp as a language is the polar opposite of imperative, Algol-based languages to which people have gravitated -- is also the reality that it's kind of stagnated while the world around it has evolved. In order to compete in the business/enterprise world nowadays, people just expect languages to have certain things -- reusability through namespaces and packaging, dependency management, a garbage collection model that doesn't suck -- and (although it's a bit of a catch-22) there isn't a community around Lisp writing, publishing, and recirculating supporting code in the same way that there are people around Perl or Ruby or even Java.

Clojure is a different story, though. It's been in development for 3 years, so it's still a young language, but it already has a conference, and a mailing list, and an established package management system, and plenty of code on GitHub where it is being used for everything from live music generation to web stuff to OpenGL. So what's its secret? Why does everyone like Clojure so much? Well, for one thing, it brings some new things to the Lisp world, such as language-level support for working with more specific data structures like vectors and hashes, and concurrency-safe data mutation with STM. The main draw, though, is that it runs on the JVM, so naturally it appeals to Java developers as well as people who, like me, are intrigued by the JVM but don't necessarily wish to write Java itself. So in a way, Clojure already ships with a wealth of features, because you can directly use any Java library you want (either the standard library or a .jar from somewhere else).

But enough about why you might want to try out Clojure. You're here because you want to quickly get a Clojure project up and running. Fortunately the process is pretty easy:

• Download Clojure to a location on your machine
• Install a wrapper script for the Clojure REPL
• Configure your editor of choice
• Install Leiningen
• Make your first project

Installing Clojure

The first thing to do, then, is to get Clojure 1.2 ¹, which is available on the Clojure site. Inside the zip file you download is a clojure.jar file. You'll want to choose a location to put this. I'm going to be using /opt/clojure/lib in this guide:

~$ mkdir -p /opt/clojure/lib
~$ cd /tmp
/tmp$ wget https://github.com/downloads/clojure/clojure/clojure-1.2.1.zip
/tmp$ unzip clojure-1.2.1.zip
/tmp/clojure-1.2.1$ cd clojure-1.2.1
/tmp/clojure-1.2.1$ cp clojure.jar /opt/clojure/lib/clojure-1.2.1.jar

Note that on the downloads page, you will also see a reference to a package called clojure-contrib. This is a user-contributed and -maintained collection of libraries that provide extra functionality you may find handy in your Clojure program. Don't worry about installing it now, though -- clojure-contrib is one of Leiningen's dependencies, so it will get installed when Leningen is installed.

(Also, ordinarily at this point I'd recommend using a package manager to prevent from having to maintain anything, but I actually prefer doing it this way because Clojure is pretty self-contained as it is, and as long as you choose a more-or-less standard location, then I don't see any issues with it. It'll also be easier to upgrade to 1.3 in the future.)

Making a wrapper script

With Clojure downloaded and installed, let's test it to make sure our setup works. If you're a Ruby or Python developer, you might be looking for a REPL so that you can quickly try out some code without having to bring out your code editor and create a standalone file. Clojure does have a REPL mode: as described on the Clojure site, you get into it with java -cp clojure.jar clojure.main. But no one uses this -- first, because it's hard to remember, and second, because the REPL isn't very usable, as it doesn't have the nice things that REPLs usually have, such as command history or tab autocompletion. There isn't really even a straightforward way to exit -- it isn't a huge deal as you can say (System/exit) or even just press Ctrl-C, but it's still kind of annoying.

Fixing these things isn't difficult, though. To provide command history and tab completion, we can use everyone's favorite C library, readline. More specifically, we can create a wrapper script, commonly called clj, that uses rlwrap to wrap the java -jar ... command in readline capabilities. The good thing is that people have already done the work for us here -- Antonio Cangiano has a version of this script that merely improves upon the version up on the Clojure wiki. So we will base ours on both of these.

That means we'll need to install rlwrap. If you are on Linux, I'm fairly sure you can get rlwrap through your package manager of choice. If you are on Mac as I am, you can simply use Homebrew:

brew install rlwrap

The wrapper script we will make is very simple. In running rlwrap, we'll tell it to do two things. First we'll give it a file, clj_completions, which contains the list of words available for autocompletion (we'll generate the completions file in a minute). Second, we'll tell it to run the java -jar clojure.jar command to start Clojure. Then, right before the REPL is loaded, we'll tell Clojure to load a second file, cljrc.clj; this file will allow us to customize the REPL, such as defining a proper (exit) command.

Here's what that all looks like:

#!/bin/bash

breakchars="(){}[],^%$#@\"\";:''|\\"
CLOJURE_DIR=/opt/clojure
CLOJURE_JAR=$CLOJURE_DIR/lib/clojure-1.2.1.jar
CONTRIB_JAR=$CLOJURE_DIR/lib/clojure-contrib-1.2.0.jar
COMPLETIONS_FILE=$CLOJURE_DIR/etc/clj_completions
RC_FILE=$CLOJURE_DIR/etc/cljrc.clj
CLOJURE_CP=$CLASSPATH:$CLOJURE_JAR:$CONTRIB_JAR:$PWD

if [ $# -eq 0 ]; then
  cmd = ""
  cmd += "rlwrap --remember -c -b $breakchars"
  if [ -f $COMPLETIONS_FILE ]; then
    cmd += " -f $COMPLETIONS_FILE"
  fi
  cmd += " java -cp $CLOJURE_CP clojure.main -i $RC_FILE --repl"
  exec $cmd
else
  exec java -cp $CLOJURE_CP clojure.main $1 -- $@
fi

Save this as /opt/clojure/bin/clj, and make it executable with chmod +x /opt/clojure/bin/clj. Since we want to be able to run this anywhere, you will also want to either add /opt/clojure/bin to your path or make a symlink to it in a directory that's already in your path (I have a ~/.bin folder just for this purpose).

Now for the files that the clj script references. clj in fact won't work without an rc file, so let's create that first. This file simply consists of:

(defn exit [] (System/exit 0))

Save this to /opt/clojure/etc/cljrc.clj.

At this point we should be able to try out clj:

~$ clj
Clojure 1.2.1
user=> (* 2 2)
4
user=> "Hello world"
"Hello world"
user=> (defn foo [x] (* x x))
#'user/foo
user=> (foo 4)
16
user=> (exit)

You should also be able to press the up arrow at any point and view past commands.

Great. We still can't tab-complete, though -- to do that we need to create a list of autocompletable symbols. This could be anything, but you probably want at least the symbols in clojure.core. Here is some Clojure code that does this (ripped straight from the Clojure wiki):

(def completions
  (reduce concat (map (fn [p] (keys (ns-publics (find-ns p))))
                      '(clojure.core))))

(with-open [f (java.io.BufferedWriter. (java.io.FileWriter. "/opt/clojure/etc/clj_completions"))]
  (.write f (apply str (interleave completions (repeat "\n")))))

The list of namespaces in line 3 (the '(clojure.core) bit) is what is iterated over to create the file. You can get an idea of what to add to this by looking at the other namespaces in the Clojure documentation.

Save this script to a file (say, /tmp/foo.clj) and run it by passing the filename to clj. This will write the completions file to /opt/local/clojure/etc/clj_completions.

Now try clj again:

~$ clj
user=> ma<TAB>
macroexpand   macroexpand-1   make-array   .......
user=> (exit)

Installing Leiningen

As you saw, we can now create Clojure files and run them with clj. What if you want to make more of a project, though? What's the recommended way of organizing the files in your project? What if your project depends on libraries -- how do you install them?

All of these concerns are easily taken care of with Leiningen. Leiningen is the standard toolkit in Clojure-land for maintaining and installing your project's dependencies. It's a bit like npm for node.js, Bundler for Ruby, or Maven for Java. In fact, it uses Maven under the covers, and Maven already tells Java where everything is, so you don't have to worry about configuring the class path or any of that -- everything just works.

Installing Leiningen is pretty simple, too. First download the lein executable and put it somewhere -- to stay consistent, we'll use /opt/clojure/bin, with a symlink in ~/.bin. Like this:

~$ cd /opt/clojure/bin
/opt/clojure/bin$ wget https://raw.github.com/technomancy/leiningen/stable/bin/lein
/opt/clojure/bin$ chmod +x lein
/opt/clojure/bin$ ln -s /opt/clojure/bin/lein ~/.bin/lein
/opt/clojure/bin$ cd
~$ which lein
~/.bin/lein

All that really downloads is a script, so to finish the installation and download the Leiningen jar file, run the self-installer:

~$ lein self-install

Now let's make a new Lein project:

~$ cd code
~/code$ lein new musicprogram
Created new project in: ~/code/musicprogram

If we inspect this folder...

~/code$ ls musicprogram
README   project.clj   src/   test/

we will see that Lein has generated a project.clj file for us:

(defproject musicprogram "1.0.0-SNAPSHOT"
  :description "FIXME: write description"
  :dependencies [[org.clojure/clojure "1.2.1"]])

This is akin to a pom.xml file in Java-land, a package.json in node-land, or a gemspec in Ruby-land.

But let's try adding a dependency to our project. Let's say we want to play around with Overtone, which is a live coding music generation library. Here is how we would tell Leiningen that our project requires it:

(defproject musicprogram "1.0.0-SNAPSHOT"
  :description "Our super awesome music program"
  :dependencies [[org.clojure/clojure "1.2.1"]
                 [overtone "0.5.0"]])

Now, let's try installing it:

~/code/musicprogram$ lein install
Downloading: overtone/overtone/0.5.0/overtone-0.5.0.pom from central
Downloading: overtone/overtone/0.5.0/overtone-0.5.0.pom from clojars
Transferring 2K from clojars
Downloading: overtone/osc-clj/0.7.1/osc-clj-0.7.1.pom from clojars
Transferring 1K from clojars
Downloading: org/clojars/technomancy/jmdns/3.2.1/jmdns-3.2.1.pom from clojars
Transferring 1K from clojars
Downloading: commons-net/commons-net/3.0.1/commons-net-3.0.1.pom from clojars
Downloading: commons-net/commons-net/3.0.1/commons-net-3.0.1.pom from central
Transferring 16K from central
Downloading: org/apache/commons/commons-parent/21/commons-parent-21.pom from clojars
Downloading: org/apache/commons/commons-parent/21/commons-parent-21.pom from central
Transferring 33K from central
[... lots of output ...]

I mentioned that Lein uses Maven for all the dependency stuff. In fact, the project.clj you write is actually a wrapper for Maven's pom format. So when you run lein install, Maven immediately knows which dependencies your project needs. For each dependency, it first checks to see if it's already been downloaded (~/.m2 is conventionally where it keeps cached jar files; technically this is a local repository). If it's not there, it goes and fetches it from an online repository. Apart from Clojure, Maven only looks in one location, the central Maven repo. Clojure adds a second repository, Clojars, which is designed specifically to read project.clj files. This is why in the output above you will sometimes see "Transferring <bytes> from central" and other times "Transferring <bytes> from clojars".

Going back to project.clj, there's other information you can put there besides your dependencies. For instance, if your project depends on packages that are only needed for development (such as testing frameworks or debugging tools), you could specify them with a :dev-dependencies property. If you were creating more of a library and you wanted to package your project and distribute it to Clojars, you would need to provide a version and :description, so that other people can easily find it when they are searching on Clojars. In this case you might also want to add a :url property so that people have a place to go to learn more about your package.

Besides these, there are a whole host of other properties available -- see here for the full list.

I could keep going about Leiningen, but you'll probably get a better understanding if you read the built-in tutorial, which explains all of this in more detail than I have knowledge of -- you can get to that by running lein help tutorial.

Setting up our editor (Vim)

Now that we have a project up and running, let's turn now to configuring our editor so that we are comfortable writing Clojure code.

As mentioned, Clojure runs on the JVM, and while this is great for production, it isn't so great for development, as it takes a fairly long time for the JVM to boot up. Hence, a lot of people will recommend you use a Java IDE or an editor that can connect to a running Clojure REPL -- that way you can send any code you want to the REPL to get executed. If you've already been doing Java development, then you are probably used to Eclipse, NetBeans, or IntelliJ, and all three do have some sort of support for Clojure (they have their pros and cons). Otherwise, if you are just getting into Clojure and don't mind learning something new, Emacs is the clear winner. A lot of people who are seriously into Clojure are using it -- it really seems like there's nothing that beats SLIME and swank-clojure, and once you install them you can get up and running and no time.

As for me, I just learned Vim and didn't really feel like switching to Emacs. The de facto plugin for Vim is VimClojure, so if you use Vim too, you'll want to install this. Like Emacs, VimClojure also has a way of starting and connecting to Clojure via Nailgun. When you install VimClojure you are totally free to not use Nailgun, but it is easy to set up, and as you will need it pretty soon anyhow, I would recommend doing so right out of the gate. There are instructions on how to install Nailgun and tell VimClojure to use it in the README for the plugin, but this guy Dave Ray has posted some instructions -- along with an example VimClojure setup as described in the post -- which I found to be super helpful.

Basically, there are two pieces involved here: one is the Nailgun server (packaged in a jar file) which is our running instance of Clojure, the other is the Nailgun client (a little C program called ng) which will hit the server with the code we want to run inside that instance. So in order to run Clojure code inside of Vim, all you really have to do is tell VimClojure where you've put ng, and then start the server outside of Vim (VimClojure assumes that the server is running on the default port of 2113 so no need to customize that).

If you want to get fancy, you can even -- as Dave describes in his post -- use screen.vim to automate starting the Nailgun server from within Vim and ensure that it gets shut down after you quit Vim. But, you can actually easily install and start the server via lein-nailgun, which adds a nailgun command to Leiningen. That method works well enough for me, so that is what I am going to go with in this guide.

Alright, so let's try this out. The first thing to do, obviously, is install VimClojure. Then, update your ~/.vimrc to tell the plugin where everything is. I have a couple of customizations I've made -- mostly to turn off VimClojure's colorization of parentheses, and since I use delimitMate, to turn off autopairing as it just ends up getting in the way. Regardless, you will also want to make sure to set maplocalleader as <LocalLeader> is used by the plugin to set up default mappings:

set mapleader = ","
set maplocalleader = ","

" Settings for the VimClojure plugin
let vimclojure#FuzzyIndent=1
let vimclojure#HighlightBuiltins=1
let vimclojure#HighlightContrib=1
let vimclojure#DynamicHighlighting=1
let vimclojure#ParenRainbow=0
let vimclojure#WantNailgun=1
let vimclojure#NailgunClient = $HOME . "/.bin/ng"

" Turn off delimateMate (which provides auto-closing parens) for Clojure files
" as they just get in the way
au! FileType clojure let b:loaded_delimitMate=1

Note: If you are using MacVim, you will also want to make sure you are on a recent version, as older versions have an annoying bug where there is a short pause when jumping to another line. You'll want to make sure you have at least snapshot 61.

Next, download the Nailgun client, compile it, and move it to a location within your path (here I have it at ~/.bin/ng):

~$ cd /tmp
/tmp$ wget http://kotka.de/projects/vimclojure/vimclojure-nailgun-client-2.3.0.zip
/tmp$ unzip vimclojure-nailgun-client-2.3.0.zip
Archive:  vimclojure-nailgun-client-2.3.0.zip
   creating: vimclojure-nailgun-client/
  inflating: vimclojure-nailgun-client/Makefile
  inflating: vimclojure-nailgun-client/ng.exe
   creating: vimclojure-nailgun-client/ngclient/
  inflating: vimclojure-nailgun-client/ngclient/ng.c
/tmp$ cd vimclojure-nailgun-client
/tmp/vimclojure-nailgun-client$ make
Building ng client.  To build a Windows binary, type 'make ng.exe'
gcc -Wall -pedantic -s -O3  -o ng ngclient/ng.c
ld: warning: option -s is obsolete and being ignored
/tmp/vimclojure-nailgun-client$ mv ng ~/.bin

Thirdly, add lein-nailgun to your project. Continuing with our musicprogram project this will look like:

(defproject musicprogram "1.0.0-SNAPSHOT"
  :description "Our super awesome music program"
  :dependencies [[org.clojure/clojure "1.2.1"]
                 [overtone "0.5.0"]]
  :dev-dependencies [[org.clojars.ibdknox/lein-nailgun "1.1.1"]])

Check that "nailgun" is listed as one of the subcommands:

~/code/musicprogram$ lein help
[... long list of tasks here ...]
nailgun     Launch a vimclojure nailgun server.

Now run the server. Note that it will just sit there waiting for input from the client:

~/code/musicprogram$ lein nailgun
NGServer started on 127.0.0.1, port 2113.

Finally, restart Vim and bring up a Clojure file. The first time you open one, it will take a few seconds for the JVM to load (as Nailgun doesn't do this right away), but after that you can open any Clojure file with no penalty.

There a whole host of mappings that VimClojure adds for doing things like requiring files or looking up documentation, but one you can try really quickly is opening a REPL, which you can do with ,sr. This should give you something that looks like this:

As long as you stay in insert mode, you can press Ctrl-Up/Down to flip through command history, and you can even type a partial word and press Tab to autocomplete it.

As useful as the REPL is, the main reason I like VimClojure is because it lets me easily run tests with it, which I'll talk about next.

Testing

I love tests. Like Wheatley, I just get this itch -- I have to do it whatever code I'm writing. Most of the time for me tests play the role of sidekick -- I'd know what code to write even if tests weren't there, but they're good at keeping me in check. In learning a new language, though, you're not sure how to express your ideas and you don't know what your code will do. In that case, tests serve as a nice exploratory tool to make sure your results are what you expect. And your explorations are recorded, too.

So what are the available options in Clojure for testing? The simplest one is the built-in library, clojure.test. It works like xUnit libraries in other languages, in that you simply create test methods and make assertions in those methods which get checked when you run the tests.

We can try it out by going back to our musicprogram project and saving the following code to test/musicprogram/test/foo.clj:

(ns musicprogram.test.foo
  (:use clojure.test))

(deftest addition
  (is (= 2 (+ 1 1))))

(run-tests)

and running it with clj:

~/code/musicprogram$ clj test/musicprogram/test/foo.clj

Testing musicprogram.test.foo

Ran 1 test containing 1 assertion.
0 failures, 0 errors.

So that's one way to write and run tests, but it isn't the best way. First, it's kind of a pain that we have to put (run-tests) at the end of all of our test files. Second, if we are working on one source file and we need to run its corresponding test file over and over again, we will quickly become frustrated because of the whole JVM load time issue again.

The best method to run one file is actually through VimClojure. If you are looking at a test file, you can run it with ,rt and that will show the results in an adjoining window.

But say we're working on a project full of source files and test files and we want to run all of the tests in one fell swoop? Well, that's where we need another way. Running all our test files directly with clj falls short because we either have to list all of our test files on the command line or come up with some Bash script. VimClojure doesn't really give us a way to do this, either, unless you make a custom mapping. ²

Since run-tests accepts a list of Clojure namespaces, we could make a function that would require the namespaces and then feed them to run-tests, perhaps something like this (drawing from a similar function here, and also from the source code for run-tests):

(defn proj-run-tests
  ([]
    (proj-run-tests *ns*))
  ([& namespaces]
    (do
      (require :reload-all :verbose & namespaces)
      (run-tests & namespaces))))

But I wouldn't worry about that. It turns out that our old friend Leiningen already gives us a way to run all of our tests with the lein test command.

Going back to the test file we created above, if we remove the (run-tests) call from it, we can re-run it with lein test like this:

~/code/musicprogram$ lein test

Testing musicprogram.test.foo

Ran 1 test containing 1 assertion.
0 failures, 0 errors.

One thing to note here is that this command assumes your project files have a particular organization -- namely, that your source files are in src/ and your test files are in test/. It also assumes that the namespace of each file matches what it's called -- so, the namespace of a file src/foo/bar.clj must be foo.bar, and the namespace of test/foo/test/bar.clj must be foo.test.bar. This is how Lein creates your project initially, but it's worth keeping in mind as you flesh out your project.

Conclusion

As long as this post is, I hope I've given you the impression that Clojure is actually pretty easy to get going with. The hard part is actually wrapping your head around Clojure itself, especially if you aren't used to programming in a functional language with immutable data structures. That's for a future blog post, though.☯

For the past couple of months, I've been attempting to make a clone of the client program that ships with Minecraft, using OpenGL (which Minecraft uses as well). At first I wanted to write it in Javascript, but quickly realized that a lot of the server-side Javascript libraries that exist now are still geared toward web stuff, which means there really aren't any mature libraries to work with OpenGL yet. Also: v8 is a royal pain to work with. So I made a decision to take a bit of a turn, go straight to the source, and learn how to work with OpenGL in Java (specifically the LWJGL library).

In my last post I figured out a basic Java project workflow, so now it was time to dive into OpenGL and LWJGL. I found the following resources to be pretty helpful:

• The OpenGL Programming Guide, aka the "Red Book" (this goes up to only OpenGL 1.1, which is totally fine for our purposes)
• LWJGL Java docs
• Basic Game class on the Game Programming wiki
• OpenGL tutorials on the Game Programming wiki
• Basic tutorials on ninjacave.com
• And, of course, the Java 1.6 docs

The first thing I learned is that LWJGL isn't exactly like the OpenGL library you'd use in C/C++. It talks to it (at least I think so), but the API differs. The API for the C library has three main components: core GL functions (glFrustum(), glEnable(), etc.), a set of utility functions (commonly called GLU), and a window system (commonly called GLUT). So what's different about LWJGL? For one, the team behind it has made decisions to keep the library easy to use and within a manageable size. So for instance, all of GLUT is out -- I guess because if you're developing a serious game, you're probably going to use the core API to implement your own stuff anyway. glColor3fv() (in the core API) is out because it's only really useful for C programmers. Also, functions that are overloaded in the C API to accept both arrays and buffers are simplified to only accept buffers. So LWJGL omits some features, but at the same time it supports multiple versions of the OpenGL API. So if you're targeting OpenGL 1.1, you use the GL11 namespace; if you're targeting OpenGL 4.0, then you use the GL40 namespace. Finally, LWJGL provides some helpful additions: classes which I consider core LWJGL, such as Display, Keyboard and Mouse; and also convenience classes, such as Cylinder, Sphere, Vector3f, Matrix4f, and amazingly, Quaternion.

To get going with LWJGL, I wanted to keep things simple and render a basic scene. I'd read somewhere that the "Utah teapot" is the object graphics people have historically used to render a test scene. There's actually a function in GLUT to quickly render this very teapot, but as I just explained, GLUT is missing from LWJGL, so I couldn't do that. So I decided to just draw a cube placed in the middle of the world.

I spent a good while revisiting some of the resources I'd found so far and studying various tutorials. I noticed that the main "Game" class that people gave tended to follow a common pattern, which I present below:

public class Game {
  private static final int WINDOW_WIDTH = ...;
  private static final int WINDOW_HEIGHT = ...;

  public static void main(String[] argv) {
    init();
    mainLoop();
    cleanup();
    System.exit(0);
  }

  private static void init() throws Exception {
    Display.setDisplayMode(new DisplayMode(WINDOW_WIDTH, WINDOW_HEIGHT));
    Display.create();

    // Initialize the projection matrix (viewbox).
    glMatrixMode(GL_PROJECTION);
    glLoadIdentity();
    // ... Use glFrustum(), gluPerspective(), glOrtho() or gluOrtho2D() to initialize the projection matrix ...

    // ... Make other display settings ...
  }

  private static void mainLoop() {
    while (true) {
      if (Keyboard.isKeyDown(Keyboard.KEY_ESCAPE) || Display.isCloseRequested()) {
        break;
      }

      if (Display.isActive()) {
        // Window is active (i.e. in the foreground / currently selected),
        // so we can go full steam.
        handleKeyboard();
        handleMouse();
        render();
      }
      else
      {
        // Window is not active, so let's go light on the CPU.
        try {
          Thread.sleep(100);
        } catch (InterruptedException e) {
          // do nothing
        }

        // Only bother rendering if the window has actually changed.
        if (Display.isVisible() || Display.isDirty()) {
          render();
        }
      }

      Display.update();
    }
  }

  private static void handleKeyboard() {
    // ... Respond to keystrokes ...
  }

  private static void handleMouse() {
    // ... Respond to mouse movement ...
  }

  private static void render() {
    // Clear the screen.
    glClear(...);

    // Rotate or translate the view.
    glMatrixMode(GL_MODELVIEW);
    glLoadIdentity();
    // Now customize it.
    glRotatef(...);
    glTranslatef(...);

    // ... Draw shapes and stuff ...
  }

  private static void cleanup() {
    Display.destroy();
  }
}

I'll briefly explain what's going on here. There are three parts to this: an initialization routine that runs when the game loads, a loop that runs continuously to redraw the game elements, and a routine that runs when the game ends to clean up stuff. In the init() function, we call Display.create(), which draws the window and attaches event handlers for the keyboard and the mouse, and then we initialize the projection matrix (which I will talk about shortly). In mainLoop(), we do three things: we call handleKeyboard() and handleMouse() to handle input events, we call render() which will set the modelview matrix (which I will also talk about shortly) and queue up the elements that need to be drawn, and then we call Display.update() to actually make those elements appear within the window. While we're at it, we optimize for the case in which the window is minimized or not active: we don't draw to the screen if we don't have to. Finally, we have a check that breaks out of the game loop if the escape key is pressed or the close button on the window is clicked. At that point, cleanup() is called, which will call Display.destroy() to close the window and perform any other necessary destruction routines.

I want to go into detail about some of the parts of this skeleton, because I like to understand how things work, and if you're reading this, I'm guessing you do, too. So let's focus on the following code for a while:

private static void init() {
  ...
  // Initialize the projection matrix (viewbox).
  glMatrixMode(GL_PROJECTION);
  glLoadIdentity();
  // ... Use glFrustum(), gluPerspective(), glOrtho() or gluOrtho2D() to customize the projection matrix ...
  ...
}

private static void render() {
  ...
  // Initialize the modelview matrix (camera / objects).
  glMatrixMode(GL_MODELVIEW);
  glLoadIdentity();
  // Now customize it.
  glRotatef(...);
  glTranslatef(...);
  ...
}

As you can tell, what we are concerned with here are the projection and modelview matrices. What are these good for, and how do you use them? Allow me to explain...

Projection and modelview matrices

The projection and modelview matrices are basically tables of values, numbers specifically, that store the transformation state of the viewing area of the scene and the objects within the scene.

Imagine that we're in a room. Perhaps there is a collection of fruit on the floor: apples, pears, bananas, etc. We can pick up an apple and place it anywhere in the scene, and we can turn it any way we like. Now let's say we have a video camera and we are filming the scene. Obviously, this means the scene is framed by what the camera is able to see. We can change what we see in four ways. Behind the camera, we can pan around, we can zoom in and out (conceivably), and we can walk around with it. However, we can also change the hardware on the camera itself -- specifically, the lens. That will distort the picture we see through the camera: a wider lens will cause us to see more, and a narrower one will cause us to see less.

The way that OpenGL works with regard to projection and modelview matrices is similar to the camera analogy. The modelview matrix allows us to apply transformations -- specifically, position and rotation offsets -- to the objects in the scene. The projection matrix allows us to apply transformations -- specifically, field of view adjustments -- to the "camera" which is "filming" the scene.

You may have noticed I didn't say anything about applying position and rotation offsets to the camera itself. The camera analogy, while useful, isn't actually quite correct. A better analogy would be this: let's say you're the Greek god of Earth, Atlas. A bit strange, admittedly, but stick with me here. For whatever reason, you've been confined to live in a box for all eternity. It's pitch black in the box, except for one window cut in the side you're facing from which you can look out to the world beyond. You could even say your view of the world is clipped by the window. So here you sit, window and world before, nowhere to go, sanity waning by the second. One day you decide you've had it -- you're tired of seeing the same thing over and over! You want to see more of the world! What do you do? Well, you can't move closer to the window. But hey, you're Atlas! You can physically move the world around you. So that's what you do. If you want to see left, you move the world right. If you want to see up, you move the world down. If you want to see more of something, you simply pull the world closer to you. In a similar way, you can even rotate the world around you. As you experiment turning the world this way and that, bringing it closer and pushing it away, moving it left and right -- for a split second, you're almost convinced that you're the one doing the moving instead of the world itself. After all, the effect is practically the same, except for the fact that you have to invert the movement and rotation directions. But hey! You're moving around! This excites you to no end, and you traverse the earth forevermore.

As this, er, story illustrates, the modelview matrix is not only used to position and rotate objects in the scene, it's also used to position and rotate the camera around too. But remember that the camera doesn't actually exist -- you just have a window to the world. So if you want to apply transformations to the "camera", you actually apply them to the world, and because of this you have to remember to invert the directions.

So that takes care of the modelview matrix. Let's go back to the projection matrix. I mentioned before that this matrix changes how you see the world, and that you can distort the field of view of the scene as though you have a camera and you can change the lens on that camera. As you can probably guess, this analogy isn't correct, either, but why? Well, in this case, it assumes that you see the world in a perspective projection, where objects farther away appear smaller and objects closer appear larger (and lines that go toward the horizon appear closer together). "Is there another way to see the world?" you ask. Yes, there is, and it's called orthographic projection. It's kind of the opposite of perspective projection: objects will appear the same size regardless of how far away or close they are. Essentially, in this mode, one dimension is removed from your three-dimensional scene, and so you end up with a two-dimensional picture (which can look quite odd if you are expecting the other dimension).

So there are different modes that change how the world is viewed, which means there must be a more accurate way to look at the OpenGL projection model. This would be the viewing area, or the "viewbox" as I like to call it. Let's take a step back and imagine that everything we see in the scene -- how far we can see and how wide we can see -- is enclosed by a box. Furthermore, imagine that how we perceive reality, specifically how near and far things appear, is governed by the shape of that box. For our everyday perspective view, what would the shape of this box be? It turns out that it is a frustum, which is kind of like a pyramid on its side, with the pointy end cut off. If "I" am looking down the box -- because there's no way I can look outside it, because it encloses whatever I see -- the smaller end of the box is closer to me, and the bigger end of the box, the end that slants outward, is farther away. So if there is an object in my view, and it starts at the smaller end of the box and moves along the box -- as it moves into the screen -- it will appear to shrink in size because it is taking up less space proportionally inside the box. (It's a bit backwards, I know, but realize that what OpenGL is trying to do here is take the location of this object and map it to a pixel location on your screen.)

So that creates a perspective effect. What would it look like if our viewbox were "normal" -- that is, if none of its edges were slanted, if it were a perfect rectangular solid? Well, then we'd have orthographic projection, because as objects moved through the box, they wouldn't grow or shrink, they'd remain exactly the same size.

For more on the projection and modelview matrices in OpenGL, see:

• Chapter 3 in the Red Book
• Song Ho Ahn's "OpenGL Projection Matrix" article
• "OpenGL Rendering" on euclideanspace.com

Matrices

Now that you know more about the projection and modelview matrices in OpenGL, let's return to the code snippet I gave earlier:

private static void init() {
  ...
  // Initialize the projection matrix (viewbox).
  glMatrixMode(GL_PROJECTION);
  glLoadIdentity();
  // ... Use glFrustum(), gluPerspective(), glOrtho() or gluOrtho2D() to customize the projection matrix ...
  ...
}

private static void render() {
  ...
  // Initialize the modelview matrix (camera / objects).
  glMatrixMode(GL_MODELVIEW);
  glLoadIdentity();
  // Now customize it.
  glRotatef(...);
  glTranslatef(...);
  ...
}

That brings us to the glLoadIdentity() method -- what does this do? To understand this, I'm going to talk for a bit about matrices.

A matrix is a mathematical concept that is nothing more than a table of values. A matrix can be of any size. 4×4 matrices are common in OpenGL programming, so here is an example of one:

Logically, a matrix is divided into rows and columns. You read and refer to values within matrices from top to bottom, left to right. So to locate 5.0 in the matrix I just gave, we first find the row, then the column; in this case it's at row 1, column 3 (which we can write numerous ways, such as {{math::inline}}M_{1,3}{{/math::inline}}).

In OpenGL, a matrix is typically represents a sequence of vectors. I may talk about vectors later, but for now, I'll say that you are probably used to seeing vectors in tuple notation, like this: {{math::inline}}(1, 2, 3){{/math::inline}}. This tuple we can also write it as a matrix -- either as a column vector like this:

or as a row vector like this:

Usually matrices are read as a sequence of row vectors, but OpenGL matrices are read as a sequence of column vectors. (The technical name for this is column-major order, as opposed to row-major order.)

There are several operations you can perform on matrices. You can add two matrices together:

or subtract them (which of course is just the inverse of adding them):

Also, you can multiply a matrix by a singular value. This has the effect of scaling the matrix (and for this reason the other value is called a scalar, and this type of multiplication is called scalar multiplication):

In the same way, you can also divide a matrix by a scalar value (which is of course the same as multiplying by the reciprocal).

Finally, you can also multiply two matrices together. This is the main way you'll work with matrices in OpenGL, so you want to make sure you have this down pat. Unfortunately the way you typically see it is rather confusing, but I'll show you it anyway so you know what to look for. So let's say we have this equation:

As you can see, multiplying two 3×3 matrices will produce a third 3×3 matrix. You'll see why in a second, but to find the product, the idea here is that you start by taking the first row vector in the first matrix and the first column vector in the second matrix, and compute the dot product between them to obtain a scalar value. So in this case it'd look like this:

To find the dot product of two vectors, we multiply corresponding elements of each vector and then sum them up, like this:

Okay, so our first value is 48, and that goes in row 1, column 1 of our resulting matrix:

At this point, I always forget what the next step is. Do I move down to the next row in the first matrix and dot-multiply it by the first column in the second matrix; or do I move across to the second column in the second matrix and dot-multiply it by the first row in the first matrix?

You're probably confused too, so let's start over, this time using a handy trick:

Okay! Now we can clearly see that row vectors in matrix 1 get dot-multiplied by column vectors in matrix 2. We don't have to worry about order -- we just have to fill in the question marks. So let's do that:

Great, now we can simplify the resulting matrix:

And now that we have our answer, we know why multiplying a 3×3 matrix by another 3×3 matrix will produce a third 3×3 matrix.

So that's pretty cool. What if we multiply a 3×3 matrix by, say, a 2×2 matrix? If we use the trick we used above, we can easily find out:

So, a 3×3 matrix times a 2×2 matrix will produce a 2×3 matrix.

Okay, now for something a bit different. Let's say we take the 3×3 matrix we just used in the last example and multiply it by the following 3×3 matrix:

Let's find out:

And simplifying:

How about that! We got the same matrix we started with. This isn't coincidence. It turns out the new matrix we used as a factor in this example has a special name: the identity matrix. As you can guess, it's called this because if you take any matrix and multiply it by the identity matrix of the same size, you'll get back an identical matrix.

Now, an identity matrix isn't just 3×3; it can be any size, as long as the diagonal values running from top-left to bottom-right are all 1's, and the rest of the values are 0's. So here's a 4×4 identity matrix:

And that's the basics of matrices. If you want to learn more, see:

• Matrix (Wikipedia)
• Matrix multiplication (Wikipedia)
• Introduction to Matrices from the Khan Academy
• Matrix multiplication from the Khan Academy (starting at 4:58)
• Matrix FAQ (Flipcode)

Details of the transformation matrices

So let's return once more to the following lines in our LWJGL skeleton code:

private static void init() {
  ...
  // Initialize the projection matrix (viewbox).
  glMatrixMode(GL_PROJECTION);
  glLoadIdentity();
  // ... Use glFrustum(), gluPerspective(), glOrtho() or gluOrtho2D() to customize the projection matrix ...
  ...
}

private static void render() {
  ...
  // Initialize the modelview matrix (camera / objects).
  glMatrixMode(GL_MODELVIEW);
  glLoadIdentity();
  // Now customize it.
  glRotatef(...);
  glTranslatef(...);
  ...
}

As I mentioned, OpenGL keeps two matrices: one for the projection and one for the modelview. These are 4×4 matrices. The projection matrix, as I've mentioned before, describes the shape of the viewbox. You don't really need to know what the values in the matrix are. The modelview matrix, which keeps track of translation and rotation, is probably what you'll be working with most of the time. Technically, you don't need to know what these values are, either, but it should be interesting to you (at least it is to me). It looks like this:

So you can visualize the modelview matrix as three vectors -- one parallel to the X axis (left), one parallel to the Y axis (up), and one parallel to the Z axis (forward) -- originating from a single point in space (pos).

Now, when you look at this matrix, you may wonder why OpenGL keeps a 4×4 matrix instead of a 3×3 matrix. After all, we have three dimensions here, so you should only need three values for each of left, up, forward, and pos, right? Well, it turns out that OpenGL coordinates actually have four dimensions: {{math::inline}}x{{/math::inline}}, {{math::inline}}y{{/math::inline}}, {{math::inline}}z{{/math::inline}}, and {{math::inline}}w{{/math::inline}}. This {{math::inline}}w{{/math::inline}} coordinate is called the homogenous coordinate. Why do we need it? Remember that one of the basic goals a rendering engine achieves is to take game coordinates and (through some complex math) convert them to screen coordinates. This is called projection. It turns out that using three dimensions is not sufficient to do this properly. In fact, doing so will fail spectacularly in the case where you have to render two parallel lines. Now in math, you know that if two lines are parallel, it means that they will never touch. However, if you ever took an art class where you were given an exercise to draw something in one-point perspective -- perhaps a downtown scene in some fictitious city, where you are looking down the street and there are buildings all along the side -- you know this is not true. The scene, at least in the real world, continues infinitely into the horizon, but obviously we are working with a finite medium, so we need a way to represent this visually. The way to do that is to pick a point on the horizon, the vanishing point, toward which all the lines in the drawing converge.

A rendering engine has to figure out how to draw this sort of scene too; the only difference is that it has to know how to refer to all the points in the scene (so that it can convert them to screen coordinates). This means we have to have a name for the vanishing point (in 3D space). If we had a line going straight down the z axis, conceptually, this would be {{math::inline}}(0, 0, -\infty){{/math::inline}}. Obviously, working with infinity from a mathematical standpoint isn't really feasible, so we need some other way to represent this. This is where the extra {{math::inline}}w{{/math::inline}} coordinate comes into play. It goes like this: if we have a four-dimensional point {{math::inline}}(x, y, z, w){{/math::inline}}, its three-dimensional equivalent is {{math::inline}}(\frac{x}{w}, \frac{y}{w}, \frac{z}{w}){{/math::inline}}. Now look what happens when {{math::inline}}w{{/math::inline}} = 0: we get a three-dimensional vector {{math::inline}}(\frac{x}{0}, \frac{y}{0}, \frac{z}{0}){{/math::inline}}, which, if converted to three dimensions, becomes {{math::inline}}(\infty, \infty, \infty){{/math::inline}}. So OpenGL works with four-dimensional vectors for as long as it can until it needs to convert them to three dimensions.

Applying transformations

So far I've been talking abstractly, but I'm sure you're dying to get to down to business. So what are the ways you can work with the transformation matrices, and what exactly happens?

As there are multiple matrices, logically, the very first thing we have to do is tell OpenGL which one we want to work with. We do this with glMatrixMode():

glMatrixMode(GL_PROJECTION);
...
glMatrixMode(GL_MODELVIEW);
...

(There are actually a couple of other matrices you can use, but we won't worry about those right now.)

Unless you're doing something special, you always want to start your matrix out with the identity matrix, which, if you forgot, looks like this:

So, you can set a transformation matrix to the identity matrix with glLoadIdentity().

At long last, you can apply transformations to each matrix. First is the projection matrix. Earlier I talked about perspective projection and orthographic projection, and how both set the shape of the viewbox in different ways. You can tell OpenGL to use orthographic projection with glOrtho(). Here you simply specify the coordinates of the vertices of the viewbox:

glOrtho(double left, double right, double bottom, double top, double near, double far)

In other words, left and right are x-values, bottom and top are y-values, and near and far are z-values. To switch to perspective projection, you use glFrustum(). Again, you specify the vertices of the viewbox:

glFrustum(double left, double right, double bottom, double top, double near, double far)

Note that in this case, you'll probably have to manually calculate the coordinates so that you end up with a box with properly slanted edges. No one really does this because it's kind of hard; instead people generally use gluPerspective(), from the GLU module, which makes this job much easier. So now instead of vertices, you specify a field-of-view angle, an aspect ratio (which will be the width of the viewbox divided by its height), and then the near and far values:

gluPerspective(double fovy, double aspect, double near, double far)

So that's the projection matrix. What about the modelview matrix? There are three operations you can perform: translation, rotation, and scaling. Translation is pretty easy, it's just the offset position. As you may be working with either floats or doubles, there are versions for each:

glTranslatef(float x, float y, float z)
glTranslated(double x, double y, double z)

Rotation is achieved by specifying an angle and an axis along which you want to perform the rotation as a unit vector (i.e., {{math::inline}}(1, 0, 0){{/math::inline}}, {{math::inline}}(0, 1, 0){{/math::inline}}, or {{math::inline}}(0, 0, 1){{/math::inline}}):

glRotatef(float angle, float x, float y, float z)
glRotated(float angle, double x, double y, double z)

Finally, scaling, where you specify a factor which every coordinate will be multiplied by:

glScalef(float x, float y, float z)
glScaled(double x, double y, double z)

To close this section, I wanted to briefly talk about what happens when you use any of these functions. You might initially think that with, for instance, gluPerspective() or glTranslate(), you are overwriting whatever is in the current matrix. This isn't true, though: you are actually multiplying what is there with other values (using matrix multiplication, which I showed you earlier). This should actually make sense for the modelview matrix, because this is what makes it possible to apply multiple transformations in succession. So you could have something like this:

float x, y, z, rotx, roty, rotz;

glTranslatef(x, y, z);
glRotatef(rotx, 1, 0, 0);
glRotatef(roty, 0, 1, 0);
glRotatef(rotz, 0, 0, 1);
// ... draw your object(s) ...

Now, something you will probably not realize is that when the modelview matrix is applied to the objects in your scene (specifically, their vertices) to calculate their final position and rotation (again using matrix multiplication) the transformation is applied in reverse. What I mean is that if you say

glTranslatef(x, y, z);
glRotatef(rotx, 1, 0, 0);
// ... draw an object ...

the rotation will be applied to your object first, then the translation. In math terms it would look like this:

This may or may not affect your program, but it's definitely something to watch out for.

For completeness, I should also talk about glPushMatrix() and glPopMatrix(). If you want to apply a transform to a single object in your scene without affecting the rest of the scene, you need a way to localize the transformation in your code. You do this with transformation stacks. Basically, the idea is that you call glPushMatrix() which copies the current matrix to a new space in memory and pushes it onto the stack. From there you can apply whatever transformations you'd like, and you can even push another matrix onto the stack if you'd like. When you're done with the matrix, you pop it off with glPopMatrix() and you are returned to the matrix you had before. (You may already be familiar with this from another rendering engine. I know for a fact that you can do this in Canvas, so I think it's pretty common.)

To learn more about the details of the transformation matrices, see:

• Song Ho Ahn's "OpenGL Transformation" article
• "Matrices" at the OpenGL Programming Wiki
• OpenGL Documentation

If you read this far, wow! You are a trooper. Hopefully it was as interesting to you as it was to me. While I can't call myself an expert on OpenGL by any means, I have done quite a bit of research (and I'm learning more every time I revisit the pages I've mentioned), so if anything in this article wasn't clear to you, feel free to contact me and I'll do my best to answer.

That does it for me. In the next post, we're going to talk about handling input and setting up an FPS-style camera/movement system. Stay tuned!☯

Hello again! I'm not dead, I'm still working on the Minecraft client. I just decided not to be quite so diligent about writing a post as I went along. In fact for the past two weeks I've been caught up in figuring out how to implement a typical FPS camera/movement system in OpenGL.

But more on that later -- let me catch you up on what I've gone through so far.

In my last post, I wondered about the possibility of using Clojure and an OpenGL wrapper library called Penumbra to give me a place to start. Unfortunately, that panned out, partly because Penumbra depended on some non-existent libraries, and partly because of my unfamiliarity with Clojure's tools for managing dependencies and, ultimately, the Java ecosystem itself.

I will probably come back to Clojure in some way, but for now, I've decided to go straight to Java and figure it out from that angle (since Java seems to be one of the two languages, the other being C++, that have the most mature bindings for OpenGL). There are two popular libraries that sit on top of OpenGL that are used for making games: JOGL and LWJGL. Minecraft itself actually uses LWJGL (there's even a tagline on the boot menu that reveals this very fact), so I figured, if it's good enough for Minecraft, it's good enough for me. Some of the goals of LWJGL -- debunking the myth that Java is and will always be slow; making it small so it can be used on portable devices; removing unneeded complexity and things game programmers don't need because it makes sense and so that developers can get up and running fast -- certainly seemed like good things. So, I decided, LWJGL was the way to go.

I started out by downloading and running some of the LWJGL demos on potatoland.org. There's a zip file you can download that contains LWJGL already, so following the README inside, in order to run a demo, all I had to do was this:

$ java -cp .:lwjgl.jar:lwjgl_util.jar gldemo/GLApp_Demo_HelloWorld

The demos are pretty decent; they're actually documented, which is much less than I can say about most OpenGL stuff I've seen. Unfortunately a lot of the code depends on convenience classes that do a lot of stuff to configure the world and set up cameras and add vectors and things -- too much if you're just starting out.

So, I went off to look for some simpler examples. There's a page on the LWJGL wiki that links to a few sites, and of those is the tutorials on ninjacave.com, which gave some example code that looked good to try out.

First, though, I wanted to set up a new project in which to run the examples. That meant, of course, I had to download LWJGL. I then made a new project and copied the files into the project. If you do it yourself, when you extract the zip file, you will see two folders, jar/ and native/ -- one contains .jar files and the other contains dynamic libraries which are specific to your OS. You will need to require both in your application -- that's just how it works). So, I just made a lib/ folder in my project and copied both jar/ and native/ into it.

Then I copied the code in the first tutorial on ninjacave ("The Display") and pasted it into a new .java file I'd created in src/. Of course, it's been a while since I've done Java stuff, so I had to re-learn how to compile a Java file and then run it, but I also learned about the classpath and the -D argument. As the page on the LWJGL wiki says, you have to tell Java about not only the LWJGL jars, but also about the path to the native library files as well. These were the magic words:

$ cd src
$ javac -cp .:../lib/jar/lwjgl.jar:../lib/jar/lwjgl_util.jar:../lib/jar/jinput.jar LWJGL_DisplayExample
$ java -cp .:../lib/jar/lwjgl.jar:../lib/jar/lwjgl_util.jar:../lib/jar/jinput.jar -Djava.library.path=../lib/native/macosx LWJGL_DisplayExample

That worked, so I tried out the rest of the ninjacave demos, which worked too. So that was great, except I didn't like the commands I had to type to run a demo -- not only were they unrememberable, but there was a noticeable duplication of the classpath. Surely there was a way I could run one short command and it would run both of these commands. I could write a shell script, I thought, but it seemed like this was one of those sorts of things for which Java people had figured out a good solution early on.

I don't know a whole lot about Java, but do know of two names that I see all the time: Maven and Ant. In working with Clojure, I'd noticed that Maven (which, as you remember, Leiningen uses as its foundation), includes a dependency manager, so right away that told me it was probably more than what I needed, which was just a simple way to run tasks. ¹ So then I looked into Ant, and sure enough, it fit the bill pretty nicely.

As you might have guessed, Ant is basically a version of make for Java (and somewhat like Ruby's version, Rake). Basically, you specify the tasks that you want to run in a file called build.xml. (Yes, you write the tasks in XML, which, coming from Ruby, is definitely an interesting way to give instructions. At the same time, the Ant team has clearly tried to make writing the build.xml short and sweet, which I appreciate.) At the top level of the XML is a <project> node which tells Ant a little about your project, and under that are <target> nodes which are like make targets in that they let you group tasks together into steps. Ant provides a plethora of tasks you can run, from things like filesystem operations to compiling source files and creating jars.

So, taking a step back, what did I want to be able to do? I basically needed a "compile" task which would call javac and a "run" task that would call java, and they needed to call javac and java exactly how I'd called them before. I also needed "clean" and "prepare" tasks which would ensure that the directory in which the .class files would be generated, build/, was cleaned out before the compile task ran. With this in mind I came up with a simple build.xml:

<project name="lwjgl_tests" default="jar">
  <target name="init">
    <property name="sourceDir" value="src" />
    <property name="libDir" value="lib" />
    <property name="outputDir" value="build" />
  </target>

  <target name="clean" depends="init">
    <deltree dir="${outputDir}" />
  </target>

  <target name="prepare" depends="clean">
    <mkdir dir="${outputDir}" />
  </target>

  <target name="compile" depends="prepare">
    <javac srcdir="${sourceDir}" destdir="${outputDir}" classpath="${libDir}/jar/lwjgl.jar:${libDir}/jar/lwjgl_util.jar:${libDir}/jar/jinput.jar" />
  </target>

  <target name="jar" depends="compile">
    <jar destfile="lwjgl_tests.jar" includes="${outputDir}/**/*.class" />
  </target>
</project>

One small problem: I'd called javac before with a -D argument. How could I specify that? Well, Java calls this a system property. The documentation for the "jar" task shows that there's a way to specify this, so my build.xml changed to this:

<project name="lwjgl_tests" default="jar">
  <target name="init">
    <property name="sourceDir" value="src" />
    <property name="libDir" value="lib" />
    <property name="outputDir" value="build" />
  </target>

  <target name="clean" depends="init">
    <deltree dir="${outputDir}" />
  </target>

  <target name="prepare" depends="clean">
    <mkdir dir="${outputDir}" />
  </target>

  <target name="compile" depends="prepare">
    <javac srcdir="${sourceDir}" destdir="${outputDir}" classpath="${libDir}/jar/lwjgl.jar:${libDir}/jar/lwjgl_util.jar:${libDir}/jar/jinput.jar" />
  </target>

  <target name="jar" depends="compile">
    <jar destfile="lwjgl_tests.jar" includes="${outputDir}/**/*.class" />
  </target>

  <target name="run">
    <java classname="lwjgl_tests" classpath="lwjgl_tests.jar" fork="true">
      <sysproperty key="java.library.path" value="${libDir}/native/macosx" />
    </java>
  </target>
</project>

Okay, cool. Now I could run ant, and that would hopefully generate the jar file:

$ ant
Buildfile: /Users/elliot/code/personal/test/java/lwjgl_tests/build.xml

init:

clean:
  [deltree] DEPRECATED - The deltree task is deprecated.  Use delete instead.

prepare:
    [mkdir] Created dir: /Users/elliot/code/personal/test/java/lwjgl_tests/build

compile:
    [javac] /Users/elliot/code/personal/test/java/lwjgl_tests/build.xml:17: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds
    [javac] Compiling 2 source files to /Users/elliot/code/personal/test/java/lwjgl_tests/build

jar:
    [jar] Building jar: /Users/elliot/code/personal/test/java/lwjgl_tests/build/jar/lwjgl_tests.jar

BUILD SUCCESSFUL

I checked build/jar/ to confirm that indeed lwjgl_tests.jar had been written to it. So it worked!

About the warnings, I wasn't sure what includeantruntime meant, or if that was even important, but the deltree task was at least easy to change:

<project name="lwjgl_tests" default="jar">
  <target name="init">
    <property name="sourceDir" value="src" />
    <property name="libDir" value="lib" />
    <property name="outputDir" value="build" />
  </target>

  <target name="clean" depends="init">
    <delete dir="${outputDir}" />
  </target>

  <target name="prepare" depends="clean">
    <mkdir dir="${outputDir}" />
  </target>

  <target name="compile" depends="prepare">
    <javac srcdir="${sourceDir}" destdir="${outputDir}" classpath="${libDir}/jar/lwjgl.jar:${libDir}/jar/lwjgl_util.jar:${libDir}/jar/jinput.jar" verbose="true" />
  </target>

  <target name="jar" depends="compile">
    <jar destfile="lwjgl_tests.jar" includes="${outputDir}/**/*.class" />
  </target>

  <target name="run">
    <java classname="lwjgl_tests" classpath="lwjgl_tests.jar" fork="true">
      <sysproperty key="java.library.path" value="${libDir}/native/macosx" />
    </java>
  </target>
</project>

Cool. So now that I had a jar generated, I could run it:

$ ant run
run:
     [java] Exception in thread "main" java.lang.NoClassDefFoundError: lwjgl_tests
     [java] Java Result: 1

Oops! That didn't work because I had told clearly told Ant in the java task that the name of the class within the jar it was to run was called "lwjgl_tests", which was not the case.

Actually, since I had a collection of Java files in my project, the class name was variant, so there wasn't any point in hard-coding it in build.xml. Unfortunately, the java task, as detailed in the Ant docs, requires that you pass either a classname or jar attribute. Obviously I couldn't use the classname attribute, but I couldn't use the jar attribute either, since that (sans the classpath attribute) would be equivalent to saying

$ java lwjgl_tests.jar

which assumes that your jar file has a Main-Class entry in the manifest inside the jar file. In other words, in order for this to work, your jar file must have one class that starts off the program, and in this case, I didn't have one. So I'd have to remove the java task. That was okay, I thought -- I'd just specify the class name on the command line.

So my build.xml now looked like:

<project name="lwjgl_tests" default="jar">
  <target name="init">
    <property name="sourceDir" value="src" />
    <property name="libDir" value="lib" />
    <property name="outputDir" value="build" />
  </target>

  <target name="clean" depends="init">
    <delete dir="${outputDir}" />
  </target>

  <target name="prepare" depends="clean">
    <mkdir dir="${outputDir}" />
  </target>

  <target name="compile" depends="prepare">
    <javac srcdir="${sourceDir}" destdir="${outputDir}" classpath="${libDir}/jar/lwjgl.jar:${libDir}/jar/lwjgl_util.jar:${libDir}/jar/jinput.jar" verbose="true" />
  </target>

  <target name="jar" depends="compile">
    <jar destfile="lwjgl_tests.jar" includes="${outputDir}/**/*.class" />
  </target>
</project>

So, I ran ant again and then tried running the resulting jar file manually:

$ ant
...
BUILD SUCCESSFUL
...
$ java -cp lwjgl_tests.jar LWJGL_DisplayExample
Exception in thread "main" java.lang.NoClassDefFoundError: LWJGL_DisplayExample

What was I doing wrong here? I tried extracting the jar file to get a sense of what was in it:

$ mkdir tmp
$ cd tmp
$ jar xf ../lwjgl_tests.jar

Upon inspection, I realized it just contained a META-INF folder. So basically, it was empty. Hmm.

After thinking about it for a bit, I realized this: in following a tutorial, I had noticed that the jar task in the tutorial had a basedir attribute, but I'd intentionally left it out, because I thought you didn't need it (surely Ant would figure out that the current directory was the base directory). As an experiment, I added it back to build.xml, re-ran ant, and checked the jar again, and it contained class files, as I expected. (So, you do need basedir after all.)

Now my build.xml looked like:

<project name="lwjgl_tests" default="jar">
  <target name="init">
    <property name="sourceDir" value="src" />
    <property name="libDir" value="lib" />
    <property name="outputDir" value="build" />
  </target>

  <target name="clean" depends="init">
    <delete dir="${outputDir}" />
  </target>

  <target name="prepare" depends="clean">
    <mkdir dir="${outputDir}" />
  </target>

  <target name="compile" depends="prepare">
    <javac srcdir="${sourceDir}" destdir="${outputDir}" classpath="${libDir}/jar/lwjgl.jar:${libDir}/jar/lwjgl_util.jar:${libDir}/jar/jinput.jar" verbose="true" />
  </target>

  <target name="jar" depends="compile">
    <jar basedir="." destfile="lwjgl_tests.jar" includes="${outputDir}/**/*.class" />
  </target>
</project>

I then tried running the program again, but got the same error as before:

$ java -cp lwjgl_tests.jar LWJGL_DisplayExample
Exception in thread "main" java.lang.NoClassDefFoundError: LWJGL_DisplayExample

So, it was back to research. After another half an hour, in which I found couple of documents on the Ant site, I basically revamped build.xml and ended up with this:

<project name="lwjgl_tests" default="jar" basedir=".">
  <property name="src.dir"     value="src" />
  <property name="lib.dir"     value="lib" />
  <property name="build.dir"   value="build" />
  <property name="classes.dir" value="${build.dir}/classes" />
  <property name="jar.dir"     value="${build.dir}/jar" />

  <path id="classpath">
    <fileset dir="${lib.dir}/jar" includes="lwjgl.jar,lwjgl_util.jar,jinput.jar" />
  </path>

  <target name="clean">
    <delete dir="${build.dir}" />
  </target>

  <target name="compile" depends="clean">
    <mkdir dir="${classes.dir}" />
    <javac srcdir="${src.dir}" destdir="${classes.dir}" classpathref="classpath" verbose="true" />
  </target>

  <target name="jar" depends="compile">
    <mkdir dir="${jar.dir}" />
    <jar basedir="${classes.dir}" destfile="${jar.dir}/${ant.project.name}.jar" />
  </target>

  <target name="run">
    <java fork="true" classname="${classname}">
      <classpath>
        <path refid="classpath"/>
        <path location="${jar.dir}/${ant.project.name}.jar"/>
      </classpath>
      <sysproperty key="java.library.path" value="${lib.dir}/native/macosx" />
    </java>
  </target>
</project>

With that, the full Ant routine now worked, and I could say:

$ ant run -Dclassname=LWJGL_InputExample

which would clear build/, compile the files in src/ and place the .class files in build/classes/, package them up into a jar within build/jar/, and then run LWJGL_InputExample.java within the jar. If you closely compare this build.xml with the previous version, you may notice the difference, which (apart from an alternate way to specify the classpath) sets the basedir of the jar file to build/classes/ instead of the current directory. I can't say for sure as I can't quite remember, but I think this is what fixed the java command so that it successfully ran.

In any case, I had a build routine working and I could forge ahead with setting up an OpenGL project, which I will talk about in the next post! ☯☯

Resources

• http://www.javaworld.com/jw-10-2000/jw-1020-ant.html?page=1
• http://ideoplex.com/focus/java
• http://ant.apache.org/manual/index.html

Footnotes

In my last post I wanted to start making something with the v8-gl, but ran into a wall because of a hole in the bridge code. In attempting to patch this hole, I got fed up with the lack of documentation that v8 has and eventually decided to ditch v8 altogether and look for another solution.

Maybe writing it in Javascript isn't the best idea. For one, I've already mentioned that v8 has zero documentation, but I'm sure SpiderMonkey ain't roses, either -- it must be a beast, judging from how long it's been around. Anyway, the whole Javascript space is still evolving, and I doubt there are many people who use it as a runtime outside of the browser for serious stuff.

So, what are some other languages that have OpenGL bindings? Well, I was mentioning to some people on Twitter the other day how I've always wanted to learn Clojure. A quick search leads to me to Penumbra, which takes an idiomatic approach to OpenGL integration. Now we're talking! And hey, it looks like more than one person has attempted to make a game in Clojure using OpenGL, anyway. This is really great. Who cares if Clojure runs on the JVM? Again, if I can get something working with relative ease, then I'll be happy. This is an experiment, after all.

So, let's play around with Penumbra. First we need to install Clojure. You can download it from the Clojure web site of course, but there's also a Homebrew formula that's up to date, so I install that. The formula tells me that if I have the repl formula installed, there's a file I might want to use that involves rlwrap. It looks like a wrapper script for something, but I'm not sure what it does, so I just skip it for now.

Immediately I wonder if there's a REPL. There is, but you have to say java -cp clojure.jar clojure.main. I probably won't remember that, so I make a shortcut:

#!/bin/bash
java -cp $(brew --prefix clojure)/clojure.jar clojure.main

I save that to ~/.bin/clj (~/.bin is in my path), chmod +x clj and now I can do:

$ clj
Clojure 1.2.0
user=> (* 12345678 12345678)
152415765279684
user=> "string"
"string"
user=> 'symbol
symbol
user=> :keyword
:keyword

Alright, that's enough of that. Let's see if we can run a Penumbra example -- say, the Asteroids one. Hmm, how do we run a file? Scanning the Clojure site, I see that clojure.main does that, too -- you just give the filename on the command line, after the clojure.main bit. So that's cool, but I don't like how the REPL doesn't have any line history. To do that, you can use JLine, like this:

$ java -cp jline-0_9_5.jar:clojure.jar jline.ConsoleRunner clojure.main

Of course we'll have to provide the full path to clojure.jar, but I imagine that we won't have to do it for JLine if we put it in Java's classpath. ~/Library/Java/Extensions is one of the locations that's already in the classpath, although checking it, I notice I already have it there (I probably did it when I installed JRuby). So, now we have to modify our shell script to support multiple arguments:

#!/bin/bash
java -cp jline-0_9_5.jar:$(brew --prefix clojure)/clojure.jar jline.ConsoleRunner clojure.main $`

And now I can try this:

$ clj

Hmm. That doesn't seem to be working; Java tells me it can't find clojure.main. A thread in the Clojure mailing list suggests that rlwrap is easier (and better) to use instead. There's a guide here. Ah, and at this point I realize that the clojure Homebrew formula already comes with a wrapper script for the REPL, located at /usr/local/Cellar/clojure/1.2.0/bin/clj (and aliased to /usr/local/bin/clj). The file it directed me to earlier replaces this with an rlwrap version (and whoever added this to the formula must've gotten it from the mailing list). So, let's just use that instead:

$ rm ~/.bin/clj

And try again:

$ clj

Unfortunately this doesn't work, either:

Exception in thread "main" java.lang.NoClassDefFoundError: clojure/main

Gragh. Okay, I'm wasting time here -- I can live without line history. Let's just recreate ~/.bin/clj and go with the following so we can get rolling:

#!/bin/bash
java -cp $(brew --prefix clojure)/clojure.jar clojure.main $`

Now we should be able to try out that Asteroids demo:

$ git clone https://github.com/ztellman/penumbra.git
$ cd penumbra
$ clj test/example/game/asteroids.clj
Exception in thread "main" java.io.FileNotFoundException: Could not locate penumbra/opengl__init.class or penumbra/opengl.clj on classpath:  (asteroids.clj:9)

Whoops! Hmm. I guess we'll need to tell Java about src/. That wrapper script wasn't very handy, was it? Let's try this:

$ java -cp $(brew --prefix clojure)/clojure.jar:src clojure.main test/example/game/asteroids.clj
Exception in thread "main" java.io.FileNotFoundException: Could not locate clojure/contrib/def__init.class or clojure/contrib/def.clj on classpath:  (core.clj:9)

Seriously?! (Insert rage face here)

On to the Penumbra mailing list, then. There Zach Tellman suggests to run lein repl. lein, as I learn, is Leiningen, which is kind of a dependency installer / toolkit for Clojure projects. So, I download it like the README says, put it in ~/.bin/lein, and run lein repl from the Penumbra directory. Leiningen goes off and does its magic, which is kind of neat to watch, until I get this error:

$ lein repl
An error has occurred while processing the Maven artifact tasks.
 Diagnosis:

Unable to resolve artifact: Missing:
----------
1) org.clojure:clojure-contrib:jar:1.1.0-master-SNAPSHOT
...

At first I think it just failed to download a file (I was in a coffee shop when I ran this command, so it was possible). Can I retry the command? Sure, with lein deps.

$ lein deps
An error has occurred while processing the Maven artifact tasks.
 Diagnosis:

Unable to resolve artifact: Missing:
----------
1) org.clojure:clojure-contrib:jar:1.1.0-master-SNAPSHOT
...

Frack. Okay, well, there's a Stack Overflow thread that mentions this issue, and one of the answerers said it worked with "today's Leiningen", which I assume means HEAD. So let's just replace ~/.bin/lein with the version at https://github.com/technomancy/leiningen/raw/master/bin/lein and try lein deps again:

$ lein deps
Failed to download https://github.com/downloads/technomancy/leiningen/leiningen-1.5.0-SNAPSHOT-standalone.jar
See README.md for SNAPSHOT build instructions.

So, I look at the README (again), but at this point I really don't want to go through the trouble of using Maven to install Leiningen manually.

Hmm, I just noticed there's a Homebrew package for Leiningen, so let's try that. First I probably want to remove the version I just tried to install, which I can do by removing ~/.lein (or, at least, that removes the version of itself that it caches). I'm going to guess Homebrew's formula installs stuff in the same place. If not, no big deal.

Okay, so I try lein deps one more time:

$ lein deps
Unable to resolve artifact: Missing:
----------
1) org.clojure:clojure-contrib:jar:1.1.0-master-SNAPSHOT
...
  Path to dependency:
    1) org.apache.maven:super-pom:jar:2.0
    2) autodoc:autodoc:jar:0.7.1
    3) org.clojure:clojure-contrib:jar:1.1.0-master-SNAPSHOT

WTF. Srsly.

So I do some more snooping, and find this thread on the Clojure mailing list. It looks like what's going on here is that autodoc 0.7.1 depends on clojure-contrib 1.1.0, which was actually removed from build.clojure.org. It suggests that autodoc 0.8.0 may work, but it's not on Clojars (for who knows what reason). So, there is actually a way to specify that certain subdependencies of dependencies you've put in project.clj that would ordinarily get installed, should not get installed. So for Penumbra, we can edit project.clj, which usually looks like this:

(defproject penumbra "0.6.0-SNAPSHOT"
  :description "An idiomatic wrapper for OpenGL"
  :dependencies [[slick-util "1.0.0"]
                 [cantor "0.2.1"]
                 [org.clojure/clojure "1.2.0"]
                 [org.clojure/clojure-contrib "1.2.0"]]
  :native-dependencies [[penumbra/lwjgl "2.4.2"]]
  :dev-dependencies [[native-deps "1.0.4"]
                     [autodoc "0.7.1"]
                     [lein-clojars "0.5.0-SNAPSHOT"]
                     [leiningen/lein-swank "1.1.0"]])

and force-remove clojure and clojure-contrib from the autodoc dependencies. Through trial and error I notice I also have to do the same for lein-swank too, so we end up with:

(defproject penumbra "0.6.0-SNAPSHOT"
  :description "An idiomatic wrapper for OpenGL"
  :dependencies [[slick-util "1.0.0"]
                 [cantor "0.2.1"]
                 [org.clojure/clojure "1.2.0"]
                 [org.clojure/clojure-contrib "1.2.0"]]
  :native-dependencies [[penumbra/lwjgl "2.4.2"]]
  :dev-dependencies [[native-deps "1.0.4"]
                     [autodoc "0.7.1" :exclusions [org.clojure/clojure-contrib org.clojure/clojure]]
                     [lein-clojars "0.5.0-SNAPSHOT"]
                     [leiningen/lein-swank "1.1.0"]])

(defproject penumbra "0.6.0-SNAPSHOT"
  :description "An idiomatic wrapper for OpenGL"
  :dependencies [[slick-util "1.0.0"]
                 [cantor "0.2.1"]
                 [org.clojure/clojure "1.2.0"]
                 [org.clojure/clojure-contrib "1.2.0"]]
  :native-dependencies [[penumbra/lwjgl "2.4.2"]]
  :dev-dependencies [[native-deps "1.0.4"]
                     [autodoc "0.7.1" :exclusions [org.clojure/clojure-contrib org.clojure/clojure]]
                     [lein-clojars "0.5.0-SNAPSHOT"]
                     [leiningen/lein-swank "1.1.0" :exclusions [org.clojure/clojure-contrib org.clojure/clojure]]])

So, that makes lein deps work. At this point we should be able to finally run the example... right? Nope. Same error as before:

$ java -cp $(brew --prefix clojure)/clojure.jar:src clojure.main test/example/game/asteroids.clj
Exception in thread "main" java.io.FileNotFoundException: Could not locate clojure/contrib/def__init.class or clojure/contrib/def.clj on classpath:  (core.clj:9)

Argh. Am I doing something wrong? Oh, scrolling up in my terminal, it looks like lein placed the dependencies in lib. So I need to add that to the classpath:

$ java -cp $(brew --prefix clojure)/clojure.jar:src:lib clojure.main test/example/game/asteroids.clj
Exception in thread "main" java.io.FileNotFoundException: Could not locate clojure/contrib/def__init.class or clojure/contrib/def.clj on classpath:  (core.clj:9)

That doesn't work neither. But wait! I remember that Zach what's-his-face said we need to run lein repl. So let's do that:

$ lein repl
REPL started; server listening on localhost:17223.
user=>

Well that's... interesting. Obviously, it worked, which is good. So what do we do now? Here the Penumbra mailing list gives me an idea:

$ lein repl
REPL started; server listening on localhost:17223.
user=> (require 'example.game.asteroids)
java.lang.ClassNotFoundException: org.lwjgl.opengl.GL11 (core.clj:9)

Okay, now we're getting somewhere -- that's more of what I was expecting. I guess it would make sense that LWJGL, which is what Penumbra uses, isn't going to get installed automatically.

Unfortunately, I'm out of time, so I'll have to pick this up later. Anyway, it's clear that I'm not going to be able to get Penumbra working until I'm more familiar with lein, or even Java. Maybe I need to see if I can get a quick OpenGL program up and running with Java first (either via JOGL or lwjgl). Then, I can take that and port it straight to Clojure (like this, or this). One way or another, I'll figure this out.☯

In my last post I started thinking about which Javascript engine to use, and concluded that since I haven't had much experience with OpenGL, I'd better start out with that. ~~~ I found a v8-to-OpenGL bridge called v8-gl, which I decided to try out. So, following the instructions I compiled it:

$ cd v8-gl
$ svn co http://v8.googlecode.com/svn/trunk/ v8
$ cd v8
$ scons mode=release
# Whoops! Don't have scons, let's use Homebrew:
$ brew install scons
# Rockin' and rollin'
$ scons mode=release
$ cd ..
$ make

Now I could run the example:

$ ./v8-gl examples/example2.js

(Wow, I thought, it actually worked. That's a first.)

I looked at the other files in examples/. There was one that drew a simple triangle, and one that attempted to use textures (but they didn't show up -- I think the files were missing), and one that drew a spinning icosahedron (which was the same example on the project page), and there's also a rather impressive demo that flew through a tunnel made of metaballs.

The spinning icosahedron was the easiest example to grok. It all looked to me like a straight port from C/C++ code, though. I decided what I really needed was some basics on OpenGL, GLUT, and possibly shaders.

There are, as you might expect, quite a few tutorials on OpenGL by now. I skimmed through a bit of this one and found it to be useful than some of the others, so that's the one I started out with.

One thing I learned is that there are actually two flavors (more like eras) of OpenGL. There's the 1.x/2.x line, in which you start by creating a window, initializing the window, initializing the frame buffer, then you draw some primitives and so forth. OpenGL 3.0 completely rehauled the API, deprecating per-vertex rendering and shading/texturing in favor of vertex buffers and shaders. (1) (2) The thing about it is, there are myriad tutorials on 1.x/2.x-style OpenGL, but not so many on 3.x/4.x. The good thing in this case, I guess, is that v8-gl targets 2.1, so getting started wouldn't be a big deal (I thought).

As I started converting one of the tutorials on the wikibooks area, I realized I didn't know that much about GLUT (which is what actually provides window and keyboard support), so I switched gears a bit and followed a tutorial from that angle. I also found the GLUT docs which were helpful.

Then I ran into a wall. I wanted to be able to close the GLUT window using the Escape key (because when the window opens, the close button is disabled). Now you should be able to do something like this:

var windowId = Glut.createWindow();
...
Glut.destroyWindow(windowId);

But that wasn't working; Glut.destroyWindow() complained that I hadn't given it a window id. So I looked at the v8-gl source, and found that when it wraps glutCreateWindow() in a Javascript function, it doesn't forward the return value of glutCreateWindow() on; Glut.createWindow() just returns nothing!

Handle<Value> GLUTCreateWindowCallback(const Arguments& args) {
  //if less that nbr of formal parameters then do nothing
  if (args.Length() < 1) return v8::Undefined();
  //get arguments
  String::Utf8Value value0(args[0]);
  char* arg0 = *value0;

  //make call
  glutCreateWindow((const char*)arg0);
  return v8::Undefined();
}
...
Glut->Set(String::NewSymbol("createWindow"), FunctionTemplate::New(GLUTCreateWindowCallback));

Ugh. Okay, maybe I could fix this. I'd have to learn more about the v8 C++ API, of course. The v8 project page felt like a good resource. I mean, you'd think that Google would have provided some helpful documentation so that people could hack on it. It's a project they've opened to the public, after all. But the project page is sparse -- there are short videos and articles about the motivation behind the project, and a small example of how to use access part of v8 in your embedding code. But wrapping C++ objects in Javascript objects? Sorry, you're on your own. And typically at this point I turn to the source code, but it's equally unhelpful, too.

Okay. Back to searching. I found this blog post which showed how to make a global context object in Javascript-land, and then take objects and types you've made in C++-land and expose them to Javascript-land. While it was insightful, I got the feeling it wasn't going to help me.

I don't know that much about C++, but I'd gathered enough to know this: I couldn't simply change GLUTCreateWindowCallback() to return glutCreateWindow((const char*)arg0), because then I'd be returning a C++-land object -- I needed to return a Javascript-land object (using v8's API). How one went about doing that, I had no idea.

I spent another half-hour skimming through the v8 mailing list, but I was quickly getting the feeling that v8 is just a bitch to work with. I kept noticing the name of the guy who wrote v8-juice, and he mentioned that v8-juice solved the problem of wrapping C++ objects, so I followed that. The nail in the coffin came when I read his commentary on v8:

Another hindrance is the relative lack of v8 documentation. Most functions and classes aren't documented or have only a single line with no real information (e.g. what exactly does "Enters the context" mean?). This means a great deal of experimentation in order to find out if something will or won't work in v8 (and another amount of guesswork as to whether it's actually legal to do it that way in v8).

Any library which is to enjoy widespread use must be well documented. It's a simple fact. History is full of libraries which thrive despite lacking docs (libexpat comes to mind), but i find that shameful.

I do too.

So, v8 is out the window. It's kind of disappointing, actually; I was looking forward to trying out the speed.

So then we come back to jslibs. I don't really care if runs on SpiderMonkey/JägerMonkey, as long as I can get something up.☯

In my last post I started doing some initial research on writing a Minecraft client using Javascript. I indicated that I might start with the network code -- getting it to send and receive packets to and from a Minecraft server. In thinking about it some more, I realized that if I start with that, it would have to be a proxy rather than a client -- after all, there isn't any point in making a client that can communicate with a server if you can't see the results of that communication.

Anyway, the network code isn't the hardest part; it's the graphics, because I'm planning on using OpenGL and I've never worked with that before. So, I'm going to start there.

Something I haven't addressed yet is which server-side Javascript solution to use. To answer that question it's worth it to know the Javascript engines people are using and the ecosystems around them. Right now it's SpiderMonkey, Rhino, and v8. Both SpiderMonkey and v8 are C/C++-based, while Rhino runs on the JVM. Binding many of the server-side solutions is something called CommonJS, which specifies an API that can be used for things like defining modules, requiring modules and the libraries they contain, and I/O access.

With that in mind, what have we got?

• Clearly the crowd favorite is node.js, which runs on v8 and implements the CommonJS module system. It's mainly targeted for applications where sending and receiving requests concurrently is core to the function of the application, as it uses an event-based concurrency model instead of the traditional thread-based model. Hence, it seems to be best suited for network-based applications that need to handle a high amount of traffic (the prime example being a web server). Even so, there's an insane amount of modules available for this thing already; some are kept in the npm registry, and there are a lot more in this giant list.
• Narwhal -- probably second -- is a Javascript platform that also implements the CommonJS module system. It runs best on Rhino, but it looks like there may also be support for v8 (not sure how mature it is, though). If your aim is to make web apps, Jack and Nitro look pretty neat, and there are some packages for doing web types of things. But I'm not, so let's move on.
• Ringo is a runtime/framework that runs on the JVM, but again, seems to be best suited for web apps. It uses the CommonJS 1.1 module system, with a custom approach to namespacing. There is a port of Processing, though, which is interesting.
• Finally, there's Jaxer, which seems to be kind of a hybrid solution, where you specify the code you want to run on the server side in client-side code. Kind of a neat idea, but otherwise, I'll pass.

I'm sure I've missed some (now that I mention it, there are several more on the CommonJS site, such as GlueScript), but the point is, it appears that all of these solutions are geared for making web applications, which makes sense, since that's where everything is heading these days. However, that's not quite what I want. I don't want the web app framework bits, but I do want the network code (because I am going to be working with the network at some point), and I also want all the CommonJS stuff (or at least some way to access the filesystem and do other basic things). Also, I'd prefer using v8 as opposed to SpiderMonkey or Rhino because let's face it, it's the fastest competitor right now. In that light, it looks like node.js is what I want to start with.

Let's say I didn't use node.js or any of that, though. Let's say I wanted to use v8 directly. What else is there? Well, v8-juice, vu8, and k7 are all very similar in that they basically provide C++ bindings to v8 along with standard library components, with the goal of making it easy to write v8 modules. Interesting, but not what I want, either.

There's also jslibs. This actually looks pretty interesting. It's not on v8 -- it uses SpiderMonkey -- but, it provides support for JägerMonkey engine, Mozilla's new competitor to v8, which is nice. And, there are wrappers for various libraries such as ODE, SDL, OpenGL, OpenAL, etc., which is even better. Maybe the only downside is that it doesn't use CommonJS, but actually, that's not a big deal -- I think the goal of CommonJS is so that, in theory, two platforms that use CommonJS can share code, but in practice I don't think that works so well. Hmm, actually, now that I look at it, the documentation is pretty horrendous. Yeah, I'm going to have a hard time starting out with this.

That brings us back to OpenGL. As I mentioned before, I've never really worked with it before. From the sample code that I've seen, it seems like there's a bit of a learning curve in terms of the API and how you set things up and how you draw things to the screen. I mean, you just have to know how stuff works. I found a v8-to-OpenGL bridge which looks pretty interesting and seems to be mature enough for playing around with. So it looks like that's my next task.☯

I have decided, at the suggestions of my friends Levi and Tim, to see if I can write a Minecraft client using Javascript and OpenGL (or something similar) that will talk to a Minecraft server. I think Javascript is really starting to become a real player, so I'd like to see how it can handle graphics. Also, I've always been interested in game programming, and I've had trouble hooking onto something so far. Minecraft is probably one of the simplest 3D games at the top of the popularity chart in terms of code and graphics complexity, so maybe I have a shot at learning a few things, at least.

Tonight I did some initial research. In poking around the Minecraft forums and other places I have become aware of the huge modding community that exists around Minecraft. People make mods for all sorts of things -- adding other crafting recipes, changing the diffusion of light and how it's rendered, making it so it's day all the time -- you name it. How do these people write these mods? Well, it's all hacking, basically -- the common way to apply a mod is to download a tool that basically inserts code into your copy of Minecraft (the minecraft.jar file, to be exact). So I got thinking, if they're writing Java code, and they know which files in minecraft.jar to patch, then they must know what the .class files in minecraft.jar actually contain, which means they're using a Java decompiler to pop them open. It isn't hard to find one. Unfortunately, even if you crack open the .class files, you'll find that all the code is obfuscated -- Java classes are called "aa", "bb", "cc", etc. and functions are similarly generically named. This is where Minecraft Coder Pack comes in. All you do is drop minecraft.jar into the MCP folder, run a little script to decompile and de-obfuscate it, and voila, you've got all the source code to Minecraft.

So, reading through that was pretty enlightening. There's definitely a lot more code there than I thought. It's a bit much to take in all at once, though.

One of the first problems I need to solve is how the client is going to communicate with the server. It all comes down to simple packets, and there's a custom protocol that client and server adhere to. This is detailed here:

• http://mc.kev009.com/Protocol
• http://www.minecraftwiki.net/wiki/Classic_Server_Protocol

That got me thinking, well, there must be people who've made Minecraft clients, or even servers, before -- maybe I could look at some code for ideas on how to implement the protocol. And of course, there are a jillion projects on the githubs. Here are just a sampling I found:

• http://github.com/nornagon/nodecraft -- server in node.js
• http://github.com/mkroman/metamine -- client in Ruby
• http://github.com/duckicraft/minerubies -- start of a client in Ruby
• http://github.com/calzoneman/uMiner -- server in C#
• http://github.com/MostAwesomeDude/bravo -- server in Python

There are also people who've made proxies that sit between a client and server:

• http://github.com/glguy/minecraft-proxy -- Haskell, ooh!
• http://github.com/aniero/golem -- Ruby client, actually written by a buddy here in town.

So maybe, just to start off, I can write a little proxy that will respond to the arrow keys, hit the server with the right packets, update the player's position on the server, at which point the server will come back and I'll let it pass through to the client, which will update the character on-screen. Something like that, anyway.☯

Let's say you're using Devise in your Rails app, and you need to override some behavior in SessionsController such that an attribute is set on the user record when the user logs in. Something like this:

class SessionsController < Devise::SessionsController
  def create
    resource = warden.authenticate!(:scope => resource_name, :recall => "new")
    resource.foo = "bar"
    set_flash_message :notice, :signed_in
    sign_in_and_redirect(resource_name, resource)
  end
end

So then you've got a controller test; maybe it looks like this:

describe SessionsController do
  describe 'POST #create' do
    it "sets the whatever" do
      user = Factory.create(:user, :email => "joe@bloe.com", :password => "secret")
      post :create, :user => {:email => "joe@bloe.com", :password => "secret"}
      user.reload
      user.foo.should == "bar"
    end
  end
end

So you go to run this test, and then you're clubbed upside the head with this monstrosity:

Failure/Error: post :create, :user => {:email => "joe@bloe.com", :password => "secret"}
AbstractController::ActionNotFound

(commence hair pulling, gnashing of teeth, etc.)

Breathe deep. The action does exist. You're not going crazy.

~~~

Where is that error coming from then? Well, it's actually Devise. See, Devise::Controllers::InternalHelpers#is_devise_resource?, which is added as a before_filter to every Devise controller, does this:

def is_devise_resource?
  raise ActionController::UnknownAction unless devise_mapping
end

Hmm, right. So what does #devise_mapping do? Let's just take a look:

def devise_mapping
  @devise_mapping ||= request.env["devise.mapping"]
end

Ah. That's the answer. In controller tests, request.env["devise.mapping"] doesn't seem to get set. So, to fix this, we just need to set this to something before each test:

before :each do
  request.env["devise.mapping"] = Devise.mappings[:user]
end

(If your resource isn't named "user", you obviously need to change that to the right thing. Consult your routes file if you forgot.)

What if we're going to be overriding multiple Devise controllers? Well, we can simply make a controller example group helper:

module ControllerExampleGroupMethods
  def it_behaves_like_a_devise_controller
    before :each do
      request.env["devise.mapping"] = Devise.mappings[:user]
    end
  end
end

RSpec.configure do |config|
  config.extend(ControllerExampleGroupMethods, :type => :controller)
end

Now we can use it in our controller spec like this:

describe SessionsController do
  it_behaves_like_a_devise_controller

  describe '#POST create' do
    # ...
  end
end

(The reason why we can't simply use a shared example group is that the before(:each) will only take effect in the shared example group, when we want it to take effect in the group we're including it into.)

Anyway, I hope that saves somebody at least two hours.☯

For some reason, I can't ever remember how to do set operations on arrays. Here are the common problems I run into and their solutions.

I want to remove the elements in array A that are present in array B

Good: Use reject + include?.

array1 = [1, 2, 3, 4, 5]
array2 = [0, 4, 5, 7]
array2.reject {|x| array1.include?(x) }  #=> [0, 7]

Better: Use the - operator.

array1 = [1, 2, 3, 4, 5]
array2 = [1, 4, 5, 7]
array2 - array1  #=> [0, 7]

I want to know whether array A contains array B

Good: Use all? + include?.

array1 = [1, 2, 3, 4, 5]
array2 = [3, 4]
array2.all? {|x| array1.include?(x) }  #=> true

Better: Use - + empty?

array1 = [1, 2, 3, 4, 5]
array2 = [3, 4]
(array2 - array1).empty?  #=> true

I want to know whether array A contains array B (partly)

Good: Use any? + include?

array1 = [1, 2, 3, 4, 5]
array2 = [3, 4, 7]
array2.any? {|x| array1.include?(x) }  #=> true

Better: Use & + !empty?.

array1 = [1, 2, 3, 4, 5]
array2 = [3, 4, 7]
!(array2 & array1).empty?  #=> true

I use Ruby 1.8.6 day-to-day. One of these days I'll get a new computer and upgrade to 1.8.7, but in the mean time, that's what I use. Evidently, though, it's not what the people who work on Rack use, because I see this error pop up way too often:

/Library/Ruby/Gems/1.8/gems/rack-1.2.1/lib/rack/utils.rb:138:in `union': can't convert Array into String (TypeError)

Here's the defective passage in Rack::Utils:

ESCAPE_HTML = {
  "&" => "&",
  "<" => "&lt;",
  ">" => "&gt;",
  "'" => "&#39;",
  '"' => "&quot;",
}
ESCAPE_HTML_PATTERN = Regexp.union(ESCAPE_HTML.keys)

The reason why Ruby chokes is that in 1.8.6 and lower, Regexp.union expects regexps or strings as arguments. The ability to pass an array (which is what ESCAPE_HTML.keys is here) was added in 1.8.7.

This bug was cleared up back in June, and here's the Github issue. Unfortunately, it was fixed after 1.2.1 came out and as of this writing, the next release of Rack hasn't been cut yet. Furthermore, nearly every library I've run across that depends on Rack does so without any sort of version dependency, which means if you install one of these libraries, you'll most likely get this error when Rack is loaded. Fortunately the fix is easy:

$ gem uninstall rack
$ gem install rack -v 1.2.0

In the case that the library you're requiring does have a dependency on Rack, well, you'll have to resort to monkey patching. Simply drop this sucker into your code and all will be well.☯

if RUBY_VERSION == "1.8.6"
  # Prior to Ruby 1.8.7, Regexp.union took String or Regexp arguments.
  # (The ability to pass in an array is new in 1.8.7.)
  # Fix this so Rack 1.2.1 will load.
  # See: <https://github.com/rack/rack/issues/issue/29>
  class Regexp
    class << self
      alias_method :__original_union, :union
      def union(*args)
        if args.size == 1 && Array === args.first
          __original_union(*args.first)
        else
          __original_union(*args)
        end
      end
    end
  end
end

I always forget which metacharacters Ruby's version of strftime accepts and what they do. Complicating this is the fact that Ruby's strftime (at least in 1.8.x) tries to be consistent with the strftime in the standard C library, but that actually varies depending on whether you're on Linux (which typically carries the GNU version of libc) or OS X (which essentially carries BSD libc). The thing I always get bit by is that compared to GNU's libc, the BSD version is kinda lame. For instance, the GNU version has a %p and %P option, whereas BSD's has only %p. Also in the GNU version, you can say e.g. %-I and that will return a version of the hour without leading zeroes or spaces, whereas with BSD's you have to go with %l, which omits a leading zero, but still leaves a leading space. To be fair, GNU actually extends strftime beyond the official standard, so there's no reason they should be in BSD, but still, once you get used to these features it's hard going back.

With that in mind I thought it would be informative to run through all the possible metacharacters on OS X and Linux, and see which ones did what. Like this:

time = Time.local(2010, 1, 1, 15, 30, 0)
chars = (("A".."Z").to_a + ("a".."z").to_a)
fmtstr = chars.map {|l| "%%#{l} = %#{l}" }.join("\n")
puts time.strftime(fmtstr)
...

And here are the results:

Out of interest I also decided to tabulate the possible metacharacters on Linux involving the special GNU libc extensions, namely the "-", "_" and "0" prefixes. I did this like so:

time = Time.local(2010, 1, 1, 15, 30, 0)
chars = (("A".."Z").to_a + ("a".."z").to_a).map {|l| ["_#{l}", "-#{l}", "0#{l}"] }.flatten
fmtstr = chars.map {|c| "%%#{c} = %#{c}" }.join("\n")
puts time.strftime(fmtstr)
...

Here are the results (skipping the ones that don't do anything):☯

I wanted to install the Ultraviolet gem, which gives you the ability to syntax-highlight code using Textmate syntax files through the Textpow gem. However textpow in turn depends on Oniguruma (a regex engine that happens to be more efficient than Ruby's built-in version). However, if you install Onigurama you will probably get a bunch of compilation errors and something about not being able to find oniguruma.h. This is because installing the gem only installs the Ruby bindings, but that assumes you already have the Oniguruma header files installed.

So I course need to do that, and I'm faced with the choice: do I a) compile it from source or b) use a package manager like MacPorts or Fink?

On Linux, of course, this is a no-brainer: apt-get install and you're done.

On Mac, neither option is really great. You can compile from source, but if the library has dependencies, you have to install those too, and then what if you want to uninstall it at some point? Good luck with that.

Okay then, what about MacPorts or Fink? Well, the problem with MacPorts is that it insists on installing everything from scratch. If you already have, say, Perl on your system (and you do), well, tough cookies, because it's going to install its own. I remember I tried to install Ghostscript one time and ended up with 15 different other packages. This gets to be unbelievably annoying. Fink is better, but they don't have every package that Debian does (naturally), and anyway, trying to shoehorn a Debian-like package management system into the Mac OS environment just seems kinda weird to me, like trying to mix oil and water.

Enter Homebrew

Homebrew appealed to me because it's so incredibly simple and flexible. Engine Yard has a good writeup here, but basically it's like MacPorts in that nobody hosts any packages -- when you install Homebrew you just install a collection of Ruby files ("formulae"). When you install a package, Homebrew finds its formula and executes the instructions therein, downloading the source and compiling it according to the formula. Where does Homebrew get the source? If you scan through some of the formulae, you'll notice that most of them come from tarballs at public URLs. However, the cool thing is that Homebrew also lets you download source from a git, svn, hg, or other repository. Evidently, this means the packages where this applies, Homebrew lets you upgrade simply by hitting the repo again.

I think the biggest thing I like about Homebrew, though, is this:

[It] doesn’t dupe stuff that comes with OS X or stuff that is provided by RubyGems, CPAN or PyPi.

Hal-le-lu-jah. I thought I would never see the end of that.

So, I'm sure that Homebrew is not perfect since it is still a maturing project, but it seems like a worthy alternative to the madness that is MacPorts or Fink.

Installing Homebrew

Normally I'd be able to follow the instructions in the wiki, but there are a few extra things that I had to do. First, Homebrew places everything in /usr/local. That's not a big deal if you don't have anything already there, but I have a few things that are, such as Ruby and MySQL. Second, Homebrew recommends you do not use sudo to manage packages. The reasoning is that you probably use it without thinking about it, and you don't need root access to install programs anyway, so why bother? I saw someone say the same thing the other day, and it's probably true. sudo was designed for multiuser setups; if you're the only person that uses your computer then it's probably not doing anything to increase safety.

Even with all that it was incredibly easy to install Homebrew. So here's a little walkthrough of what I did.

Pre-flight

First we remove MacPorts:

$ sudo port -f uninstall installed
$ sudo rm -rf \
    /opt/local \
    /Applications/DarwinPorts \
    /Applications/MacPorts \
    /Library/LaunchDaemons/org.macports.* \
    /Library/Receipts/DarwinPorts*.pkg \
    /Library/Receipts/MacPorts*.pkg \
    /Library/StartupItems/DarwinPortsStartup \
    /Library/Tcl/darwinports1.0 \
    /Library/Tcl/macports1.0 \
    ~/.macports

Then Fink:

$ sudo rm -rf /sw

As for the stuff in /usr/local, we can just move it out of the way:

$ cd /usr
$ sudo mv local local.old

Moving /usr/local caused a few problems so let's deal with them now. The immediate problem is that upon opening a new bash session I get an error since ~/.bash_profile is failing to load (it depended on the GNU version of ls which I'd also had installed to /usr/local). So let's get that out of the way:

$ mv ~/.bash_profile ~/.bash_profile.old

Re-creating /usr/local doesn't matter since the Homebrew installation script will do that for us.

Fixing RubyGems

I am back to a stock Ruby installation since my custom Ruby was previously in /usr/local. So, I need to upgrade RubyGems, which is woefully out of date:

$ which gem
/usr/bin/gem
$ gem -v
1.0.1

Yikes.

$ sudo gem update --system
$ gem -v
1.3.6

Much better ;)

I also want RubyGems to install any executables that may come with gems to /Library/Ruby/Gems/1.8/bin instead of /usr/bin. We can do this by placing gem: -n/Library/Ruby/Gems/1.8/bin in our ~/.gemrc file (it's just a hash represented in a YAML file; gem is the key in this case). Then, we need to make sure to add /Library/Ruby/Gems/1.8/bin to our PATH.

Okay, for reals this time

I think that's all. Let's install Homebrew! To do that, simply copy the recommended script, paste it into a file, and run it (remember though: not as root!).

Okay, let's try out some packages:

$ brew install readline
$ brew install git
$ brew install wget

Impressive! I notice that I keep getting a warning about XCode needing to be upgraded to 3.1.4, but it doesn't seem to matter, so I don't worry about it.

Here's one that was pretty painful with MacPorts:

$ brew install imagemagick

Oh man. It caught the dependencies (libjpeg, libgd, libwmf, libtiff, and jasper) automatically without installing a crapload of other things too. I love this thing already.

MySQL

I realize that MySQL had been in its own directory before, /usr/local/mysql. So I find I can simply move that over to the new /usr/local:

$ mv /usr/local.old/mysql /usr/local

The mysql client program will complain because it can't read or write to the database so we need to do:

$ chown -R mysql:mysql /usr/local/mysql

coreutils

One of the essential things I needed immediately was GNU coreutils (because I'm used to GNU ls options). So I simply reinstalled it:

$ brew install coreutils

However, it told me to then run brew install coreutils --aliases > ~/.bashrc. I found that didn't work, so if you install this too, DON'T DO THAT!! Besides, what this will do is add lines like

alias ls="/usr/local/bin/gls"

which I put in ~/.bash_profile, but it didn't work for some reason. So I decided to install symlinks instead. Because I don't want to muddy up /usr/local/bin this time around, I opted to install them to ~/.bin, and then add ~/.bin to my PATH. Then I ran:

$ for command in base64 basename cat chcon chgrp chmod chown chroot cksum comm cp csplit cut date dd df dir dircolors dirname du echo env expand expr factor false fmt fold groups head hostid id install join kill link ln logname ls md5sum mkdir mkfifo mknod mktemp mv nice nl nohup od paste pathchk pinky pr printenv printf ptx pwd readlink rm rmdir runcon seq sha1sum sha224sum sha256sum sha384sum sha512sum shred shuf sleep sort split stat stty sum sync tac tail tee test touch tr true tsort tty uname unexpand uniq unlink uptime users vdir wc who whoami yes "["; do ln -s "/usr/local/bin/g$command" "$command"; done

Anyway...

If you recall we were trying to install the Oniguruma gem. First let's try installing Oniguruma proper. Do we have it?

$ brew search oniguruma
oniguruma

Yes indeedy. Let's do it:

$ brew install oniguruma
==> Downloading http://www.geocities.jp/kosako3/oniguruma/archive/onig-5.9.1.tar.gz
######################################################################### 100.0%
==> ./configure --prefix=/usr/local/Cellar/oniguruma/5.9.1 --disable-debug --disable-dependency-tracking
==> make install
/usr/local/Cellar/oniguruma/5.9.1: 9 files, 800K, built in 25 seconds

Lovely! Now the moment of truth:

$ gem install oniguruma
Building native extensions.  This could take a while...
Successfully installed oniguruma-1.1.0
1 gem installed

Holy crap. That is awesome.☯

Resources

• http://www.engineyard.com/blog/2010/homebrew-os-xs-missing-package-manager/
• http://hugofrappier.wordpress.com/2010/01/01/rails-1-2-x-ruby-1-8-6-snow-leopard-the-missing-link/
• http://wiki.github.com/mxcl/homebrew/gems-eggs-and-perl-modules
• http://blog.zenspider.com/2009/08/gem-path-rubygems-and-you.html

So you've got Home and End working properly in Terminal.app. Now you go to use vi, start editing a file, switch to insert mode, and quickly find out that Home and End don't work.

Yeah, go ahead. Curse Apple. Let it all out.

Feel better? Right then. Let's fix vi.

First, type vi ~/.vimrc. It is essential we edit this file in vi as it gives us the ability to type keystrokes without actually doing what those keystrokes do.

Next, switch to insert mode and type map followed by a space. Now press Ctrl-V, and then press the Home key. Depending on what you've set Home to be in your Terminal.app preferences, you will see a character sequence. I have Home set to Escape-[-H (or \033[H), so I see a blue ^\[ followed by \[H. Now type space, and then type <Home>. This is literally the word "Home" in angle brackets, not the Home key.

Got that? The first "Home" is Ctrl-V, Home. The second "Home" is <Home>.

Once you type that, you will see something like this on-screen (provided you have syntax highlighting on):

map ^[[H <Home>

So what does this do? Well, as you can see here, map allows you to map keys in normal, visual, and operator-pending mode. Really the only thing we're concerned about is normal mode, which is the first mode you're in when you launch vi. So, this line tells vi, "when I press Home, this really means Home and not Escape-[-H."

However that's not really what we want here --- what we want is to map Home in insert mode. We do that with imap. So now type imap, space, Ctrl-V, Home, space, <Home>. This is what I saw after I typed that:

imap ^[[H <Home>

Okay. Let's test it out. Save and exit (:wq) and re-enter vim. Now go into insert mode and try pressing Home and End.

Now repeat the same thing for End. You will also want to add another rule for forward delete. Logically, <Delete> represents the delete key.

This is the full contents of my ~/.vimrc file:

map ^[[H <Home>
imap ^[[H <Home>
map ^E <End>
imap ^E <End>
imap ^D <Delete>

Hope that helped!☯

Recently at work we migrated our code repository from Subversion to Git. You can find numerous guides all over the internet to do this, but none of them covered migrating a repo where the conventional directory structure (trunk, branches, tags) had been introduced halfway into the project. So, I figured it out and wrote a script to do it, and then decided I'd write a walkthrough so that you can follow along if you ever need to do this. Skip down to the script or continue reading...

The repository

To keep things simple, I'm going to be working against a sample SVN repository. Let's set that up now:

$ sudo svnadmin create /var/svn/svn2git-test

Here's the first few commits:

$ svn co file:///var/svn/svn2git-test svn
Checked out revision 0.
$ cd svn
$ echo "Lorem ipsum dolor sit amet" > test.txt
$ svn add test.txt
A         test.txt
$ svn commit -m "Initial commit"
Adding         test.txt
Transmitting file data .
Committed revision 1.
$ echo "Here's another line we added in rev 2" >> test.txt
$ svn commit -m "Update test.txt"
Sending        test.txt
Transmitting file data .
Committed revision 2.
$ echo "And another line we added in rev 3" >> test.txt
$ svn commit -m "Update test.txt, again"
Sending        test.txt
Transmitting file data .
Committed revision 3.

Up to now we don't have a conventional structure yet, we are just putting everything in root. But now we want to make a branch so let's change that:

$ svn mkdir trunk branches tags
A         trunk
A         branches
A         tags
$ svn commit -m "Add trunk, branches, tags"
Adding         branches
Adding         tags
Adding         trunk

Committed revision 4.
$ svn mv test.txt trunk
A         trunk/test.txt
D         test.txt
$ svn commit -m "Move everything to trunk"
Deleting       test.txt
Adding         trunk/test.txt

Committed revision 5.

Before we make the branch, though, we decide to commit something to trunk:

$ echo "Add another line from trunk, rev 6" >> trunk/test.txt
$ svn commit -m "Update test.txt from trunk"
Sending        trunk/test.txt
Transmitting file data .
Committed revision 6.

We also need to make sure revision the trunk folder is associated with is up to date, otherwise our branch will appear in the wrong place when we convert it:

$ svn up
At revision 6.

And now we make the branch:

$ svn copy trunk branches/foo
A         branches/foo
$ svn commit -m "Make 'foo' branch"
Adding         branches/foo
Adding         branches/foo/test.txt

Committed revision 7.

make some changes in the branch:

$ echo "Add another line from foo, rev 8" >> branches/foo/test.txt
$ svn commit -m "Update test.txt from foo"
Sending        branches/foo/test.txt
Transmitting file data .
Committed revision 8.

and go back and make some changes to trunk:

$ echo "Add another line from trunk, rev 9" >> trunk/test.txt
$ svn commit -m "Update test.txt finally, from trunk"
Sending        trunk/test.txt
Transmitting file data .
Committed revision 9.

If you're like me, all you want to see is a picture, so here's a graph of the commit history:

The problem

At some point down the road, we decide that we want to convert this repository to Git. Now most guides these days will tell you to use svn2git. I agree, it's a great little script that takes care of details you don't want to have to worry about to ensure that your repo is converted properly.

However, if we try to use it on the sample repo we created above, we will quickly find that we don't have all of the commits. First let's do the conversion. Note that my version of Git doesn't let you pass a file:// URL to git-svn so we have to use svnserve so we can refer to the repository using svn://:

$ svnserve -d
$ mkdir git1 && cd git1
$ svn2git svn://localhost/var/svn/svn2git-test
Found possible branch point: svn://localhost/var/svn/svn2git-test/trunk => svn://localhost/var/svn/svn2git-test/branches/foo, 4
Found branch parent: (refs/remotes/foo) 2b09a1ad55e4b0b1bf826d3718821f9b065d198c
Following parent with do_switch
Successfully followed parent
Checked out HEAD:
  svn://localhost/var/svn/svn2git-test/trunk r9
Note: moving to 'foo' which isn't a local branch
If you want to create a new branch from this checkout, you may do so
(now or later) by using -b with the checkout command again. Example:
  git checkout -b <new_branch_name>
HEAD is now at 42679bc... Update test.txt from foo
Switched to a new branch 'foo'
Note: moving to 'trunk' which isn't a local branch
If you want to create a new branch from this checkout, you may do so
(now or later) by using -b with the checkout command again. Example:
  git checkout -b <new_branch_name>
HEAD is now at dadf507... Update test.txt finally, from trunk
Switched to a new branch 'master'
Counting objects: 15, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (10/10), done.
Writing objects: 100% (15/15), done.
Total 15 (delta 4), reused 0 (delta 0)

Now let's see open the repository in GitX to see what we've got:

As you can see, the history stops at the point that we split the structure, because git-svn assumes the structure has been the same from the beginning.

The research

I spent a good two days just trying to figure out where to start. A lot of the different things I read involved combining three pieces of history into one repository using grafting and merging, assembling history using git-archive and git_load_dirs, converting an inconsistent trunk using svn2git's --rootistrunk option, a temporary branch, and git-format patch. I also found plenty of solutions for merging one Git repository into another one as a subtree, but that wasn't really what I wanted to do.

However, what ended up being the best resource was a post by this guy, David Wheeler, who'd taken a CVS repo, SVN repo, converted both of them to Git and then mashed them together into one repository. There's a lot of good information on some of the problems he faced, how he solved them, and a few scripts that he wrote to automate the whole process.

Putting aside svn2git for the moment, I knew that I could use git svn init and git svn fetch -r REV1:REV2 to make two copies of the SVN repository, one repository that contained the first half before the split to trunk/branches/tags, and another that contained the last half. The trick was figuring out how to merge these halves into one continuous Git repository.

So, armed with new information, I did a couple of experiments on a sample repository (similar to the one I presented earlier). First I tried using git fetch and git rebase:

$ mkdir git.pre && cd git.pre
$ git svn init svn://localhost/var/svn/svn2git-test
Initialized empty Git repository in /tmp/svn2git/migration_with_rebase/git.pre/.git/
$ git svn fetch -r 1:3
        A    test.txt
r1 = 0a500d79a303cd0a0153e2208d42e825d9f90504 (refs/remotes/git-svn)
        M    test.txt
r2 = 00bfdbf380411ad86ed274aef02e9a28528e0211 (refs/remotes/git-svn)
        M    test.txt
r3 = 6b79e39d6e19256fdbb0a95210570972b349c30b (refs/remotes/git-svn)
Checked out HEAD:
  svn://localhost/var/svn/svn2git-test r3
$ cd ..
$ mkdir git.post && cd git.post
$ git svn init svn://localhost/var/svn/svn2git-test -s
Initialized empty Git repository in /tmp/svn2git/migration_with_rebase/git.pre/.git/
$ git svn fetch -r 5:HEAD
        A    test.txt
r5 = f8a9998b0bd00144c28a82d74a3099d719ef33f4 (refs/remotes/trunk)
        M    test.txt
r6 = a3a5c0896cac7cd8b4d26b48855a7314e4d30d1f (refs/remotes/trunk)
Found possible branch point: svn://localhost/var/svn/svn2git-test/trunk => svn://localhost/var/svn/svn2git-test/branches/foo, 4
        A    test.txt
r7 = 005cc439b5b81c9fcc2ab26ed8d8c60c11993061 (refs/remotes/foo)
        M    test.txt
r8 = 83904e2fd2afae9db7af372c2b3154c33c0a58e7 (refs/remotes/foo)
        M    test.txt
r9 = 28d31c35c2696fb477b2c27bded270a14b3dce56 (refs/remotes/trunk)
Checked out HEAD:
  svn://localhost/var/svn/svn2git-test/trunk r9
$ git fetch ../git.pre master:pre
warning: no common commits
remote: Counting objects: 9, done.
remote: Compressing objects: 100% (5/5), done.
remote: Total 9 (delta 2), reused 0 (delta 0)
Unpacking objects: 100% (9/9), done.
From ../git.pre
 * [new branch]      master     -> pre
$ git branch
* master
  pre
$ git branch -r
  foo
  trunk
$ git rebase pre
First, rewinding head to replay your work on top of it...
Applying: Move everything to trunk
Using index info to reconstruct a base tree...
Falling back to patching base and 3-way merge...
No changes -- Patch already applied.
Applying: Update test.txt from trunk
Applying: Update test.txt finally, from trunk

Basically, I'm attempting to copy all the commits in the first half to the last half, then replaying the commits in the last half on top of the ones in the first half. However, as you can see something was off:

If I'd copied the foo branch to a local branch it might have worked, but I didn't know how to do that yet.

I also tried something similar:

$ mkdir git.post && cd git.post
$ svn2git svn://localhost/var/svn/svn2git-test
Found possible branch point: svn://localhost/var/svn/svn2git-test/trunk => svn://localhost/var/svn/svn2git-test/branches/foo, 4
Found branch parent: (refs/remotes/foo) 2b09a1ad55e4b0b1bf826d3718821f9b065d198c
Following parent with do_switch
Successfully followed parent
Checked out HEAD:
  svn://localhost/var/svn/svn2git-test/trunk r9
Note: moving to 'foo' which isn't a local branch
If you want to create a new branch from this checkout, you may do so
(now or later) by using -b with the checkout command again. Example:
  git checkout -b <new_branch_name>
HEAD is now at 42679bc... Update test.txt from foo
Switched to a new branch 'foo'
Note: moving to 'trunk' which isn't a local branch
If you want to create a new branch from this checkout, you may do so
(now or later) by using -b with the checkout command again. Example:
  git checkout -b <new_branch_name>
HEAD is now at dadf507... Update test.txt finally, from trunk
Switched to a new branch 'master'
Counting objects: 15, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (10/10), done.
Writing objects: 100% (15/15), done.
Total 15 (delta 4), reused 0 (delta 0)
$ cd ..
$ mkdir git.merged && cd git.merged
$ git svn init svn://localhost/var/svn/svn2git-test
Initialized empty Git repository in /tmp/svn2git/migration_with_fetch/git.merged/.git/
$ git svn fetch -r 1:3
        A    test.txt
r1 = 0a500d79a303cd0a0153e2208d42e825d9f90504 (refs/remotes/git-svn)
        M    test.txt
r2 = 00bfdbf380411ad86ed274aef02e9a28528e0211 (refs/remotes/git-svn)
        M    test.txt
r3 = 6b79e39d6e19256fdbb0a95210570972b349c30b (refs/remotes/git-svn)
Checked out HEAD:
  svn://localhost/var/svn/svn2git-test r3
$ cp ../git.post/.git/config ./.git/config # make sure fetch knows where to put trunk, branches, and tags
$ git svn fetch -r 4:HEAD
r4 = 2b09a1ad55e4b0b1bf826d3718821f9b065d198c (refs/remotes/trunk)
        A    test.txt
r5 = 020e7ff72c87b3711486bbffa4230b2fd9ccc266 (refs/remotes/trunk)
        M    test.txt
r6 = 80fdfdd4fc0f69f79f10c114219da4aae97aabce (refs/remotes/trunk)
Found possible branch point: svn://localhost/var/svn/svn2git-test/trunk => svn://localhost/var/svn/svn2git-test/branches/foo, 4
Found branch parent: (refs/remotes/foo) 2b09a1ad55e4b0b1bf826d3718821f9b065d198c
Following parent with do_switch
        A    test.txt
Successfully followed parent
r7 = 56356325a446aa2539a6f78f0e11673972a8a8b6 (refs/remotes/foo)
        M    test.txt
r8 = 42679bc779f0842de4a83e99c9fa718df88e4cc6 (refs/remotes/foo)
        M    test.txt
r9 = dadf507389f0342456e8e5e7367f2aa862ec1b4a (refs/remotes/trunk)

However, this didn't work either, because what were formerly revision 3 and 4 were now disconnected:

What I ended up doing in the end was taking the script David Wheeler wrote to stitch repositories together, running it piece by piece against my sample SVN repository and working out any problems. Once I had that working, I could then run it against the real SVN repository. Before I did this, however, because our repository is rather large, I first copied it to my computer using svnsync:

$ sudo svnsync init /var/svn/store svn+ssh://my@server.com/path/to/real/repo
$ sudo svnsync sync /var/svn/store

I should say that GitX was invaluable in confirming that the new Git repo was complete and that all the history was intact.

The breakdown

I'll have the script that I ended up with at the very end, but because you might like to see how it works, I'm going to break it apart for you, using the sample SVN repo I gave at the very beginning. Note that I'm simplifying what the script does using shell script, but the final script is in Perl.

Here's what we'll be doing in a nutshell:

1. Run master for both halves of the SVN repo (pre-repo and post-repo)
2. Copy commits from post-repo to pre-repo
3. Copy remote branches to local ones on pre-repo
4. Graft pre and post together
5. Copy remote branches to local ones on the final repo
6. Relocate the master branch
7. Clean up (add remote, etc.)

One more thing. I ended up going with the Perl port of svn2git and then modifying it, converting it to an object-oriented version so I could require it in my script and adding a few things like a --revision option. I'll have this at the end too.

One

The first thing we do is run svn2git twice:

$ svn2git svn://localhost/var/svn/svn2git-test git.pre --root-is-trunk -r 1:3 --authors /path/to/authors.txt
$ svn2git svn://localhost/var/svn/svn2git-test git.post --authors /path/to/authors.txt

This will create two Git repositories: the first repo has the part of the SVN repo before the split occurred, the other has the part afterward. I'll be referring to them as "pre-repo" and "post-repo" (or simply "pre" and "post").

At this point here's what the git repos look like:

Two

The next thing we want to do is copy commits (or, in Git parlance, objects) from post-repo to pre-repo. We do this using git fetch:

$ cd git.pre
$ git fetch ../git.post
warning: no common commits
remote: Counting objects: 11, done.
remote: Compressing objects: 100% (4/4), done.
remote: Total 11 (delta 3), reused 11 (delta 3)
Unpacking objects: 100% (11/11), done.
From /tmp/git.post
 * branch            HEAD       -> FETCH_HEAD

Now, this is supposed to copy objects in local branches from post-repo to pre-repo. However, for whatever reason it didn't do that for me. However, that command isn't totally useful. As git has told you, FETCH_HEAD now points to HEAD in the post-repo, which happens to point to post-repo's master branch. This will prove to be valuable later. So let's save that in a temporary branch:

$ git checkout -b post-master FETCH_HEAD
Switched to a new branch 'post-master'
$ git checkout master
Switched to branch 'master'

Since the first git fetch didn't do anything, let's try copying objects in remote branches from post to pre. Basically we get a list of the remote branches from post-repo using git branch -r, and for each branch we say something like this:

git fetch ../git.post refs/remotes/$branch:refs/remotes/$branch

Let's do that for the sample repo:

$ git fetch ../git.post refs/remotes/foo:refs/remotes/foo
remote: Counting objects: 6, done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 4 (delta 1), reused 2 (delta 0)
Unpacking objects: 100% (4/4), done.
From /tmp/git.post
 * [new branch]      foo        -> foo

I can't find any place that explains quite what that colon syntax does better than man git-fetch does, but I think here we're telling Git, "Hey, be a doll and look at post-repo, find a remote branch called foo, and copy all the commits inside to the remote branch foo in pre-repo."

At this point here's what the pre-repo looks like:

Three

If you take a closer look at the screenshot above, you will notice that master is colored orange and foo is colored blue. This is because foo is a remote branch and master is a local branch:

$ cd git.pre
$ git branch
* master
  post-master
$ git branch -r
  foo

So, we need to copy the remote branches over to local branches. We can do that by looping through the remote branches, stripping "origin/" from the branch names if it's present, and saying something like:

git branch --no-track $branch refs/remotes/$branch

Let's do that for the sample repo:

$ git checkout foo
Note: moving to 'foo' which isn't a local branch
If you want to create a new branch from this checkout, you may do so
(now or later) by using -b with the checkout command again. Example:
  git checkout -b <new_branch_name>
HEAD is now at 23a450b... Update test.txt from foo
$ git checkout -b foo

At this point here's what pre-repo looks like:

As you can see, there's now a green "foo" next to the blue "foo", since foo is now a local branch. We can double-check this through the command line:

$ cd git.pre
$ git branch
  foo
* master
  post-master

Four

You might think that everything is okay with the repository, judging by the screenshot above. There's another problem with it, however. Do you see it? There isn't a line connecting "Update test.txt, again" and "Add trunk, branches, tags". This is because, in fact, they aren't connected. The first half is the master branch -- you can see that master points to the end of it. The second half, though, isn't on the master branch, it's just kind of floating around in the repository. (post-master points to it, but forget that's there.) We can confirm this with git log:

$ cd git.pre
$ git checkout master
Already on 'master'
$ git log
commit ae7596f38fefc8a53bf6f2018391a8c5ff3c0379
Author: elliot <elliot@445a800c-2d3f-4b84-95f0-4d0e8c338740>
Date:   Sun Jan 31 03:02:26 2010 +0000

    Update test.txt, again

commit bead1ec713a5361b0988cc71cbaf08b9c4a1462b
Author: elliot <elliot@445a800c-2d3f-4b84-95f0-4d0e8c338740>
Date:   Sun Jan 31 03:02:25 2010 +0000

    Update test.txt

commit e89a208a2b1aa2bf5b569a4d4e3fbfd23e138615
Author: elliot <elliot@445a800c-2d3f-4b84-95f0-4d0e8c338740>
Date:   Sun Jan 31 03:02:24 2010 +0000

    Initial commit

So, that's obviously not good. How do we fix this? We can graft the two histories together using a file that's special to Git, .git/info/grafts. Each line in the file specifies a connection that looks something like this:

<commit id> <parent id> [<parent id>]*

So if we were to take another look at our repo in GitX and click on the commit where post-repo starts, we would see that its commit id is d2874583232277a9e8e425a80e1670e9b1193013. If we click on the commit below, we would see that its commit id is ae7596f38fefc8a53bf6f2018391a8c5ff3c0379. So in order to create the connection between them, all we have to do is open .git/info/grafts and add this:

d2874583232277a9e8e425a80e1670e9b1193013 ae7596f38fefc8a53bf6f2018391a8c5ff3c0379

The cool thing is that if we save this file, refresh the view in GitX, scroll down and find the split again, we should see that the commits are connected now:

This connection is temporary, however; we won't be able to push the repository until we apply the change to the repository. To do that, we use git filter-branch. Not only will this create the connections from .git/info/grafts, it will also regenerate all the commit ids for every subsequent commit after the new connections. ¹

Something worth noting is that git filter-branch by itself only applies the change to the branch that you're on. If you're only concerned about master, that's all right, but if you have any branches that come off of master after the split, like we do, those won't be properly rewritten. So to rewrite everything we say this:

git filter-branch --tag-name-filter cat -- --all

So we run that, and then we make sure to

rm .git/info/grafts

so that it doesn't interfere with future changes to pre-repo.

Now, it turns out that git filter-branch saves old commits as it rewrites everything (I suppose in case you want to roll it back or something, I'm not sure). As you can see it looks pretty funky:

To correct this, all we have to do is clone the repo:

git clone file:///tmp/git.pre /path/to/git.final

As David Wheeler points out, it's important to use file:// here to get a copy of everything, otherwise some of the copied files end up as hard links.

The cloned repo will look like:

Five

Since we've cloned the repository, we need to copy remote branches to local ones again. However, this time we're going to actually remove the remote branches since we don't want them to interfere with anything when we push the repository to its final place at the end.

This works pretty much the same way as step two (git branch --no-track $branch refs/remotes/$branch), except that we need to make sure we aren't copying over the master branch, since we already have that locally.

Six

So let's review: we've created two Git repos from the pre-split and post-split parts of the SVN repo, copied over commits from post to pre, copied remote branches to local branches on pre, grafted pre and post together, cloned that into a final repository, and re-copied remote branches to local branches in the final repo.

Okay, so let's inspect the repo:

$ cd git.final
$ git checkout master
Already on 'master'
$ git log
commit ae7596f38fefc8a53bf6f2018391a8c5ff3c0379
Author: elliot <elliot@445a800c-2d3f-4b84-95f0-4d0e8c338740>
Date:   Sun Jan 31 03:02:26 2010 +0000

    Update test.txt, again

commit bead1ec713a5361b0988cc71cbaf08b9c4a1462b
Author: elliot <elliot@445a800c-2d3f-4b84-95f0-4d0e8c338740>
Date:   Sun Jan 31 03:02:25 2010 +0000

    Update test.txt

commit e89a208a2b1aa2bf5b569a4d4e3fbfd23e138615
Author: elliot <elliot@445a800c-2d3f-4b84-95f0-4d0e8c338740>
Date:   Sun Jan 31 03:02:24 2010 +0000

    Initial commit

Hmm. That's weird. The history still only goes up to pre-split. But we grafted the two halves together -- that should have been enough, right?

Just to make sure, let's pull up the repo in GitX and see what we have:

Okay, now we can see that everything's there, but master still points to the commit before the split. Is there a way we can move it? Yes -- but we have to tell git exactly where it should move master to. You might think it's easy -- can't you just tell Git to find the very latest commit and move master to that? Unfortunately not, because what if the latest commit is on a branch, and the branch has been modified beyond master?

So we have to go with a different tactic. At some point, something must have pointed to the last commit on post-repo. Of course! It was post's master branch that pointed to its last commit. Is there a way we can find out which commit that points to? Probably, but unfortunately, even if we had that commit id, it would be invalid due to git filter-branch having rewritten the history. However, that doesn't mean we can't have stored it in a branch before said history is rewritten. If you remember way back in step one, we did this:

$ git fetch ../git.post
$ git checkout -b post-master FETCH_HEAD

Now you know why. We've have had access to that through the post-master branch all along (it's been carried over through the steps). Now we can simply write:

$ git checkout post-master
Switched to branch 'post-master'
$ git branch -D master
Deleted branch master (was ae7596f).
$ git checkout -b master
Switched to a new branch 'master'
$ git branch -D post-master
Deleted branch post-master (was 2ca81cf).

And just like that, now master points to the very end:

Seven

All that's left now is some cleanup to reduce the size of the repository:

$ git gc
$ git repack -a -d -f --depth 50 --window 50

While we're at it we can add the URL for the remote repo on our server:

$ git remote rm origin
$ git remote add origin ssh://you@yourserver.com/path/to/repository

And we're done!

Pushing the final repo

I wanted to say a few words about our Git setup. If you don't want to have to pay for Github it's actually very easy to host your Git repository right on your server. We followed this guide to set up a git user and set permissions on the directory that holds our Git repos. Assuming you have that, any time you create a new project, you can initialize a Git repo on your server using the following commands (taken partially from that and also this post):

su git
cd /path/to/git/repos_dir
umask 007
mkdir proj.git && cd proj.git
git init --bare
git config core.sharedrepository 1
git config receive.denyNonFastforwards true
find objects -type d -exec chmod 02770 {} \;
chmod -R o-rwx .
chmod -R g=u .

Notice that we initialize a bare repository. In a non-bare repository, all of the files that hold your project's history is stored in a special .git folder inside your project. In a bare repository, the folder is the .git folder. It's not totally necessary, but since we'll never be modifying the repo on the server, it saves some disk space.

Now that we have that, all we have to do to push our newly converted repository to the server is this:

$ git push origin --all

The script(s)

You can find my stitch script as well as my fork of svn2git in this gist. The stitch script I've put in a gist because it's probably very likely you'll need to change it to fit your repository. For instance, we didn't have any tags in our repository so the stitch script doesn't account for that. Also the svn2git fork should probably go on CPAN, but I may do that at a later time.

That's all! If you have any questions, leave them in the comments. ☯

Footnotes ☯

tl;dr -- If you're (still) using MacPorts, and you have LIBRARY_PATH and LD_LIBRARY_PATH to /opt/local, you may want to temporarily (or permanently) unset these when you install a ruby using RVM, especially if you've customized which architecture MacPorts uses.

I'm using RVM 0.0.99 on OS X Leopard. I want to install Rubinius, so I try running rvm install rbx, and this is what I get:

<e> Error running '/Users/elliot/.rvm/ruby-1.8.7-p248/bin/gem install rake --no-rdoc --no-ri', please check /Users/elliot/.rvm/log/ruby-1.8.7-p248/gems.install.error.log </e>

Hmm. That's weird. Let's see what gems.install.error.log says:

[2010-01-10 16:52:20] /Users/elliot/.rvm/ruby-1.8.7-p248/bin/gem install rake --no-rdoc --no-ri
/Users/elliot/.rvm/ruby-1.8.7-p248/lib/ruby/1.8/openssl/ssl.rb:31: [BUG] Bus Error
ruby 1.8.7 (2009-12-24 patchlevel 248) [i686-darwin9.8.0]

/Users/elliot/.rvm/scripts/utility: line 127: 11569 Abort trap             /Users/elliot/.rvm/ruby-1.8.7-p248/bin/gem install rake --no-rdoc --no-ri

What? A bus error? Caused by running gem install rake? Sure enough:

$ rvm 1.8.7
$ gem install rake
/Users/elliot/.rvm/ruby-1.8.7-p248/lib/ruby/1.8/openssl/ssl.rb:31: [BUG] Bus Error
ruby 1.8.7 (2009-12-24 patchlevel 248) [i686-darwin9.8.0]

It's weird that I'd have a problem all of a suddenly installing something via RVM, because I already have 1.8.7, 1.9.1 and 1.9.2 and I didn't have any problems installing those.

Looking around the interwebs, some people say to remove thin or eventmachine. Unfortunately I don't have either since this is a fresh Ruby install. Wayne Seguin, the author of RVM, suggested that this points to an architecture mismatch, which kind of makes sense, because in ~/.rvmrc I have rvm_archflags set to -arch i386 (although that really shouldn't cause any issues). In any case unsetting that doesn't help.

I land on this post and suddenly have an idea:

$ port installed | grep openssl
   openssl @0.9.8l_0+darwin+universal (active)

Oh. I remember now a while back I was having issues with Nokogiri and libxml2, so I went back and re-installed all my ports as +universal. I don't think that's quite the issue, though --- I think it's that when RVM goes to compile Ruby 1.8.7, it's using MacPorts' openssl, instead of (I guess) compiling it from scratch. A quick check in ~/.bashrc and I realize now that I also modified LIBRARY_PATH and LD_LIBRARY_PATH to include /opt/local/. I'll remove that and then recompile Ruby 1.8.7:

$ rvm install 1.8.7
[...]
Error running 'make ', please check /Users/elliot/.rvm/log/ruby-1.8.7-p248/make.error.log

Um, hmm. make.error.log says:

readline.c: In function ‘filename_completion_proc_call’:
readline.c:703: error: ‘filename_completion_function’ undeclared (first use in this function)
readline.c:703: error: (Each undeclared identifier is reported only once
readline.c:703: error: for each function it appears in.)
readline.c:703: warning: assignment makes pointer from integer without a cast
readline.c: In function ‘username_completion_proc_call’:
readline.c:730: error: ‘username_completion_function’ undeclared (first use in this function)

Ugh. Right. I get this quite a lot, and I always forget how I fix it (it probably has to do with MacPorts too). Fortunately there's something on the RVM troubleshooting page about this:

$ rvm install readline

Now:

$ rvm install 1.8.7
[...]
Error running 'make ', please check /Users/elliot/.rvm/log/ruby-1.8.7-p248/make.error.log

Gah! Okay, one last resort:

$ rvm install 1.8.7 -C --with-readline-dir=/Users/elliot/.rvm/usr

Yay -- that works, and now I can install Rubinius.☯