JoelPM

Curl with Cookies and Headers

2010-06-17T00:00:00-07:00

Every now and then I find myself needing to make an http request with specific cookies and headers to help debug an issue. And every time I resort to the curl manpage and try a few times until I get the right incantation. So, to save myself time in the future, here is an example:

curl -v --cookie "cookieName=cookieValue" --header "Accept-Language: en" --header "X-Forwarded-For: 123.123.123.123" "http://localhost:8080/somepage"

-v
Tells curl to be verbose, useful for seeing if your headers are being sent.
--cookie "cookieName=cookieValue"
Tells curl to send the cookie header with the given value. Multiple cookies can be sent using "cookieName1=cookieValue1;cookieName2=cookieValue2".
--header "headerName: headerValue"
Tells curl to add the header line you specified. You can send multiple headers by giving multiple --header arguments (see above).

The ability to hand-craft HTTP requests is a great feature of curl and makes it a great debugging tool.

EC2 Storms

2010-02-12T00:00:00-08:00

Below is a graph of latency for requests coming into three load balancers (the dashed line is the latency for requests to the domain that fronts the three LBs, which does DNS round-robin to the three load balancers). As you can see, requests to two of the three load balancers are extremely slow (the orange and blue lines), while requests to the third load balancer (the green line) are consistently fast.

All three load balancers front the same set of back-end machines. The only difference is that the green load balancer is in the same 'availability zone' as the back-ends, while the blue and orange load balancers are in different availability zones. Normally this isn't a problem, but earlier this week we began to see this type of deterioration. At the time, though, we only tracked latency to the domain, not individually for the three load balancers. The degradation also happened to correspond with a spike in traffic, so initially I assumed the problem was with my back-end code.

Lots of investigation ruled out the back-ends, though, at which point we started looking elsewhere, eventually coming to the conclusion that it had to do with cross-availability-zone traffic. Lots of time down the drain, all because something changed in EC2 land.

This is one of the drawbacks of deploying at large scale on EC2 - the network is a complete black box. Over the last six months I've spent lots of time fighting fires. I've come to call events like these "EC2 Storms" because out of the blue, with absolutely no changes to the software or configuration, everything will suddenly tank. A few hours or days later things clear up again and everything looks fine.

Maven and YUICompressor

2009-07-15T00:00:00-07:00

Here's how I use YUICompressor to minify/obfuscate our Javascript during Maven's compile phase.

First, add YUICompressor to your pom as dependency:

Once YUICompressor is listed as a dependency we can use the ant-run plugin to invoke it during the build:

What we're doing above is hooking the ant-run plugin to the compile phase, creating an ant property that contains the path to yuicompressor.jar, and then invoking a task in an Ant build file called yuicompress.xml. Make sure you specify version 1.3 of the ant-run plugin (or newer) - older versions didn't make the Maven dependency paths available to Ant. The ant file looks like this:

In my case I'm compressing Freemarker templates that contain JS, which is why I specified the type as JS. These are in a subdirectory of my resources directory called 'templates.' I have YUICompressor output the compressed version to the corresponding location in the build directory with a slightly modified name so they get packaged with my jar file and are available (along with the uncompressed versions) at runtime.

One thing to note is that the version of YUICompressor available on the Maven repo is slightly outdated. You may want to install the latest jar in your local repo and use that instead.

Thanks to Julien Lecomte and Yahoo for making YUICompressor freely available.

Compiling HAProxy on OS X

2009-06-26T00:00:00-07:00

If you want to use PCRE when building HAProxy on your mac (recommended by the docs) and you've installed PCRE using MacPorts (sudo port install pcre), you may need to tweak Makefile.osx to help it find the headers, otherwise you'll see a whole slew of errors starting with this:

In file included from include/types/proxy.h:34,
                 from include/common/cfgparse.h:29In file included from include/types/proxy.h:34,
                 from include/common/cfgparse.h:29,
                 from src/haproxy.c:63:
include/common/regex.h:28:18:,
                 from src/haproxy.c:63:
include/common/regex.h:28:18: error:  error: pcre.h: No such file or directory
include/common/regex.h:29:23: error: pcreposix.h: No such file or directory

To fix that, open up Makefile.osx and find the two lines that define PCREDIR:

PCREDIR!= pcre-config --prefix 2>/dev/null || :
#PCREDIR=/usr/local

I commented out the first line, uncommented the second line and modified it to point at /opt/local instead of /usr/local:

PCREDIR=/opt/local

After that the compile worked. While I was at it I also set ARCH=x86_64 and TARGET=darwin, though I'm not sure if it actually accomplished anything.

Thrift: Bidirectional Async RPC

2009-04-03T00:00:00-07:00

Source on GitHub: http://github.com/JoelPM/BidiThrift

A reader posted to the thrift-user mailing list wondering if it was possible for a Thrift RPC server to send messages to the client. The responses indicated that this could be accomplished by polling the server for updates or hosting another Thrift server in the client that could receive RPCs from the server (requires opening another port on the client and handling firewall issues). I responded with a technique I'd used for accomplishing something similar and the responses made me think maybe this would be worth writing up and posting some example code.

Here's my email to the mailing list that describes what I'm doing:

I think I've done something similar to what you're trying to do, and as long as you can commit to using only async messages it's possible to pull it off without having to start a server on the client to accept RPCs from the server.

When your RPC is marked as async the server doesn't send a response and the client doesn't try to read one. So, if all your RPC calls from the client to the server are async you have effectively freed up the inbound half of the socket connection. That means that you can use it for receiving async messages from the server - the only catch is that you have to start a new thread to read and dispatch the incoming async RPC calls.

In a typical Thrift RPC system you'd create a MyService.Processor on your server and a MyService.Client on your client. To do bidirectional async message sending you'll need to go a step further and create a MyService.Client on your server for each client that connects (this can be accomplished by providing your own TProcessorFactory) and then on each client you create a MyService.Processor. (This assumes that you've gone with a generic MyService definition like you described above that has a bunch of optional messages, another option would be to define separate service definitions for the client and server.) With two clients connected the objects in existence would look something like this:

Server:
MyService.Processor mainProcessor - handles incoming async RPCs
MyService.Client clientA - used to send outgoing async RPCs to ClientA
MyService.Client clientB - used to send outgoing async RPCs to ClientB

ClientA:
MyService.Client - used to send messages to Server
MyService.Processor clientProcessor - used (by a separate thread) to process incoming async RPCs

ClientB:
MyService.Client - used to send messages to Server
MyService.Processor clientProcessor - used (by a separate thread) to process incoming async RPCs

Hopefully that explains the concept. If you need example code I can try and pull something together (it will be in Java). The nice thing about this method is that you don't have to establish two connections, so you can get around the firewall issues others have mentioned. I've been using this method on a service in production and haven't had any problems. When you have a separate thread in your client running a Processor you're basically blocking on a read, waiting for a message from the server. The benefit of this is that you're notified immediately when the server shuts down instead of having to wait until you send a message and then finding out that the TCP connection was reset.
Cheers,
Joel

Here's an example app that sends messages between clients connected to a server. It's similar to a chat app.

Thrift Definition

First, define your Thrift objects and the service. Our object and service are extremely simple:

In this case the service is generic enough that both the client and server will use the same service definition. We could also create a ClientMessageService and a ServerMesageService if we needed different functionality.

Server

On the server side, when a client connection is accepted we want to create a MessageService.Client object that we'll use to send messages back to the client. We can accomplish this by creating our own TProcessorFactory and using the getProcessor method as an opportunity to get access to the transport being used between the client and server:

As you can see above, we're using the same MessageDistributor for each new processor that we create. Before we create and return the processor we create a new client and add it to the list of clients that the MessageDistributor is aware of. The server is pretty simple and you can take a look at the code to see how the MessageDistributor uses the clients to send messages back.

Client

On the client side things are a little more complex because we have to create a separate thread to read incoming messages (this is handled by the TThreadPoolServer on the server side). Here's the class that reads incoming messages:

It extends a utility class called ConnectionRequiredRunnable that provides utility methods for handling server disconnects and reconnects, but on the whole it's pretty simple because we pass in a separate class that actually handles the incoming messages. We also create a MessageService.Client, but we wrap it in a separate thread and use a blocking queue so that other components in the system wanting to send a message can do so very quickly - or at least, have the message handed off for delivery extremely quickly.

Here's the class that handles our message sending:

This class also extends ConnectionRequiredRunnable since it can't send messages without a connection. Here's the main method of the Client that establishes the connection to the server:

It actually looks pretty simple since all the different pieces are organized in separate classes. The ConnectionStatusMonitor class is responsible for opening the actual connection and notifying the MessageSender and MessageReceiver when the connection has been established, at which point they'll start sending and receiving messages. If the server dies both of those processes will stop and wait until a connection has been re-established (a task the ConnectionStatusMonitor is responsible for). Here's sample output from the server:

And here's output from client1:

And here's from client2:

You can see that client1 and client2 paused when the server was terminated. Had the server restarted the clients would have reconnected and begun sending messages again.

The source is built using Maven and requires that you've installed libthrift.jar in your local maven repo (see the README for details). I'm also including a tgz with the compiled jar files for those who can't build the source.

Source: BidiMessages.tgz Jars: BidiMessagesJars.tgz

Thrift: Writing Thrift Objects to Disk with Java

2009-02-05T00:00:00-08:00

If you're using files to pass data around between different components of a system it may make sense to use Thrift. Doing so means you get smaller file sizes (assuming you do binary serialization) but still have files that can be easily read in many programming languages. In Java, writing Thrift objects to disk is straightforward. Here's a utility class that I use for doing so:

import java.io.BufferedOutputStream;
import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;

import com.facebook.thrift.TBase;
import com.facebook.thrift.TException;
import com.facebook.thrift.protocol.TBinaryProtocol;
import com.facebook.thrift.transport.TIOStreamTransport;

/**
 * Simple class that makes it easy to write Thrift objects to disk.
 * @author Joel Meyer
 */
public class ThriftWriter {
  /** File to write to. */
  protected final File file;
  
  /** For writing to the file. */
  private BufferedOutputStream bufferedOut;

  /** For binary serialization of objects. */
  private TBinaryProtocol binaryOut;
  
  /**
    * Constructor.
    */
  public ThriftWriter(File file) {
    this.file = file;
  }
  
  /**
    * Open the file for writing.
    */
  public void open() throws FileNotFoundException {
    bufferedOut = new BufferedOutputStream(new FileOutputStream(file), 2048);
    binaryOut = new TBinaryProtocol(new TIOStreamTransport(bufferedOut));
  }
  
  /**
    * Write the object to disk.
    */
  public void write(TBase t) throws IOException {
    try {
      t.write(binaryOut);
      bufferedOut.flush();
    } catch (TException e) {
      throw new IOException(e);
    }
  }
  
  /**
    * Close the file stream.
    */
  public void close() throws IOException {
    bufferedOut.close();
  }
}

If you have a Thrift object called Album and you'd like to write a list of them to file you'd do it like so:

// Create writer

ThriftWriter thriftOut = new ThriftWriter(new File("/some/thrift/file.thrifty"));

// Open writer

thriftOut.open();

// Write the objects to disk

for (Album album : albums) {
  thriftOut.write(album);
}

// Close the writer

thriftOut.close();

ThriftWriter uses the interface com.facebook.thrift.TBase, which defines the methods needed for reading and writing a Thrift object to a Thrift protocol. We could use a single ThriftWriter to write any number of different objects to disk, since they all implement TBase, but then you'd have to make sure you knew what order the objects came in when reading the file.

Thrift: Reading Thrift Objects from Disk with Java

2009-02-05T00:00:00-08:00

Reading Thrift objects is as easy as writing them to disk. Here's the utility class I use for reading one or more Thrift objects (of the same type) serialized to disk:

import java.io.BufferedInputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;

import com.facebook.thrift.TBase;
import com.facebook.thrift.TException;
import com.facebook.thrift.protocol.TBinaryProtocol;
import com.facebook.thrift.transport.TIOStreamTransport;

/**
 * A simple class for reading Thrift objects (of a single type) from a file.
 *
 * @author Joel Meyer
 */
public class ThriftReader {
  /**
    * Thrift deserializes by taking an existing object and populating it. ThriftReader
    * needs a way of obtaining instances of the class to be populated and this interface
    * defines the mechanism by which a client provides these instances.
    */
  public static interface TBaseCreator {
    TBase create();
  }

  /** File containing the objects. */
  protected final File file;

  /** Used to create empty objects that will be initialized with values from the file. */
  protected final TBaseCreator creator;
  
  /** For reading the file. */
  private BufferedInputStream bufferedIn;

  /** For reading the binary thrift objects. */
  private TBinaryProtocol binaryIn;
  
  /**
    * Constructor.
    */
  public ThriftReader(File file, TBaseCreator creator) {
    this.file = file;
    this.creator = creator;
  }
  
  /**
    * Opens the file for reading. Must be called before {@link read()}.
    */
  public void open() throws FileNotFoundException {
    bufferedIn = new BufferedInputStream(new FileInputStream(file), 2048);
    binaryIn = new TBinaryProtocol(new TIOStreamTransport(bufferedIn));
  }

  /**
    * Checks if another objects is available by attempting to read another byte from the stream.
    */
  public boolean hasNext() throws IOException {
    bufferedIn.mark(1);
    int val = bufferedIn.read();
    bufferedIn.reset();
    return val != -1;
  }

  /**
    * Reads the next object from the file.
    */
  public TBase read() throws IOException {
    TBase t = creator.create();
    try {
      t.read(binaryIn);
    } catch (TException e) {
      throw new IOException(e);
    }
    return t;
  }

  /**
    * Close the file.
    */
  public ThriftReader close() throws IOException {
    bufferedIn.close();
    return this;
  }
}

This class is useful if you've got a file containing a list of Thrift objects of the same type. Let's say you had a Thrift object called Album and a file containing a list of these Album objects. To read the file you'd do this:

// Create the reader

ThriftReader thriftIn = new ThriftReader(new File("/some/thrift/file.thrifty", new ThriftReader.TBaseCreator() {
  @Override TBase create() {
    return new Album();
  }
});

// Open it

thriftIn.open();

// Read objects

List<Album> albums = new ArrayList<Album>();
while (thriftIn.hasNext()) {
  albums.add((Album)thriftIn.read());
}

// Close reader

thriftIn.close();

As mentioned in the post on writing Thrift objects, TBase is the interface all Thrift objects implement that provides the read(...) and write(...) methods. When you're dealing with generated files you're limited to the code that's generated, so in this case I opted to use the TBase interface since that at least guarantees that the read and write methods will exist. I thought about making the class generic and casting to TBase, but then it's not clear (well, not contractually enforced) that the class is expecting a Thrift object. The downside of my method is that you have an unchecked cast, which I'm not a fan of.

Invoking the Thrift Compiler from Maven

2009-01-28T00:00:00-08:00

Thrift and Protocol Buffers require you to invoke a custom compiler during the build process to generate source code from the message definitions in your .thrift or .proto files. You can accomplish this with Maven by using the antrun plugin. Here's the relevant section from pom.xml:

  <profiles>
    <profile>
      <id>profile-buildthrift</id>
      <activation>
        <file>
          <exists>/usr/local/bin/thrift</exists>
        </file>
      </activation>
      <build>
        <plugins>
          <plugin>
            <artifactId>maven-antrun-plugin</artifactId>
            <executions>
              <execution>
                <id>generate-sources</id>
                <phase>generate-sources</phase>
                <configuration>
                  <tasks>
                    <mkdir dir="target/generated-sources" />
                    <exec executable="${thrift.executable}">
                      <arg value="--gen" />
                      <arg value="java:beans"/>
                      <arg value="-o"/>
                      <arg value="target/generated-sources"/>
                      <arg value="src/main/resources/MyThriftMessages.thrift"/>
                    </exec>
                    <delete>
                      <fileset dir="src/main/java/com/joelpm/generated" includes="**/*"/>
                    </delete>
                    <copy todir="src/main/java/com/joelpm/generated">
                      <fileset dir="target/generated-sources/gen-javabean/com/joelpm/generated"/>
                    </copy>
                  </tasks>
                </configuration>
                <goals>
                  <goal>run</goal>
                </goals>
              </execution>
            </executions>
          </plugin>
        </plugins>
      </build>
    </profile>
  </profiles>

You'll notice that the whole thing is wrapped in a profile with an activation. This is done so machines that don't have the Thrift compiler installed can still build using the most recent checked-in version of the generated files. Inside the antrun plugin we use four tasks to accomplish what we want:

Create the output directory for the generated sources under the target directory.
Invoke the thrift compiler and tell it to generate Java Bean style getters & setters, to write the output to the directory we just created, and to look for our Thrift file in the src/main/resources directory.
Delete all files under src/main/java/com/joelpm/generated
Copy all generated source to src/main/java/com/joelpm/generated

You may be wondering why the generated source is added to the src tree (and as a result gets checked into the SCM) instead of being passed to the Java compiler for compilation. I did it this way because I want machines that don't have the Thrift compiler installed to be able to build the project. To be honest, I have mixed feelings about checking the generated source into Subversion, but for now this is what I've settled on. The last thing to note is that the thrift compiler is going to put the generated java source in com/joelpm/generated because my thrift message definition contains the namespace directive:

namespace java com.joelpm.generated

I use 'generated' in the package name as a way to call out the fact that the source in that package is generated and should not be modified by hand.

Rest In a Jar: Maven, Spring, Jetty, and Jersey

2009-01-22T00:00:00-08:00

Here's a quick example of how to use Maven, Spring, embedded Jetty, and Jersey to build an application that provides a RESTful interface (in a single Jar file). There are four main parts to this project:

First, create and configure the project using Maven.
Second, create the resources.
Third, create/configure the Jetty server.
Finally, tie it all together with Spring.

First, create your project using Maven:

$ mvn archetype:create -DarchetypeGroupId=org.apache.maven.archetypes -DgroupId=com.joelpm -DartifactId=restInAJar

Now we need to configure the project - this includes adding the dependencies and telling Maven where they can be found (the repositories). We also add two plug-ins - one to tell Maven we're using Java 1.6 and the second (the Assembly plug-in) to build a jar file that includes our code as well as all dependencies. Here's what the finished file looks like:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>com.joelpm</groupId>
  <artifactId>restInAJar</artifactId>
  
  <packaging>jar</packaging>
  
  <version>1.0-SNAPSHOT</version>
  <name>restInAJar</name>
  <url>http://maven.apache.org</url>
  
  <properties>
    <gson.version>1.2.3</gson.version>
    <jersey.version>1.0.1</jersey.version>
    <jetty.version>7.0.0.pre5</jetty.version>
    <junit.version>3.8.1</junit.version>
    <spring.version>2.5.6</spring.version>
  </properties>
  
  <dependencies>
    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>${junit.version}</version>
      <scope>test</scope>
    </dependency>
    
    <dependency>
      <groupId>com.google.code.gson</groupId>
      <artifactId>gson</artifactId>
      <version>${gson.version}</version>
      <scope>compile</scope>
    </dependency>
    
    <dependency>
      <groupId>com.sun.jersey</groupId>
      <artifactId>jersey-server</artifactId>
      <version>${jersey.version}</version>
    </dependency>
    
    <dependency>
      <groupId>com.sun.jersey.contribs</groupId>
      <artifactId>jersey-spring</artifactId>
      <version>${jersey.version}</version>
    </dependency>
    
    <dependency>
      <groupId>org.mortbay.jetty</groupId>
      <artifactId>jetty</artifactId>
      <version>${jetty.version}</version>
    </dependency>

    <dependency>
      <groupId>org.springframework</groupId>
      <artifactId>spring</artifactId>
      <version>${spring.version}</version>
    </dependency>
  </dependencies>
  
  <build>
    <plugins>
      <!-- Tell Maven we're using Java 1.6 -->
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-compiler-plugin</artifactId>
        <version>2.0.2</version>
        <configuration>
          <source>1.6</source>
          <target>1.6</target>
        </configuration>
      </plugin>
    
      <!-- Configure the assembly plugin to build a single jar with all dependecies -->
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-assembly-plugin</artifactId>
        <configuration>
          <descriptorRefs>
            <descriptorRef>jar-with-dependencies</descriptorRef>
          </descriptorRefs>
          <archive>
            <manifest>
              <mainClass>com.joelpm.restInAJar.App</mainClass>
            </manifest>
          </archive>
        </configuration>
        <executions>
          <execution>
            <id>simple-command</id>
            <phase>package</phase>
            <goals>
              <goal>attached</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>
  
  <repositories>
    <!-- Repository for the GSON code -->
    <repository>
      <id>gson</id>
      <url>http://google-gson.googlecode.com/svn/mavenrepo</url>
      <snapshots>
        <enabled>true</enabled>
      </snapshots>
      <releases>
        <enabled>true</enabled>
      </releases>
    </repository>
    
    <!-- Repository for the Jersey code -->
    <repository>
      <id>maven2-repository.dev.java.net</id>
      <name>Java.net Repository for Maven</name>
      <url>http://download.java.net/maven/2/</url>
      <layout>default</layout>
    </repository> 
    
    <!-- Repository for the Jetty code -->
    <repository>
      <releases>
        <enabled>true</enabled>
        <updatePolicy>never</updatePolicy>
        <checksumPolicy>warn</checksumPolicy>
      </releases>
      <id>codehaus</id>
      <name>Codehaus Maven2 Repository</name>
      <url>http://repository.codehaus.org/</url>
      <layout>default</layout>
    </repository>
  </repositories>
</project>

I've set the main class to be com.joelpm.restInAJar.Launcher, which we'll create later on. Now let's move on to creating a resource. We'll put resources in their own package at com.joelpm.restInAJar.resources. We'll just create a simple resource that provides an in-memory hashmap:

package com.joelpm.restInAJar.resources;

import java.lang.reflect.Type;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;

import javax.ws.rs.Consumes;
import javax.ws.rs.DELETE;
import javax.ws.rs.GET;
import javax.ws.rs.POST;
import javax.ws.rs.PUT;
import javax.ws.rs.Path;
import javax.ws.rs.PathParam;
import javax.ws.rs.Produces;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.Response;

import org.springframework.context.annotation.Scope;
import org.springframework.stereotype.Component;

import com.google.gson.Gson;
import com.google.gson.reflect.TypeToken;

/**
 * This class provides the resource manipulated through the /map path.
 * We use Spring annotations to declare it as a Singleton and Jersey/JAX-RS
 * annotations to describe the REST interface.
 * 
 * This class is designed to work with Dojo's JsonRest and JsonRestStore,
 * which is why the PUT/POST methods function the way they do.
 * 
 * @author Joel Meyer
 *
 */
@Component
@Scope("singleton")
@Path("/map")
@Produces(MediaType.APPLICATION_JSON)
public class MapResource {
  // Map to store the values

  private final Map<String,String> map = new ConcurrentHashMap<String,String>();
  
  // Used to serialize/deserialize JSON

  private final Gson gson = new Gson();
  
  // Needed by Gson to deserialize a Map<String,String>

  Type stringMapType = new TypeToken<Map<String,String>>(){}.getType();
  
  @GET
  public String getEntireMapJson() {
    return gson.toJson(map);
  }
  
  @GET
  @Path("{key}")
  public String getValueForKeyJson(@PathParam("key") String key) {
    return gson.toJson(map.get(key));
  }
  
  @POST
  @Consumes(MediaType.APPLICATION_JSON)
  public Response addKeyValueJson(String json) {
    Map<String,String> keyValues = gson.fromJson(json, stringMapType);
    map.putAll(keyValues);
    return Response.ok().build();
  }
  
  @PUT
  @Consumes(MediaType.APPLICATION_JSON)
  @Path("{key}")
  public Response putValueJson(@PathParam("key") String key, String json) {
    String value = gson.fromJson(json, String.class);
    map.put(key, value);
    return Response.ok().build();
  }
  
  @DELETE
  @Path("{key}")
  public Response deleteKeyValue(@PathParam("key") String key) {
    map.remove(key);
    return Response.ok().build();
  }
}

I've omitted error checking for the sake of brevity in the example above - consider that an exercise for the reader. With our resource defined we need a way to serve it up, which is where Jetty comes in. Jersey-server provides the com.sun.jersey.spi.spring.container.servlet.SpringServlet which is designed to work with Spring web-apps, but we're going to use a very simple embedded Jetty that we configure programmatically so we need to extend that servlet and create our own:

package com.joelpm.restInAJar;

import java.util.logging.Level;
import java.util.logging.Logger;

import org.springframework.beans.BeansException;
import org.springframework.context.ApplicationContext;
import org.springframework.context.ApplicationContextAware;
import org.springframework.context.ConfigurableApplicationContext;
import org.springframework.context.annotation.Scope;
import org.springframework.stereotype.Component;

import com.sun.jersey.api.core.ResourceConfig;
import com.sun.jersey.spi.container.WebApplication;
import com.sun.jersey.spi.spring.container.SpringComponentProviderFactory;
import com.sun.jersey.spi.spring.container.servlet.SpringServlet;

/**
 * Extends {@link SpringServlet} so we can control what context gets passed to the
 * {@link SpringComponentFactory} and implements {@link ApplicationContextAware} so
 * that Spring can give us a reference to the application context when it instantiates
 * this class. We then pass the application context that Spring gave us to the
 * SpringComponentFactory.
 * 
 * @author Joel Meyer
 *
 */
@Component
@Scope("singleton")
public class EmbeddedJettySpringServlet extends SpringServlet implements ApplicationContextAware {
  private static final long serialVersionUID = 1L;
  private static final Logger LOGGER = Logger.getLogger(EmbeddedJettySpringServlet.class.getName());
  
  private ApplicationContext springContext;
  
  public EmbeddedJettySpringServlet() {
    super();
  }
  
  @Override
  protected void initiate(ResourceConfig rc, WebApplication wa) {
    try {
      wa.initiate(rc, new SpringComponentProviderFactory(rc, (ConfigurableApplicationContext) springContext));
    } catch( RuntimeException e ) {
      LOGGER.log(Level.SEVERE, "Exception occurred when intializing", e);
      throw e;
    }
  }

  /**
   * @see org.springframework.context.ApplicationContextAware#setApplicationContext(org.springframework.context.ApplicationContext)
   */
  @Override
  public void setApplicationContext(ApplicationContext applicationContext)
      throws BeansException {
    springContext = applicationContext;
  }
}

Our EmbeddedJettySpringServlet is ApplicationContextAware so that Spring gives it a reference to the application context when Spring instantiates this class. Now that we have our modified servlet we just need to create the Jetty server to host it, which we do in the Launcher:

package com.joelpm.restInAJar;

import org.mortbay.jetty.Server;
import org.mortbay.jetty.handler.ContextHandlerCollection;
import org.mortbay.jetty.servlet.Context;
import org.mortbay.jetty.servlet.ServletHolder;
import org.springframework.beans.factory.annotation.Autowired;

/**
 * Configures Jetty.
 * 
 * @author Joel Meyer
 */
public class Launcher {
  private final Server server;
  
  @Autowired
  public Launcher(EmbeddedJettySpringServlet restServlet) {
    server = new Server(8080);

    ContextHandlerCollection contexts = new ContextHandlerCollection();
    server.setHandler(contexts);

    Context rest = new Context(contexts, "/");
    rest.getServletContext();
    rest.addServlet(new ServletHolder(restServlet), "/*");
  }
  
  public void start() {
    try {
      server.start();
    } catch (Exception e) {
      e.printStackTrace();
    }
  }

  public void stop() {
    try {
      server.stop();
    } catch (Exception e) {
      e.printStackTrace();
    }
  }
}

We're now done with coding, we just need to set up our application-context.xml:

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:aop="http://www.springframework.org/schema/aop"
       xmlns:context="http://www.springframework.org/schema/context"
       xmlns:tx="http://www.springframework.org/schema/tx"
       xsi:schemaLocation="
       http://www.springframework.org/schema/aop http://www.springframework.org/schema/aop/spring-aop-2.5.xsd
       http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.5.xsd
       http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-2.5.xsd
       http://www.springframework.org/schema/tx http://www.springframework.org/schema/tx/spring-tx-2.5.xsd">

  <!-- This initializes all the classes that we annotated with @Component -->
  <context:component-scan base-package="com.joelpm.restInAJar"/>
  
  <bean name="launcher" class="com.joelpm.restInAJar.Launcher">
  </bean>
</beans>

Because we annotated the classes with @Component, @Scope, and @Autowire as we went along Spring takes care of most of the work for us. All we have to do is tell Spring to scan our classes looking for those annotations and create the Launcher. In our static main all we have to do is:

package com.joelpm.restInAJar;

import org.springframework.context.ApplicationContext;
import org.springframework.context.support.ClassPathXmlApplicationContext;

/**
 * 
 */
public class App {
  public static void main(String[] args) {
    ApplicationContext applicationContext = new ClassPathXmlApplicationContext("application-context.xml");
    Launcher restInAJarLauncher = (Launcher)applicationContext.getBean("launcher");
    restInAJarLauncher.start();
  }
}

Now all that remains is to build and run our server:

$ mvn clean package
$ java -jar target/restInAJar-1.0-SNAPSHOT-jar-with-dependencies.jar

Well, I wish it was that easy. Unfortunately there's a gotcha. When the assembly plugin creates the combined jar it overwrites some files instead of combining them. The Jersey ServiceProvider looks in META-INF/services/ for a text file that lists the classes that provide services of a given type (denoted by the name of the text file). Both jersey-core.jar and jersey-server.jar provide copies of some of these files, but the final assembly only contains the files from jersey-server.jar. The right way to solve this problem is with a ContainerDescriptorHandler that merges these files, but I couldn't find any documentation on how to create one and have Maven use it so I resorted to a hack, which is manually merging these files and placing them in my local resources/META-INF/services directory - these copies then overwrite any provided by the dependencies.

Add new files to SVN

2009-01-09T00:00:00-08:00

I frequently find myself needing to add a bunch of new files to svn. Here's how to do it for Java files:

svn status | grep ^?.*\.java$ | awk '{print $2}' | xargs svn add

And here's how it works:

svn status |

Gets the status from Subversion and pipes it to:

grep ^?.*\.java$ |

Which matches all lines starting with a question mark (meaning they aren't in the svn repo) and ending in .java and pipes them to:

awk '{print $2}' |

Which grabs the second column of the line grep returned (the filename including path, relative to the current directory) and pipes it to:

xargs svn add

Which uses xargs to make sure the length of the command doesn't exceed the allowable limits. To get just the list of new Java files you could omit the xargs bit:

svn status | grep ^?.*\.java$ | awk '{print $2}'

From the Archives: Monitors

2007-07-11T00:00:00-07:00

This is another entry from the Mephisto era that still seems to get a lot of traffic, so I'm bringing it back in the hopes that it will help someone trying to decide what the right monitor strategy is for their needs. For the record, I still favor a single 30" over dual 24".

My first internship was with a company called Tellabs that made telecom products - large, powerful boxes that did telephone switching. These big huge boxes generated big huge log files and my task was to write a Perl program that made it easier for engineers to search through and get what they needed from the log files. My computer was an aging Sun something-or-other.

My second internship was with SAIC where I spent the first part of my summer writing code to parse various binary file types and the second part implementing terrain rendering algorithms in C++/OpenGL. My computer was again an aging relic with FPS measurements usually less than one (no textures, just tri-strips and simple shading): an SGI Indigo.

When I started at IBM I was given an older T21 laptop and a decent workstation. By the time I left I had a top of the line T42, a top of the line T41, and a very powerful workstation with a pro-grade graphics card that was unfortunately never utilized. I also had a 19" CRT monitor.

After my first two internships my experience at IBM was a welcome change - finally a place that provided its engineers with really good equipment. From them I learned the value of having the best tools available and three years later when I was interviewing at Xanga one of the things I evaluated them on was the type of equipment their developers had. As it turned out, they gave their devs very good equipment, which came in handy when we played Counter-Strike or Battlefield 2 in the evenings, as well as when programming, of course. It was at Xanga that I was first able to experience a dual-monitor setup. I started with two 19" monitors and graduated to a 24" and a 20" a little later.

While I was at Xanga experiencing my new found screen real-estate wealth I couldn't help but wonder what it would be like to have dual 24" monitors, a curiosity that was satisfied when I started at Google, where dual 24" monitors was the default for engineers. The amount of screen real-estate you get with dual 24" monitors is amazing. I can have Eclipse maximized in one screen and have a large browser window, a terminal, and a text editor on the other screen, like this:

What I hadn't expected, though, was the amount of head turning it would take to shift my gaze from one monitor to the next. And, believe it or not, even with Eclipse maximized on one monitor things still felt cramped when I had two code panes open side-by-side (my default mode of operating). So when the opportunity arose to trade in my 24"s for a 30" monitor I jumped at it. After using the 30" for a couple weeks I think I made the right decision. Eclipse maximized on a 30" monitor is a beautiful thing. I can see two full code panes, stack the Package Explorer view on top of the Outline view and both are still usable. The benefit is also apparent in the Debug perspective when there are even more views, all with useful information, visible at the same time. I can see the call stack, the variables currently in scope, the console, and the editor view is actually usable when you're stepping through code:

Another advantage of the 30" monitor is that I don't have to move my head as much, especially to look at something on my laptop, which I use for email and messaging. A drawback is that I can't see Eclipse and Firefox at the same time so I end up alt-tabbing between them, but it's not as much of a drawback as I had thought it might be. For reference, that's a 15" MacBook Pro on the left and a Lenovo X60 on the right.