As I’ve been blogging about recently, we have been developing an exam-taking web application at Terracotta to demonstrate the Session Clustering capabilities of Terracotta. Since one of the requirements of this web app is that we support 40,000 concurrent users, we thought we’d better cache the hottest exams (using Ehcache) rather than fetch them from Hibernate each time. Since modifying an exam should occur far, far less frequently than taking an Exam, and since Terracotta already supports Ehcache, this was a no-brainer.

There is an ExamService service, configured as a Spring bean, with DaoExamService being the default implementation:

The findById method is expected to be the most frequently-invoked method. The other two methods shown are administrative functionality, not expected to be used frequently. We want to cache the results of all three methods in a single, clustered Exam cache.

My approach was to write a new service, CachingWrapperExamService, a proxy which owned the cache and which delegated to another ExamService instance:

The following were straightforward changes:

• Once again, our use of interfaces have reaped dividends: Introducing this new Caching ExamService was as easy as tweaking the Spring XML file – the change was completely transparent to all users of the ExamService bean. It also made unit testing easy, as I could create a mock ExamService to test caching with.
• The Maven pom.xml had to be changed to note that Ehcache is now a compile-time dependency, not just a runtime dependency.
• The Terracotta config file tc-config.xml did not have to be modified, as we were already using the Ehcache TIM, and so our CacheManager was automatically clustered.

And just like that, we have a clustered cache of the hottest exams being used.

Okay, granted, I’m biased, I work for Terracotta. Be that as it may, I’d like to share some experiences my teammates and I have had using Hibernate recently while developing a web app.

First, some brief background. We are developing a “reference” web app at Terracotta to use to promote and explore the Sessions Clustering Use Case which we are working to nail. The app is an online exam-taking application, with the goal of supporting 40,000 concurrent users. I’ve blogged about this before, and you can read about the technology stack we settled on. Development has been done primarily by myself and my teammates Geert Bevin, Abhishek Sanoujam, and our supervisor Alex Miller.

Hibernate is wonderful, and it is an integral part of our web app. It feels to me like we got moving pretty quickly using Hibernate for persistence of our domain objects. For ORM, it’s unbeatable.

But the thing I noticed is, there’s just no avoiding the fact that whatever your domain POJO’s are that need to be persisted, chances are good that the use of ORM will impose some constraints on how you must write those POJO’s. I have two examples of this to share.

Example One – Generics

First, we have an exam Section class which, conceptually, is a container for either multiple sub sections, or Questions, but not both. The ideal solution would be to define Section as this (JPA annotations omitted):

where TestContent is an interface implemented by both Section and Question. Thus, an instance of Section could be declared as having type of either Section

When my manager Alex first started at Terracotta, he blogged a simple Hello Terracotta example to demonstrate POJO clustering in action. When I first started last month, he walked me through this same example and went into quite a bit more depth, decompiling some code and showing me a glimpse of how client POJO’s are instrumented by Terracotta and thus plug into our clustering framework. Here’s what we did.

First, if you haven’t yet, please go and run through Alex’s Hello Terracotta example. I’ll be starting right where he left off.

Now, as Alex pointed out to me, I need to make a slight correction to the example POJO code he is using. Here is the code:

The correction is the commented out bit. The compiler complains about that code, since the root field is final. And anyway, it is not necessary – if an L1 node starts up and Terracotta sees that the clustered root already exists, Terracotta will ignore the assignment in the source code (Item 2 above) and instead set that field to the pre-existing root. That is why, if you run this sample program twice in a row without bouncing the Terracotta server, you will see that the counter continues to increment from where it left off the first time. Terracotta clustered objects persist.

To continue, today when I run this example I am adding the special super-secret -Dtc.classloader.writeToDisk=true flag to the vm flags when I start up the dso process. My full command looks like this:

dso-java.sh -Dtc.config=config/tc-config.xml -Dtc.classloader.writeToDisk=true -classpath bin test.HelloTerracotta

This additional flag causes Terracotta to dump the modified classes under ~/adapted/. Under that folder you should find a test/ directory containing the instrumented HelloTerracotta.class file.

At this point you can examine the raw bytecode to see what we’ve added – Terracotta uses ASM to instrument bytecode on the fly in order to cluster arbitrary client code.

Or, you can decompile the file like I did using Jad. This likely won’t be able to decompile everything all the way, but it’s close enough that you can see what’s going on.

When I decompile HelloTerracotta using Jad, after cleaning up the go() method by hand, this is what I see:

First thing to notice, the instrumented version of the class now implements two interfaces, Manageable and TransparentAccess. These are both in the dso-l1-api project if you download the Terracotta source code. The api is our internal api for dealing with all clustered objects in the L1 nodes.

Next, notice that all direct reads of the root field have been replaced with calls to a new dynamically-generated Set __tc_getroot() method, and the one spot in the constructor where we were setting the root field directly is now replaced by a call to a new __tc_setroot(Set) method. And you can see that those getter/setter methods are doing the work of ensuring any pre-existing root is set on this object, or else creating the new clustered root.

You can see that there are a lot of calls into ManagerUtil – this is a dynamically clustered object’s hook into the Terracotta runtime. ManagerUtil can be thought of as a facade of static methods, encapsulating a more interesting system which I won’t go into here.

The go() method didn’t decompile entirely correctly, so I’ve had to tweak it by hand into what you see here. What I find most interesting is that now, in addition to the synchronized block, there are calls (in a try-finally block) to ManagerUtil to acquire/release a clustered lock. Remember – since the root field is actually a clustered object, modifying it requires obtaining a cluster-wide exclusive write lock. It’s also interesting to note that a chunk of that tc-config.xml file, specifically the lock information including the method expression, is being passed to the monitorEnterWithContextInfo method.

In conclusion, this example illustrates the pattern for any POJO being clustered by Terracotta. Terracotta dynamically instruments a class to adhere to the dso api, an api for handling all of the distributed objects and locks within Terracotta. Of course that barely scratches the surface of all that Terracotta does, but at least you can see that conceptually what Terracotta is doing behind the scenes to cluster your code is really not all that complicated.

For further reading, the Terracotta website offers some very straightforward articles about how Terracotta works, and how Terracotta scales. Additionally, there are quite a few other Terracotta employees who blog, and their blogs are listed here.

This week at Terracotta we accomplished something that two weeks ago I thought was both impossible and dangerous – we instrumented java.lang.String to be compressible.

What, you ask, the heck is going on? String doesn’t implement any interface called JavaLangString. It doesn’t have a __tc_decompress() method. String is final and immutable! It has to be for thread-safety! Are you mad?

I offered these same objections to my boss two weeks ago, but the fact is that something like the above code will be in an upcoming Terracotta release. What makes this all possible is that we instrument Java bytecode on the fly.

Terracotta already does some large String compression when clustering Strings across cluster nodes, but we wanted to improve on that. What if, when we decoded a compressed String data from across the cluster and constructed a String instance on a remote node, we actually kept the String instance compressed until it absolutely needed to decompress to be read?

Through the voodoo black magic of bytecode instrumentation, we have accomplished this and made it completely transparent to the application’s use of String. Here’s a basic outline of what happens:

• data about clustered compressed String is sent over the wire (we call it “hydration”) and decoded at one of the cluster nodes. The following data is encoded:
  • actual compressed String data, in byte[] array form
  • uncompressed String length (int)
  • String hashcode (int)
• when decoded, the compressed byte[] array is encoded into a char[] array – basically two bytes can fit into a single char.
• a String instance is constructed with that char[] array, and also the uncompressed length and original hashcode

So, if the resulting String were actually read, it would be gibberish (if it were displayable at all). (By “read” I mean, if it’s internal character content needed to be accessed.) That’s why we need to instrument the String class – if and when the String needs to be read, we have instrumented it so that it knows how to decompress itself. We have basically added a new private boolean field (indicating if compressed or not) a new constructor and some additional methods.

There’s a lot going on here so l’ll point a few things out. First, we intercept any field-level access of the private internal char[] value and we route it through this __tc_getvalue() getter method. This is how we transparently decompress the contents no matter how the String instance is accessed.

Secondly, there are important concurrency issues in play here. String is thread-safe because it is immutable, or was until we got our grubby mits on it. We need for String to remain both thread-safe and also highly concurrent, so we wanted to avoid synchronized blocks. Our solution?

We have instrumented value to be volatile, as well as our new $__tc_compressed field. Now if you once more examine the __tc_getvalue() method above, you’ll see that there is a benign race condition. It’s possible two threads could both decompress and set the value field. That’s fine, since the decompression is deterministic and outputs the same result each time. Because the fields are volatile, once they are set those changes should be visible to all other threads. What should not ever happen is, no thread should attempt to decompress the characters once they are already decompressed – the call to StringCompressionUtil inspects the data to see if it has already been decompressed, and if it has it returns null.

The above methods don’t exist anywhere in source code per se. Rather, we use ASM to add the bytecode for these new methods dynamically to the String class as it is loaded at runtime. So what we actually have somewhere is code that looks like this:

This creates the bytecode to implement the __tc_decompress() method.

When does an instrumented String instance need to decompress itself? Basically, when it’s internal char[] array needs to be accessed. It does not need to decompress when length() method is called, because we have set the uncompressed length when we constructed the String. It does not need to decompress when hashCode() is called, since we have preset the hash code, so that means a compressed String can sit around in a Map all day long without needing to decompress. We even have plans to muck with the equals() method – if two Strings are unequal due to hash code we can exit early and avoid decompressing them to compare character content.

Let’s look once more at the code way at the top:

Because we have instrumented String, at runtime instances of String do implement the interface JavaLangString. So, within Terracotta we can cast an instance of String to JavaLangString, which is an interface containing the __tc_decompress() method. In this way we avoid using reflection. One interesting thing to note is that we first have to upcast to Object to trick the compiler – at compile time the compiler knows (or thinks it knows) that String does not implement this interface, so it errors.

In conclusion, a bunch of us were talking after Alex’s Terracotta talk last night, and someone made the point that, with any good technology, your users will inevitably take advantage of it in a way that you never dreamed of. And that, I think, is what Terracotta is doing.

Wanna learn more about Terracotta? Sure you do! Do you live in the St. Louis area? Then you’re in luck, because Alex Miller is giving a presentation about Terracotta tonight at the St. Louis Java User’s Group.

Alex is the manager and lead developer of the Terracotta Transparency team, so he knows his stuff. He is also an experienced presenter, and in fact has been accepted to speak at JavaOne in May.

Oh, I almost forgot – Terracotta is giving away one iPod Shuffle tonight.

Hope to see you there!

Somehow my first month at Terracotta has already flown by. Here’s what I’ve been up to.

Mostly I’ve been testing. We have a very cool distributed testing framework called “Droid”, and using it we’ve written a number of test scripts (in Groovy) to simulate some real-world use cases and see where we can make improvements. Droid, in a nutshell, allows us to easily and automatically run tests on multiple nodes, and provides means of synchronizing those nodes. In fact, Droid achieves this clustered synchronization using Terracotta, which just goes to show you that we’re not afraid to eat our own dog food.

In the process of working on this I’ve gotten to dive fairly deep into both Groovy and the java.util.concurrent classes, both of which I’ve enjoyed immensely. My humble contribution to Droid was to add some Groovy utilities enabling us to set up multiple test “phases”, which are simply points at which the nodes all synchronize by waiting on a CyclicBarrier. Our test scripts can now register closures to run during a phase, which cleans up the scripts themselves somewhat. Droid already had the ability to “kill” a test node, which was implemented rather ingeniously by using a cluster-wide (via Terracotta) Object to wait on and notify all – basically a daemon thread in each node would wait on this object, and when notified would check to see if it had been killed and should do a System.exit(-1). Building upon this, in my Groovy utilities I added methods to easily set up a kill phase, in which all testing would be suspended while a node was killed off – this leverages the Runnable CyclicBarrier action, which is run before any threads are released from the barrier.

Also as part of all of this, I’ve become reacquainted with Unix. Terracotta has an impressive array of “perf” machines dedicated entirely to continuous performance testing. Some are tuned to act as client L1 nodes, other more powerful multicore machines are tuned to act as the L2 servers. These machines can be reserved, scheduled and used all remotely. So my third week was largely spent actually running some of these distributed tests on multiple perf machines, which involved ssh and sudo.

This week, based in part on some findings from running these tests, I have finally begun (along with Alex) diving into actual production code to see what improvements can be made. In particular, we are looking into the case where cache values are comprised of very large strings, for example large XML documents, and we are investigating how we can improve our string compression code, or whether we can avoid unnecessarily faulting String values into an L1 node.

I love working at Terracotta. I love it! I could go on and on about other cool stuff that’s happening, but I won’t. Suffice to say, the technologies we use and the problems we are tackling are fascinating, to say the least. All of my colleagues are both very sharp and very down-to-earth – no power trips going on that I can tell. Working at home is wonderful – it’s so nice to just take a 15 minute break and walk my son to preschool, or have lunch with my family. Or, in theory, work in my boxers, not that I’ve done that, yet. And working with Alex again rocks.

Nearing the end of my second week on the job at Terracotta. I’m going to try to document here the software that I use, much of which I had to install. Hopefully this will streamline things for the next newbie. If you’re like me and are making the transition from a PC, you may find my handy-dandy Mac transition guide useful.

Terracotta

• Terracotta.org is where the main software can be downloaded, including source code. Terracotta has been open source since December of ‘06.
• The Terracotta Forge is, among other things, a Maven repository of Terracotta-related projects such as Terracotta Integration Modules (TIMs)

General

• Firefox (download)
• Eclipse Java IDE
• Adium for IM (supports Yahoo, GTalk, etc)
• Skype for video chat
• Eyebeam soft phone (this is the non-free version of the free X-Lite soft phone, apparently with Leopard we must use this. You’ll need to get a download URL and license from the Terracotta help desk.)
• TextWrangler text editor
• TextMate not free but great text editor
• NetNewsWire feed reader
• Growl (download) system alerts, integrates with Adium, NetNewsWire, Elluminate and other Mac software
• NeoOffice (download) free OpenOffice for Mac (was preinstalled on my MacBook)
• iWork office suite, not free
• iStat menu bar meters for cpu, etc.
• Terminal (preinstalled) which as of Leopard is now tabbed
• Quicksilver
• OmniGraffle – graphics editor du jour
• Cisco AnyConnect VPN Client (preinstalled)

Java

• Jad (download) is a Java byte code decompiler

Eclipse Plugins (all of these were installed via Eclipse’s Update Manager)

• Terracotta Plugin (URL for update manager) – eases integrating Terracotta into a project
• ASM (URL) – you’ll want to install the ASM Framework and Bytecode Outline
• Groovy (URL) – syntax highlighting and whatnot
• Subclipse (URL) – Subversion support
• Maven (URL) – needed to resolve Eclipse inter-project dependencies using Maven pom.xml’s rather than Eclipse .classpath or .plugin files. (Not to be confused with Maven Eclipse plugin, a Maven plugin for generating Eclipse project files.)
• FindBugs (URL)
• QuickREx (URL) – regex
• Implementors (URL) – quickly navigate between interfaces and implementations, or super and sub classes

Firefox Plugins

• DZone plugin (download)

Today, February 5th, 2008, is my first day as a Terracotta developer.

I intend to blog about both Terracotta and my progress as a developer. So far two ideas I have is to maintain a separate account of my transition from a PC to a MacBook Pro, and to document a checklist for new TC developers.

Terracotta developers are currently given the choice of windows, Mac OS or Linux. As I said, I’ve chosen a MacBook Pro as my computer. Mostly this is because my friend and boss Alex is already using one. It’s also because I wanted to try something other than windows, which is the only OS I’ve used professionally.

As you might expect, this first day was mostly about setting up my computer. Alex and I did talk about what I would work on first, which is the distributed testing framework that is in development and a set of test use cases centered around Terracotta’s distributed cache functionality. These use cases are based on a number of real world use cases. I’m particularly excited about one aspect of this in-house test framework, which is that the tests are scripted using Groovy.

I’ll be working from home, which is also another first. Frequently I’ll be working at Alex’s home, as I did this morning. This was a surreal but enjoyable experience and I’m looking forward to it. Last Friday was my last day of my 1.5 year stint as a contractor at Monsanto, so it was quite a contrast to go from that business casual environment to sitting on a folding chair in Alex’s basement with my shoes off.

Alex capped off my first morning nicely by treating me to lunch at Penn Station.