As I’ve been blogging about recently, we have been developing an exam-taking web application at Terracotta to demonstrate the Session Clustering capabilities of Terracotta. Since one of the requirements of this web app is that we support 40,000 concurrent users, we thought we’d better cache the hottest exams (using Ehcache) rather than fetch them from Hibernate each time. Since modifying an exam should occur far, far less frequently than taking an Exam, and since Terracotta already supports Ehcache, this was a no-brainer.

There is an ExamService service, configured as a Spring bean, with DaoExamService being the default implementation:

The findById method is expected to be the most frequently-invoked method. The other two methods shown are administrative functionality, not expected to be used frequently. We want to cache the results of all three methods in a single, clustered Exam cache.

My approach was to write a new service, CachingWrapperExamService, a proxy which owned the cache and which delegated to another ExamService instance:

The following were straightforward changes:

• Once again, our use of interfaces have reaped dividends: Introducing this new Caching ExamService was as easy as tweaking the Spring XML file – the change was completely transparent to all users of the ExamService bean. It also made unit testing easy, as I could create a mock ExamService to test caching with.
• The Maven pom.xml had to be changed to note that Ehcache is now a compile-time dependency, not just a runtime dependency.
• The Terracotta config file tc-config.xml did not have to be modified, as we were already using the Ehcache TIM, and so our CacheManager was automatically clustered.

And just like that, we have a clustered cache of the hottest exams being used.

Okay, granted, I’m biased, I work for Terracotta. Be that as it may, I’d like to share some experiences my teammates and I have had using Hibernate recently while developing a web app.

First, some brief background. We are developing a “reference” web app at Terracotta to use to promote and explore the Sessions Clustering Use Case which we are working to nail. The app is an online exam-taking application, with the goal of supporting 40,000 concurrent users. I’ve blogged about this before, and you can read about the technology stack we settled on. Development has been done primarily by myself and my teammates Geert Bevin, Abhishek Sanoujam, and our supervisor Alex Miller.

Hibernate is wonderful, and it is an integral part of our web app. It feels to me like we got moving pretty quickly using Hibernate for persistence of our domain objects. For ORM, it’s unbeatable.

But the thing I noticed is, there’s just no avoiding the fact that whatever your domain POJO’s are that need to be persisted, chances are good that the use of ORM will impose some constraints on how you must write those POJO’s. I have two examples of this to share.

Example One – Generics

First, we have an exam Section class which, conceptually, is a container for either multiple sub sections, or Questions, but not both. The ideal solution would be to define Section as this (JPA annotations omitted):

where TestContent is an interface implemented by both Section and Question. Thus, an instance of Section could be declared as having type of either Section

After two weeks of debugging, on Friday I added five magic lines of code to make the last three automated Spring tests pass. With this change we can say we support Spring up to 2.0.8 in our upcoming 2.6.2 release. We still have targeted Spring 2.5.x support for an upcoming summer release.

Two weeks of debugging, five lines of code, one bug fix. This has to be improved on.

Just for my own amusement, I’m going to try to list and describe all of the things that accounted for all the time spent on this.

By midweek last week, I had finally gotten into a groove with the Eclipse debugger, stepping through one of the failing Spring container tests. By “container” test, we mean parts of the test were actually running in a web app container, Tomcat in this case. Even so, this was not a very good, tight feedback loop. The basic pattern was for me to start the test, then attach to each of the two processes running in tomcat with the debugger. (Luckily, our code was already set up to allow for debugging, although it took me awhile to find and enable the magic property.) Then I just had to step through code in the debugger, not really sure what I was looking for. I had to alternate doing this, first with Spring 2.0.1 in the classpath, since our test was passing for that version, then with Spring 2.0.8 in the classpath, and look for what was going wrong. (We have a way of easily varying the version of a library such as Spring. However, I spent basically all of Tuesday helping Hung debug a problem in our build process concerning those variants.) Each time I alternated Spring variants, I also had to tweak the source lookup in my Eclipse remote debug configuration, to pull up source from the right Spring version.

The test uses our DSO functionality, it’s basically a semi-complete running instance of the DSO client. At one point I found I needed to hit breakpoints that were occuring during DSO bootstrapping, which meant I had to set yet another magic property (in ClassProcessorHelper) to enable debugging which is normally not available prior to TC instrumentation. I had to dig through my IM chat transcripts to find the name and location of that property, which my boss had mentioned once weeks ago, and then enable it, and find through painful trial and error that it didn’t work unless I did a full clean recompile (but it worked like a charm after that).

The code I was stepping through was instrumented using AspectWerkz, although for the most part the breakpoints seemed to work fine. But the amount of code was just vast. All I can say is, I spent probably twelve to fourteen hours of straight debugging on Thursday and Friday, just hitting breakpoints, comparing state, following hunches and wild goose chases and red herrings. In the end I found the Spring class whose source had changed between 2.0.1 and 2.0.5, and again in 2.0.8. (It was org.springframework.aop.config.ScopedProxyBeanDefinitionDecorator, deep in the guts of Spring’s aop framework.)

So in the final tally, I spent unexpected amounts of time

• helping debug a build problem which prevented us from using the “variants” feature, to vary the Spring library
• setting up Eclipse projects for different versions of Spring, so I could browse 2.0.1 and 2.0.8 source code side by side
• tweaking the Spring source lookup for the Eclipse remote debugging configuration, to alternate between 2.0.1 and 2.0.8
• figuring out how to enable debugging of our container tests
• figuring out how to enable debugging during DSO client startup
• trying (in vain) to write a more lightweight unit test to tackle the problem with
• debugging for many, many hours once it was all working

And that’s not even counting actually learning AspectWerkz and writing a new pointcut and advice, which I would have had to do anyway, but which itself involved some painful trial and error.

And at the end of all of this, I sort of feel like I’ve just patched some big honking beast that I don’t fully understand. I feel like our code as it stands now (the AspectWerkz pointcuts and advices for clustering Spring) are still very tightly coupled to Spring source code in a very fragile way, and it’s only a matter of time before a new minor Spring version comes along and breaks it again. All week I was thinking, there has got to be a better way.

Our code for instrumenting and clustering Spring is really cool and mind-blowing, don’t get me wrong. I never would have thought of it in a million years. But it is a classic case of code that is tightly coupled, not modular at all. There is an impressive amount of test coverage through automated integration and system tests, which we need. But the Spring test suite takes about an hour to run, and that’s not even every test. There is a lot of AspectWerkz advices and pointcuts being used to cluster Spring, and none of them can be run and tested in isolation. You just have to start up the whole shebang and debug.

I consider myself a TDD and refactoring proponent. All week long, the pain of trying to debug this was screaming out to me that this part of the code needed refactoring and unit tests. Honestly, I’m not sure I could have done any worthwhile refactoring and still figured out the problem in the same amount of time. That’s the classic TDD/refactoring chicken and egg problem – when you’re in a time crunch, the prospect of trying to clean up some code in order to make things easier at some undermined future point seems insurmountable, and meanwhile you always tend to mentally downplay the amount of time it will take to just debug and fix the code as is. That’s the fear that comes with TDD.

In an interesting twist, I was talking to my boss Steve, and he asked me why I wasn’t refactoring. My boss wants me to refactor! He didn’t order me to or anything, but he sounds very much in favor of it. He said he almost always regrets not refactoring as he goes.

I think keeping the code clean and testable is so important to the long term health of a large, constantly evolving code base. I mean, we can’t just keep having software developers spend two weeks debugging one bug.

There’s a lot more I want to think about along these lines, but I’ve written enough for now.

This weekly summary actually encompasses the last three weeks. Sigh.

Lots of activity throughout dev is centered around the Terracotta 2.6 and 2.6.1 releases, as well as the upcoming 2.6.2 release.

Primarily I’ve been working on updating Terracotta’s Spring support to 2.5.x. Currently we only support up to 2.0.5. I had thought I had gotten it working up through Spring 2.0.8, but late last week we fixed a bug in our build process which then revealed three failing automated TC Spring tests which were previously (incorrectly) passing. So Spring 2.0.8 is not quite there…but close. Meanwhile, my compadre Nitin had made some changes that got TC working with Spring 2.5, but those changes are not backwards compatible to Spring 2.0.x, so I’m investigating whether they can be merged together somehow. Since we are dependent on the Spring source code in order to instrument their code (by using Aspectwerkz), we are subject to the whims of whatever source code changes occur between even minor releases (such as differences between Spring 2.0.5 and 2.0.8).

The other thing of note that I got accomplished was to respond to this post on our forums about a deadlock occurring in Terracotta L1. The poster had nicely laid it all out for us, with a stack trace excerpt clearly showing the deadlock. My teammates and I reviewed the pertinent class, and I cleaned up a number of synchronization bugs or missing synchronization. The deadlock itself was cleaned up by moving to a CopyOnWriteArrayList for a collection, which previously was being locked while iterating through it (read-only) and doing expensive stuff. The fix will be in 2.6.2 release.

I was without internet connection at my house a couple weeks ago for a few days. I had to do bloody battle with Charter to get that fixed. Ultimately a technician came and found that the line to my house had been put on a splitter at some undetermined point in the past, and so my signal strength was no longer strong enough. Meanwhile, luckily, I was able to go to my parents’ house and get some work done there. Have I mentioned that I love my MacBook Pro, and wireless internet?

My previous post was about whether Guice should be used in unit testing, and Crazy Bob himself commented:

I prefer the simpler unit test. I think the real moral of this story is that unit tests alone are never enough.
Also, you did get the error right away when you started your app. That’s like an automatic test that you needn’t replicate manually (comparable to a compiler check).

I started to reply and my comment quickly grew long enough that I decided to just write another post. So, should Guice be used in unit tests? I hate to be wishy washy but I think the answer is it depends. As happens so often in software development, there are conflicting forces at work. In this case I think two conflicting forces are ease of unit testing versus fast feedback loop.

One thing I do completely agree with Bob on is that unit tests alone are never enough. Alex Miller wrote a great post about that called Weaving the Test Fabric.

Fast feedback loop

Different parts of the test fabric perform differently and are intended to be run at different frequencies. It’s common to only run all of the automated tests once during a nightly build, especially if any of the system tests are long-running and/or resource-intensive enough. It’s common to have a medium-weight suite of tests which a developer is expected to run once prior to committing the code, but which may still take quite a few minutes to run. And I prefer to have a suite of as many automated tests as possible which run fast. If you have a fast running suite of automated tests, then you have a fast feedback loop. I’m talking under twenty seconds, the sort of suite you can run in between every single change, as you work. Such tests most likely make heavy use of mocks and have no dependencies on external resources like db’s, file systems or network connections. Such tests are most likely unit tests.

With Guice, as with any code, the question is: how long can you stand to wait to find out if you implemented, or broke, something?

I would not be able to stand having to start the whole application up every time I was debugging my Guice usage. Bob Lee reminded me that you can still have automated tests that test your usage of Guice, they could be component or system level automated tests, not necessarily unit tests. They could even be included in the fast running suite.

Ease of unit testing

It is so important to keep unit tests, and unit testing, simple. For the sake of all current and future developers working on the code, the barriers to unit testing need to be as few as possible – basically just know JUnit. It’s hard enough as it is to get people to write unit tests. So, having thought about it, now I’m not so sure it’s worth having Guice in unit tests if it burdens my peers. I know how overwhelmed I feel when I work on a project and find I have to learn additional in-house unit test conventions and special TestCase subclasses that I’m expected to start with. It’s an awfully attractive idea to keep unit testing confined to simple JUnit tests, and leave fancier stuff for more complicated integration-style tests.

This is all predicated on the idea that Guice is not mainstream yet, as JUnit is, and would still be a burden to have to learn just to unit test with. If Guice ever becomes more widely used and understood, then I might revert back to my earlier conclusion. After all, Guice is the new new – why not embrace it in unit test code if it is being embraced in production? I like the API very much and think it could easily become part of the unit testing vernacular.

Conclusion

The two forces I mentioned above do not have to be diametrically opposed. Thinking about it now, I could envision component-level or subsystem-level automated tests that do nothing except test the Guice dependency injection for that component. Such a test could still be fast-running if mocks are used as appropriate – Guice itself is performant enough thanks to it’s pure Java implementation. And such tests would naturally force the developer to place the proper Guice annotations in the proper place. Unit tests could remain simple JUnit tests.

Last week was shortened by jury duty on Monday. Fortunately, I was never selected from the pool, and on Tuesday I was back to work.

There are (still) a number of monkey failures (such as this one) that I need to get working on.

However, I discovered I could procrastinate tackling those by checking the forums. I decided to try to answer this post (which has since been addressed by a couple of my teammates). Almost two days later, I conceded that it’s really really hard to try to cluster the underlying javax.swing.text.AbstractDocument of a JTextField. I still haven’t got it. (See gkeim’s response for a clever workaround.)

I finished up the week by working on droid. One of my peers was having trouble running a test in which he wanted one of the spawned workers to have a different tc-config than all the others. There are some weird subtleties in passing vm arguments through the agent which are intended for the worker, but as it turned out, I believe the functionality is already there and didn’t require any changes on my part, just an explanation of how to do it.

At my Guice presentation recently, I used some small code exercises to demonstrate the basics of Guice. Here is my simplest example, involving a Service, a Client needing a Service injected, a test case, and an Application.

First, the Service:

Next, the Client, and it’s test (JUnit 4):

And finally, an Application that ties it all together using Guice:

During the talk, Jeff Grigg had a question followed by an interesting comment. For his question, he asked me to remove the @Inject attribute from Client and re-run the test. I did, thinking I was about to demonstrate a typical helpful Guice Exception. But the test passed. I was momentarily lost, until I remembered that the unit test did not use Guice at all. I ran the Application just as a sanity check, and it was indeed broken.

Jeff was on it like a hobo on a ham sandwich – I had broken the application but the test still passed. My test did not justify the use of the @Inject attribute.

He was absolutely right. I realize now that I had been following the example from the Guice Developer Day Slides, which had taken a non-Guice example and migrated it to Guice, leaving the JUnit test unmodified so that, like my test above, it continued to manually pass a mock to the client code and did not use Guice. This, I realize now, is fine for a tutorial but it not truly Test Driven Development, where every line of code has some test to justify it’s existence.

Naturally, Jeff raised the question – should Guice then be used in unit tests?

I think the answer is yes, why not? After all, Guice suggests it be thought of as the new new, and that using Guice is no more work than plain old factories.

If Guice were used in unit tests, then the tests would naturally require that the code have the necessary @Inject attributes, or whatever other attributes are necessary for Guice. The only difference would be that different Modules would have to be created for the tests, which contained bindings which differed slightly from the production Module(s).

I tried implementing a new client, strictly adhering to TDD and using Guice in my unit tests. Here is what I came up with.

There are two things I notice. One is that my mission is accomplished – my Client’s @Inject attribute is now truly unit tested – the test will fail without it. As a bonus, Client is slightly simplified in that it no longer needs a constructor which accepts a Service. The other thing I notice is that the test is somewhat more complicated, but really it’s only the addition of a Module and Injector.

My feeling is that it’s worth the extra effort to use Guice in unit tests, if you’re going to embrace Guice in your production code anyway.

In a future post, I’d like to investigate whether there is a better way to integrate Guice with JUnit, alleviating the need to write boilerplate unit testing Modules over and over again. Just tonight I noticed both GuiceBerry and AtUnit which potentially address this and look somewhat promising.

I gave my Guice presentation tonight. The best part of the talk for me is that a lot of people chimed in with comments and questions, which is just the way I like it. Thanks to everyone for attending and for generating some good discussion. My talk was comprised mostly of code samples, the presentation materials will soon be available on the JUG knowledge base.

It was great to see some old compadres (Greg, Rasesh, Jeff C and Dennis) as well as introduce myself to some familiar faces in the St. Louis Java community (Michael, Mark, Jeff G, Weiqi, Kyle).

The differences between Guice and Spring were definitely a frequent line of questions and comments. Some open questions and comments that came up related to this are:

• Is it ever preferable to have your application’s configuration truly outside of the source code (a la Spring xml)? Is there value in reconfiguring the app without rebuilding, or is that just a pipe dream? Is external configuration desirable so that non-developers can reconfigure a system, or is that fraught with peril?
• What would be a reason to integrate Guice into an existing code base which uses Spring, especially if you are already leveraging Spring 2.5’s annotation-based dependency injection? I conceded there may be no reason. Kyle Cordes pointed out that in a pristine, pie-in-the-sky world this may be true, but so often in the real world there are large clunky heterogeneous revenue-generating apps and components which need integrating, and you may be in a situation where you have no choice but to integrate Guice with Spring. In that department, Guice scores points for having integration with Spring built in.
• Is there a Guice parallel to Spring’s named instances? That is, in Spring you can configure beans which are named multiple instances of a single concrete type, and said beans can be injected. Jeff C suggested that a Guice Provider might supply that behavior. Kyle pointed out that, by Guice’s very nature you may not need to declaratively configure named instances because Guice entirely operates as Java, and you should be able to just programmatically get or create whatever instances you want. It’s definitely a different mindset than Spring.

Weiqi urged some of us to read the Guice source code, particularly to be impressed by some clever use of generics. I plan to read a little of it right now.

This just in – Weiqi has already written a writeup of my talk on his own blog. Thanks Weiqi, I’m flattered!

This Thursday night I’m giving a presentation on Guice at the St. Louis Java Users Group. Come on out and see!

My talk is shaping up to be mostly code and very few slides. In the first half I will be demonstrating the basics of Guice, what it is and what it can do, via the usual simple contrived code samples that you would expect. In the second half I hope to address how you might introduce Guice to an existing codebase. Specifically, I want to look at Guice’s integration with Spring, and also look at some basic strategies for refactoring out smelly code like Singletons or tightly coupled spaghetti code.

Boy, I hope it turns out as good as the above paragraph suggests!

If you’ve never been to the JUG before I encourage you to come check it out. It’s very informal and relaxed, it’s a great place to network and plug in to the St. Louis Java community. Plus there’s food and refreshments, and almost always cool swag being given away.

One last thing – I would be remiss if I didn’t mention that my company, Terracotta, has support for clustering both Spring and Guice at the JVM level. (There – now I can expense this blog.)

Last week I had the opportunity to use the Extract Method refactoring. I’m generally a proponent of this refactoring – I think it’s a worthy goal to have more numerous, shorter methods, if that means that any given method is more quickly understood at a glance.

The code I had, after I had made some further complicating modifications, looked something like this.

I didn’t like the nearly-duplicate logic duplicated in the if-else block. I felt there was an extract method in there somewhere. However, it took me three times before I came up with a solution I liked, which is basically this:

I won’t bore you with the first two tries, but I’ll just say that for some reason the use of the third defaultTo arg didn’t occur to me right away, but was just what I needed to make the method calls nice and clean.

After I looked at the finished product, I was reminded that another benefit of Extract Method is that differing levels of abstraction are separated. In this case, the determination of whether WebLogic or Tomcat is in use, and what delimiter and default to use to determine the web app name, is a sort of mid-level policy. The nuts and bolts of parsing that app name out of a String instance is the lowest level of abstraction and, in my opinion, is better off not inlined into the policy code. In a similar vein, we would not want to see the implementation of String.substring() inlined into every single invocation of that method.

I don’t think my finished product is perfect. I think maybe there’s a better method name that could be used, which communicates the intent better, because it’s not perfectly obvious at a glance why the delimiter and default need to be passed to the method and what happens. But then again that’s part of the point, we don’t want to know the details. Meanwhile, on the plus side, we see that the same method is invoked in both cases, and only the arguments differ, so it’s obvious that there’s a pattern there which wasn’t necessarily obvious before.