|
|
|
An Interview with TestNG's Cedric Beust
|
Article Topics
An Interview with TestNG's Cedric Beust · JUnit 4 · Dynamic Languages Page 1 of 3
TestNG: The Past, Present, and Future
Cedric Beust's myriad contributions to the Java Community span the gamut from EJBGen to working on JSR175 (Annotations). Not only has his code and his contributions been valuable resources, but his blog is a constant source of ideas and entertainment.
When Cedric isn't coding for his new employer, Google, he spends his time managing TestNG, an innovative developer testing framework designed to overcome some of JUnit's shortcomings. In this interview, Cedric shares his experiences creating a testing framework from the ground up, his thoughts on the future of TestNG and JUnit, and his feelings on dynamic languages.
What is the TestNG elevator pitch? I'm using JUnit, why should I consider another testing framework?
JUnit is a good testing framework, but suffers from severe limitations for non-trivial testing. TestNG offers the following benefits:
- It is non-intrusive (via annotations)
- TestNG cleanly separates the programming model from the runtime (no need to recompile if all you want is to run a different set of tests)
- It makes your daily life easier with pragmatic features such as rerunning failed tests automatically
- TestNG covers more than unit testing with advanced functionalities such as dependent methods, groups, parameters or partial failures
- It has a powerful plug-in API allowing one to generate custom reports or even modify how TestNG works internally
How are people using TestNG?
TestNG offers a lot of functionalities that are quite orthogonal to each other and that users started using in very creative ways. The discussions on the mailing-lists and forums are a constant eye-opener to me and they show me ways in which I had never thought TestNG would be used.
For example, I noticed a certain pattern in the way TestNG users convert from JUnit: whether they use the various converter tools we provide with TestNG (command line, Eclipse or IDEA plug-in) or not, it usually takes a lot less time than they initially planned. It's so quick, in fact, that they start using more advanced functionalities of TestNG. As a result, they end up spending more time thinking about their testing infrastructure than coming up with creative ways to work around limitations of their older framework.
The first TestNG feature that users typically dive into is test groups. Once you realize that categorizing your tests is no longer limited to class and package naming, a whole new world opens up. Your tests are no longer constrained physically (test methods can belong to more than one group while they obviously can't belong to more than one class), but your tests become "fluid": the class barrier is deconstructed and the focus shifts back to the individual test methods and how they relate to each other.
For example, you realize that two different test methods belonging to the "database" and the "web" groups share a common attribute: they take a long time to run. So you also make them belong to the "long" group. Of course, you also have a "short" group which you can run whenever you make a change and which returns almost instantly, while you save the "long" groups for check-in or maybe only for nightly tests.
Once you GO; down this path, you start wondering about strategies for naming your groups and some very interesting discussions have already taken place many times on the forum on this particular topic.
Another popular feature is the fine granular configuration options that TestNG provides. JUnit only allows you to configure your test around test methods (setUp() / tearDown()) but TestNG lets you specify this around classes (beforeTestClass / afterTestClass), around groups beforeGroup / afterGroup) and around entire suites (beforeSuite / afterSuite).
This level of granularity is essential for complex test scenarios when you need to keep your resource consumption as low as possible because it tends to run in excess of megabytes of memory. Being able to allocate and clean up these resources at very precise points adds a lot of flexibility to your testing infrastructure. This precise scoping also makes it easy to track down bugs as only the required state and resources are alive at a given point.
Another TestNG feature that users like a lot is method parameters.
This is actually another consequence of JUnit being the only testing framework and Java programmers have ever used: to a lot of them, testing is synonymous with JUnit, and after so many years using JUnit, they think of testing in terms of JUnit's functionalities.
It was quite obvious to me that being able to have parameters in my test methods was not only a very natural thing to do, it was also much more practical than the cumbersome workarounds I was using until then.
I could GO; on and on but I'll just mention one last feature that I am particularly fond of: rerunning failed tests. I have always found it odd that a testing framework would tell me that a certain test failed and where, but wouldn't let me rerun just this particular test without forcing me to GO; in my code and comment out Java code to insulate the offending methods. Since TestNG automatically generates a testng-failed.xml file whenever it encounters failures, the typical TestNG session is now:
java org.testng.TestNG testng.xml
java org.testng.TestNG testng-failed.xml
|
Users love this feature and can't do without it once they start using it.
Rerunning of failed tests is quite handy! You've been coming out with updated versions on a regular basis- what is your release cycle like? When you decide to push out a new version of TestNG?
It's done in a fairly ad hoc manner: I keep a list of the various changes that GO; into the CVS head and when I believe that we have reached a quorum, I create a new release. Whether it's a minor or a major release depends on the breadth of the changes, but I make a point of keeping the TestNG core and the IDE plug-ins synchronized, so that users don't get any surprise.
What is your take on test coverage tools? Do you use any?
I feel a bit ashamed to admit that I'm not using any. While I recognize their value, my experience with coverage tools is that they tend to make you obsessive about things that don't necessarily matter. Coverage tools can be a distraction- you obsess over the last 10% of code that isn't covered and you forget to test some essential functionality that your users depend on every day.
Don't get me wrong- coverage tools are like all other tools, they can be used well and they can be abused. So far, I just haven't felt that I needed a coverage tool to guarantee that all the features of TestNG are covered adequately and that no regressions will occur.
Was TestNG all based on what JUnit didn't have or did you see admirable features in other frameworks?
JUnit was an obvious inspiration, and I can't discuss the roots of TestNG without paying due respect to Erich Gamma and Kent Beck, who have forever changed the Java world.
The idea of dependent methods was present in a proprietary testing framework that I used at one of my previous jobs. It was created out of necessity: when you test complex systems and each test run involves several thousand test methods, you need to avoid doing extra work, and there are some clear cases where it doesn't make sense to run 100 test methods if one test method failed before that. It's necessary not just to save time, but to get a clear message: I much prefer to read "1 test failed, 100 tests skipped" than "101 tests failed".
Except for these two examples, all the other features came from problems that I faced in the last ten years and that I never found a clean solution to.
For example, I have always thought that using class and package names to simulate some kind of categorization in your tests was ill-formed and didn't scale. At the same time, products like GMail and del.icio.us made tagging popular, and suddenly it hit me: why would I not be able to "tag" my test methods? Assign names (which I later renamed "groups") to them? Once my tests started falling into categories, I realized that I could use these categories to invoke groups selectively and have "laser-focused test runs". It was one of those moments when you realize that things just fall into place naturally and they make so much sense that you wonder why nobody thought of doing that before.
The idea of using annotations was a no-brainer. First of all, because I have been using this technology for more than four years (EJBGen was one of the first frameworks to use the idea of annotations), and also because my involvement in JSR-175 had opened my eyes to the wonderful possibilities that annotations could offer.
Of course, a lot of features came from users and developers as well, and I can't thank them enough for helping us take TestNG to the level where it is now. I'd like to extend a very personal thanks to Alexandru Popescu, who joined the project about six months after I created it, and who has been a tireless contributor to TestNG ever since. Within a few weeks, he implemented the JDK 1.4 portion of TestNG and after that, he was the main developer of the Eclipse plug-in, which has considerably boosted TestNG's popularity. Even as of today, Alexandru is responding to every single email that comes on the mailing-list and he keeps cranking up fixes and improvements at a stunning pace. I don't sleep a lot myself, but I think I have found someone who sleeps even less and spends every waking hour thinking about coding.
The last few releases of the 2.5 line have a ton of features- what's TestNG 3.0 look like?
It will probably not be very different from the current version of TestNG. Over the past year and a half, we have been working hard on the TestNG core and my impression is that it's relatively stable from a feature standpoint right now. All the major features that I wanted in a testing framework are implemented, tested and used a lot.
Recently, I noticed that more and more questions on the forums and the mailing-list gravitate around the tools more than TestNG itself. This makes me pretty happy because it shows that TestNG has reached a very mature stage and that it is ready to move to the next stage.
As a matter of fact, we have spent most of our time these past weeks working on the Eclipse and IDEA plug-ins (the IDEA plug-in is new and now officially available). Some contributors have also joined the development team and released the first versions of a Maven plug-in and a Spring integration module. DBUnit is also present and more frameworks are coming up.
Speaking of frameworks, another very satisfying observation was that adding integration to TestNG ended up being not only easy, but very lightweight. If you look at the spring/ or maven/ directories in the CVS depot, you will be surprised to find only a few classes and XML files that amount to just a few kilobytes.
The credit is not just TestNG's, though: the fact of the matter is that JUnit extensions usually tie into JUnit through their setUp/tearDown methods, and this is obviously a trivial thing to port to TestNG. As a consequence, pretty much any existing JUnit extension is absolutely trivial to port over to TestNG.
|
|
