A few months ago we took a closer look at our testing strategy. We assessed testng and noticed it had a number of features we thought we'd find useful. e.g. data driven testing, ability to run only the tests the failed, configurable testing in xml based on annotations and/or xdoclet tags.
Based on the features and a little prototyping we decided to use testng as a standard test framework. We setup the tests to run on our CI server (hudson) and we were good to go...
And things moved a long smoothly, we used a lot of the extra features of testng and also found that having an xml file that indicated what was a test was quite useful.
But then issues started to surface...
1. Testng cannot run each test on a different VM. The makers of testng do not give you the option to run your tests in a totally new VM so you can potentially run into issues when doing integration like tests requiring the hibernate session factory and database. In junit I have the option of runnning each test case in a totally new VM.
2. Testng doesn't really work with 1.4 xdoclet annotations. Unfortunately we are still on 1.4 and thus had to use the xdoclet annotation mechanism for tagging a test. This was problematic almost all the time. I wouldn't be surprised if the issues we had with testng would be a lot less if we were able to use class annotations.
3. Testng tries to skip tests when the setup fails, i.e. it doesn't fail fase.This was the major issue. The problem was that when a test's setup does fail and thus the test is skipped, testng struggles to identify how many tests were supposed to run. Thus in our CI server we found that the number of tests kept on changing. Permanently. Thus it because very difficult to get predictable results and a good idea of what is going on in the tests.
4. The Eclipse plugin for testng is not as good as junit's. The developers in my shop were unhappy partly because of this. When I announced that we were moving back to junit they were happy. Some of them had continued to subclass TestCase because they felt the assertions provided were a lot richer when this is done (and they are). With Testng, because it is not required that you subclass when creating a test, you generally use the java keyword "assert".
So now we are moving off testng back onto the old faithful, junit. There are however tests which are built with the data provider and/or parameter subsystems which testng offers and these will stay testng tests. We have no plans to move these off testng. And if our developers really need the extra functionality provided by testng, they can use it, within reason - our testng CI run has not totally gone away.
The approach now is, if the tests only requires the basic features which junit3 has, then use junit. If you need testng functionality, get permission for it before you use it.
Based on the features and a little prototyping we decided to use testng as a standard test framework. We setup the tests to run on our CI server (hudson) and we were good to go...
And things moved a long smoothly, we used a lot of the extra features of testng and also found that having an xml file that indicated what was a test was quite useful.
But then issues started to surface...
1. Testng cannot run each test on a different VM. The makers of testng do not give you the option to run your tests in a totally new VM so you can potentially run into issues when doing integration like tests requiring the hibernate session factory and database. In junit I have the option of runnning each test case in a totally new VM.
2. Testng doesn't really work with 1.4 xdoclet annotations. Unfortunately we are still on 1.4 and thus had to use the xdoclet annotation mechanism for tagging a test. This was problematic almost all the time. I wouldn't be surprised if the issues we had with testng would be a lot less if we were able to use class annotations.
3. Testng tries to skip tests when the setup fails, i.e. it doesn't fail fase.This was the major issue. The problem was that when a test's setup does fail and thus the test is skipped, testng struggles to identify how many tests were supposed to run. Thus in our CI server we found that the number of tests kept on changing. Permanently. Thus it because very difficult to get predictable results and a good idea of what is going on in the tests.
4. The Eclipse plugin for testng is not as good as junit's. The developers in my shop were unhappy partly because of this. When I announced that we were moving back to junit they were happy. Some of them had continued to subclass TestCase because they felt the assertions provided were a lot richer when this is done (and they are). With Testng, because it is not required that you subclass when creating a test, you generally use the java keyword "assert".
So now we are moving off testng back onto the old faithful, junit. There are however tests which are built with the data provider and/or parameter subsystems which testng offers and these will stay testng tests. We have no plans to move these off testng. And if our developers really need the extra functionality provided by testng, they can use it, within reason - our testng CI run has not totally gone away.
The approach now is, if the tests only requires the basic features which junit3 has, then use junit. If you need testng functionality, get permission for it before you use it.
No comments:
Post a Comment