Introducing RichFaces Framework Tests
June 8, 2013
This post is a wrap-up for the Testing JSF and RichFaces with Arquillian series which I started with:
As we announced earlier, we are undergoing several changes in our approach to how RichFaces is tested. We already have an extensive functional test suite authored by a top-notch RichFaces QE team, but this suite doesn’t allow us to react quickly on regressions introduced and so we usually find some of them during release testing.
The series was divided to four parts
Still, there are few points I would like to share with you in relation to the architecture of the test suite which is rather specific for testing the RichFaces framework.
Test Architecture Requirements
The special requirements for running integration tests in variety of supported environments:
JSF implementations: Mojarra, MyFaces, Mojarra in JBoss
browsers: Chrome, Firefox, Internet Explorer
application servers: JBoss EAP 6.1, WildFly, JBoss AS 7, TomEE, GlassFish
servlet containers: Tomcat
Additionaly, some related requirements:
reusing our test harness across subprojects
issue reproduction & debugging
testing against older versions of the framework
testing against different JSF implementations
post commit verification
pull request verification
limiting the scope of execution
for tests failing globally or in a specific environment
resolution of a container binary from Maven
I have to say that to cover all those aspects is quite a challenge. :-)
There is a complete Framework Tests documentation which covers all the aspects of authoring, executing and maintaining the test suite.
You will also find there all the information needed to give those tests a try.
As our QE team does a really good job with test automation, they are getting a new toy and challenge at the same time: test everywhere.
Thanks to extensive Arquillian support for various browsers and application servers in different run modes (managed/remote), it is pretty straight forward to run on different implementations.
The matrix jobs are run on a Jenkins continuous integration server.
Issue Reproduction & Debugging
From the point an issue is reproduced, it needs to be reproduced in a sandbox environment.
In order to be efficient during issue reproduction, we prepare framework tests in every case where it seems beneficial.
In case you are interested in how we write those tests, I have dedicated a special post to this topic.
Some tests aren’t guaranteed to work on every supported environment for two reasons:
the test is not able to run in a target environment
the test hits a bug which is reproducible only in a given environment
We could just accept the fact that, in some environments, certain tests are failing. However we found that in order to verify results of continous jobs in a straight-forward manner, we should only run tests which are known to be stable on the target environment.
One of our many supported categories are "smoke tests".
Smoke tests help us to evaluate that a certain commit is stable enough so that it can be integrated into the
master branch and it can be further assessed by more advanced test suites.
Before we introduced framework tests, we simply used unit tests for this assessment which themselves didn’t sufficiently guard the gramework.
Fortunately we are able to run a limited subset of framework tests on a headless browser to ensure that framework and components fundamentally work.
Developers are encouraged to run smoke tests before they push commits upstream (it takes just 10 minutes to run locally). Additionally it is beneficial to run a subset of all tests which could be potentially affected by changes (e.g. for a family of components).
Continous Integration in a Cloud
For the sake of a peaceful sleep, we run smoke tests in a cloud on post-commit basis.
After evaluating several cloud providers, we have decided to go with Travis CI.
In order to build in Travis CI, you just need to add
.travis.yml file and enable a post-commit hook in your GitHub repository configuration.
With both tools, you can now just send an patch (pull request) to the repository and you will get feedback within half an hour!
RichFaces Framework Tests and their architecture really goes further than the usual Arquillian user’s needs.
However the setup is very flexible and allows us to do test-driven development in as complex environment as JSF.
We strongly believe that framework tests are the way to deliver an even more stable library.