Introducing RichFaces Framework Tests

This post is a wrap-up for the Testing JSF and RichFaces with Arquillian series which I started with:

As we announced earlier, we are undergoing several changes in our approach to how RichFaces is tested. We already have an extensive functional test suite authored by a top-notch RichFaces QE team, but this suite doesn’t allow us to react quickly on regressions introduced and so we usually find some of them during release testing.

The series was divided to four parts

which pretty much covers what we have achieved in last couple of months. And every bit of that effort we are sharing with our community in form of Arquillian extensions - Warp and Graphene.

Still, there are few points I would like to share with you in relation to the architecture of the test suite which is rather specific for testing the RichFaces framework.

Test Architecture Requirements

The special requirements for running integration tests in variety of supported environments:

JSF implementations: Mojarra, MyFaces, Mojarra in JBoss
browsers: Chrome, Firefox, Internet Explorer
application servers: JBoss EAP 6.1, WildFly, JBoss AS 7, TomEE, GlassFish
servlet containers: Tomcat

Additionaly, some related requirements:

reusing our test harness across subprojects
issue reproduction & debugging
- testing against older versions of the framework
- testing against different JSF implementations
smoke testing
- build stability
- post commit verification
- pull request verification
test categorization
- limiting the scope of execution
- for tests failing globally or in a specific environment
resolution of a container binary from Maven

I have to say that to cover all those aspects is quite a challenge. :-)

TL;DR

There is a complete Framework Tests documentation which covers all the aspects of authoring, executing and maintaining the test suite.

You will also find there all the information needed to give those tests a try.

Matrix Jobs

As our QE team does a really good job with test automation, they are getting a new toy and challenge at the same time: test everywhere.

Thanks to extensive Arquillian support for various browsers and application servers in different run modes (managed/remote), it is pretty straight forward to run on different implementations.

The matrix jobs are run on a Jenkins continuous integration server.

Issue Reproduction & Debugging

From the point an issue is reproduced, it needs to be reproduced in a sandbox environment.

In order to be efficient during issue reproduction, we prepare framework tests in every case where it seems beneficial.

In case you are interested in how we write those tests, I have dedicated a special post to this topic.

Test Categorization

Some tests aren’t guaranteed to work on every supported environment for two reasons:

the test is not able to run in a target environment
the test hits a bug which is reproducible only in a given environment

We could just accept the fact that, in some environments, certain tests are failing. However we found that in order to verify results of continous jobs in a straight-forward manner, we should only run tests which are known to be stable on the target environment.

One of our many supported categories are "smoke tests".

Smoke Testing

Smoke tests help us to evaluate that a certain commit is stable enough so that it can be integrated into the master branch and it can be further assessed by more advanced test suites.

Before we introduced framework tests, we simply used unit tests for this assessment which themselves didn’t sufficiently guard the gramework.

Fortunately we are able to run a limited subset of framework tests on a headless browser to ensure that framework and components fundamentally work.

Developers are encouraged to run smoke tests before they push commits upstream (it takes just 10 minutes to run locally). Additionally it is beneficial to run a subset of all tests which could be potentially affected by changes (e.g. for a family of components).

Continous Integration in a Cloud

For the sake of a peaceful sleep, we run smoke tests in a cloud on post-commit basis.

After evaluating several cloud providers, we have decided to go with Travis CI.

In order to build in Travis CI, you just need to add .travis.yml file and enable a post-commit hook in your GitHub repository configuration.

Travis CI will make sure to verify not only the master branch, but also all feature branches and pull requests.

With both tools, you can now just send an patch (pull request) to the repository and you will get feedback within half an hour!

Conclusion

RichFaces Framework Tests and their architecture really goes further than the usual Arquillian user’s needs.

However the setup is very flexible and allows us to do test-driven development in as complex environment as JSF.

We strongly believe that framework tests are the way to deliver an even more stable library.