Core principles of good tests
When you start automating your tests, there a few important principles to stick to.
Reliable
First your tests must be reliable. This means that they must repeatable and always give the same results. They should not contain any environment dependencies, and be executable on any environment. To get this working, all hard-coded settings must be configurable, a maven or ant script is required to build and configure the test environment, and all code, test date and configuration scripts must be checked in source code repository. Having reliable tests that run everywhere and always, also means that your tests must take care to setup necessary test data, and clean it afterward again.
Fast
Next, it would be nice when your tests are fast. In the lifetime of a project you will build up ever more tests; with every next sprint you will need to run more tests. You can't run all your tests on check-in, and for certain groups of tests need to schedule test execution times at less regular intervals.
Self explaining tests
Also your tests should be easy to understand, and have clean reports. Easy to understand means that an outsider can quickly understand what the test is about. Proper naming helps a lot here, additionally you can add document tags to the comments. It also helps to stick to the domain language so that it is easy to map test cases to use cases.
Clean reports
Test reports should have a clean baseline. Try to avoid including known bugs, as they clog up the test results, and make the news bugs less visible. Also in your asserts add meaningful log statements that help finding the root cause of the failure.
Continous integration
Further you want your tests be integrated in continous integration build process. You will probably want to create several groups and have some of these more frequently executed than others. Once you have them running automatically, you are sure that they actually get executed, and you also make sure that they are really reproducible. Before you add your tests to CI you must make sure though that your test reports have a clean baseline. Else people will ignore the test reports.
Continuous integration of test environment setup
Deployment and execution of tests in CI environments is the easy part. Far more difficult it may be to set up and configure your test environment. This includes setting up your application server, initialising your database, starting your application server, deploying your application and configuration of appl server and application. This will require a lot of specialist knowledge.
One step further you may also want to mock your connection proxies to external connections. This may be for instance a credit card provider. You don't want to send your payments that you generate in tests to a real provider. So you will need to mock the provider's functionality and have your test environment set up with this mock interface instead of the real one.
Testability
A last point of attention concerns testability. Some of your application's functionality may be difficult to test as you can't reach it with your testing code. Examples could be a status of an object that you have just changed, and can only be checked by a UI application. As a tester you would like to have direct access to this functionality, so that you can use it for the asserts of the test. These testability functionalities are additional requests to the developers. They will need to build it in the code, just for the sake of testing. Other important testability examples are: fixed IDs for html elements, or some deleteFunctionality to clean up test generated objects.
When you start automating your tests, there a few important principles to stick to.
Reliable
First your tests must be reliable. This means that they must repeatable and always give the same results. They should not contain any environment dependencies, and be executable on any environment. To get this working, all hard-coded settings must be configurable, a maven or ant script is required to build and configure the test environment, and all code, test date and configuration scripts must be checked in source code repository. Having reliable tests that run everywhere and always, also means that your tests must take care to setup necessary test data, and clean it afterward again.
Fast
Next, it would be nice when your tests are fast. In the lifetime of a project you will build up ever more tests; with every next sprint you will need to run more tests. You can't run all your tests on check-in, and for certain groups of tests need to schedule test execution times at less regular intervals.
Self explaining tests
Also your tests should be easy to understand, and have clean reports. Easy to understand means that an outsider can quickly understand what the test is about. Proper naming helps a lot here, additionally you can add document tags to the comments. It also helps to stick to the domain language so that it is easy to map test cases to use cases.
Clean reports
Test reports should have a clean baseline. Try to avoid including known bugs, as they clog up the test results, and make the news bugs less visible. Also in your asserts add meaningful log statements that help finding the root cause of the failure.
Continous integration
Further you want your tests be integrated in continous integration build process. You will probably want to create several groups and have some of these more frequently executed than others. Once you have them running automatically, you are sure that they actually get executed, and you also make sure that they are really reproducible. Before you add your tests to CI you must make sure though that your test reports have a clean baseline. Else people will ignore the test reports.
Continuous integration of test environment setup
Deployment and execution of tests in CI environments is the easy part. Far more difficult it may be to set up and configure your test environment. This includes setting up your application server, initialising your database, starting your application server, deploying your application and configuration of appl server and application. This will require a lot of specialist knowledge.
One step further you may also want to mock your connection proxies to external connections. This may be for instance a credit card provider. You don't want to send your payments that you generate in tests to a real provider. So you will need to mock the provider's functionality and have your test environment set up with this mock interface instead of the real one.
Testability
A last point of attention concerns testability. Some of your application's functionality may be difficult to test as you can't reach it with your testing code. Examples could be a status of an object that you have just changed, and can only be checked by a UI application. As a tester you would like to have direct access to this functionality, so that you can use it for the asserts of the test. These testability functionalities are additional requests to the developers. They will need to build it in the code, just for the sake of testing. Other important testability examples are: fixed IDs for html elements, or some deleteFunctionality to clean up test generated objects.
Comments