In this guide you will know how to Integrate your recorded/coded tests using CircleCI Pipelines
Background:
This tutorial explains how to integrate TestProject with CircleCI CI/CD pipeline with the following scenarios
1.
Running TestProject recorded tests with CircleCI CI/CD Pipelines
2.
Running TestProject coded python tests with CircleCI CI/CD Pipelines
3.
Running TestProject coded Java tests with CircleCI CI/CD Pipelines
Running TestProject recorded tests with CircleCI CI/CD Pipelines
Today’s development cycles include fast releases and require agile testing methods, developers and testers need to collaborate to release tested features faster. Integration of test automation within the software CI/CD plays a significant with agile development cycle and better software to be released to production faster.
TestProject cloud offers seamless integration with any CI/CD flow by using its Restful API which allows performing numerous operations, as well as execute Tests on-premise and on remote agents, both ad hoc and permanent agents,.
Below is a diagram that explains high-level TestProject architecture and the prerequisites needed for CI/CD :
Click on the Authorize button and paste your API key and click on Authorize.
By using TestProject Restful API, we will create a curl command that will execute the desired recorded tests using a CURL command in the Pipeline.
Generating the Curl Command
In the API Swagger, we will scroll down to the Endpoint we wish, in this example we will use the Run Job Endpoint which can be found here.
With this Endpoint, we can execute a Test on any registered agent in the account associated with our API key.
All values that are passed in the body of this request will override the DEFAULT job values
In this example, we will run a Recorded Web test on Chrome using the Run Job endpoint.
Setting up the Endpoint:
In the “Run” Job endpoint, click on ‘Try It Out’
We will insert the Project ID:
And the Job ID:
Next, we can edit the body of the request, here we can insert values that will override the default job values on runtime.
To just run the Job as is, with the default values, you can just keep the body empty like so:
In our example, we will edit the values of the body to override default values from the Job.
We will override the default params username and password in the test with custom values.
The job will have two tests, both have two parameters in them called username and password.
We will override those parameters on runtime using the testParameters JSON which we pass in the body of the request.
This is how the parameters are defined in the tests:
This is how we override them using the request body:
We will now generate the CURL command by pressing Execute:
And we will see the CURL command generated:
This is the curl command we will be used in our pipeline.
Now we can set up our pipeline using the above command to execute the Job every time we push a new change to the defined branch.
Setting up the Pipeline in CircleCI on a Permanent Agent
1.
In your Dashboard, add a Project, this will be the project which will have the pipeline, once we push a new change to this project, we will execute the pipeline.
2. Select the project
3. In the following window, we will have the option to choose pre-configured YAML files for our pipeline, in this example, I will select the Python file.
4. Paste the curl command in the proper step.
5. Click on Commit and Run to configure the pipeline in the selected branch
Now every time we push a change to the selected branch, the Pipeline will trigger the Curl command which in the following will trigger the tests.
command: curl -X POST "Request URL" -H "accept:application/json" -H "Authorization:YOUR API KEY" -H "Content-Type:application/json" -d "{'testParameters': [{'data': [{'username': 'TestProject','passowrd': $PASSWORD}]}]}"
14
name: Run Test On Build
15
workflows:
16
main:
17
jobs:
18
- build-env
19
- run-recorded-tests
Copied!
You can view the pipeline execution in the Dashboard:
Running TestProject recorded tests with CircleCI CI/CD Pipelines using Docker Ephemeral Agent
In the previous example, we saw how to execute tests on a permanent agent by supplying a Job ID and API key. This example will explain how to execute Web tests on Docker agents, which will allow you to spin up agents on demand. We will utilize docker-compose to create agents on the fly for a specific job to run and terminate them afterward.
It will also allow you to execute multiple tests in Parallel within your pipeline.
Below is a diagram on the flow of the process:
First, we will need to config the correct environment variables.
Configuring Environment Variables in CircleCI:
The following config below uses these Environment variables:
You can set them in CircleCI Dashboard:
1.
Go to project settings:
2. Click on Environment Variables:
3. Add Environment Variable:
Configuring Docker Compose YML file
You will need a docker-compose.yml file that will spin up the Docker agent.
In this example we are overriding the username/password parameters that are used in the tests by using this YAML file:
Using the above compose file, we will spin a docker agent which will execute the Job stated in $TP_JOB_ID.
The variable TP_JOB_PARAMS holds the JSON which holds the username and password parameters that will be overridden in runtime.
Using this config in the pipeline to execute the Job on the Docker agent.
1
version: 2.1
2
jobs:
3
run-recorded-tests:
4
machine:
5
image: ubuntu-2004:202010-01
6
steps:
7
- checkout
8
- run:
9
name: Run tests
10
command: |
11
set -x
12
docker-compose up --abort-on-container-exit
13
14
workflows:
15
main:
16
jobs:
17
- run-recorded-tests
18
Copied!
You can read more on TestProject Docker agent here.
Running TestProject coded Python tests with CircleCI CI/CD Pipelines
Using TestProject Python SDK, you can seamlessly integrate existing/new Python code to your Pipeline.
With a simple process, you will be able to run your tests with every new change that is pushed and enjoy Cloud and Local reports directly to your TestProject account.
Below is a diagram on the flow of the process:
Before integrating your coded tests with CircleCI, we need to make sure you are using our Python SDK.
Once we have set up our code, we can integrate our coded python test using CircleCI.
In CircleCI, open a new project like shown previously, and select the Project for your coded test.
Like previously, we will select a pre-configured Python yml file, and configure it to run our Tests.
Here is a complete example of a config.yml that does the following:
1.
Creates a clean ubuntu 20.04 environment.
2.
Spins up docker container using docker-compose from a public example’s repo from TestProject.
3.
Runs the Example python tests on the Docker Agent using Pytest.
Note: When using Docker Agent on CircleCI, you must use Machine config and Not Docker Executor.
Config.yml:
1
version: 2
2
jobs:
3
build:
4
machine:
5
image: ubuntu-2004:202010-01
6
steps:
7
- checkout
8
- run:
9
name: Install Requirements
10
command: pip3 install -r requirements.txt
11
12
- run:
13
name: Start Agent and wait for agent to register
14
command: |
15
set -x
16
docker-compose up -d
17
trap 'kill $(jobs -p)' EXIT
18
attempt_counter=0
19
max_attempts=45
20
mkdir -p build/reports/agent
21
docker-compose -f docker-compose.yml logs -f | tee build/reports/agent/log.txt&
22
until curl -s http://localhost:8585/api/status | jq '.registered' | grep true; do
23
if [ ${attempt_counter} -eq ${max_attempts} ]; then
24
echo "Agent failed to register. Terminating..."
25
exit 1
26
fi
27
attempt_counter=$(($attempt_counter+1))
28
sleep 1
29
done
30
31
- run:
32
name: Run Tests
33
command: pytest python_example.py
34
Copied!
The following config above uses these environment variables:
You can set them in CircleCI Dashboard:
1.
Go to project settings:
2. Click on Environment Variables:
3. Add Environment Variable:
Now you are good to go running your Python Coded tests using TestProject SDK in your pipeline.
Running TestProject coded Java tests with CircleCI CI/CD Pipelines
Integrating your Coded Java tests is very similar to Integrating Python tests like shown above.
You need the save Environment variables, just need to change the config.yml to work with your Java environment.
We will use Gradle in this example.
First, we set up our build.gradle to point to the classes which hold our tests.
In this simple example, we will use one class called JavaExample which holds the test using JUNIT, and we will be using Gradle wrapper(gradlew) to execute the tests.