TestProject Agent CLI

TestProject Agent CLI (Command-Line Interface)
TestProject Agent CLI
TestProject Agent CLI is a cross operating system command line interface utility.  It is made for execution of test packages generated using the TestProject platform.
Exported test packages are stand alone textual YAML files. These file can be stores outside of the TestProject cloud, including in an on-premises version control such as git.
Test Packages
A package can be either a single YAML file that contains a test, and its auxiliary (nested and recovery) tests, or an archive (ZIP) that is called a bundle and will contain additional files besides the test itself.
A single YAML file doesn't contain any dependencies like apps, coded tests, or addons that are used in the test.  The assumption is that they were already downloaded by the executing Agent or will be downloaded on demand from the TestProject cloud over the internet by the executing Agent before an execution.
A package bundle, on the contrary to a single file, will contain all the dependencies (apps, coded tests or addons) and additional files such as settings and parameters for overriding default values and data-driven execution.
Package Bundle Structure
The following files tree demonstrates a package bundle structure:
package.zip
├── addons
│   ├── {addon-guid}
│   ├── {addon-guid}
│   ├── {addon-guid}
├── apps
│   ├── {app-guid}.{ipa/apk}
│   ├── {app-guid}.{ipa/apk}
│   ├── {app-guid}.{ipa/apk}
├── coded-tests
│   ├── {test-guid}.{jar/dll/zip}
│   ├── {test-guid}.{jar/dll/zip}
│   ├── {test-guid}.{jar/dll/zip}
├── package.{yaml}
├── project-parameters.{yaml}
├── settings.{yaml}
└── test-parameters.csv
Running the CLI
TestProject Agent CLI allows interaction with the Agent and specifically execution of packages.
CLI can be started from the Agent installation folder or from any other location by executing the following command from the shell:
[email protected]:~$ testproject-agent
Or via command prompt / PowerShell in Windows:
testproject-agent.exe
When started without parameters or with the help parameter, CLI will display self-explanatory usage instructions, e.g.
[email protected]:~$ testproject-agent help
Will print:
Usage: testproject-agent [-hv] [-A=<agent>] [-C=<grpcTimeout>] [COMMAND]
  -A, --agent=<agent>   Target Agent (host:port)
  -C, --connect-timeout=<grpcTimeout>
​
  -h, --help            Print usage information.
  -v, --version         Print version information and exit
Commands:
  start     Starts the Agent
  connect   Connects to an Agent and prints its version
  register  Registers an Agent using provided API or Development token
  browsers  List available browsers
  devices   List and query available devices
  validate  Inspects and validates an execution package file(s)
  run       Runs an execution package file(s)
Running it with the -v or --version option will print the TestProject Agent CLI version, e.g.
[email protected]:~$ testproject-agent -v
Will print:
3.0.0-RELEASE
Commands
TestProject Agent CLI allows invocation of various features using a command parameter.  The following sections describe all the supported commands.
Start
This command allows starting a local Agent. Before CLI attempts to start the Agent, it verifies that no other Agent process is running.
[email protected]:~$ testproject-agent start
The command above with no options specified will start the Agent from within the CLI. CLI process will keep running until the Agent quits or is shut down.
Here's the command help:
Usage: testproject-agent start [-fhjsV] [-A=<agent>] [-C=<grpcTimeout>]
                               [-D=<dataPath>] [--grpc-address=<grpcAddress>]
                               [--max-workers=<maxWorkers>]
                               [--rest-address=<restAddress>]
Starts the Agent
  -A, --agent=<agent>   Target Agent (host:port)
  -C, --connect-timeout=<grpcTimeout>
                        Connection timeout in seconds
  -D, --data-path=<dataPath>
                        Path to Agent's data folder
  -f, --fork            Fork current process and exit after Agent is started
      --grpc-address=<grpcAddress>
                        Agent's spawned gRPC server address (host:port)
  -h, --help            Show this help message and exit.
  -j, --json            Change default output format to JSON
      --max-workers=<maxWorkers>
                        Agent's maximum amount of workers.
      --rest-address=<restAddress>
                        Agent's REST API address (host:port)
  -s, --headless        Run Agent headless (do not show system tray icon)
  -V, --version         Print version information and exit.
Fork
Using the -f or --fork option CLI process can be forked when the Agent process is started.  In this mode, no STD will be attached and CLI will not wait for the Agent to exit.  This might be useful in CI / CD pipelines and various configurations.
External Connectivity
By default, Agent starts its gRPC and Web servers, binding them to localhost.  Starting from Agent version 2.2.0, to allow external connectivity for the CLI or the SDK incoming requests, additional parameters should be used:
Using --grpc-address, Agent can be instructed for binding its gRPC to other host and port.
Using --rest-address, Agent can be instructed for binding its Web Server to other hosts and port.
[email protected]:~$ testproject-agent start --grpc-address 10.0.0.2:65000 --rest-address 0.0.0.0:65001
Command above will start the Agent, instructing it for binding its:
gRPC server to a specific IP 10.0.0.2 and port 65000
Web server to all the available interfaces on the machine it's running on, and port 65001
This way, CLI can communicate with the started Agent via 10.0.0.2:65000,  and SDK can communicate with the Agent using any of the available hosts and port 65001.
Max Workers
Using the --max-workers option, it is possible to set the maximum amount of workers for an Agent. Workers amount is the number of tasks an Agent can perform in parallel.
A task can be a test/job execution, a recording session, or an OpenSDK development session. Make sure this is a reasonable value. Too many concurrent executions can impact automation performance due to CPU and Memory starvation. Typically this value should not exceed 50% of the available CPU cores and 2GB of RAM per worker.
For example, if the host has 8 CPU cores, it is recommended to not exceed 4 Agent workers.
Connect
This command connects to an Agent and prints its version:
[email protected]:~$ testproject-agent connect
Will print:
Agent Version: [1.0]
If there is no Agent running, a relevant error message will appear.
Register
This command allows registering an Agent when it is not registered.
Agent must be running before a registration attempt.  It can be started using the start command if necessary.
Here's the command usage help:
[email protected]:~$ testproject-agent register --help
​
Usage: testproject-agent register [-hV] -a=<alias> [-A=<agent>]
                                  [-C=<grpcTimeout>] -t=<token>
Registers an Agent using provided API or Development token
  -a, --alias=<alias>   Alias to use when naming the Agent
  -A, --agent=<agent>   Target Agent (host:port)
  -C, --connect-timeout=<grpcTimeout>
                        Connection timeout in seconds
  -h, --help            Show this help message and exit.
  -t, --token=<token>   Token to use for registration
  -V, --version         Print version information and exit.
Notice the -a or --alias and the -t or --token required options that should be used to invoke the command properly.  For example right after installation, you can run the following command to register the Agent as MyAgent:
[email protected]:~$ testproject-agent register -a MyAgent -t ***
Token
The token used in the command above (masked as triple asterisks) is actually an API or a Development token.  Any of them can be obtained from the Integrations page on the TestProject platform.
If the Agent is already registered, a relevant error message will appear.
Browsers
This command queries the Agent and provides a list of browsers that are installed on the machine where Agent is running.  It can be used to efficiently configure the execution settings with a list of available browsers for Web test execution.
[email protected]:~$ testproject-agent browsers
Running the command shown above will result in an output similar to the below:
+=========+===============+
| type    | version       |
|=========|===============|
| EDGE    | 88.0.705.81   |
| CHROME  | 88.0.4324.192 |
| SAFARI  | 14.0.3        |
| FIREFOX | 85.0          |
+=========+===============+
It can be formatted as JSON if an additional option --json is set:
[email protected]:~$ testproject-agent browsers --json
Will result in:
{
  "browsers": [{
    "type": "EDGE",
    "version": "88.0.705.81"
  }, {
    "type": "CHROME",
    "version": "88.0.4324.192"
  }, {
    "type": "SAFARI",
    "version": "14.0.3"
  }, {
    "type": "FIREFOX",
    "version": "85.0"
  }]
}
Information about available browsers can be used to configure execution settings.
Chrome and Firefox have a headless operation mode.  When configuring execution settings, use CHROME_HEADLESS and FIREFOX_HEADLESS when required.
Devices
This command allows listing connected devices (also Android emulators or iOS simulators) to an Agent.  It also allows listing Apps installed on a device.
Here's a usage help printout:
[email protected]:~$ testproject-agent devices list --help
​
Usage: testproject-agent devices list [-hjV] [-A=<agent>] [-C=<grpcTimeout>]
                                      [--android | --ios] [--physical |
                                      --virtual]
List available devices
  -A, --agent=<agent>   Target Agent (host:port)
  -C, --connect-timeout=<grpcTimeout>
                        Connection timeout in seconds
  -h, --help            Show this help message and exit.
  -j, --json            Change default output format to JSON
  -V, --version         Print version information and exit.
Mobile Platform
      --android         List only Android devices
      --ios             List only iOS devices
Device Type
      --physical        List only physical devices
      --virtual         List only virtual (emulator or simulator) devices
List
When the list parameter specified, CLI will list devices:
[email protected]:~$ testproject-agent devices list
Will result in a similar output to the following:
+======+==========+==========+=======================================+=======================================+=========+
| udid | platform | type     | name                                  | model                                 | version |
|======|==========|==========|=======================================|=======================================|=========|
| ***  | IOS      | PHYSICAL | iPad                                  | MP2G2                                 | 14.4    |
| ***  | IOS      | VIRTUAL  | iPad Pro (11-inch)                    | iPad Pro (11-inch)                    | 12.4    |
| ***  | IOS      | VIRTUAL  | iPhone SE                             | iPhone SE                             | 12.4    |
| ***  | IOS      | VIRTUAL  | iPhone 5s                             | iPhone 5s                             | 12.4    |
| ***  | IOS      | VIRTUAL  | iPhone 6                              | iPhone 6                              | 12.4    |
| ***  | IOS      | VIRTUAL  | iPhone 6 Plus                         | iPhone 6 Plus                         | 12.4    |
| ***  | IOS      | VIRTUAL  | iPhone 6s                             | iPhone 6s                             | 12.4    |
| ***  | IOS      | VIRTUAL  | iPhone 6s Plus                        | iPhone 6s Plus                        | 12.4    |
| ***  | IOS      | VIRTUAL  | iPhone 7                              | iPhone 7                              | 12.4    |
| ***  | IOS      | VIRTUAL  | iPhone 7 Plus                         | iPhone 7 Plus                         | 12.4    |
| ***  | IOS      | VIRTUAL  | iPhone 8                              | iPhone 8                              | 12.4    |
| ***  | IOS      | VIRTUAL  | iPhone 8 Plus                         | iPhone 8 Plus                         | 12.4    |
| ***  | IOS      | VIRTUAL  | iPhone X                              | iPhone X                              | 12.4    |
| ***  | IOS      | VIRTUAL  | iPhone Xs                             | iPhone Xs                             | 12.4    |
| ***  | IOS      | VIRTUAL  | iPhone Xs Max                         | iPhone Xs Max                         | 12.4    |
| ***  | IOS      | VIRTUAL  | iPhone Xʀ                             | iPhone Xʀ                             | 12.4    |
| ***  | IOS      | VIRTUAL  | iPad (5th generation)                 | iPad (5th generation)                 | 12.4    |
| ***  | IOS      | VIRTUAL  | iPad (6th generation)                 | iPad (6th generation)                 | 12.4    |
| ***  | IOS      | VIRTUAL  | iPad Air                              | iPad Air                              | 12.4    |
| ***  | IOS      | VIRTUAL  | iPad Air (3rd generation)             | iPad Air (3rd generation)             | 12.4    |
| ***  | IOS      | VIRTUAL  | iPad Air 2                            | iPad Air 2                            | 12.4    |
| ***  | IOS      | VIRTUAL  | iPad Pro (10.5-inch)                  | iPad Pro (10.5-inch)                  | 12.4    |
| ***  | IOS      | VIRTUAL  | iPad Pro (12.9-inch)                  | iPad Pro (12.9-inch)                  | 12.4    |
| ***  | IOS      | VIRTUAL  | iPad Pro (12.9-inch) (2nd generation) | iPad Pro (12.9-inch) (2nd generation) | 12.4    |
| ***  | IOS      | VIRTUAL  | iPad Pro (12.9-inch) (3rd generation) | iPad Pro (12.9-inch) (3rd generation) | 12.4    |
| ***  | IOS      | VIRTUAL  | iPad Pro (9.7-inch)                   | iPad Pro (9.7-inch)                   | 12.4    |
| ***  | ANDROID  | PHYSICAL | google-pixel_4a-0A301JEC217496        | Pixel 4a                              | 30      |
| ***  | ANDROID  | VIRTUAL  | emulator-5554                         | Android SDK built for x86_64          | 29      |
+======+==========+==========+=======================================+=======================================+=========+
There are additional options available to narrow down the list:
Using the --android or --ios options, list can be filtered by a mobile platform.
Using the --physical or --virtual options, either physical or virtual devices only can be requested.
Another useful option is the --json, that will output results as JSON, for example:
[email protected]:~$ testproject-agent devices list --physical --json
Will result in a similar output:
{
  "devices": [{
    "udid": "***",
    "platform": "IOS",
    "type": "PHYSICAL",
    "name": "iPad",
    "model": "MP2G2",
    "version": "14.4"
  }, {
    "udid": "***",
    "platform": "ANDROID",
    "type": "PHYSICAL",
    "name": "google-pixel_4a",
    "model": "Pixel 4a",
    "version": "30"
  }]
}
Apps
When the apps parameter specified, CLI will list apps installed on a device.  It requires an additional argument -d or --device to be specified, for example:
[email protected]:~$ testproject-agent devices apps -d ***
Assuming the triple asterisks above are an actual UDID of a device, this command will print a list of apps installed on a device with this UDID, for example here's a printout of apps on an Android device:
+============+=========================================+===============================================================+
| name       | packageName                             | activityName                                                  |
|============|=========================================|===============================================================|
| Calculator | com.google.android.calculator           | com.android.calculator2.Calculator                            |
| Calendar   | com.google.android.calendar             | com.android.calendar.AllInOneActivity                         |
| Camera     | com.google.android.GoogleCamera         | com.android.camera.CameraLauncher                             |
| Chrome     | com.android.chrome                      | com.google.android.apps.chrome.Main                           |
| Clock      | com.google.android.deskclock            | com.android.deskclock.DeskClock                               |
| Contacts   | com.google.android.contacts             | com.android.contacts.activities.PeopleActivity                |
| Drive      | com.google.android.apps.docs            | com.google.android.apps.docs.app.NewMainProxyActivity         |
| YouTube    | com.google.android.youtube              | com.google.android.youtube.app.honeycomb.Shell$HomeActivity   |
+============+=========================================+===============================================================+
Or something similar for an iOS device:
+===========+===================+=======================================+
| name      | version           | bundleId                              |
|===========|===================|=======================================|
| App Store | 1                 | com.apple.AppStore                    |
| Books     | 4698              | com.apple.iBooks                      |
| Calendar  | 1.0               | com.apple.mobilecal                   |
| Camera    | 3732.0.210        | com.apple.camera                      |
| FaceTime  | 36                | com.apple.facetime                    |
| Family    | 1                 | com.apple.family                      |
| Feedback  | 454               | com.apple.appleseed.FeedbackAssistant |
| Files     | 200.4.1           | com.apple.DocumentsApp                |
| Mail      | 3654.60.0.1.23    | com.apple.MailCompositionService      |
| Maps      | 2608.33.11.29.4   | com.apple.Maps                        |
| YouTube   | 16.05.9           | com.google.ios.youtube                |
+===========+===================+=======================================+
Using the --json option same output will be presented as a JSON:
[email protected]:~$ testproject-agent devices apps -d *** --json
Validate
This command inspects the contents of a backup package or bundle, validates it, and prints basic details about it:
[email protected]:~$ testproject-agent validate backup.yaml
Running the command shown above will print similar details:
Package Protocol: 1
Project
+========================+==================+
| ID                     | Project Name     |
|========================|==================|
| ***                    | My First Project |
+========================+==================+
​
Tests
+========================+==================+
| ID                     | Test Name        |
|========================|==================|
| ***                    | My First Test    |
+========================+==================+
Protocol version indicates what is the TestProject Agent CLI compatible version for this backup file.
Same data can be presented as a JSON if the --json the option is used:
[email protected]:~$ testproject-agent validate backup.yaml --json
Will output similar details:
{
  "protocol": 1,
  "project": {
    "***": "My First Project"
  },
  "tests": {
    "***": "My First Test"
  }
}
Run
This command should be used to execute a backup package.  Here are the usage instructions following the execution of the command with the --help option:
[email protected]:~$ testproject-agent run --help
​
Usage: testproject-agent run [-hV] [--parallel-targets] [--parallel-tests]
                             [--plain] [-rd] [-a=<alias>] [-A=<agent>]
                             [-C=<grpcTimeout>] [-D=<dataPath>]
                             [--grpc-address=<grpcAddress>] [-o=<outputPath>]
                             [-p=<parametersFile>] [-r=<report>]
                             [--rest-address=<restAddress>] [-s=<settingsFile>]
                             [-t=<token>] [-d=<servers>]...
                             [--browser=<browsers> [--browser=<browsers>]... |
                             --device=<devices> [--device=<devices>]...]
                             <files>...
Runs an execution package file(s)
      <files>...             Path to execution package file(s)
  -a, --alias=<alias>        Alias to use when naming the Agent
  -A, --agent=<agent>        Target Agent (host:port)
  -C, --connect-timeout=<grpcTimeout>
                             Connection timeout in seconds
  -d, --driver-server=<servers>
                             Driver server - local within the Agent (default)
                               or cloud.
  -D, --data-path=<dataPath> Path to Agent's data folder
      --grpc-address=<grpcAddress>
                             Agent's spawned gRPC server address (host:port)
  -h, --help                 Show this help message and exit.
  -o, --output-path=<outputPath>
                             Output folder path for the local report file(s)
  -p, --parameters=<parametersFile>
                             Parameters file (data-source CSV/YAML)
      --parallel-targets     Execute package(s) in parallel on all configured
                               browsers/devices
      --parallel-tests       Execute all tests in package(s) in parallel on all
                               configured browsers/devices
      --plain                Produce plain output (without progress bars).
  -r, --report=<report>      Report type - local, cloud or both (default)
      -rd, --restart-driver  Restart driver before each test
      --rest-address=<restAddress>
                             Agent's REST API address (host:port)
  -s, --settings=<settingsFile>
                             Execution settings file (targets, etc.)
  -t, --token=<token>        Token to use for registration
  -V, --version              Print version information and exit.
Targets
      --browser=<browsers>   Target browsers (multi-value)
      --device=<devices>     Target devices (multi-value)
As stated earlier a package can be either a single YAML file, or an archive (ZIP).
Settings
A bundle (ZIP) is a self-sufficient archive that contains a settings file with a pre-selected list of browsers or devices for execution.  While a stand-alone YAML file requires to specify these settings.
In order to override execution settings included in a bundle or to specify execution settings for a stand-alone YAML file, the following options are available:
-s or --settings should be used to provide an external setting file.
--browser or --device to provide a list of target browsers or devices as command arguments.
The following example demonstrates the execution of a bundle:
[email protected]:~$ testproject-agent run MyFirstTestBackup.zip
And this is an example of an unbundled stand-alone YAML:
[email protected]:~$ testproject-agent run MyFirstTestBackup.yaml
Since no setting are bundled, and none are provided, default target will be used.  Chrome or any other installed browser for Web test and any connected device for Mobile.
To override bundled settings using a file, the following command can be executed with -s or --settings option:
[email protected]:~$ testproject-agent run MyFirstTestBackup.zip --settings settings.yaml
The command above will use settings configured in settings.yaml instead of the settings inside the archive.  The same could be achieved using the --browser option:
[email protected]:~$ testproject-agent run MyFirstTestBackup.zip --browser CHROME --browser FIREFOX
The command above will override bundled settings and use provided list of browsers.  In case of a mobile test, device(s) UDID(s) should be provided using --device option:
[email protected]:~$ testproject-agent run MyFirstTestBackup.zip --device *** --device ***
Command arguments can also be used to provide settings for unbundled files such as stand-alone YAML:
[email protected]:~$ testproject-agent run MyFirstTestBackup.yaml --browser SAFARI --browser EDGE
Parameters
Similar to settings, test parameters can be overridden or provided externally.
In order to override bundled test parameters using an external file, -p or --parameters option should be used:
[email protected]:~$ testproject-agent run MyFirstTestBackup.zip --parameters parameters.csv
The same syntax should be used to supply parameters values for an unbundled stand-alone YAML backup files:
[email protected]:~$ testproject-agent run MyFirstTestBackup.yaml --parameters parameters.csv
Report
Following an execution, a local report generated.  By default, it will be created under Agent's data folder, inside reports folder.
Running the following command:
[email protected]:~$ testproject-agent run MyFirstTestBackup.zip
Will produce a similar output:
Executing target 1/1: [Chrome (Headless) 88.0.4324.192]
Tests 100% │██████████████████████████████████████████│ 1/1 (0:00:00 / 0:00:00) 
Steps 100% │██████████████████████████████████████████│ 1/1 (0:00:00 / 0:00:00) 
Execution Report Path: /Users/testproject/Desktop/{TIMESTAMP}.html
Execution Report URL: http://localhost:8585/api/report/{TIMESTAMP}.html
Execution is complete.
Notice the Execution Report Path and Execution Report URL log lines.  URL provided might be useful in combination with the -A or --agent parameter that is used to specify a remote Agent.  Using this URL, a report that is generated by the remote Agent can be retrieved from the remote machine.
Custom Report Location
To specify a different location for the report file, -o or --output-path parameter can be used:
[email protected]:~$ testproject-agent run MyFirstTestBackup.zip -o ~/Desktop
the command above will run MyFirstTestBackup.zip bundle and generate a report in ~/Desktop, for example:
Executing target 1/1: [Chrome (Headless) 88.0.4324.192]
Tests 100% │██████████████████████████████████████████│ 1/1 (0:00:00 / 0:00:00) 
Steps 100% │██████████████████████████████████████████│ 1/1 (0:00:00 / 0:00:00) 
Execution Report Path: /Users/testproject/Desktop/{TIMESTAMP}.html
Execution is complete.
Specifying a custom report file location will result in no report URL link appearing in logs.
In the example above, the report file created in /Users/testproject/Desktop/MyFirstTest-{TIMESTAMP}.html
Progress
Execution progress is represented using progress bars by default.  For consoles that do not support progress bars representation, please use --plain option to output plain text progress.
Progress does not indicate steps results such as passed or failed, but only a periodic indication of the execution advancement. Therefore, when using the --plain option, the same step might appear more than once or do not appear at all if execution advances too quickly.
It is also possible that when using progress bars, it will not reach 100% if the test fails on one of the steps before reaching the end. To inspect the actual execution results, refer to the report file generated a the end of the execution.
Parallel Execution Settings
Using the --parallel-targets or --parallel-tests options, package execution can be parallelized. By default the execution is serial, meaning that tests will be executed one by one on each defined browser/device.
Enabling --parallel-targets will result in tests being executed one by one on all defined browsers/devices, in parallel.
Enabling --parallel-tests will result in all tests being executed on all defined browsers simultaneously. If a test has a data source, all data iterations will run at the same time as well.
These options override any similar settings set via the package's bundle settings YAML file.
Global Options
Starting from Agent version 2.2.0, there are several global options that can be used with all the commands that interact with an Agent.  For example, connect, devices, browsers, etc.
Remote Agent
By default, CLI uses the local Agent, running on the same machine where the CLI is running.  Using the -A or --agent option, allows targeting a remote Agent, in the same network, on a different machine:
[email protected]:~$ testproject-agent --agent 10.0.0.2:9999 browsers list
For example, command above will list available browsers on the Agent at 10.0.0.2:9999.  The accepted format for the target Agent address is {HOST}:{PORT}.
Running packages on a remote Agent, is currently not supported.
Connection Timeout
Sometimes, it might be necessary to allow an extended timeout connecting to the Agent.  This can be done using the -C or --connect-timeout option, specified in seconds, as shown below:
[email protected]:~$ testproject-agent --connect-timeout 10 run MyFirstTestBackup.zip
The default value is 10 seconds, and using this option can be changed.
CLI Versions & Backwards Compatibility
CLI versions are not backward compatible with previous Agents.  Therefore CLI version 2.x should be used with Agents 2.x and CLI 3.x should be used with Agents 3.x
Summary
TestProject CLI is a powerful tool allowing the execution of backup packages.  It can be used on-premises and integrated into any CI / CD pipelines.
TestProject CLI is available as part of the TestProject Agent v2.0.0 installation.
TestProject Agents - Previous
TestProject Agent on Kubernetes
Next - TestProject Agents
Silent Deployment
Last modified 1yr ago