PerfCake is a command line tool so it provides a CLI (command line interface) for executing
a PerfCake scenario (see Section 2.2, “Performance scenario definition” ). The
main script can be found in $PERFCAKE_HOME/bin/perfcake.sh
for Linux OS or
%PERFCAKE_HOME%\bin\perfcake.bat
in case of Windows OS.
If the script is run without arguments, from the following help that is printed you can see the arguments that can be used to control the scenario execution.
[PERFCAKE_HOME]$ ./bin/perfcake.sh === Welcome to PerfCake 7.5 === usage: ScenarioExecution -s <SCENARIO> [options] [-D<property=value>]* -d,--debug start debug JMX agent for external monitoring -D <property=value> system properties -dn,--debug-name <AGENT_NAME> debug agent name in the JMX tree -h,--help prints help/usage -log,--log-level <LOG_LEVEL> logging level -md,--messages-dir <MESSAGES_DIR> directory for messages -pd,--plugins-dir <PLUGINS_DIR> directory for plugins -pf,--properties-file <PROPERTIES_FILE> custom system properties file -r,--replay <RAW_FILE> raw file to be replayed -s,--scenario <SCENARIO> scenario to be executed -sd,--scenarios-dir <SCENARIOS_DIR> directory for scenarios -skip,--skip-timer-benchmark skip system timer benchmark
The only mandatory argument is -s
(or --scenario
), that
specifies the name of the scenario to be executed. PerfCake will look for the scenario
definition in the file called <SCENARIO>.xml
that is placed under the
directory that is located at $PERFCAKE_HOME/resources/scenarios
. The path to the
directory with scenarios can be specified by the -sd
(or
--scenarios-dir
) argument.
If the scenario is configured to send any message (see Section 4.5, “What - Messages” ), it will look under the directory that is by
placed at $PERFCAKE_HOME/resources/messages
. That location can be specified by
the -md
(or --messages-dir
) argument.
If you need to set one or more system properties for scenario (see Section 2.2.5, “Filtering properties” ), there are two
ways. You can either use -Dproperty=value
like arguments or specify a path to the
property file using the -pf
(or --properties-file
) argument.
Following table describes all the PerfCake CLI arguments.
Argument | Description | Required | Default value |
---|---|---|---|
-s, --scenario | Name of the scenario to be executed. | Yes | - |
-log, --log-level <LOG_LEVEL> | Default PerfCake logging level. Can be trace ,
debug , info , warn ,
error . | No | info |
-D<property>=<value> | Sets a single system property with a given name and value. Multiple property arguments are allowed. | No | - |
-sd, --scenarios-dir | Path to the directory where scenarios are located. | No | $PERFCAKE_HOME/resources/scenarios |
-md, --messages-dir | Path to the directory where message files are located. | No | $PERFCAKE_HOME/resources/messages |
-pd, --plugins-dir | Path to the directory where plugins are located (see Section 6.2, “Custom components - Plugins” ). | No | $PERFCAKE_HOME/resources/plugins |
-r, --replay <RAW_FILE> | Replay the previously raw data with the given scenario. No test is
executed, just the reporters are used to create reports from the
<RAW_FILE> data file. | No | - |
-pf, --properties-file | Path to the file containing system properties. | No | - |
-skip, --skip-timer-benchmark | Skip system timer benchmark during PerfCake startup. This can speedup the startup a little bit. | No | false |
-d, --debug | Starts an internal debug agent that publishes status of PerfCake components via JMX. This is an experimental feature suited for performance scenario development. It has a significant performance drawback and must not be used for real testing. | No | false |
-dn, --debug-name | Name of the debug agent for the case when there were multiple PerfCake instances in the same JVM sharing the JMX data. | No | perfcake-1 |
-h, --help | Prints PerfCake help/usage information. | No | - |
Table 2.3. PerfCake CLI arguments
Some useful properties can be configured using the -Dproperty=value
parameter
. These are not so common so there is no dedicated short parameter variant.
The following table lists the other properties recognized by PerfCake. None of them is
required.
Property name | Description | Default value |
---|---|---|
perfcake.scenario | An alternative way to specify the PerfCake scenario. It is equivalent to the
-s parameter. | - |
perfcake.encoding | The encoding used for all inputs and outputs. | UTF-8 |
perfcake.scenarios.dir | Equivalent of the -sd command line parameter. | $PERFCAKE_HOME/resources/scenarios |
perfcake.messages.dir | Equivalent of the -md command line parameter. | $PERFCAKE_HOME/resources/messages |
perfcake.plugins.dir | Equivalent of the -pd command line parameter. | $PERFCAKE_HOME/resources/plugins |
perfcake.properties.file | Equivalent of the -pf command line parameter. | - |
perfcake.logging.level | Equivalent of the -log command line parameter. | info |
perfcake.fail.fast | Allows preliminary termination of PerfCake in the case there is a failure during sending a message. PerfCake needs to wait for all the threads to finish, so you might see an error to be shown multiple times actually. In case this is set to true, no more requests for sending a messages will be created. | false |
perfcake.templates.disabled | Completely disables the templating engine, no placeholders will be replaced in scenarios and messages. Normally, it is sufficient just not to use the templates. Having the engine on and not using it does not bring any performance overhead. By setting this to true, PerfCake can also run in JRE (the templating is the only part of PerfCake depending on JDK). | false |
perfcake.skip.timer.benchmark | Equivalent of the -skip command line parameter. | false |
perfcake.replay | Equivalent of the -r command line parameter. | - |
perfcake.debug | Equivalent of the -d command line parameter. | false |
perfcake.debug.name | Equivalent of the -dn command line parameter. | perfcake-1 |
Table 2.4. PerfCake recognized properties
PerfCake is provided with a Maven plugin which allows to run PerfCake scenario within Maven build. This makes perfomance test automation more easy and encourages you to run performance tests on a regular basis (e.g. within your favourite CI server). Running performance test on a regular basis allows you to spot performance drops very soon and thus makes it much easier to identify which commit has caused performance regression.
Currently, the plugin has only one goal scenario-run
, which runs specified
PerfCake scenario. By default, this goal is executed in integration-test
phase (i.e. assumes, that you deploy/start your application in
pre-integration-test phase and shut it down in
post-integration-test phase).
The only mandatory parameter is <scenario>
, which specifies the name of the
scenario to be run. Optionally, you can also specify <scenarios-dir>
,
<messages-dir>
and <plugins-dir>
, which specify paths to
directories containing scenarios, messages and plugins, respectively. If you don't setup these
parameters, plugin assumes, that appropriate directories (scenarios
,
messages
, plugins
) are in
src/test/resources/perfcake
. This can be switched to
src/main/resources/perfcake
by configuring
<use-test-resources>
to false
. If any of these directories
does not exists, plugin will use src/test/resources
(or
src/main/resources
) as a fallback value for missing parameter.
The PerfCake Maven plugin requires log4j2 configuration file for PerfCake to report
properly. By default it assumes the file to be located in project root and named
log4j2.xml
. This can be changed by the <log4j2-config>
tag.
The PerfCake logging level can be configured with <log-level>
tag and a file
with additional configuration properties can be specified by
<properties-file>
.
The PerfCake version used with the plugin is always the same as plugin version. Both jar files are released together.
Example 2.6. PerfCake Maven plugin sample configuration
<properties> <perfcake.version>7.5</perfcake.version> ... </properties> ... <build> <plugins> <plugin> <groupId>org.perfcake.maven</groupId> <artifactId>perfcake-maven-plugin</artifactId> <version>${perfcake.version}</version> <configuration> <scenario>my_perfcake_scenario</scenario> <log4j2-config>src/test/resources/log4j2.xml</log4j2-config> </configuration> <executions> <execution> <id>perfcake-scenario-run</id> <goals> <goal>scenario-run</goal> </goals> </execution> </executions> <dependencies> <dependency> <groupId>org.perfcake</groupId> <artifactId>perfcake</artifactId> <version>${perfcake.version}</version> <scope>test</scope> </dependency> </dependencies> </plugin> ... </plugins> </build>
By default, PerfCake comes configured for a production environment. Logging level is set to
info and validation messages are logged in a separate file. You can
change the default logging level for PerfCake by a command line parameter -log
(see Section 2.3.1, “Command line parameters” for more details). Possible
logging levels are: all, trace,
debug, info, warn,
error, fatal, off.
More tweaking can be done in the log4j2.xml
file. It is out of the scope of
this guide to describe this configuration as this is a standard Log4j2 configuration file and
one can find its own documentation.
It is possible to record raw results data from a performance test run and later replay them. This can be useful when we use a remote machine to run the test and we need to process the data on a different location or when we want to fine-tune the reporting facilities and we cannot or don't want to rerun the real test many times.
To record the test data, you can use the section called “RawReporter”. This will create a single output file with the recorded data. The file can be of a significant size depending on the number of test iterations. From our experience, a test with 100,000 iterations results in afile of size approx 4-8MB (depending on test meta-data available, sequences used etc.).
The file can be later replayed by providing the corresponding scenario with a different
configuration of reporting section and passing the recorded data file to PerfCake in the
--replay
parameter.
Example 2.7. Sample usage of the results replay mode
> $PERFCAKE_HOME/bin/perfcake.sh -s http-reporting -r perfcake-measurement-1465935794498.raw 2016-06-14 22:37:20,359 INFO {org.perfcake.ScenarioExecution} === Welcome to PerfCake 7.5 === 2016-06-14 22:37:20,699 INFO {org.perfcake.scenario.ScenarioBuilder} Scenario configuration: file:/home/perfcake/perfcake-7.5/resources/scenarios/http-reporting.xml 2016-06-14 22:37:20,660 INFO {org.perfcake.ScenarioExecution} Replaying raw results recorded in perfcake-measurement-1465935794498.raw. [0:00:00][1 iterations][3%] [319.848264 ms] [warmUp => false] [Threads => 10] [ResponseSize => 0.00 B] [failures => 0] [RequestSize => 0.00 B] [Average => 319.848264 ms] [0:00:01][59 iterations][14%] [179.383135 ms] [warmUp => false] [Threads => 10] [ResponseSize => 0.00 B] [failures => 0] [RequestSize => 0.00 B] [Average => 207.6054971694915 ms] [0:00:02][128 iterations][24%] [148.827863 ms] [warmUp => false] [Threads => 10] [ResponseSize => 0.00 B] [failures => 0] [RequestSize => 0.00 B] [Average => 179.01238675000008 ms] [0:00:03][198 iterations][34%] [138.458243 ms] [warmUp => false] [Threads => 10] [ResponseSize => 0.00 B] [failures => 0] [RequestSize => 0.00 B] [Average => 167.52745448989901 ms] [0:00:04][265 iterations][44%] [172.478225 ms] [warmUp => false] [Threads => 10] [ResponseSize => 0.00 B] [failures => 0] [RequestSize => 0.00 B] [Average => 161.4592350792453 ms] [0:00:05][315 iterations][54%] [232.630756 ms] [warmUp => false] [Threads => 10] [ResponseSize => 0.00 B] [failures => 0] [RequestSize => 0.00 B] [Average => 167.65647934285724 ms] [0:00:06][369 iterations][64%] [170.956762 ms] [warmUp => false] [Threads => 10] [ResponseSize => 0.00 B] [failures => 0] [RequestSize => 0.00 B] [Average => 170.2701377127373 ms] [0:00:07][421 iterations][74%] [205.153517 ms] [warmUp => false] [Threads => 10] [ResponseSize => 0.00 B] [failures => 0] [RequestSize => 0.00 B] [Average => 173.20733971496443 ms] [0:00:08][470 iterations][85%] [201.284832 ms] [warmUp => false] [Threads => 10] [ResponseSize => 0.00 B] [failures => 0] [RequestSize => 0.00 B] [Average => 176.6280006106384 ms] [0:00:09][520 iterations][95%] [232.02807 ms] [warmUp => false] [Threads => 10] [ResponseSize => 0.00 B] [failures => 0] [RequestSize => 0.00 B] [Average => 179.5895076057694 ms] [0:00:10][536 iterations][100%] [196.740767 ms] [warmUp => false] [Threads => 10] [ResponseSize => 0.00 B] [failures => 0] [RequestSize => 0.00 B] [Average => 181.79292323694048 ms] 2016-06-14 22:37:21,058 INFO {org.perfcake.ScenarioExecution} === Goodbye! ===
Please not the time difference between the start and the end of the replay (22:37:20,660 - 22:37:21,058). This means that all the results from a test that originally took 10 seconds was replayed in less than a second.
The Debug Agent is an experimental feature suited for performance scenario development. It has a significant performance drawback and must not be used for real testing.
This agent publishes status of PerfCake components via JMX and can be read by utilities
like jvisualvm
(with MBean plugin).
The agent can be started by configuring the system property perfcake.debug
to
true
or by passing the -d
or --debug
command line
options. For the agent to work properly, the JDK's tools.jar
must be on the
classpath. This necessity will be removed in Java 9. The PerfCake startup scripts try to
configure the classpath accordingly but they still might fail. As a workaround, you might copy
the tools.jar
file to ${PERFCAKE_HOME}/lib/ext
.
When there are multiple PerfCake instances running in the same JVM, they can be
differentiated by setting perfcake.debug.name
property (or -dn
,
--debug-name
command line options). The default agent name is
perfcake-1
.
The main purpose of the Debug Agent is for IDE plugin integration and debugging performance
scenarios. The feature state will be claimed stable once there is enough testing with the IDE
plugins and probably when we can remove the tools.jar
dependency.