Scenario is a receipt for telling PerfCake what to do. You can specify how PerfCake would generate load by configuring a generator, where and what to send by defining a sender and messages. To get any measured results such as an average throughput or a memory usage you can use reporting capabilities. To check that the responses are correct a validation is available for you to set in the scenario. There is also a possibility to specify scenario meta-data by setting the scenario's properties.
As you can see from the following listing the XML scenario is defined by
urn:perfcake:scenario:7.0
namespace
[3]. The scenario is divided into several sections: Properties, Run, Generator,
Sequences, Sender, Receiver, Reporting, Messages and Validation.
1 <?xml version="1.0" encoding="utf-8"?> 2 <scenario xmlns="urn:perfcake:scenario:7.0"> 3 <!-- Scenario properties (optional) --> 4 <properties> 5 <property name="..." value="..."/> 6 ... 7 </properties> 8 9 <!-- Run section (required) --> 10 <run ... > 11 ... 12 </run> 13 14 <!-- Generator section (required) --> 15 <generator ... > 16 ... 17 </generator> 18 19 <!-- Sequences section (optional) --> 20 <sequences> 21 ... 22 </sequences> 23 24 <!-- Sender section (required) --> 25 <sender ... > 26 ... 27 </sender> 28 29 <!-- Receiver section (optional) --> 30 <receiver> 31 ... 32 </receiver> 33 34 <!-- Reporting section (optional) --> 35 <reporting> 36 ... 37 </reporting> 38 39 <!-- Messages section (optional) --> 40 <messages> 41 ... 42 </messages> 43 44 <!-- Validation section (optional) --> 45 <validation> 46 ... 47 </validation> 48 </scenario>
Let's take a look at particular sections of the scenario.
This optional section allows you to add some meta-data about your scenario. It can contain multiple properties.
All the scenario properties are set as Java System properties so can be used further in scenario (See Section 2.2.5, “Filtering properties” for more details.).
The run section specifies the duration for what the scenario will run. It is mandatory since PerfCake needs to know how long to generate load.
The scenario run configuration is described in more details in Section 4.1, “How - Generating load” .
The generator section specifies the way how the load is generated. It is mandatory since PerfCake needs to know how to generate load.
The generators are described in more details in Section 4.1, “How - Generating load” .
The sender section is about the transport (e.g. HTTP, MQTT, JMS, ...) and the target where the load is directed. It is required to be specified in the scenario.
More information about the senders can be found in Section 4.2, “Where - Sending messages” .
The receiver section defines a component used to receive responses from a separate message channel. Because it is possible to send requests to some protocol and use a completely different protocol to receive responses.
A receiver always needs to know which response matches which original request. This is why it needs a correlator to be specified.
More information about the receivers and senders can be found in their corresponding chapters Section 4.3, “Receiving messages” and Section 4.4, “Correlating messages”.
Reporting module is responsible for gathering metrics and reporting the results to various places in specified moments. It is not required to configure the reporting in the scenario but without it the PerfCake has no way of measuring and reporting results.
The reporting abilities are described in Section 4.7, “Reporting” .
The messages represent the payload that is transferred by senders to the tested system. It is optional since there can be situations where there is no actual message being send.
The Section 4.5, “What - Messages” describes the messages in more details.
Inside of the messages and scenario configuration properties, you can use sequences of values. The sequences can be configured separately in the scenario and just provide a next value in the sequence each time they are queried. This allows you to have unique messages or even send the message to various targets.
The Section 4.6, “Sequences” describes the sequences in more details.
Validation module allows to validate the response messages.
The validation capabilities are described in Section 4.8, “Validation” .
There is a possibility to specify scenarios in the form that resembles a natural language. This is typically stored in a file with the .dsl suffix. It is useful to open these scenarios in an editor that supports Groovy syntax as the DSL language is actually developed in Groovy.
The DSL scenarios can use the same features, properties and constructs as the XML scenario. The language is just different. Following is a sample DSL scenario.
1 scenario "my cool scenario" 2 qsName "test" propA "hello" 3 run 10.s with 4.threads 4 generator "DefaultMessageGenerator" senderTaskQueueSize 3000 5 sender "TestSender" target "httpbin.org" delay 12.s 6 reporter "WarmUpReporter" 7 reporter "ThroughputStatsReporter" minimumEnabled false 8 destination "CsvDestination" every 3.s path '${perfcake.scenario}-stats.csv' enabled 9 destination "ConsoleDestination" every 5.percent disabled 10 reporter "ResponseTimeStatsReporter" 11 destination "ConsoleDestination" every 10.percent 12 message file:"message1.xml" send 10.times 13 message content:"Hello World" values 1,2,3 14 message "file://message2.txt" validate "text1","text2" 15 message "Simple text" propA "kukuk" headers name:"Franta",count:10 validate "text1, text2" 16 validation fast disabled 17 validator "RegExpValidator" id "text1" pattern "I am a fish!" 18 validator "RegExpValidator" id "text2" pattern "I was a fish!" 19 end
The language format is typically in the form of <keyword> <attribute>
.
Mandatory keywords used are scenario
, run
, generator
and sender
.
In general, strings are in quotes or apostrophes, an array and a map are specified by the
elements separated by commas. The map elements are specified by the pairs
key:value
separated by a colon. There are some special units defined that can
be used with numbers. The format is a number followed by a dot and the unit name. Following is
the list of supported units, in the scenario source, they are represented by the abbreviations
shown in the parentheses: milliseconds (ms), seconds (s), minutes (m), hours (h), days (d),
iterations (iteration, iterations), percents (percent, percents), threads (thread, threads),
and times (times).
Scenario
is followed by the scenario name and mandatory properties and their
values.
Run
is followed by the time specification and the number of threads after the
with
keyword.
Generator
, sender
, reporter
,
destination
, validator
and sequence
are followed by
the class name implementing the component. Then there are properties and their values for the
given component. A single component configuration cannot be split to multiple lines. It must
all be present at a single line.
Message
is followed by the location of the message or its content. Either
there is a differentiator in the form file:
or content:
, or there is
a string with the protocol specification (file://
, http://
…),
or a string with message content. Following are more configuration properties.
It is also possible to run PerfCake by utilizing its API. The simplest way is to load a scenario specified in an external file (or even URL location) and execute it.
Example 2.1. Loading of a scenario definition and its execution from API
import org.perfcake.scenario.Scenario; import org.perfcake.scenario.ScenarioLoader; ... final Scenario scenario = ScenarioLoader.load("/full/path/http.xml"); scenario.init(); scenario.run(); scenario.close();
A call to the run()
method is blocking and does not return until all messages
are sent or a fatal error occurs. It is however possible to stop the scenario execution from
another thread by a call to the stop()
method of the Scenario
class.
The termination is not immediate, rather a graceful shutdown is performed.
In a case when you do not require access to the scenario control object (org.perfcake.scenario.Scenario) and you just want to run a test (for example in TestNG or jUnit), there is a simplified method of the above.
Example 2.2. Executing a scenario with a single method call using PerfCake API
import org.perfcake.scenario.ScenarioExecution; ... ScenarioExecution.execute("scenario-name", new Properties());
As a scenario name, the same value is passed as in the -s parameter on the command line. The properties can configure anything that would normally go into -D parameter. Most other command line parameters can be configured via these properties (see ug.perfcake-features.scenario-definition.filtering-properties).
It is also possible to build the complete scenario from scratch using PerfCake API. Most objects are designed with fluent API so you can easily join multiple setter methods.
Example 2.3. Complete scenario definition and execution using its API
import org.perfcake.PerfCakeException; import org.perfcake.RunInfo; import org.perfcake.common.BoundPeriod; import org.perfcake.common.Period; import org.perfcake.common.PeriodType; import org.perfcake.message.Message; import org.perfcake.message.MessageTemplate; import org.perfcake.message.correlator.Correlator; import org.perfcake.message.correlator.GenerateHeaderCorrelator; import org.perfcake.message.generator.DefaultMessageGenerator; import org.perfcake.message.generator.MessageGenerator; import org.perfcake.message.receiver.HttpReceiver; import org.perfcake.message.receiver.Receiver; import org.perfcake.message.sender.HttpSender; import org.perfcake.message.sender.MessageSender; import org.perfcake.message.sequence.PrimitiveNumberSequence; import org.perfcake.message.sequence.Sequence; import org.perfcake.reporting.destination.ConsoleDestination; import org.perfcake.reporting.destination.Destination; import org.perfcake.reporting.reporter.IterationsPerSecondReporter; import org.perfcake.reporting.reporter.Reporter; import org.perfcake.validation.MessageValidator; import org.perfcake.validation.RegExpValidator; ... final Period period = new Period(PeriodType.TIME, 30_000); final RunInfo runInfo = new RunInfo(period); final MessageGenerator generator = new DefaultMessageGenerator(); generator.setThreads(10); final MessageValidator validator = new RegExpValidator().setCaseInsensitive(true).setPattern(".*"); final Message message = new Message(); message.setPayload("Hello world no. @{intSeq}!"); final MessageTemplate messageTemplate = new MessageTemplate(message, 1, Collections.singletonList("regExp")); final MessageSender sender = new HttpSender().setMethod(HttpSender.Method.POST).setTarget("http://httpbin.org/post"); final Correlator correlator = new GenerateHeaderCorrelator(); final Receiver receiver = new HttpReceiver().setSource("localhost:8282").setThreads(10); final Sequence sequence = new PrimitiveNumberSequence(); final Destination destination = new ConsoleDestination(); final Reporter reporter = new IterationsPerSecondReporter(); reporter.registerDestination(destination, new Period(PeriodType.TIME, 1000)); final ScenarioBuilder builder = new ScenarioBuilder(runInfo, generator, sender); builder.setReceiver(receiver).setCorrelator(correlator); builder.putMessageValidator("regExp", validator).addMessage(messageTemplate). putSequence("intSeq", sequence).addReporter(reporter); final Scenario scenario = builder.build(); scenario.init(); scenario.run(); scenario.close();
The API is slightly counter-intuitive because the scenario needs to be composed from the
bottom to top. For example, we first need to define a Message Validator
to validate a message and then use it to create a Message Template. Next,
we set the Message Validator reference ID later when passing it to the
ScenarioBuilder
.
Similar concept works for Sequences and Message Reporters and others.
So far we have prepared preliminary versions of plugins for Eclipse and IntelliJ Idea. Some efforts were done for NetBeans as well. None of the plugins is stable enough at the moment. We are in a search of a brave contributor who could consolidate all the plugins and make them stable and user friendly.
If you want to go wild, try to search the default repositories of your IDE. This comes with no guarantees at the moment.
It is possible to use property placeholders in scenarios (and in other components of the scenario, see later). The placeholders are replaced by the actual value of the particular property or by the default value if specified in a process called property filtering.
This proces is quite complex and allows many useful manipulations with the values provided in the scenario.
At first, when the scenario is loaded from a file (no matter which format is used), all the placeholders of the following format are loaded and replaced by appropriate values if they are found.
${<property name>:<default value>}
The colon and the default value are optional. Also, any properties specified in the scenario are loaded and put into system properties.
During this phase, PerfCake checks for properties in the system properties and then in environment properties. We can also limit it to check just one of the sources by providing a prefix to the property name. The prefix can be env. for environment properties and props. for system properties.
The properties found in the scenario can be used throughout the scenario as well and can be also used as in other properties. However, circular definitions do not work.
Example 2.4. An example of a scenario with properties
1 <?xml version="1.0" encoding="utf-8"?> 2 <scenario xmlns="urn:perfcake:scenario:7.0"> 3 <properties> 4 <property name="composedProperty" value="${defaultProperty}-2"/> 5 <property name="defaultProperty" value="${test.missing.property:default-property-value}"/> 6 </properties> 7 ... 8 <messages> 9 <message uri="unfiltered-message.txt"> 10 <header name="testHeader" value="${defaultProperty}"/> 11 </message> 12 </messages> 13 </scenario>
The property filtering process is performed in a moment, when a scenario file is loaded by PerfCake, before it is parsed. There are no advanced features, just a simple string replacement for scenario files. The placeholders can be at any place in the scenario file because PerfCake approaches the scenario as a simple string without any syntax meaning.
The filtering tries to find the property by name. If the property is found, the whole placeholder is replaced by property's actual value. If the specified property does not exist, it looks for the default value if it is specified. If so the whole placeholder is replaced by the default value. Otherwise it leaves the placeholder in place intact.
More complex filtering is available for some components of the system. The advanced filtering works for: Sender's target attribute, any part of Message including content loaded from an external file, and for patterns in RegExpValidator.
The following types of placeholders are supported:
${<property name>:<default value>} @{<property name>:<default value>}
The following characters can be escaped by a backslash: \
(backslash itself), $
, @
,
{
, and }
. These characters should not appear unescaped if they
are not supposed to act as control characters. However, the default value of a property can
contain anything except for }
. Also, backshlash needs to be escaped. Colon is not
allowed in the property name and no characters can be escaped in it.
The following table lists a few examples of strings and how they are rendered in the end.
Suppose we have the following system properties set: ab = 1
, cd =
2
.
Template | Result |
---|---|
${ab:cd} | 1 |
\\${cd} | \2 |
\${ab} | ${ab} |
${non.existing:\}$@\\} | }$@\ |
\\\@{cd} | \@{cd} |
${env.JAVA_HOME} | System dependant, for example: /opt/jdk |
Table 2.1. Examples of templates and how they are rendered
Properties with the dollar sign ($) are replaced just once, while the properties with the at sign (@, so called dynamic properties) are alsways rendered again with every single use and thus containing a fresh value. This is mainly useful for Section 4.6, “Sequences”. Please note that rendering values of the placeholders with the at sign can introduce a slight performance impact. PerfCake tries to switch off the placeholder rendering whenever possible (when there are no dynamic properties present in the particular string).
In the case of advanced property filtering, the environment and system properties need to
be strictly separated and referenced with the corresponding prefix. The environment properties
are accessed by ${env.<property name>}
. The system properties can be accessed
by ${props.<property value>}
, ${props[<property value>]}
, and
${props['<property value>']}
.
Also, these placeholders are considered ultimate and no more replacing is possible. So in
the case when there is no property available and there is no default value, they are replaced
by the null
string.
Both of the replacement strategies can be combined to achieve interesting results. For example, to define a property in a scenario the value of which is derived from another property defined there, and to use it in the meesage content, we need to use the following approach.
Example 2.5. An example of a scenario with combined property replacement strategies
1 <?xml version="1.0" encoding="utf-8"?> 2 <scenario xmlns="urn:perfcake:scenario:7.0"> 3 <properties> 4 <property name="composedProperty" value="${props.defaultProperty}-2"/> 5 <property name="defaultProperty" value="${test.missing.property:default-property-value}"/> 6 </properties> 7 ... 8 <messages> 9 <message content="${props.composedProperty}"/> 10 </messages> 11 </scenario>
As you can see, we had to use the advanced approach to refer to the
composedProperty
and to the defaultProperty
. This is because
PerfCake first reads the scenario, defines the values of both composedProperty
and defaultProperty
. But at that time, we did not know the value of the
defaultProperty
and it could not be used to replace the value in the
composedProperty
. So the composedProperty
remains exactly as seen
the example. Now PerfCake parses the scenarion by a syntax parser. The value of the
defaultProperty
is now known but in this stage, only the advanced templating
is supported. So the props.
prefix worked and we got the correct value
default-property-value-2
in the message content.
To provide PerfCake the actual value of the property you can just pass it using an ordinary way:
-D<property name>=<property value>
In Windows command shell, the parameter specifying the property name and value might
need to be placed in double quotes: "-D<property name>=<property
value>"
There are several properties that exist in PerfCake and that might be usefull in the scenarios or messages (e.g. a timestamp of the scenario execution start). The following table describes all available internal properties.
Property name | Description |
---|---|
perfcake.encoding | Default encoding |
perfcake.messages.dir | Messages directory |
perfcake.plugins.dir | Plugins directory |
perfcake.properties.file | Custom properties file |
perfcake.run.timestamp | A Unix timestamp of the moment of the scenario execution start |
perfcake.run.nice.timestamp | A timestamp of the moment of the scenario execution start in a human readable
format (yyyyMMddHHmmss ). |
perfcake.scenario | A name of the scenario |
perfcake.scenarios.dir | Scenarios directory |
perfcake.logging.level | Logging level of the PerfCake core |
Table 2.2. Available PerfCake internal properties