2.2. Performance scenario definition

Scenario is a receipt for telling PerfCake what to do. You can specify how PerfCake would generate load by configuring a generator, where and what to send by defining a sender and messages. To get any measured results such as an average throughput or a memory usage you can use reporting capabilities. To check that the responses are correct a validation is available for you to set in the scenario. There is also a possibility to specify scenario meta-data by setting the scenario's properties.

2.2.1. XML scenario

As you can see from the following listing the XML scenario is defined by urn:perfcake:scenario:7.0 namespace [3]. The scenario is divided into several sections: Properties, Run, Generator, Sequences, Sender, Receiver, Reporting, Messages and Validation.

Scenario structure

  1 <?xml version="1.0" encoding="utf-8"?>
  2 <scenario xmlns="urn:perfcake:scenario:7.0">
  3    <!-- Scenario properties (optional) -->
  4    <properties>
  5       <property name="..." value="..."/>
  6       ...
  7    </properties>
  8 
  9    <!-- Run section (required) -->
 10    <run ... >
 11       ...
 12    </run>
 13 
 14    <!-- Generator section (required) -->
 15    <generator ... >
 16       ...
 17    </generator>
 18 
 19    <!-- Sequences section (optional) -->
 20    <sequences>
 21       ...
 22    </sequences>
 23 
 24    <!-- Sender section (required) -->
 25    <sender ... >
 26       ...
 27    </sender>
 28 
 29    <!-- Receiver section (optional) -->
 30    <receiver>
 31       ...
 32    </receiver>
 33 
 34    <!-- Reporting section (optional) -->
 35    <reporting>
 36       ...
 37    </reporting>
 38 
 39    <!-- Messages section (optional) -->
 40    <messages>
 41       ...
 42    </messages>
 43 
 44    <!-- Validation section (optional) -->
 45    <validation>
 46       ...
 47    </validation>
 48 </scenario>

Sections of the scenario

Let's take a look at particular sections of the scenario.

Scenario properties

This optional section allows you to add some meta-data about your scenario. It can contain multiple properties.

All the scenario properties are set as Java System properties so can be used further in scenario (See Section 2.2.5, “Filtering properties” for more details.).

Run

The run section specifies the duration for what the scenario will run. It is mandatory since PerfCake needs to know how long to generate load.

The scenario run configuration is described in more details in Section 4.1, “How - Generating load” .

Generator

The generator section specifies the way how the load is generated. It is mandatory since PerfCake needs to know how to generate load.

The generators are described in more details in Section 4.1, “How - Generating load” .

Sender

The sender section is about the transport (e.g. HTTP, MQTT, JMS, ...) and the target where the load is directed. It is required to be specified in the scenario.

More information about the senders can be found in Section 4.2, “Where - Sending messages” .

Receiver and Correlator

The receiver section defines a component used to receive responses from a separate message channel. Because it is possible to send requests to some protocol and use a completely different protocol to receive responses.

A receiver always needs to know which response matches which original request. This is why it needs a correlator to be specified.

More information about the receivers and senders can be found in their corresponding chapters Section 4.3, “Receiving messages” and Section 4.4, “Correlating messages”.

Reporting

Reporting module is responsible for gathering metrics and reporting the results to various places in specified moments. It is not required to configure the reporting in the scenario but without it the PerfCake has no way of measuring and reporting results.

The reporting abilities are described in Section 4.7, “Reporting” .

Messages

The messages represent the payload that is transferred by senders to the tested system. It is optional since there can be situations where there is no actual message being send.

The Section 4.5, “What - Messages” describes the messages in more details.

Sequences

Inside of the messages and scenario configuration properties, you can use sequences of values. The sequences can be configured separately in the scenario and just provide a next value in the sequence each time they are queried. This allows you to have unique messages or even send the message to various targets.

The Section 4.6, “Sequences” describes the sequences in more details.

Validation

Validation module allows to validate the response messages.

The validation capabilities are described in Section 4.8, “Validation” .

2.2.2. DSL scenario

There is a possibility to specify scenarios in the form that resembles a natural language. This is typically stored in a file with the .dsl suffix. It is useful to open these scenarios in an editor that supports Groovy syntax as the DSL language is actually developed in Groovy.

The DSL scenarios can use the same features, properties and constructs as the XML scenario. The language is just different. Following is a sample DSL scenario.

  1 scenario "my cool scenario"
  2    qsName "test" propA "hello"
  3    run 10.s with 4.threads
  4    generator "DefaultMessageGenerator" senderTaskQueueSize 3000
  5    sender "TestSender" target "httpbin.org" delay 12.s
  6    reporter "WarmUpReporter"
  7    reporter "ThroughputStatsReporter" minimumEnabled false
  8       destination "CsvDestination" every 3.s path '${perfcake.scenario}-stats.csv' enabled
  9       destination "ConsoleDestination" every 5.percent disabled
 10    reporter "ResponseTimeStatsReporter"
 11       destination "ConsoleDestination" every 10.percent
 12    message file:"message1.xml" send 10.times
 13    message content:"Hello World" values 1,2,3
 14    message "file://message2.txt" validate "text1","text2"
 15    message "Simple text" propA "kukuk" headers name:"Franta",count:10 validate "text1, text2"
 16    validation fast disabled
 17       validator "RegExpValidator" id "text1" pattern "I am a fish!"
 18       validator "RegExpValidator" id "text2" pattern "I was a fish!"
 19 end

The language format is typically in the form of <keyword> <attribute>. Mandatory keywords used are scenario, run, generator and sender.

In general, strings are in quotes or apostrophes, an array and a map are specified by the elements separated by commas. The map elements are specified by the pairs key:value separated by a colon. There are some special units defined that can be used with numbers. The format is a number followed by a dot and the unit name. Following is the list of supported units, in the scenario source, they are represented by the abbreviations shown in the parentheses: milliseconds (ms), seconds (s), minutes (m), hours (h), days (d), iterations (iteration, iterations), percents (percent, percents), threads (thread, threads), and times (times).

Scenario is followed by the scenario name and mandatory properties and their values.

Run is followed by the time specification and the number of threads after the with keyword.

Generator, sender, reporter, destination, validator and sequence are followed by the class name implementing the component. Then there are properties and their values for the given component. A single component configuration cannot be split to multiple lines. It must all be present at a single line.

Message is followed by the location of the message or its content. Either there is a differentiator in the form file: or content:, or there is a string with the protocol specification (file://, http://…), or a string with message content. Following are more configuration properties.

2.2.3. Scenario specified through API

It is also possible to run PerfCake by utilizing its API. The simplest way is to load a scenario specified in an external file (or even URL location) and execute it.

Example 2.1. Loading of a scenario definition and its execution from API

import org.perfcake.scenario.Scenario;
import org.perfcake.scenario.ScenarioLoader;
...
      final Scenario scenario = ScenarioLoader.load("/full/path/http.xml");

      scenario.init();
      scenario.run();
      scenario.close();

A call to the run() method is blocking and does not return until all messages are sent or a fatal error occurs. It is however possible to stop the scenario execution from another thread by a call to the stop() method of the Scenario class. The termination is not immediate, rather a graceful shutdown is performed.

In a case when you do not require access to the scenario control object (org.perfcake.scenario.Scenario) and you just want to run a test (for example in TestNG or jUnit), there is a simplified method of the above.

Example 2.2. Executing a scenario with a single method call using PerfCake API

import org.perfcake.scenario.ScenarioExecution;
...
      ScenarioExecution.execute("scenario-name", new Properties());

As a scenario name, the same value is passed as in the -s parameter on the command line. The properties can configure anything that would normally go into -D parameter. Most other command line parameters can be configured via these properties (see ug.perfcake-features.scenario-definition.filtering-properties).

It is also possible to build the complete scenario from scratch using PerfCake API. Most objects are designed with fluent API so you can easily join multiple setter methods.

Example 2.3. Complete scenario definition and execution using its API

import org.perfcake.PerfCakeException;
import org.perfcake.RunInfo;
import org.perfcake.common.BoundPeriod;
import org.perfcake.common.Period;
import org.perfcake.common.PeriodType;
import org.perfcake.message.Message;
import org.perfcake.message.MessageTemplate;
import org.perfcake.message.correlator.Correlator;
import org.perfcake.message.correlator.GenerateHeaderCorrelator;
import org.perfcake.message.generator.DefaultMessageGenerator;
import org.perfcake.message.generator.MessageGenerator;
import org.perfcake.message.receiver.HttpReceiver;
import org.perfcake.message.receiver.Receiver;
import org.perfcake.message.sender.HttpSender;
import org.perfcake.message.sender.MessageSender;
import org.perfcake.message.sequence.PrimitiveNumberSequence;
import org.perfcake.message.sequence.Sequence;
import org.perfcake.reporting.destination.ConsoleDestination;
import org.perfcake.reporting.destination.Destination;
import org.perfcake.reporting.reporter.IterationsPerSecondReporter;
import org.perfcake.reporting.reporter.Reporter;
import org.perfcake.validation.MessageValidator;
import org.perfcake.validation.RegExpValidator;
...
      final Period period = new Period(PeriodType.TIME, 30_000);
      final RunInfo runInfo = new RunInfo(period);

      final MessageGenerator generator = new DefaultMessageGenerator();
      generator.setThreads(10);

      final MessageValidator validator = new RegExpValidator().setCaseInsensitive(true).setPattern(".*");

      final Message message = new Message();
      message.setPayload("Hello world no. @{intSeq}!");

      final MessageTemplate messageTemplate = new MessageTemplate(message, 1, Collections.singletonList("regExp"));

      final MessageSender sender = new HttpSender().setMethod(HttpSender.Method.POST).setTarget("http://httpbin.org/post");

      final Correlator correlator = new GenerateHeaderCorrelator();

      final Receiver receiver = new HttpReceiver().setSource("localhost:8282").setThreads(10);

      final Sequence sequence = new PrimitiveNumberSequence();

      final Destination destination = new ConsoleDestination();

      final Reporter reporter = new IterationsPerSecondReporter();
      reporter.registerDestination(destination, new Period(PeriodType.TIME, 1000));

      final ScenarioBuilder builder = new ScenarioBuilder(runInfo, generator, sender);
      builder.setReceiver(receiver).setCorrelator(correlator);
      builder.putMessageValidator("regExp", validator).addMessage(messageTemplate).
            putSequence("intSeq", sequence).addReporter(reporter);

      final Scenario scenario = builder.build();

      scenario.init();
      scenario.run();
      scenario.close();

The API is slightly counter-intuitive because the scenario needs to be composed from the bottom to top. For example, we first need to define a Message Validator to validate a message and then use it to create a Message Template. Next, we set the Message Validator reference ID later when passing it to the ScenarioBuilder.

Similar concept works for Sequences and Message Reporters and others.

2.2.4. IDE plugins

So far we have prepared preliminary versions of plugins for Eclipse and IntelliJ Idea. Some efforts were done for NetBeans as well. None of the plugins is stable enough at the moment. We are in a search of a brave contributor who could consolidate all the plugins and make them stable and user friendly.

If you want to go wild, try to search the default repositories of your IDE. This comes with no guarantees at the moment.

2.2.5. Filtering properties

It is possible to use property placeholders in scenarios (and in other components of the scenario, see later). The placeholders are replaced by the actual value of the particular property or by the default value if specified in a process called property filtering.

This proces is quite complex and allows many useful manipulations with the values provided in the scenario.

At first, when the scenario is loaded from a file (no matter which format is used), all the placeholders of the following format are loaded and replaced by appropriate values if they are found.

${<property name>:<default value>}

The colon and the default value are optional. Also, any properties specified in the scenario are loaded and put into system properties.

During this phase, PerfCake checks for properties in the system properties and then in environment properties. We can also limit it to check just one of the sources by providing a prefix to the property name. The prefix can be env. for environment properties and props. for system properties.

The properties found in the scenario can be used throughout the scenario as well and can be also used as in other properties. However, circular definitions do not work.

Example 2.4. An example of a scenario with properties

  1 <?xml version="1.0" encoding="utf-8"?>
  2 <scenario xmlns="urn:perfcake:scenario:7.0">
  3    <properties>
  4       <property name="composedProperty" value="${defaultProperty}-2"/>
  5       <property name="defaultProperty" value="${test.missing.property:default-property-value}"/>
  6    </properties>
  7    ...
  8    <messages>
  9       <message uri="unfiltered-message.txt">
 10          <header name="testHeader" value="${defaultProperty}"/>
 11       </message>
 12    </messages>
 13 </scenario>

The property filtering process is performed in a moment, when a scenario file is loaded by PerfCake, before it is parsed. There are no advanced features, just a simple string replacement for scenario files. The placeholders can be at any place in the scenario file because PerfCake approaches the scenario as a simple string without any syntax meaning.

The filtering tries to find the property by name. If the property is found, the whole placeholder is replaced by property's actual value. If the specified property does not exist, it looks for the default value if it is specified. If so the whole placeholder is replaced by the default value. Otherwise it leaves the placeholder in place intact.

More complex filtering is available for some components of the system. The advanced filtering works for: Sender's target attribute, any part of Message including content loaded from an external file, and for patterns in RegExpValidator.

The following types of placeholders are supported:

${<property name>:<default value>}
@{<property name>:<default value>}

The following characters can be escaped by a backslash: \ (backslash itself), $, @, {, and }. These characters should not appear unescaped if they are not supposed to act as control characters. However, the default value of a property can contain anything except for }. Also, backshlash needs to be escaped. Colon is not allowed in the property name and no characters can be escaped in it.

The following table lists a few examples of strings and how they are rendered in the end. Suppose we have the following system properties set: ab = 1, cd = 2.

TemplateResult
${ab:cd}1
\\${cd}\2
\${ab}${ab}
${non.existing:\}$@\\}}$@\
\\\@{cd}\@{cd}
${env.JAVA_HOME}System dependant, for example: /opt/jdk

Table 2.1. Examples of templates and how they are rendered


Properties with the dollar sign ($) are replaced just once, while the properties with the at sign (@, so called dynamic properties) are alsways rendered again with every single use and thus containing a fresh value. This is mainly useful for Section 4.6, “Sequences”. Please note that rendering values of the placeholders with the at sign can introduce a slight performance impact. PerfCake tries to switch off the placeholder rendering whenever possible (when there are no dynamic properties present in the particular string).

In the case of advanced property filtering, the environment and system properties need to be strictly separated and referenced with the corresponding prefix. The environment properties are accessed by ${env.<property name>}. The system properties can be accessed by ${props.<property value>}, ${props[<property value>]}, and ${props['<property value>']}.

Also, these placeholders are considered ultimate and no more replacing is possible. So in the case when there is no property available and there is no default value, they are replaced by the null string.

Both of the replacement strategies can be combined to achieve interesting results. For example, to define a property in a scenario the value of which is derived from another property defined there, and to use it in the meesage content, we need to use the following approach.

Example 2.5. An example of a scenario with combined property replacement strategies

  1 <?xml version="1.0" encoding="utf-8"?>
  2 <scenario xmlns="urn:perfcake:scenario:7.0">
  3    <properties>
  4       <property name="composedProperty" value="${props.defaultProperty}-2"/>
  5       <property name="defaultProperty" value="${test.missing.property:default-property-value}"/>
  6    </properties>
  7    ...
  8    <messages>
  9       <message content="${props.composedProperty}"/>
 10    </messages>
 11 </scenario>

As you can see, we had to use the advanced approach to refer to the composedProperty and to the defaultProperty. This is because PerfCake first reads the scenario, defines the values of both composedProperty and defaultProperty. But at that time, we did not know the value of the defaultProperty and it could not be used to replace the value in the composedProperty. So the composedProperty remains exactly as seen the example. Now PerfCake parses the scenarion by a syntax parser. The value of the defaultProperty is now known but in this stage, only the advanced templating is supported. So the props. prefix worked and we got the correct value default-property-value-2 in the message content.

To provide PerfCake the actual value of the property you can just pass it using an ordinary way:

-D<property name>=<property value>

Note

In Windows command shell, the parameter specifying the property name and value might need to be placed in double quotes: "-D<property name>=<property value>"

There are several properties that exist in PerfCake and that might be usefull in the scenarios or messages (e.g. a timestamp of the scenario execution start). The following table describes all available internal properties.

Property nameDescription
perfcake.encodingDefault encoding
perfcake.messages.dirMessages directory
perfcake.plugins.dirPlugins directory
perfcake.properties.fileCustom properties file
perfcake.run.timestampA Unix timestamp of the moment of the scenario execution start
perfcake.run.nice.timestampA timestamp of the moment of the scenario execution start in a human readable format (yyyyMMddHHmmss).
perfcake.scenarioA name of the scenario
perfcake.scenarios.dirScenarios directory
perfcake.logging.levelLogging level of the PerfCake core

Table 2.2. Available PerfCake internal properties