1. Query Test Framework

There is a test framework available specifically designed to test Viatra Queries. It was developed with the following use cases in mind:

  • Testing the Viatra Query Engine itself

  • Provide a regression testing framework for users to test patterns

1.1. Basic concepts

The framework allows the user to compare the results of a pattern execution using different engine implementation or to a predefined result set (so-called snapshot). It defines a convenient internal DSL to define test cases. A description of a test case consists of the following parts:

  • What to test

    • Generated query specifications

    • Pattern groups

    • Generic pattern groups (possible parsed directly from .vql files)

  • Input models

  • Execution methods

    • by Rete or LocalSearch engines

    • from snapshot

  • Assumption (optional)

    • Checks whether all given execution method supports the given patterns (i.e. the test case is applicable)

  • Assertion

    • Checks whether the results provided by all execution methods are the same for each patterns.

Example
ViatraQueryTest. //Entry point
  test(SomeQuerySpecification::instance).
  and(AnotherQuerySpecification). // Patterns under test
  on(modelURI). // Instance models (optional; snapshot model references the input model)
  with(snapshot). // Compare prepared results stored by a snapshot model
  with(new ReteBackendFactory). // Compare results produced by the Rete engine
  assumeInputs. // checks whether the given snapshots and backend factories are valid for the patterns under test. Throws JUnit assumption error otherwise
  assertEquals // compute difference of each given snapshot and pattern executions. Throws JUnit assertion failure if differences occur

1.2. Incremental Scenarios

The framework supports testing scenarios in which the results can be checked again after a model modification using the modify method:

Recheck after model manipulation
ViatraQueryTest. //Entry point
  test(SomeQuerySpecification::instance).
  and(AnotherQuerySpecification). // Patterns under test
  on(modelURI). // Instance models (optional; snapshot model references the input model)
  with(snapshot). // Compare prepared results stored by a snapshot model
  with(new ReteBackendFactory). // Compare results produced by the Rete engine
  assertEqualsThen. // assertEqualsThen does not return void
  modify(Type, [name=="John"], [age=35]). // The given operation is executed on each instance of the given type on which the given condition evaluates to true.
  with(snapshotAfterModification). // Any modify operation causes all previously loaded snapshots to be invalidated.
  assertEquals

1.3. Supporting plain Java objects in substitutions

In some cases plain Java objects need to be added to the Query Snapshot model. However, the serialization and comparison of such elements might be relevant on the domain in which the testing framework is used. In this case, the framework allows the user to define how certain plain Java types should be handled, through JavaObjectAccess elements.These elements enable the framework to serialize, deserialize and compare certain typed POJOs. The following example demonstrates how these Access objects should be registered into the framework.

In this example, the metamodel contains a 'CustomInteger' typed attribute. 'CustomInteger' is a java type that extends 'Integer'. The following fragment shows how 'Access' type definition.

Accessing a Custom Attribute
public class CustomIntegerAccess extends JavaObjectAccess{

	public CustomIntegerAccess() {
		super(CustomInteger.class);
	}

        //Create Substitution object based on the CustomInteger object
	@Override
	public SerializedJavaObjectSubstitution toSubstitution(Object value) {
		SerializedJavaObjectSubstitution sub = SnapshotFactory.eINSTANCE.createSerializedJavaObjectSubstitution();
		if(value instanceof CustomInteger){
			sub.setType(getType().getName());
			sub.setValue(((CustomInteger) value).integerValue()+"");
		}
		return sub;
	}

        //Calculate hash code (needed for equality checking)
	@Override
	public int calculateHash(SerializedJavaObjectSubstitution substitution) {
		return Objects.hashCode(Integer.parseInt(substitution.getValue()));
	}

        //Check if two substitutions are equal (assuming they each define a 'CustomInteger')
	@Override
	public boolean equals(SerializedJavaObjectSubstitution a, SerializedJavaObjectSubstitution b) {
		if(a.getType().equals(getType().getName()) && b.getType().equals(getType().getName())){
			int aVal = Integer.parseInt(a.getValue());
			int bVal = Integer.parseInt(b.getValue());
			return aVal == bVal;
		}
		return false;
	}

}
Using CustomIntegerAccess
    ...
    Map<String, JavaObjectAccess> objectAccess = Maps.newHashMap();
    map.put(CustomInteger.class.getName(),new CustomIntegerAccess());
    ViatraQueryTest.test(specs, new SnapshotHelper(objectAccess)).on(new EMFScope([MODEL])).with([SNAPSHOT]).assertEquals();
    ...

1.4. Specifying custom serialization for EMF objects

In the default configuration of the snapshot generator, any EMF object is converted to a generic EMF substitution. However, in some scenarios, the user might want to specify a (list of) custom function(s), which define how EMF objects should be represented in the snapshot.

The framework allows for registering such custom functions, in form of a map indexed by the types (EClass) desired to be handled in a user-defined way, where the corresponding map value stores the substitution function. Such functions are expected to receive an EObject and return a String.

The following example shows how to specify a simple custom function for the class ApplicationType from the CPS metamodel, and how to set up a corresponding test case.

customMap.put(CyberPhysicalSystemPackage.Literals.APPLICATION_TYPE, [at | (at as ApplicationType).identifier])

ViatraQueryTest.test(ApplicationTypesQuerySpecification.instance, snapshotHelper(customMap))
          .on(new EMFScope([MODEL]))
          .with([SNAPSHOT])
          .assertEquals

1.4.1. Coverage analysis and reporting

Starting with VIATRA 1.6 (see bug 514628), you can add analyzers to a test object which measure various metrics of query execution. For example, you can analyze coverage during testing:

static var CoverageAnalyzer coverage;

@BeforeClass
static def void before(){
    coverage = new CoverageAnalyzer();
}

@Test
def void testApplicationTypes() {
    ViatraQueryTest.test(ApplicationTypesQuerySpecification.instance)
        .analyzeWith(coverage) // Analyze coverage
        .with(new ReteBackendFactory) // First set of matches should come from query evaluation with Rete backend
        .with(snapshot) // Second set of matches should come from a snapshot
        .assertEquals // Assert that the match sets are equal
    }

Then after running the tests, you can get the analyzed coverage with CoverageAnalyzer#getCoverage(), or report it with CoverageReporter:

@AfterClass
static def void after(){
    CoverageReporter.reportHtml(coverage, new File("coverage.html"))
}

For a complete example, see the CPS Framework tests.

1.4.1.1. Interpreting the coverage report

A coverage report looks like this: CPS Framework tests coverage report

An element (pattern, pattern body or a constraint) can be:

Covered

the Rete node which belongs to it had at least one match during the query executions

Uncovered

the Rete node which belongs to it had no matches during any query execution

Not represented

it is not represented in the Rete network, which usually means that the optimizer removed it because it is redundant.

Not represented by error

it should be represented in the Rete network, but it was removed for an unknown reason; if you encounter this, please report an issue, including your query file and the coverage report.

Note that a pattern body can be uncovered although each of its constraints is covered, because the Rete nodes belonging to the constraints could have matches during different query executions, which means that the constraints were not fulfilled at once.

A pattern’s aggregated coverage metric is calculated the following way: number of covered elements / number of represented (covered/uncovered) elements

Known limitations in 1.6:

  • coverage measurement is supported only with the Rete backend

  • the results might be indeterministic because of the indeterminism of the Rete evaluation

  • the constraints are displayed in their internal PQuery representation (see http://bugs.eclipse.org/515723)

2. Debugging and Profiling Graph Patterns

2.1. Rete Visualizer

Installation
  • Install the VIATRA Query and Transformation SDK Extensions feature from the VIATRA update site.

    • e(fx)clipse runtime and GEF5 is required. They are both available from the release train.

Usage
  • Go to Window | Show View | Other…​ and choose Rete Visualizer.

  • Load the instance model and the pattern in Query Results view.

  • Click on the pattern name to visualize it.

  • To change the visualized Rete net, unload the pattern and load with the green start button.

  • You may change the layout by clicking the downward pointing triangle and going to the Layout menu. rete visualizer

2.2. Local Search Debugger

Installation
  • Install the VIATRA Query and Transformation SDK Extensions feature from the VIATRA update site.

    • e(fx)clipse runtime and GEF5 is required. They are both available from the release train.

Usage
  • Open the Local Search Debugger view in Eclipse

  • Load the model and query definitions in the Query explorer

  • Select the the query in the Query explorer

  • Run the local search debugger by invoking the command on the toolbar of the Local Search Debugger view. The command reads the selection from the query explorer and initiates the matching: initializes and shows the plan.

  • Step or run the matching using the designated commands on the view’s toolbar. Breakpoints can also be added when needed.

  • The state of the shown search plan shows the state of the execution, while the already found matches and the current variable substitutions can be seen in the view’s provided Zest viewer.

2.3. Memory Optimization of Rete Networks

If you have a large number of patterns and you find that the incremental query evaluation seems very slow or uses more memory than expected, it is useful to take an instance model where the issue occurs and evaluate the footprint of each pattern independently. Of course, patterns that call other patterns will also incorporate the footprint of those patterns, but the difference can be seen nonetheless.

We provide a simple JUnit test for performing this hotspot evaluation easily using the Testing Framework of VIATRA Query: QueryPerformanceTest.xtend. Read the JavaDoc for more details, while an example on its usage can be found here for our UML surrogate queries: UMLSurrogateQueryPerformanceTest.java

Note that if you run the test and use profilers, such as YourKit or Java Mission Control, the problematic patterns can be easily identified by looking at the heap size graphs, selecting the exceeding parts and looking at the current trace to see which query is being built.

3. Maven integration

VIATRA Query supports building VIATRA Query projects in a Maven-based builds by generating the pattern matcher code from the vql files.

Requirements
  • The maven compiler requires Maven 3.1 to function correctly. In some cases Maven 3.0.5 is enough, but there are some dependency issues that are problematic for this version. Versions before Maven 3.0 will not work at all. See http://bugs.eclipse.org/478437 for details.

Known limitations
  • Code generation for integration components (e.g. validation framework, derived features) is not supported. See http://bugs.eclipse.org/434794

  • The VIATRA Query project is not available from Maven Central, only from repo.eclipse.org.

  • There is no maven archetype support: it is not possible to generate an Eclipse-less project automatically, that works with VIATRA Query. However, manually created projects can be built with the existing compiler.

3.1. Repository

Maven components are available from the Eclipse maven repository with the following urls:

The following maven projects are available for use:

org.eclipse.viatra:viatra-query-runtime

dependency for the VIATRA Query runtime, without support for generic API

org.eclipse.viatra:viatra-query-language

dependency for the VIATRA Query with support for generic API (requires many more dependencies - only use if required)

org.eclipse.viatra:viatra-maven-plugin

Maven code generator, should not be added to the compile classpath

To use VIATRA Query features, add the repository and the required dependencies to your Maven project:

<!-- use this in your project's pom.xml file -->
<properties>
    <!-- It is a good idea to specify VIATRA framework version once as
    a property (e.g. in the parent pom) and use that throughout the build -->
	<viatra.version>1.2.1</viatra.version>
</properties>

<dependencies>
	<dependency>
		<groupId>org.eclipse.viatra</groupId>
		<artifactId>viatra-query-runtime</artifactId>
		<version>${viatra.version}</version>
	</dependency>

	<!-- requires many more dependencies - only use if required -->
	<dependency>
		<groupId>org.eclipse.viatra</groupId>
		<artifactId>viatra-query-language</artifactId>
		<version>${viatra.version}</version>
	</dependency>
</dependencies>

<repositories>
	<repository>
		<id>viatra</id>
		<url>https://repo.eclipse.org/content/groups/viatra2/</url>
	</repository>
</repositories>

3.2. viatra-maven-plugin

The maven plugin requires information from the used EMF packages and additionally the EMF packages should be able loaded as well. For this reason, it is important to add references to EPackages and Genmodels together with the corresponding dependencies.

Since 1.5, it is possible to use the project dependencies without declaring them explicitly. This helps when the metamodels you are using are not available as Maven artifacts.

Additional notes
  • Package reference is added either by file path to the .genmodel file (typically by platform:/resource URI) or fully qualified name of the Ecore Package class that is available on the classpath. Note that if the class is in the same plugin as the query file, the class based reference will not work as compilation will happen at a later build phase than the generation. Also pay attention not to mix the package class-based and the genmodel-based mechanism in one reactor because it can lead to strange errors.

  • Each package that is imported must be listed in the metamodels section. The packages that are used transitively by the explicitly listed packages do not need to be listed.

  • Explicit dependency declarations are transitive, so you don’t need to specify all dependencies in all POM.XML files. Note that in some cases you need to add extra dependencies with specific versions (e.g. emf.core) if your genmodel requires a higher version than what is provided by the viatra-maven-plugin. This way you can redefine the EMF versions used by the generator.

  • If a pattern file in a Maven project imports patterns from another project on which it depends, make sure that the files containing the imported patterns are included in the dependency’s Maven artifact.

Example POM.XML (based on CPS example)
<pluginRepositories>
  <pluginRepository>
    <id>viatra</id>
    <url>https://repo.eclipse.org/content/groups/viatra/</url>
  </pluginRepository>
</pluginRepositories>
<build>
<plugins>
<!-- Using maven-clean-plugin to remove previously generated code -->
<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-clean-plugin</artifactId>
  <version>2.5</version>
  <executions>
    <execution>
      <phase>clean</phase>
      <goals>
        <goal>clean</goal>
      </goals>
      <configuration>
        <filesets>
          <fileset>
            <!-- Generated code folder -->
            <directory>src-gen</directory>
            <includes>
              <include>**/*</include>
            </includes>
          </fileset>
        </filesets>
      </configuration>
    </execution>
  </executions>
</plugin>
<!-- Setting up generator -->
<plugin>
  <groupId>org.eclipse.viatra</groupId>
  <artifactId>viatra-maven-plugin</artifactId>
  <version>${viatra.version}</version>
  <!-- Binding execution to the code generation lifecycle phase -->
  <executions>
    <execution>
      <goals>
        <goal>generate</goal>
      </goals>
    </execution>
  </executions>
  <configuration>
    <!-- Output directory - required -->
    <outputDirectory>src-gen</outputDirectory>
    <metamodels>
      <metamodel>
        <!-- Select one of the following depending on where is your metamodel defined -->

        <!-- (a) Java class for the EMF EPackage - use this if generated EMF code is in the classpath -->
        <packageClass>org.eclipse.viatra.examples.cps.cyberPhysicalSystem.CyberPhysicalSystemPackage</packageClass>
        <!-- (b) genmodel file used for generating the EMF model classes - use this if EMF model is in the same project -->
        <!-- <genmodelUri>model/model.genmodel</genmodelUri> -->
      </metamodel>
    </metamodels>
    <!-- Since 1.5, you can use the project dependencies instead of specific Maven dependencies - optional -->
    <useProjectDependencies>true</useProjectDependencies>
  </configuration>
  <dependencies>
    <!-- Dependency required for the cps domain project (that contains the generated EPackage), unless you set useProjectDependencies -->
    <dependency>
      <groupId>org.eclipse.viatra.examples.cps</groupId>
      <artifactId>org.eclipse.viatra.examples.cps.model</artifactId>
      <version>1.2.0</version>
    </dependency>
  </dependencies>
</plugin>
</plugins>
</build>

3.2.1. Language extension support

Starting with VIATRA 2.0, the viatra-maven-compiler can understand custom annotations, and validate them. To support these features, the new viatra-query-language-extensions maven module has to be added to the dependencies of the project.

Custom annotation support
<plugins>
<!-- Using maven-clean-plugin to remove previously generated code -->
<!-- Setting up generator -->
<plugin>
  <groupId>org.eclipse.viatra</groupId>
  <artifactId>viatra-maven-plugin</artifactId>
  <version>${viatra.version}</version>
  <configuration>
    ...
  </configuration>
  <dependencies>
    <dependency>
      <groupId>org.eclipse.viatra</groupId>
      <artifactId>viatra-query-language-extensions</artifactId>
      <version>${viatra.version}</version>
    </dependency>
  </dependencies>
</plugin>
</plugins>
</build>
Code generator features bound to annotations, e.g. in case of the validation framework or the query-based derived feature support are still not available in the Maven compiler.

3.2.2. Troubleshooting

Cyclic linking

In a project that contains multiple pattern definition files, there is a slight chance you get an error message like follows:

ERROR:Cyclic linking detected : PatternCall.patternRef→PatternCall.patternRef

This is a known issue we are working to fix; see http://bugs.eclipse.org/464120 and http://bugs.eclipse.org/480652 for details about the underlying issues.

As workaround, we suggest to explicitly define types for all parameters, because that avoids this issue. If that is not enough, you have to ensure that the vql files are processed in calling order: all patterns are to be processed before they are used in a pattern call. This can be achieved either by moving patterns between vql files, or by renaming your .vql files so that the files that call patterns from other files should have names lexicographically greater than the referenced files.

For example, let’s suppose that you have a query file with two queries:

// util.vql

pattern utilityPattern(...) {
  find anotherUtilityPattern(...);
}

pattern anotherUtilityPattern(...) {
  ...
}

The cyclic linking occurs where utilityPattern calls anotherUtilityPattern. Search for callers of utilityPattern! Let’s suppose it is in this file:

// logic.vql

pattern myPattern(...) {
  find utilityPattern(...);
}

You have to rename util.vql to a_util.vql and logic.vql to b_logic.vql, so that the former is processed before the latter.

Ambiguous types

If you get type errors during validation such as ERROR:foo cannot be resolved. or ERROR:Ambiguous variable type defintions: [Foo, Bar], type cannot be selected although the query files are valid in Eclipse, check the cross-references in your ecore/genmodel files by opening them with a text editor. If their URIs are workspace-based, i.e. they start with platform:/resource, you have to map those URIs to absolute file: URIs by including URI mappings in the viatra-maven-plugin configuration (since 1.6):

<plugin>
	<groupId>org.eclipse.viatra</groupId>
	<artifactId>viatra-maven-plugin</artifactId>
	<configuration>
...
		<uriMappings>
			<uriMapping>
				<sourceUri>platform:/resource/school/model/school.ecore</sourceUri>
				<targetUri>file:/${project.basedir}/school/model/school.ecore</targetUri>
			</uriMapping>
			<uriMapping>
				<sourceUri>platform:/resource/school/model/school.genmodel</sourceUri>
				<targetUri>file:/${project.basedir}/school/model/school.genmodel</targetUri>
			</uriMapping>
		</uriMappings>
	</configuration>
</plugin>
Multiple definitions of a type

If you get the error ERROR:Variable foo has a type Foo which has multiple definitions: 'file://C:\project\model/../../anotherProject/model/usedMetamodel.ecore' —  'file://C:\project\../anotherProject/model/usedMetamodel.ecore' make sure that the genmodelUri in the viatra-maven-plugin’s configuration is exactly the same as the URI in your .genmodel file, e.g. if your metamodel.ecore resides in the model subfolder, this will be the configuration with the correct relative URI:

<plugin>
	<groupId>org.eclipse.viatra</groupId>
	<artifactId>viatra-maven-plugin</artifactId>
	<configuration>
		<metamodels>
			<metamodel>
				<genmodelUri>model/../../anotherProject/model/usedMetamodel.genmodel</genmodelUri>
			</metamodel>
		</metamodels>
	</configuration>
</plugin>
The URIs should be the same after URI mapping if used.