The test infrastructure, currently geared exclusively at Erlang, for the Perpetuum system. C is bound to follow sometime later.
Tests are available in test under the project directory.
The builtin tests of the package can be run after configuring with
CMake and building with your local build system. You can simply
run ctest to see the test outputs. The first few tests are very
slow, and if your interest is with later tests you might want to
speedup with some concurrency, and use ctest -j 25 instead.
The remainder of this document deals with creating and running your own tests, possibly against your own modules.
We like to use GreatSPN as an editor and above all as a simulator for Petri Nets. It is available for free download and runs on all the major desktops.
GreatSPN has its own file format named PNPRO. It is an XML format, but not compatible with the standard PNML documents. To give you as users a choice we settled on the standard, but we do supply the PNPRO where possible because this wonderful tool won't import PNML files. We ask you to return the favour when you happen to use the same tool, as it greatly improves the usability of the workflow. But don't force people to use this tool; please always supply PNML as well, like we do too.
We coined a simple format for describing test runs, basically as a single Erlang expression. The format is a list where two consequtive elements belong together:
- A marking, as a mapping of places to a number of tokens
- What can fire, as a list of transition names
The list therefore always has an even number of elements, because these paired elements were not brought together in a tuple; mostly to simplify manual entry.
Also as a simplification, you need not specify places with zero tokens.
The first transition in the list is fired at the Petri Net to come to the next marking and transitions that are armed to fire.
A few oddities of Erlang to be mindful about:
- End the expression with a dot
- Comments are prefixed with
%and not the common# - Names of places and transitions need not be quoted when styled like an identifier in C, but you may use single quotes to get more expressive freedom
You should write your test files manually because it makes little sense to test automated output against automated input. Of course this is not as important when you generate a test file from completely distinct software. I can hear you asking, but no, the GreatSPN simulator has no export facility for this. We may ask them, though, they have been willing to help out before.
We have devised a rather extensive list of tests. We hope it suits your needs, but of course it cannot be complete.
All tests run against a compiled Erlang module, the gen_perpetuum
behaviour and your .test file (although certain tests may ignore the
file or make improper use of it).
This test starts a process through your generated module and then tests
if it responds well to a stop() invocation. It will wait for 5 seconds
until your module terminates, and check if it does so with some
dignity.
This test starts a process through your generated module and then passes
through the process laid out in the .test file. The markings will
be checked and so will the list of possible transitions; after this is
done, the first transition will be fired to reach to the next marking
and transition list.
Things tested are:
- Compare the marking in the test file against the output of
marking/1, where undefined places are set to 0 and the resulting markings must match exactly. - Ask the Petri Net through
canfire/1which transitions are ready to fire; compare to the list given in the test file. (script) - Iterating over all transitions known to the Petri Net, use
canfire/2to see what transitions can fire, and be sure that it contains no more and no less than the list given in the test file. (others) - Iterating over all transitions known to the Petri Net that
are not ready to fire according to the test file, try to
fire them at the Petri Net, and harvest the result
{retry,marking}to indicate that it is not currently possible. (probed) - Invocation of an operation through
event/3. - A
noreplyarranged answer indeed responds withnoreply.
This test is very similar to syncrun, except that the calls are
made through the asynchronous method signal/3 and that no
return codes are expected or evaluated.
The reason that we can test this is that introspection functions
such as marking/1 and canfire/1,2 are also sent as messages
to the process for the Petri Net instance. The process may well
accept asynchronous messages, but it handles them internally in
a completely ordered fashion. So, by the time it gets to handle
the marking/1 message, it has finished processing our
asynchronous submission through signal/3.
The main reason for asynchronicity is to decouple the client from whatever the Petri Net needs to do in the background.
This test is yet another variation of syncrun, but this time with application logic attached that collects state, and reportes whether the collected state results in output that is expected.
This is also the first test against enquire/2, a generic
mechanism to request data from the application logic that
lies behind the Petri Net. These requests are simply
passed in and out by the Petri Net, and are run as part of
its overall process.
This has no purpose for the test script, other than rather
ungratefully seeding a predictable series of seemingly
random values from it. Once again it is like syncrun,
but it will makes an effort to pass a simple calculation
through the Petri Net, based on its events. Again, this
uses enquire/2 to pickup the result.
As the name indicates, this is yet another variation on
the syncrun test. It arranges for the transitions to not
fire directly, but rather with a delay. The application
logic calls for this, as it would do in a real application
where timeouts need to be triggered. If need be, an
application can return a value that defers an event or
signal to a later time for processing. This test also
tests the event/4 call, which allows a maximum timeout
to be specified. Without this timeout, attempts of
deferral are rejected; with this timeout, there is a
maximum waiting time before attempted deferrals get
rejected. Note that we are talking of blocked waiting.
Similar to syncrun_delayed, except based on the
asynchronous calls, this time testing the signal/4
function which adds a timeout.
Unlike syncrun_delayed, there is no blocked waiting involved here; the signal is sent and control returns.
We used similar trickery to the plain asyncrun to ensure that we're not sending signals before the Petri Net is ready for them. Hadn't we done this, as we initially didn't by the way, then the Petri Net will take in signals that match the initial state but drop anything else. And it looks really silly when only certain transition names get through!
This is somewhat similar to the asyncrun_delayed, but it is setup differently. All but the first and last transitions in the test file are bundled and sent to the application logic, which arranges for the transitions to fire in order. The client has no idea what is going on until it tries the last transition. It cannot check intermediate states, but the first and last it can check.
Your generated module holds a function sentinel/1
that provides values for the first few numbers of
storage bits per place. After five or so, it
changes to using a dynamic reflow_sentinal/4
operation that starts from the initial Sentinel.
We compare this compiler-supported sequence with
one of our own making.
The other sequence starts with the same initial
Sentinel but then makes a stepwise extension.
It compares the next ten values. Although this
sequence is also based on the dynamic
reflow_sentinel/4 operation, it increments in
steps, and therefore even uses a different
computation for the last five in the sequence.
This test does not use the test file, although it will be read in. Read about Sentinels in the Bit Field Vector documentation but keep in mind that this test only addresses the general Sentinel, not the Transition Sentinel.
Your generated module holds a function transmap/1
that provides values for the first few numbers of
storage bits per place. After five or so, it
changes to using a dynamic reflow_transmap/4
operation that starts from the initial Transition
Map. We compare this compiler-supported sequence
with one of our own making.
The other sequence starts with the same initial
Transition Map but then makes a stepwise extension.
It compares the next ten values. Although this
sequence is also based on the dynamic
reflow_transmap/4 operation, it increments in
steps, and therefore even uses a different
computation for the last five in the sequence.
This test does not use the test file, although it will be read in. Read about Transition Maps in the Bit Field Vector documentation and note that the tests include reflowing of a Transition Sentinel, unlike the general Sentinel tested in the reflow_sentinel test.
The tool to start a test run is an Erlang script
named run_tests.erl which can be started like
any other executable. As arguments, it expects
a Petri Net name and a sequence of test names
(which are the subsection headers of the
previous section).
The gen_perpetuum module is found somewhere
along $ERL_PATH; the Erlang module that was
generated for the Petri Net must be translated
to a .beam file and will be found by adding
that extension after the Petri Net name and
will be looked for in a number of locations,
usually including the current directory; the
test file is found by adding .test after the
Petri Net name and looking for it in a number
of locations, usually including the current
directory. Test names are looked up in the
internal list, and run one by one.
The output
can be dramatically informative (that's a
euphemism, I suppose) but is easily summarised
in SUCCESS or FAILURE for each test, as
well as an overall conclusion at the end,
which is also reflected in the usual exit codes
zero for SUCCESS and nonzero for FAILURE,
which are precisely what CTest expects.
We are aware that run_tests.erl is functional,
but not very friendly. Improvements will be
made, also in an attempt to simplify the build
setup which now requires us to copy all files
into the test/ subdirectory of the build
directory. It's a matter of priorities, really.
For now, functionel felt as good enough.
There are a few tests that come from the
erlang directory underneath the project root.
These are the common tests ct and the type
validation test dialyzer. The latter one is
slow, it may take a minute or so on a decent
machine. No idea why, but it made us choose
to run this test as part of the CTest run
instead of at every build.
Please help us to improve Perpetuum if you found a bug! A great way of doing this is described in the INSTALL document.