Category: Test Design Techniques

Approach 2.0

(This is improved approach in comparison to the idea of state transition testing described HERE.)

I was presenting in my past articles the approach which allows to generate test cases for combinatorial problems. This is very large group of aspects we encouter when dealing with problem of assuring the quality.

Still, there is an area where we need more general approach. Let’s think about simple GUI application which allows you to log in and fill in some form which can be saved. We have combinatorial aspect when filling in and saving it as we can do it in many ways.
But what about the situation we just cancel form filling? Or else we will fill in many form in the row? As you most probably know state transition diagram comes in hand. This is test design technique focusing on most general aspect of the application which is application state and transition context.
I would like to show in this article how to practically model application using this technique and most importantly how to automatically generate test cases with specific coverage which will be instantly executable.

State transition diagram coverage

Speaking about the coverage: according to my idea the coverage for diagrams is basing on how many times each transition is used: I call it N-tn coverage. So when I say 2-tn coverage it means each transition which is present in the diagram will be used at least twice. It is worth to notice this is something different in comparison to what you can find in QA literature where you can find N-switch coverage. As you probably now, 0-switch coverage means you test single transition (no states), 1-switch coverage means you test 2 transitions (the piece of diagram with 2 transitions and 1 state) and so on. This is nice but I think hard to use in pratice. Why? Because to test specific part of diagram you have to render the application into specific state: you have to execute all the states and transitions which lead you to the state you choose as starting point (setting the state of application without executing the path – like updating database, caches and other stuff manually – is very risky in my opinion and should be avoided). It is just better to avoid complex setup process.

The complexity

Unfortunately the complexity which is hidden behind the diagram is enormous: it is actually infinite. Let’s imagine application which has only 2 states and 1 transition:

How many test cases can we have? Infinite…
1. A-B-A (1-switch coverage, 1-tn coverage)
2. A-B-A-B (2-switch coverage, 1-tn coverage as T2 is used only once)
3. A-B-A-B-A (3-switch coverage, 2-tn coverage)
4. A-B-A-B-A-B (4-switch coverage, 2-tn coverage as T2 is used only twice)
5. A-B-A-B-A-B-A (5-switch coverage, 3-tn coverage)
…
and so on until infinity is reached which is never of course…
Repeating transition once, twice and thousand times are all different test cases. Here you can clearly see how many test cases you miss to reach 100% confidence your application is working as expected.
Anyway, theory is very nice but let’s apply it in practice to make it useful finally.

Practical example

In general, we need just the same as what was the case for combinatorial problems: we need a model, generated test cases in xml format and generated test cases in domain language.
Let’s assume we would like to test Notepad’s functionality related to tabs and text direction. Let’s start with plain old diagram:

It looks very nice but we cannot do anything useful with it right now. Let’s write it in XML format with domain language part:

<System name="State-Transition-Diagram">
    <!-- this is the model input for technical blog -->
    <State name="Tab1_is_selected" id="1" property="START">
        <Transition name="createNewTab" nextstate="Tab2_is_selected" whenNot="tab2exists" setGlobalProperty="tab2exists">
            <Has name="command" value="tab(new2).isOpened()"/>
            <Has name="expected" value="notepad().guiIsVisible()"/>
            <Has name="gherkinType" value="WHEN"/>
        </Transition>
        <Transition name="select_2_tab" nextstate="Tab2_is_selected" when="tab2exists">
            <Has name="command" value="tab(new2).isSelected()"/>
            <Has name="expected" value="notepad().guiIsVisible()"/>
            <Has name="gherkinType" value="WHEN"/>
        </Transition>
        <Transition name="writeTab1" nextstate="Tab1_is_selected">
            <Has name="command" value="tab(new1).isWritten(#DQ#tab1text#DQ#)"/>
            <Has name="expected" value="notepad().guiIsVisible()"/>
            <Has name="gherkinType" value="WHEN"/></Transition>
            <State name="Text_LTR" id="100" start="true">
                <Transition name="changeTextDirectionToRTL" nextstate="Text_RTL">
                    <Has name="command" value="textDirectionIsChanged(RTL)"/>
                    <Has name="expected" value="notepad().guiIsVisible()"/>
                    <Has name="gherkinType" value="WHEN"/>
                </Transition>
            </State>
            <State name="Text_RTL" id="101">
                <Transition name="changeTextDirectionToLTR" nextstate="Text_LTR">
                    <Has name="command" value="textDirectionIsChanged(LTR)"/>
                    <Has name="expected" value="notepad().guiIsVisible()"/>
                    <Has name="gherkinType" value="WHEN"/>
                </Transition>
            </State>
    </State>
    <State name="Tab2_is_selected" id="2">
        <Transition name="deleteTab" nextstate="Tab1_is_selected" unsetGlobalProperty="tab2exists">
            <Has name="command" value="tab(new2).isClosed()"/>
            <Has name="expected" value="notepad().guiIsVisible()"/>
            <Has name="gherkinType" value="WHEN"/>
        </Transition>
        <Transition name="select_1_tab" nextstate="Tab1_is_selected">
            <Has name="command" value="tab(new1).isSelected()"/>
            <Has name="expected" value="notepad().guiIsVisible()"/>
            <Has name="gherkinType" value="WHEN"/>
        </Transition>
        <Transition name="writeTab2" nextstate="Tab2_is_selected">
            <Has name="command" value="tab(new2).isWritten(#DQ#tab2text#DQ#)"/>
            <Has name="expected" value="notepad().guiIsVisible()"/>
            <Has name="gherkinType" value="WHEN"/>
        </Transition>
            <State name="Text_LTR" id="200" start="true">
                <Transition name="changeTextDirectionToRTL" nextstate="Text_RTL">
                    <Has name="command" value="textDirectionIsChanged(RTL)"/>
                    <Has name="expected" value="notepad().guiIsVisible()"/>
                    <Has name="gherkinType" value="WHEN"/>
                </Transition>
            </State>
            <State name="Text_RTL" id="201">
                <Transition name="changeTextDirectionToLTR" nextstate="Text_LTR">
                    <Has name="command" value="textDirectionIsChanged(LTR)"/>
                    <Has name="expected" value="notepad().guiIsVisible()"/>
                    <Has name="gherkinType" value="WHEN"/>
                </Transition>
            </State>
    </State>
</System>

Now it is becoming unreadable for humans but it is much better for a machine…

Please note, expected results for each transition is “notepad GUI is visible” which is quite trivial. This should be more meaningful when doing real diagram model.
At this point we need some software to find valid paths through the diagram with given N-tn coverage. I couldn’t find anything useful in the internet so I wrote myself the piece of software. You can view the code under automatic-tc-generation-from-diagram-another-approach branch HERE. This is: src/test/java/com/passfailerror/diagram2sequence_generator/Diagram2SequenceGenerator.java class.

The algorythm is quite simple:
– diagram is converted into state transition table
– starting state row is chosen as 1. item
– state transition table is shuffled and scanned; when matching row is found (according to diagram logic) it is appended to valid path and transition table is reshuffled
– this process repeates until diagram path is built with specific N-tn coverage (each transition is visited at least N times)
– notice, it makes sense to generate more than one diagram case as each time specific N-tn coverage is generated different path is chosen.

After running Diagram2SequenceGenerator there is result XML file which I call diagram cases generated which we need to convert into executable diagram cases as it is not executable yet:

<?xml version="1.0"?>
<DiagramCases system="State-Transition-Diagram">
<DiagramCase id="0">
<Transition name="createNewTab">
<Has name="command" value="tab(new2).isOpened()" />
<Has name="expected" value="notepad().guiIsVisible()" />
<Has name="gherkinType" value="WHEN" />
</Transition>
<Transition name="changeTextDirectionToLTR">
<Has name="command" value="textDirectionIsChanged(LTR)" />
<Has name="expected" value="notepad().guiIsVisible()" />
<Has name="gherkinType" value="WHEN" />
</Transition>
<Transition name="select_1_tab">
<Has name="command" value="tab(new1).isSelected()" />
<Has name="expected" value="notepad().guiIsVisible()" />
<Has name="gherkinType" value="WHEN" />
</Transition>
<Transition name="select_2_tab">
<Has name="command" value="tab(new2).isSelected()" />
<Has name="expected" value="notepad().guiIsVisible()" />
<Has name="gherkinType" value="WHEN" />
</Transition>
<Transition name="writeTab2">
<Has name="command" value="tab(new2).isWritten(#DQ#tab2text#DQ#)" />
<Has name="expected" value="notepad().guiIsVisible()" />
<Has name="gherkinType" value="WHEN" />
</Transition>
<Transition name="changeTextDirectionToLTR">
<Has name="command" value="textDirectionIsChanged(LTR)" />
<Has name="expected" value="notepad().guiIsVisible()" />
<Has name="gherkinType" value="WHEN" />
</Transition>
<Transition name="writeTab2">
<Has name="command" value="tab(new2).isWritten(#DQ#tab2text#DQ#)" />
<Has name="expected" value="notepad().guiIsVisible()" />
<Has name="gherkinType" value="WHEN" />
</Transition>
<Transition name="select_1_tab">
<Has name="command" value="tab(new1).isSelected()" />
<Has name="expected" value="notepad().guiIsVisible()" />
<Has name="gherkinType" value="WHEN" />
</Transition>
<Transition name="writeTab1">
<Has name="command" value="tab(new1).isWritten(#DQ#tab1text#DQ#)" />
<Has name="expected" value="notepad().guiIsVisible()" />
<Has name="gherkinType" value="WHEN" />
</Transition>
<Transition name="writeTab1">
<Has name="command" value="tab(new1).isWritten(#DQ#tab1text#DQ#)" />
<Has name="expected" value="notepad().guiIsVisible()" />
<Has name="gherkinType" value="WHEN" />
</Transition>
<Transition name="select_2_tab">
<Has name="command" value="tab(new2).isSelected()" />
<Has name="expected" value="notepad().guiIsVisible()" />
<Has name="gherkinType" value="WHEN" />
</Transition>
<Transition name="select_1_tab">
<Has name="command" value="tab(new1).isSelected()" />
<Has name="expected" value="notepad().guiIsVisible()" />
<Has name="gherkinType" value="WHEN" />
</Transition>
<Transition name="changeTextDirectionToLTR">
<Has name="command" value="textDirectionIsChanged(LTR)" />
<Has name="expected" value="notepad().guiIsVisible()" />
<Has name="gherkinType" value="WHEN" />
</Transition>
<Transition name="select_2_tab">
<Has name="command" value="tab(new2).isSelected()" />
<Has name="expected" value="notepad().guiIsVisible()" />
<Has name="gherkinType" value="WHEN" />
</Transition>
<Transition name="select_1_tab">
<Has name="command" value="tab(new1).isSelected()" />
<Has name="expected" value="notepad().guiIsVisible()" />
<Has name="gherkinType" value="WHEN" />
</Transition>
<Transition name="changeTextDirectionToLTR">
<Has name="command" value="textDirectionIsChanged(LTR)" />
<Has name="expected" value="notepad().guiIsVisible()" />
<Has name="gherkinType" value="WHEN" />
</Transition>
<Transition name="changeTextDirectionToRTL">
<Has name="command" value="textDirectionIsChanged(RTL)" />
<Has name="expected" value="notepad().guiIsVisible()" />
<Has name="gherkinType" value="WHEN" />
</Transition>
<Transition name="changeTextDirectionToLTR">
<Has name="command" value="textDirectionIsChanged(LTR)" />
<Has name="expected" value="notepad().guiIsVisible()" />
<Has name="gherkinType" value="WHEN" />
</Transition>
<Transition name="changeTextDirectionToRTL">
<Has name="command" value="textDirectionIsChanged(RTL)" />
<Has name="expected" value="notepad().guiIsVisible()" />
<Has name="gherkinType" value="WHEN" />
</Transition>
<Transition name="changeTextDirectionToRTL">
<Has name="command" value="textDirectionIsChanged(RTL)" />
<Has name="expected" value="notepad().guiIsVisible()" />
<Has name="gherkinType" value="WHEN" />
</Transition>
<Transition name="writeTab1">
<Has name="command" value="tab(new1).isWritten(#DQ#tab1text#DQ#)" />
<Has name="expected" value="notepad().guiIsVisible()" />
<Has name="gherkinType" value="WHEN" />
</Transition>
<Transition name="changeTextDirectionToLTR">
<Has name="command" value="textDirectionIsChanged(LTR)" />
<Has name="expected" value="notepad().guiIsVisible()" />
<Has name="gherkinType" value="WHEN" />
</Transition>
<Transition name="select_2_tab">
<Has name="command" value="tab(new2).isSelected()" />
<Has name="expected" value="notepad().guiIsVisible()" />
<Has name="gherkinType" value="WHEN" />
</Transition>
<Transition name="changeTextDirectionToLTR">
<Has name="command" value="textDirectionIsChanged(LTR)" />
<Has name="expected" value="notepad().guiIsVisible()" />
<Has name="gherkinType" value="WHEN" />
</Transition>
<Transition name="changeTextDirectionToRTL">
<Has name="command" value="textDirectionIsChanged(RTL)" />
<Has name="expected" value="notepad().guiIsVisible()" />
<Has name="gherkinType" value="WHEN" />
</Transition>
<Transition name="writeTab2">
<Has name="command" value="tab(new2).isWritten(#DQ#tab2text#DQ#)" />
<Has name="expected" value="notepad().guiIsVisible()" />
<Has name="gherkinType" value="WHEN" />
</Transition>
<Transition name="writeTab2">
<Has name="command" value="tab(new2).isWritten(#DQ#tab2text#DQ#)" />
<Has name="expected" value="notepad().guiIsVisible()" />
<Has name="gherkinType" value="WHEN" />
</Transition>
<Transition name="writeTab2">
<Has name="command" value="tab(new2).isWritten(#DQ#tab2text#DQ#)" />
<Has name="expected" value="notepad().guiIsVisible()" />
<Has name="gherkinType" value="WHEN" />
</Transition>
<Transition name="deleteTab">
<Has name="command" value="tab(new2).isClosed()" />
<Has name="expected" value="notepad().guiIsVisible()" />
<Has name="gherkinType" value="WHEN" />
</Transition>
</DiagramCase>
</DiagramCases>

We can do the convertion with enhanced version of testcase generator which was used in my previous articles which were dealing with combinatorial problems. You can see the source in :
src/test/java/com/passfailerror/testcases_generator/Generator.java class which receives extra parameter TestcaseSourceType which in turn allows to generate test cases both for TCases output file and diagram cases output file.

The executable test case is:

@Test
public void testStates_dc0(){
WHEN().tab(new2).isOpened().
THEN().notepad().guiIsVisible().
WHEN().textDirectionIsChanged(LTR).
THEN().notepad().guiIsVisible().
WHEN().tab(new1).isSelected().
THEN().notepad().guiIsVisible().
WHEN().tab(new2).isSelected().
THEN().notepad().guiIsVisible().
WHEN().tab(new2).isWritten("tab2text").
THEN().notepad().guiIsVisible().
WHEN().textDirectionIsChanged(LTR).
THEN().notepad().guiIsVisible().
WHEN().tab(new2).isWritten("tab2text").
THEN().notepad().guiIsVisible().
WHEN().tab(new1).isSelected().
THEN().notepad().guiIsVisible().
WHEN().tab(new1).isWritten("tab1text").
THEN().notepad().guiIsVisible().
WHEN().tab(new1).isWritten("tab1text").
THEN().notepad().guiIsVisible().
WHEN().tab(new2).isSelected().
THEN().notepad().guiIsVisible().
WHEN().tab(new1).isSelected().
THEN().notepad().guiIsVisible().
WHEN().textDirectionIsChanged(LTR).
THEN().notepad().guiIsVisible().
WHEN().tab(new2).isSelected().
THEN().notepad().guiIsVisible().
WHEN().tab(new1).isSelected().
THEN().notepad().guiIsVisible().
WHEN().textDirectionIsChanged(LTR).
THEN().notepad().guiIsVisible().
WHEN().textDirectionIsChanged(RTL).
THEN().notepad().guiIsVisible().
WHEN().textDirectionIsChanged(LTR).
THEN().notepad().guiIsVisible().
WHEN().textDirectionIsChanged(RTL).
THEN().notepad().guiIsVisible().
WHEN().textDirectionIsChanged(RTL).
THEN().notepad().guiIsVisible().
WHEN().tab(new1).isWritten("tab1text").
THEN().notepad().guiIsVisible().
WHEN().textDirectionIsChanged(LTR).
THEN().notepad().guiIsVisible().
WHEN().tab(new2).isSelected().
THEN().notepad().guiIsVisible().
WHEN().textDirectionIsChanged(LTR).
THEN().notepad().guiIsVisible().
WHEN().textDirectionIsChanged(RTL).
THEN().notepad().guiIsVisible().
WHEN().tab(new2).isWritten("tab2text").
THEN().notepad().guiIsVisible().
WHEN().tab(new2).isWritten("tab2text").
THEN().notepad().guiIsVisible().
WHEN().tab(new2).isWritten("tab2text").
THEN().notepad().guiIsVisible().
WHEN().tab(new2).isClosed().
THEN().notepad().guiIsVisible();

}

The sequence of WHENs and THENs is diagram case, while single WHEN-THEN pair would be a test case according to my terminology. Just to repeat: it is valid to have more than one diagram case for given N-tn coverage as the sequence of transitions which is generated is always different.
Now, it is just the matter of running the output as it is directly executable:

Approach 2.0 sum up

First we draw a diagram:

Then we translate it into XML (unfortunately manually):

Then diagram cases are generated (automatically):

Finally executable test cases are generated (automatically):

The DSL which is used here (internal domain language implemented in Java) as well as framework (Sikuli) doesn’t really matter. They are used only as an example. Most often it is Selenium, or maybe some kind of strange things like Protractor which will be used in practice and Cucumber or other behaviour driven development library on top of this. The most important thing is that when using approach 2.0 the only important thing is to use any kind of domain language so that it can be used in HAS elements in model file in order to generate diagram cases automatically.

Conclusions

State transition diagram test design technique starts to be useful finally – I have never seen anybody applying this in pratice which is weird as this is about all applications which have at least 2 transitions. Or maybe I didn’t see much?
There are a few important points behind all this: it was very simple problem illustrated here where only few states and few transitions resulted in so many actions. It means the complexity hidden behind simple application is very large and so when modelling more complex applications we have to focus on small coverage or choose only part of application to be tested in this way. Also, I didn’t say anything about invalid paths through the diagram: we should also be checking if invalid paths are really invalid and how system behaves in such situation.

Anyway, I am sure this is very useful technique to deal with problems which are modelled by state transition diagrams.