Pass Fail Error

Category: Pipeline

  • Write your own pipeline testing framework

    Introduction

    I had a need recently to start using pipeline testing framwework in order to be able to test number of pipeline files I am using on daily basis. I tell you, it is bad feeling when number of pipeline scripts is growing and the only testing you can do is to make some sandboxing inside Jenkins…
    So I started to look around and at first I found this framework for testing which looked quite promising:
    https://github.com/jenkinsci/JenkinsPipelineUnit
    Unfortunately, almost immediately I stumbled upon problem of malfunctioning mock for local functions: I was not able to mock internal function – it was only possible to mock steps, so I would have to mock all of the atomic steps of all auxiliary functions I was using. Imagine function with 50 steps which you just need to mock but you cannot do it and the only workaround is to mock most of that 50 steps…
    For me lesson learned is: open source solution is good when it is working propery or all defects can be worked around easily. Project has to be very active I guess with lot of developers involved. This is not the case here – I would guess this even may be abandoned in the future.
    Anyway, because workaround was hard and I found yet another problem with that framework, I decided to try to create my own solution.

    Understanding the problem

    When looking at typical declarative or scripted pipeline one can see it reminds groovy. Let’s look at the example:

    package com.passfailerror

pipeline {
    agent { label 'test' }
    stages {
        stage('First stage') {
            steps {
                echo 'First'
                sh 'mvn --version'
                env.TEST_GLOBAL_VAR='global_value'
            }
        }
        stage('Second stage') {
            steps {
                echo 'Second'
                sh 'java -version'
            }
        }
    }
}

    Trying to launch simplePipeline.groovy outside of Jenkins as regular groovy script ends up with error like this:

    groovy.lang.MissingMethodException: No signature of method: com.passfailerror.example2.pipeline() is applicable for argument types: (com.passfailerror.example2$_run_closure1) values: [com.passfailerror.example2$_run_closure1@77b21474]

    So, the problem with running it with standard groovy is for sure pipeline. But what does it mean in terms of syntax? Well, it is just a function named pipeline which accepts closure as parameter:

    pipeline { … } 

    The other items are other groovy functions as well, they just have 2 arguments like:

    stage(‘Second stage’) { … } 

    This one is stage function which has String as first argument and closure as the second one.

    It is clear now that our goal is just to mock all pipeline specific functions so that they are runnable by groovy. This will allow to run whole pipeline file outside of Jenkins ! It sounds complicated but it is not. Read on.

    Making it runnable

    Let’s see the example of how can we write first version of code which will run pipeline example:

    import groovy.test.GroovyTestCase
import java.nio.file.Paths

class runPipelineOnly extends GroovyTestCase {


    Script pipelineScript


    def mockJenkins(pipelineScript) {
        pipelineScript.metaClass.pipeline = { Object... params ->
            log.info("pipeline")
            params[0].call()
        }
        pipelineScript.metaClass.agent = { Object... params ->
            log.info("agent")
            params[0].call()
        }

        pipelineScript.metaClass.label = { Object... params ->
            log.info("label "+params[0].toString())
        }

        pipelineScript.metaClass.stages = { Object... params ->
            log.info("stages")
            params[0].call()
        }

        pipelineScript.metaClass.stage = { Object... params ->
            log.info("stage("+params[0].toString()+")")
            params[1].call()
        }

        pipelineScript.metaClass.steps = { Object... params ->
            log.info("steps")
            params[0].call()
        }

        pipelineScript.metaClass.echo = { Object... params ->
            log.info("echo")
            log.info(params[0])}

        pipelineScript.metaClass.sh = { Object... params ->
            log.info("sh")
            log.info(params[0])}
    }

    void setUp() {
        def pipelineFile = Paths.get("src/main/groovy/com/passfailerror/simplePipeline.groovy").toFile()
        def binding = new Binding()
        binding.setProperty("env", [:])
        GroovyShell shell = new GroovyShell(binding)
        pipelineScript = shell.parse(pipelineFile)
        mockJenkins(pipelineScript)
    }

    void testPipeline() {
        pipelineScript.run()
    }

}

    If you run it you will see it works now! All jenkins related syntax items are mocked and so groovy can execute the file. How does it work?

    I use expando class mechanism here which is part of runtime metaprogramming in groovy (You can read about it here:

    https://www.groovy-lang.org/metaprogramming.html#metaprogramming_emc).

    In highlighted lines you can observe how “pipeline” function is mocked. It says: whenever you meet function named pipeline with matching signature (which is array of one or more objects), run the assigned closure (run the code in curly braces).

    So, during the runtime the code is run and it means that the closure:

    • logs word “pipeline”
    • treats first parameter as a closure and runs it (which means in this example that “agent” function will be run and so next mock will be executed)

    This is the core of the pipeline testing framework: mocking jenkins syntax which is expressed in pipeline file in curly braces as closures.

    Of course not all of the syntax items can be mocked this way, as for example there are functions like echo which have String parameter only. Their mock is just printing it out.

    Improving the solution

    The first version contains much redundancy. We can do some improvements here:

    import com.passfailerror.ResultStackProcessor
import groovy.test.GroovyTestCase

import java.nio.file.Paths

class runPipelineImproved extends GroovyTestCase {

    Script pipelineScript
    ResultStackProcessor stackProcessor

    def steps = ["label", "echo", "sh"]
    def sections = ["pipeline", "agent", "stages", "stage", "steps"]


    def mockJenkins(pipelineScript, mockedSteps, mockedSections) {
        mockSteps(pipelineScript, mockedSteps)
        mockSections(pipelineScript, mockedSections)
    }

    def mockSections(pipelineScript, mockedSections) {
        mockedSections.each {
            section ->
                def currentSection = section
                pipelineScript.metaClass."$currentSection" = { Object... params ->
                    log.info(currentSection)
                    if (params.length > 1) {
                        params[1].call() // stage("name"){closure}
                    }
                    else{
                        params[0].call() // steps{closure}
                    }
                }

        }
    }


    def mockSteps(pipelineScript, mockedSteps) {
        mockedSteps.each {
            step ->
                def currentStep = step
                pipelineScript.metaClass."$currentStep" = { Object... params ->
                    log.info(currentStep + " " + params[0].toString())
                }
        }
    }


    void setUp() {
        def pipelinePath = Paths.get("src/main/groovy/com/passfailerror/simplePipeline.groovy")
        def pipelineFile = pipelinePath.toFile()

        def binding = new Binding()
        binding.setProperty("env", [:])
        GroovyShell shell = new GroovyShell(binding)
        pipelineScript = shell.parse(pipelineFile)
        mockJenkins(pipelineScript, steps, sections)

    }

    void testPipeline() {
        pipelineScript.run()
    }

}

    It looks much better, we separated steps and sections and only one piece of code is responsible for mocking each of them. We can also easily add new mocks now. We even have here this crazy feature of dynamic method names. It is awesome!

    However, we still cannot do any assertions. Let’s solve this problem as well. Firstly, we need to find a way to trace the script during the runtime and store its states in some memory structure. We need to know for each step in the pipeline:

    1. the caller hierarchy (which syntax item called which one, “did stage 1 called echo ?” )
    2. what was the state of global variables (“was env.TEST_GLOBAL_VAR set in “First stage” and did it have value “global_value” in the “Second stage”?)
    3. what exactly was called (“was echo step called with “test1″ parameter ?”)

    Ad. 1 We can achieve by extracting information from stack trace using for example these 2 methods:

    def getPipelineFileSTELements() {
        return Thread.currentThread().getStackTrace().findAll { item -> item.getFileName() == pipelineFile.getName(); }
    }

def createStackLine() {
        def lineNumbers = getLineNumbersFromSTElements(getPipelineFileSTELements())
        def stackLineAsList = getLinesFromPipelineFile(lineNumbers)
        return stackLineAsList.join(",")
    }

    Numbers of lines from pipeline file are caught and translated into text – we know what is the caller hirarchy now.

    Ad. 2 We can get information about state of global variables by using: pipelineScript.getBinding().getVariables()

    when mocking.

    Ad. 3 To get exact syntax item with parameters we can report them during mocking as well.

    This is example for 2 and 3:

    def mockSteps(pipelineScript, mockedSteps) {
        mockedSteps.each {
            step ->
                def currentStep = step
                pipelineScript.metaClass."$currentStep" = { Object... params ->
                    log.info(currentStep + " " + params[0].toString())
                    ResultStackProcessor.getInstance().storeInvocation(currentStep, params, pipelineScript.getBinding().getVariables())
                }
        }
    }

    Method “storeInvocation” is called with 3 parameters:

    • syntax item name
    • its parameters
    • current state of variables
    void storeInvocation(syntaxItem, parameters, runtimeVariables) {
        def stackLine = createStackLine()
        def syntaxItemInvocation = [:]
        syntaxItemInvocation.put(syntaxItem, parameters);
        resultStack.getInvocationStack().add(new ResultStackEntry(stackLine, syntaxItemInvocation, getDeepCopy(runtimeVariables)));
    }

    In this method we have 2, 3 coming as parameters, and 1 created by “createStackLine” function. I store it in ResultStackEntry class and name it: 1-stackLine, 2-invocations and 3-runtimeVariables.

    Take a look on github for details.

    Putting it all together, this code:

    import com.passfailerror.ResultStack
import com.passfailerror.ResultStackProcessor
import groovy.test.GroovyTestCase

import java.nio.file.Files
import java.nio.file.Paths

class runPipelineWithStacktrace extends GroovyTestCase {

    Script pipelineScript
    ResultStackProcessor stackProcessor

    def steps = ["label", "echo", "sh"]
    def sections = ["pipeline", "agent", "stages", "stage", "steps"]


    def mockJenkins(pipelineScript, mockedSteps, mockedSections) {
        mockSteps(pipelineScript, mockedSteps)
        mockSections(pipelineScript, mockedSections)
    }

    def mockSections(pipelineScript, mockedSections) {
        mockedSections.each {
            section ->
                def currentSection = section
                pipelineScript.metaClass."$currentSection" = { Object... params ->
                    log.info(currentSection)
                    if (params.length > 1) {
                        params[1].call() // stage("name"){closure}
                    }
                    else{
                        params[0].call() // steps{closure}
                    }
                    ResultStackProcessor.getInstance().storeInvocation(currentSection, params, pipelineScript.getBinding().getVariables())
                }

        }
    }


    def mockSteps(pipelineScript, mockedSteps) {
        mockedSteps.each {
            step ->
                def currentStep = step
                pipelineScript.metaClass."$currentStep" = { Object... params ->
                    log.info(currentStep + " " + params[0].toString())
                    ResultStackProcessor.getInstance().storeInvocation(currentStep, params, pipelineScript.getBinding().getVariables())
                }
        }
    }


    void setUp() {
        def pipelinePath = Paths.get("src/main/groovy/com/passfailerror/simplePipeline.groovy")
        def pipelineFile = pipelinePath.toFile()

        ResultStackProcessor.setPipelineFile(pipelineFile)
        ResultStackProcessor.setPipelineFileContent(Files.readAllLines(pipelinePath))
        ResultStackProcessor.setResultStack(new ResultStack())

        def binding = new Binding()
        binding.setProperty("env", [:])
        GroovyShell shell = new GroovyShell(binding)
        pipelineScript = shell.parse(pipelineFile)
        mockJenkins(pipelineScript, steps, sections)

    }

    void testPipeline() {
        pipelineScript.run()
        ResultStackProcessor.getResultStack().print()
    }

}

    produces stack trace in the output:

    pipeline,agent  label 'test',agent  label 'test'===[label:[test]]===[env:[:]]
pipeline,agent  label 'test'===[agent:[com.passfailerror.simplePipeline$_run_closure1$_closure2@4917d36b]]===[env:[:]]
pipeline,stages,stage('First stage'),steps,echo 'First'===[echo:[First]]===[env:[:]]
pipeline,stages,stage('First stage'),steps,sh 'mvn --version'===[sh:[mvn --version]]===[env:[:]]
pipeline,stages,stage('First stage'),steps===[steps:[com.passfailerror.simplePipeline$_run_closure1$_closure3$_closure4$_closure6@35c09b94]]===[env:[TEST_GLOBAL_VAR:global_value]]
pipeline,stages,stage('First stage')===[stage:[First stage, com.passfailerror.simplePipeline$_run_closure1$_closure3$_closure4@2d0bfb24]]===[env:[TEST_GLOBAL_VAR:global_value]]
pipeline,stages,stage('Second stage'),steps,echo 'Second'===[echo:[Second]]===[env:[TEST_GLOBAL_VAR:global_value]]
pipeline,stages,stage('Second stage'),steps,sh 'java -version'===[sh:[java -version]]===[env:[TEST_GLOBAL_VAR:global_value]]
pipeline,stages,stage('Second stage'),steps===[steps:[com.passfailerror.simplePipeline$_run_closure1$_closure3$_closure5$_closure7@c3fa05a]]===[env:[TEST_GLOBAL_VAR:global_value]]
pipeline,stages,stage('Second stage')===[stage:[Second stage, com.passfailerror.simplePipeline$_run_closure1$_closure3$_closure5@7b44b63d]]===[env:[TEST_GLOBAL_VAR:global_value]]
pipeline,stages===[stages:[com.passfailerror.simplePipeline$_run_closure1$_closure3@4a699efa]]===[env:[TEST_GLOBAL_VAR:global_value]]
pipeline===[pipeline:[com.passfailerror.simplePipeline$_run_closure1@38499e48]]===[env:[TEST_GLOBAL_VAR:global_value]]

    The last important thing here is global environment handling. In Jenkins, there is environment map named ENV where keys can be set by assigning values like this: env.TEST_VARIABLE=”test1″

    Groovy by default doesn’t know anything of such map, so it is neccesary to prepare it so that pipeline can populate it during runtime:

    def binding = new Binding()
binding.setProperty("env", [:])

    Assertions

    We have working framework with result stack but we still cannot do any assertions. We have to add this ability to be able to test anything, don’t we ?

    Let’s add some result stack validator (look at the Github for full code):

    class ResultStackValidator {

    static getInstance(){return new ResultStackValidator()}

    boolean stageCallsStep(String stageName, String stepName){
        return resultStackHasStageWithStep(stageName, stepName)
    }

    boolean resultStackHasStageWithStep(String stageName, String stepName){
        def result = ResultStackProcessor.getInstance().getResultStack().getInvocationStack().findAll(item->item.stackLine.contains(stageName))
        if(result.size()>0){
            return resultStackHasStep(result, stepName)
        }
        return false
    }

    And assertion class to be mini DSL for asserting result stack when writing tests:

    package com.passfailerror

class Assertion {

    static Assertion stage(String stageName){
        return new Assertion(stageName)
    }

    String stageName;

    public Assertion(String stageName){
        this.stageName = stageName
    }

    boolean calls(String stepName){
        return ResultStackValidator.getInstance().stageCallsStep(stageName, stepName)
    }

    boolean hasEnvVariable(String variableName){
        return ResultStackValidator.getInstance().stageHasEnvVariable(stageName, variableName)
    }
}

    Only now can we start doing some assertions:

    import com.passfailerror.Assertion
import com.passfailerror.ResultStack
import com.passfailerror.ResultStackProcessor
import groovy.test.GroovyTestCase

import java.nio.file.Files
import java.nio.file.Paths

class testPipelineWithAssertions extends GroovyTestCase {

    Script pipelineScript
    ResultStackProcessor stackProcessor

    def steps = ["label", "echo", "sh"]
    def sections = ["pipeline", "agent", "stages", "stage", "steps"]


    def mockJenkins(pipelineScript, mockedSteps, mockedSections) {
        mockSteps(pipelineScript, mockedSteps)
        mockSections(pipelineScript, mockedSections)
    }

    def mockSections(pipelineScript, mockedSections) {
        mockedSections.each {
            section ->
                def currentSection = section
                pipelineScript.metaClass."$currentSection" = { Object... params ->
                    log.info(currentSection)
                    if (params.length > 1) {
                        params[1].call() // stage("name"){closure}
                    }
                    else{
                        params[0].call() // steps{closure}
                    }
                    ResultStackProcessor.getInstance().storeInvocation(currentSection, params, pipelineScript.getBinding().getVariables())
                }

        }
    }


    def mockSteps(pipelineScript, mockedSteps) {
        mockedSteps.each {
            step ->
                def currentStep = step
                pipelineScript.metaClass."$currentStep" = { Object... params ->
                    log.info(currentStep + " " + params[0].toString())
                    ResultStackProcessor.getInstance().storeInvocation(currentStep, params, pipelineScript.getBinding().getVariables())
                }
        }
    }


    void setUp() {
        def pipelinePath = Paths.get("src/main/groovy/com/passfailerror/simplePipeline.groovy")
        def pipelineFile = pipelinePath.toFile()

        ResultStackProcessor.setPipelineFile(pipelineFile)
        ResultStackProcessor.setPipelineFileContent(Files.readAllLines(pipelinePath))
        ResultStackProcessor.setResultStack(new ResultStack())

        def binding = new Binding()
        binding.setProperty("env", [:])
        GroovyShell shell = new GroovyShell(binding)
        pipelineScript = shell.parse(pipelineFile)
        mockJenkins(pipelineScript, steps, sections)

    }

    void testPipeline() {
        pipelineScript.run()
        ResultStackProcessor.getResultStack().print()
        assert Assertion.stage("First stage").calls("sh")
        assert !Assertion.stage("First stage").hasEnvVariable("test1")
        assert Assertion.stage("First stage").hasEnvVariable("TEST_GLOBAL_VAR")
    }

}

    E voila. So finally we have very simple working example which can run simple pipeline and assert simple things.

    We can use as starting point for real framework with mocking all aspects of pipeline file: sections, steps, functions, objects and properties.
    This is all possible with using just 2 aspects of metaprogramming in Groovy: expando metaclasses and binding.

    Full code:

    https://github.com/grzegorzgrzegorz/pipeline-testing/tree/master

    Interesting links:
    https://www.groovy-lang.org/metaprogramming.html
    http://events17.linuxfoundation.org/sites/events/files/slides/MakeTestingGroovy_PaulKing_Nov2016.pdf