24 07 2018

Concourse lazy pipeline templating for free ?

gaelL | concourse, continuous integration, devops, programming, tools | 0

I’ve been using Concourse CI since quite a long time and I wanted to share with you a little trick to natively improve flexibility of pipelines.

A Concourse pipelines looks like the following one :

resources:
- name: booklit
  type: git 
  source:
    uri: https://github.com/vito/booklit
    branch: master

jobs:
- name: unit
  plan:
  - get: booklit
    trigger: true
  - task: say-hello
    config:
      platform: linux
      image_resource:
        type: docker-image
        source: {repository: alpine}
      run:
        path: /bin/sh
        args:
        - -ec
        - | 
          echo ${MSG}
    params:
      MSG: "Hello world !"

- name: build
  plan:
  - get: booklit
    passed: [unit]
    trigger: true
  - task: say-hello
    config:
      platform: linux
      image_resource:
        type: docker-image
        source: {repository: alpine}
      run:
        path: /bin/sh
        args:
        - -ec
        - | 
          echo ${MSG}
    params:
      MSG: "Hello world 2 !"

Pipeline

https://concourse-ci.org/pipelines.html

This pipeline example contains 2 jobs and a resource. Pretty simple, but you can imagine a bigger one with several resources, jobs and tasks. The pipeline could end up in a huge bit YAML file. Additionally to that, Concourse will fail to update a pipeline if you declare a resource not used by a job. This could be boring when you just want to start from a pipeline sample and start to play with it.

Because of that, you usually end up using templating. There are several ways to do it, for example using engine like Jinja2.

But for smaller needs usually you don’t want to write some code and call it to render the template before using it in Concourse.

That’s where the following tricks could interest you. Have you ever heard about YAML Anchors and aliases ?

Well, everybody know YAML, but few people already used Anchors and aliases. The nice things about that, it’s native to most of YAML parser, you are free to use it everywhere (not only For Concourse).

Anchors and aliases let you identify an item with an anchor in a YAML document (identified by a & char), and then refer to that item with an alias (identified by a * char)

But it also lets you do some cool stuff like override or use only specific part of an Anchor when you call it with the Alias.

Which in our example could give the following pipeline

templating:
  - &say-hello
    task: say-hello
    config:
      platform: linux
      image_resource:
        type: docker-image
        source: {repository: alpine}
      run:
        path: /bin/sh
        args:
        - -ec
        - | 
          echo ${MSG}

resources:
- name: booklit
  type: git 
  source:
    uri: https://github.com/vito/booklit
    branch: master

jobs:
- name: unit
  plan:
  - get: booklit
    trigger: true
  - <<: *say-hello
    params:
      MSG: "Hello world !"

- name: build
  plan:
  - get: booklit
    passed: [unit]
    trigger: true
  - <<: *say-hello
    params:
      MSG: "Hello world 2 !"

Improved pipeline

Really simple right ?

I created a fake YAML section templating which is not read by Concourse which define the task of my jobs (Anchor). Then I simply can use the Alias to call it.

Note the override of params field to allow me to give specific configuration to my task.

I hope it will help you. You can see more about YAML Anchors and aliases here https://blog.daemonl.com/2016/02/yaml.html

gaelL