Mastering Automation

Automation is the backbone of my teams work at EscherCloud AI, having the responsibility of standing up and configuring a significant amount of different hardware and software at scale means that any task that we do more than once has to be automated, otherwise we can not move at speed, and efficiency suffers.

With this in mind I thought I would quickly describe the process that I go through when attempting to automate a new process, hopefully you find this helpful.

The Stages.

Understand the workflow you are being asked to automate.

Now this sounds simple, but I’m talking about REALLY understanding the workflow. At this point you need to understand all the inputs which you will need, where you can get that input from, which systems you will need to touch etc. Ideally you need to sit with people who are doing this manually and ask them to walk you though the process. Once you have been walked through the process you need to go off and have a go yourself, as many times as you can until you fully understand how everything fits together. Without this knowledge you are setting yourself up for failure.

Visualise the workflow.

Once you have a solid understanding of the workflow, I really like to diagram this out on paper. I firstly write out all the steps, and then taking the knowledge gained in step one I draw a simple flow diagram which shows the when the inputs are needed, what outputs are generated, if they need to be stored, where they get stored, or if they are passed into another stage. At this point you also need to be thinking about the error handling, as this is going to automated you need to be able to inform people if the flow breaks or goes wrong.

Visualising workflows on paper, will help you to spot edge cases and failure modes that can occur when moving from a manual process to an automated one.

At the end of this stage, you should have a diagram that explains all the steps, the inputs and outputs and any odd edge cases you can think off.

Tool Selection.

At this point I usually have a good idea of the tools I’m going to use to automate this workflow. If your not sure, I find this stage useful to really start narrowing down what your going to need to do. Don’t be afraid to mix and match tools to take advantage of their strengths and avoid their weaknesses. Usually tools land in one of two camps. Camp 1, tools that actually do a thing, so something like Ansible, or Terraform, and camp two, tools used to glue things together, this might be Bash, Python or another programming language, or it might be a process automation tool like StackStorm or Rundeck.

If you going to automate a process and glue it together with something like Python, use this step to also look at libraries that exist which will help you achieve what your looking to automate. Also use this time to think about how you want to structure the tools together to achieve the end goal of end to end workflow automation.

Draft a Skeleton flow.

Usually I will now start to layout pseudo-code and start structuring my code layout. This helps me to understand if I need to write any modules or if I can make use of any modules that have been previously written by the team. By the end of this stage, you should have a good understanding of the functions you will need to create, as well as the overall flow of the code.

Coding, testing and iteration time.

Now its onto the fun part, take your time and write good quality code, break the code down into testable chunks, and test those chunks as much as you can. Unit tests are great, but sometimes you need to just test against what your trying to automate, this should be done against a staging environment.

Keep testing and extending your code, until the entire workflow is automated. Remember its fine to release sections of the workflow if they can be release individually, and if releasing them will provide some benifit to the organisation.

This is how I go about automating a workflow, and hopefully it will help you the next time you need to automate something.

Until next time, Steve.