Our DevOps pipeline for a heterogeneous fleet of autonomous robots

At Black Coffee Robotics we help companies build software for their autonomous robots. We have developed software stacks for various applications like industrial flooring cleaning robots, intra-logistics robots, lawn-mowing robots and even autonomous boats. One of our major focus areas is dealing with fleets of robots. When working with a large number of robots, deploying and updating software quickly becomes a challenge. Having a simple way to deploy software to multiple machines enables a fast software iteration cycle and is imperative for a mature product.

‍

Use Case

In one of our use-case, we are controlling a heterogeneous fleet of robots. Robots with different capabilities and form factors operate in the same location and have the same software running on them. The compute platforms on some robots are Intel NUCs which have amd64 architecture and the rest are running on NVIDIA Jetson AGX Xavier compute modules. NUCs run amd64 architecture, which is one of the most common architectures for Laptops and development machines. Xavier is based on aarch64 (also known as arm64) architecture which is also used by single board computers like Raspberry Pi. The software that runs on these different robots is exactly the same and the robots’ different capabilities are managed using parameters and namespaces. A naive way of deploying software for the fleet would be to have dedicated build machines of each architecture. Setting up cross-compilation toolchains can also be tricky. We found docker buildx to be the perfect solution for us to build once, deploy anywhere.

‍

Why we use dockers to deploy code

Managing dependencies for various sensors for a robot is tricky. Sensors like IntelRealsense cameras, or GPU accelerated stereo cameras like ZED camera can require specific version of libraries as dependencies. Different sensors can have different versions of the same library as a dependency for their drivers. Separate teams can also be specializing on different software subsystems, and bringing together their work on a single machine can be challenging due to integration issues. We have found using containers as the best way to deploy code on our robots. With docker containers, we can develop a driver once and keep it isolated and agnostic of Ubuntu or ROS versions of the host system.

‍

We use dockers both during development and production. In development dockers, we pull code into the docker and compile it, whereas in production we can debianize the software and install it inside the docker that is supposed to be deployed on the robot.

‍

Leveraging CI tools

In absence of any tooling, a typical way to deploy on different machines would be to manually build docker images for each update/release and then push to a container registry or build tarballs that are manually copied to respective machines. A quick turnaround time is the essence of DevOps so reducing the time taken from pushing the code onto a git repository to having it compiled and running on robot hardware is essential for shorter iteration cycles and faster testing of features. We host our git repos on Gitlab and use Gitlab CI for our build-test-deploy pipeline.

Let’s look at an example configuration to understand the process. In this demo repository I have set up Gitlab CI to execute a pipeline every time someone commits to the default branch. The Dockerfile is supposed to build a docker image that would be deployed on robot hardware. For your application you could pull and compile code from the same repo and install other required dependencies into the docker container. After the build pipeline succeeds, the images are uploaded to Gitlab’s container registry linked to the same repo.

Let us have a look at the .gitlab-ci.yml file where the CI pipeline is defined.

‍

docker-build:
 variables:
   MULTIARCH_PLATFORMS: "linux/amd64,linux/arm64"
   DOCKER_BUILDKIT: 1
   RELEASE_TAG: "latest"
 image: docker:latest
 stage: build
 services:
   - docker:20.10.16-dind
 before_script:
   - echo "$CI_REGISTRY_PASSWORD" | docker login -u "$CI_REGISTRY_USER" --password-stdin $CI_REGISTRY
 script:
   - IMAGE_TAG=${CI_REGISTRY_IMAGE}:latest
   - docker run --rm --privileged tonistiigi/binfmt --install all
   - docker context create multiarch-context
   - docker buildx create --name multiarch-builder --driver docker-container --use multiarch-context
   - docker buildx inspect --bootstrap
   - docker buildx build --platform $MULTIARCH_PLATFORMS --cache-from $IMAGE_TAG --build-arg BUILDKIT_INLINE_CACHE=1 -t $IMAGE_TAG --push .rules:
   - if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'
   - if: '$CI_COMMIT_TAG'

‍

Breaking down what’s happening at a higher level:

• qemu based emulators are installed in a docker container which will build the final docker images.

- docker run — rm — privileged tonistiigi/binfmt — install all

• Setup for docker buildx and setting up caching using Buildkit

- docker context create multiarch-context
- docker buildx create --name multiarch-builder --driver docker-container --use multiarch-context
- docker buildx inspect --bootstrap

• building docker images for the Dockerfile in this repository for the architectures specified in MULTIARCH_PLATFORMS variable (in this case amd64 and arm64) and pushing them to the gitlab repo’s container registry .

- docker buildx build --platform $MULTIARCH_PLATFORMS --cache-from $IMAGE_TAG --build-arg BUILDKIT_INLINE_CACHE=1 -t $IMAGE_TAG --push .

• Using the updated docker image any computer architecture is as simple as running a docker pull as the same image is supported on all the architectures that we built the image for. For our demo, the image can be pulled using,

docker pull registry.gitlab.com/zubinp/multiarch_demo:latest

‍

Conclusion

DevOps and related tooling ecosystem has a vast scope, especially when applied to robotics software development. In this article, I walked through the pipeline that we have been using for our robotics projects. We’ve highly simplified building and deploying code for non-trivial applications. If you need engineering help for your autonomous robots, feel free to reach out to us at Black Coffee Robotics!