Skip to main content

Deployment procedure

This guide aims to guide you through the deployment procedure of the entire ACUBE stack:

  • ACUBETotal
  • OpenCTI
  • CAPEv2 Sandbox
  • Correlation engine
tip

This guide will also provide some useful debugging/troubleshooting tips. However, if you encounter an error that is not shown here: Google is your best friend.

note

Please read through the entire post before deploying.

General knowledge​

Systemd services​

Most of the services are deployed as systemd unit services

Useful commands​

  • systemctl
    • systemctl status <service_name> - check the current state of the service
  • systemctl [start | restart | stop] <service_name> - change the state of the service
  • systemctl [enable | disable] <service_name> - enables/disables service start-up on boot
  • journalctl - useful for checking the logs of systemd services

Docker​

Knowledge of docker is extremely useful, including (but not limited to):

  • images
  • containers
  • volumes
  • networking (extremely useful)

Useful commands​

  • docker exec -it <running_container_name> <command> - useful for sanity checking to ensure commands run fine on the container
    • e.g. docker exec -it acubetotal-service_floss /bin/bash

First-time deployment procedure​

  1. Ensure you have all dependencies ready
    1. Includes apt packages and pip packages
  2. Install docker and docker-compose
  3. Load in all docker images
# You can manually pull each tar file per docker image
docker pull ghcr.io/csitgovsg-acube/acubetotal-frontend:main

# A gzip tar file of multiple docker images works too
docker load -i acubetotal-airgap-images-main.tar.gz

# Alternatively, just load in all docker tar / gzip files in current directory
for f in *; do docker load -i $f; done
  1. Install ansible
sudo apt install python3-pip
pip3 install ansible
  1. Install sshpass on the host running ansible (if using ssh connections with passwords)
sudo apt install sshpass

  1. Add local bin directory to PATH

    1. Add the following line to ~/.bashrc file:
    export PATH="$USER/.local/bin:$PATH"
    1. Activate the ~/.bashrc file, either by restarting the shell or running the following command:
    source ~/.bashrc

  2. Run the ansible playbook

    ansible-playbook --ask-become-pass --inventory inventory-local.yml --extra-vars setup_virt=yes setup.yml
    1. extra-vars specifies additional variables that you want to manually override
      1. Notably, setup_virt should only be run when setting up virutal machines via ansible, e.g. on the first run or together with reset_virt enabled
      2. This will take an additional 20-30 minutes to spin up the virtual machines for CAPEv2 sandbox
    2. ask-become-pass will prompt the user to specify the sudo password

  3. If the playbook errors out, debug the error and repeat

  4. Debug connectivity issues between docker containers, if any

What does the ansible-playbook do?​

  • Installs all the relevant dependencies
  • Deploys systemd services that run the docker containers
    • All systemd units are located at /etc/systemd/system as *.service files

Common deployment issues​

General debugging tips​

  • Check service status: systemctl status <service>

  • Read journalctl logs of the service: journalctl -fu <service> -n 100

  • Execute code directly in the running docker container: docker exec -it <container_name> /bin/sh

  • List all docker images: docker image ls

  • List all running docker containers: docker ps

  • List all docker containers (both running and stopped): docker ps -a

  • After editing the system files, you need to reload them to take effect before restarting

    systemctl daemon-reload
    systemctl restart <service>

Unable to find docker image​

Error​

Systemd services fail to start with the following error message in journalctl:

Unable to find image <docker_name> locally
docker: Error response from daemon: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io: Temporary failure to name resolution.

Solution​

  1. Check that the docker name is correct / spelt correctly
    1. Run docker image ls to see the list of loaded docker images, and check that the docker instance that you are trying to run is available locally
  2. Check that the version specified is correct (note that docker defaults to version latest)
    1. hello:main ≠ hello:v1 ≠ hello

Unable to bind to address​

Error​

Systemd services fail to start with the following error message in journalctl:

[Errno 98] error while attempting to bind an address ('0.0.0.0', 8000): address already in use

Solution​

  1. Check that the port you are trying to bind to is not in use
    1. netstat -tulpn will list all current ports that are open, check that the port you are trying to bind to is not inside the output
  2. Note the difference between a port in the host, and a port in the docker container

Docker container doesn’t start with no error messages​

Error​

In journalctl logs, the systemd service simply fails with no proper error messages:

Started <docker_image>.
<docker_image>: Main process exited, code=exited, status=254\n\a
<docker_image>: Failed with result `exit-code`.

Solution​

  1. Check that the --read-only flag is not specified in the systemd service files
    1. E.g. if acubetotal-frontend.service is failing, edit the following file: /etc/systemd/system/acubetotal-frontend.service
    2. Note the back-slashes at the end of each argument for multi-line commands

Connectivity issues between each docker container​

Error​

All docker containers can start and run fine, but they cannot communicate between each docker container.

Solution​

  1. Often a result of firewall (disable if possible)
    1. ufw disable
  2. Check if packets can reach the app using Wireshark or tcpdump
    1. tcpdump -i <interface or 'any'> port <port_number>
  3. Note that host IP is different from docker container IP
    1. Check using ifconfig or ip addr
    2. List docker networks with docker network ls
    3. List the container IPs in the docker network with docker network inspect <docker_network>
    4. Tip: Instead of specifying the container IP itself, docker can resolve the container IP using the container name.
      1. Say there are 2 docker containers, foo and bar in the docker network test
      2. From within the docker container foo, you are able to access bar's webserver using http://bar
  4. Note the difference between internal_url and external_url in the .env files

CAPE service failure - migrations not applied​

Error​

The cape and cape-processor services fail with the message migrations not applied

Solution​

  1. Enter the CAPEv2 db_migration directory
    1. Default directory: /opt/CAPEv2/utils/db_migration
  2. Run the migrate command: sudo -u cape alembic upgrade head

CAPE service failure - OSError or PermissionError​

Error​

The cape and cape-processor services fail with the message OSError or PermissionError. You may have accidentally written to the directory using another user, causing cape to lose write access to the folder.

Solution​

  1. Ensure that the cape user has read and write permissions to all files and subdirectories in /opt/CAPEv2
    1. Ensure all files in /opt/CAPEv2 are owned by the cape user: sudo chown -R cape:cape /opt/CAPEv2

CAPEv2 sandbox tasks are stuck at “pending” (should no longer be an issue)​

Error​

After submitting tasks, the tasks are always stuck at “pending” and do not become “running” or “completed”

Solution​

Run this command: sudo -u postgres psql -c "update tasks set machine='' where status='pending';" capedata

Offline installation of CAPEv2 dependencies​

  1. Copy out the entire git repository, especially pyproject.toml and poetry.lock, to an internet-connected PC (with the same folder CAPEv2)
  2. Run the following commands:
pip3 install -U poetry
poetry update
  1. Copy the entire .cache folder from the internet PC to the airgapped machine and place it in the folder /opt/CAPEv2
  2. The pip dependencies should now be mirrored successfully

Steps to mirror pip packages for CAPEv2 (if not using poetry)​

danger

We highly recommend using poetry, as it handles the fixing of package dependencies, as well as fixing all the installed packages in a fixed directory. It does not have a steep learning curve, and is extremely easy to pick up.

You’re on your own. Refer to requirements.txt. You may choose to create a virtualenv and install these dependencies inside a virtual environment.

Dependencies​

  • APT and python dependencies (a full list can be found here)
  • VM base images for CAPEv2 sandbox (both x64 and x86)
  • Docker images and services (found here)