Deploying a self-hosted SourceCraft worker on a Yandex Compute Cloud VM

Self-hosted workers are users’ personal servers, both virtual and physical, on which SourceCraft CI/CD processes run. These processes will have access to the user server environment.

Restrictions on the amount of computing resources do not apply to self-hosted workers, and they also do not consume the total CI/CD runtime quota.

If you have a computer with a suitable configuration, you can set up a self-hosted worker for SourceCraft all by yourself. Otherwise, you can create a VM in Compute Cloud and install a self-hosted worker on it. Even more, you can automate not only the processes of creating a VM, installing, configuring, and starting a self-hosted worker, but also the process of deleting a VM after the CI/CD workflow is completed. This way you can optimize the use of your Compute Cloud resources.

Here we discuss an example of how to implement such automation with the help of SourceCraft CI/CD processes using a service connection.

Self-hosted worker use cases:

  • In the same repository the worker was created from. In cases like this, the VM can be automatically deleted as soon as the workflow is completed.

    test-task-and-delete-vm-async workflow example
    test-task-and-delete-vm-async:
  runs_on: self-hosted
  tasks:
    - test-task
    - name: delete
      uses: delete-vm-async
      needs: [test-task]

    The test-task performs useful work, such as compiling a project.

    The delete task gets started only after test-task (the needs: [test-task] parameter) is executed and asynchronously deletes the VM on which the self-hosted worker was run.

    Before executing the test-task-and-delete-vm-async workflow, you need to execute the create-vm workflow the self-hosted worker will be created in.

    Note

    You cannot set a precise workflow sequence, so workflows must be started manually one by one.

  • In any other repository of the organization. In this case, you need to delete the VM with the self-hosted worker manually.

To deploy a self-hosted SourceCraft worker on a Compute Cloud VM:

  1. Get ready.
  2. Deploy the infrastructure.
  3. Create a repository with a CI/CD configuration.
  4. Create secrets.
  5. Set up a service connection.
  6. Set up a CI/CD process.
  7. Test the worker.

If you no longer need the resources you created, delete them.

Get ready for work

  1. Sign up for Yandex Cloud and create a billing account:

    1. Navigate to the management console and log in to Yandex Cloud or create a new account.
    2. On the Yandex Cloud Billing page, make sure you have a billing account linked and its status is ACTIVE or TRIAL_ACTIVE. If you do not have a billing account, create one and link a cloud to it.

    If you have an active billing account, you can create or select a folder for your infrastructure on the cloud page.

    More on clouds and folders

  2. Authenticate in SourceCraft on the service home page or sign up.

The cost of supporting a self-hosted worker infrastructure includes:

Deploy the infrastructure

  1. If you do not already have a network infrastructure in the Yandex Cloud folder you want to have your self-hosted worker VM in, create a cloud network, e.g., default, and a subnet, e.g., default-ru-central1-d.

    Note

    The example CI/CD configuration discussed here uses a network named default and a subnet named default-ru-central1-d.

  2. In the given example, the default security group is used for the VM. Add the following rules to it:


    Traffic
    direction

    Description

    Port range

    Protocol

    Source /
    Destination name

    CIDR blocks

    Inbound

    Access to VM from
    the internet

    0-65535

    Any

    CIDR

    0.0.0.0/0

    Outbound

    VM access to
    the internet

    0-65535

    Any

    CIDR

    0.0.0.0/0

  3. In the Yandex Cloud folder you want to have your self-hosted worker VM in, create a service account with the compute.editor role for the folder.

    This service account will be used to create VMs in Compute Cloud and establish a SourceCraft service connection.

Create a repository with a CI/CD configuration

Tip

This example shows how to create a new repository based on a self-hosted-worker repository.

Instead of creating a new repository, you can also fork it from the self-hosted-worker repository.

  1. Create a new empty repository in the organization whose cloud and folder will be used for the self-hosted worker VM.

    On the page that opens, copy the link for access to the new repository formatted as: https://git@git.sourcecraft.dev/<organization_slug>/<new_repository_slug>.git.

    Note

    In the example below, you work with git via HTTPS. You can also use SSH.

  2. Clone the self-hosted-worker repository with an example of a self-hosted worker deployment:

    git clone https://git@git.sourcecraft.dev/examples/self-hosted-worker.git
cd self-hosted-worker

    The self-hosted-worker repository contains the following files:

    ├── .sourcecraft
│   └── ci.yaml                      # SourceCraft CI/CD configuration
├── .gitignore
├── compute-instance-create.sh       # script for creating a VM
├── compute-instance-delete-async.sh # script for deleting a VM in asynchronous mode
├── compute-instance-delete.sh       # script for deleting a VM
├── metadata.yaml                    # metadata for a VM
└── README.md

  3. Use the self-hosted-worker repository as the basis for your new repository:

    git remote remove origin
git remote add origin https://git@git.sourcecraft.dev/<organization_slug>/<new_repository_slug>.git
git branch -M main

  4. Push your changes to the new repository:

    git push -u origin main

Create secrets

  1. Get a personal authentication token (PAT) with the Repository administrator role and access to the repository you want to be using the self-hosted CI/CD worker in.

    Tip

    If you want to use the worker for more than just the repository it was created from, specify the appropriate permissions for the personal token.

    The worker uses this token to get authenticated in SourceCraft.

  2. Generate SSH keys for access to the worker VM.

  3. Create the following SourceCraft secrets:

    • With the PAT name and the token value.
    • With the SSH_PUB name and a public SSH key.

Set up a service connection

Service connections allow you to securely integrate your SourceCraft projects with the Yandex Cloud resources.

Create a service connection named default-service-connection, use the previously created service account with the compute.editor role for the folder.

Warning

The SourceCraft grant you get when creating a service connection in a SourceCraft personal organization, does not apply to Yandex Compute Cloud and Yandex Virtual Private Cloud.

Set up a CI/CD process

Set up a CI/CD process to deploy a self-hosted worker.

Configure basic CI/CD settings

The CI/CD process is configured in the .sourcecraft/ci.yaml file.

The presented CI/CD configuration consists of the following workflows:

  • create-vm: Creates a VM with a self-hosted worker.
  • delete-vm: Deletes a VM with a self-hosted worker.
  • test-task-and-delete-vm-async: Runs a test task on a self-hosted worker and deletes the VM asynchronously.

Note

The create-vm and delete-vm workflows run on standard SourceCraft cloud workers. The test-task-and-delete-vm-async process runs on a self-hosted worker.

Adjust the CI/CD configuration for your needs or use an existing one.

Set your VM parameters

The VM configuration for a self-hosted worker is specified in the env variable section in the .sourcecraft/ci.yaml file:

  • COMPUTE_INSTANCE_NAME: VM name and host name, e.g., self-hosted-worker.
  • VPC_SUBNET_NAME: Name of the subnet created earlier the VM will reside in, e.g., default-ru-central1-d.
  • CREATE_BOOT_DISK: Disk image from which the VM will be created, e.g., image-folder-id=standard-images,image-family=ubuntu-2204-lts (Ubuntu 22.04 LTS). See also Getting a list of public images.
  • PLATFORM: VM platform, e.g., standard-v3.
  • MEMORY: Amount of RAM, e.g., 64GB.
  • CORES: Number of vCPU cores, e.g., 8.
  • CORE_FRACTION: Guaranteed share of vCPUs, e.g., 100.
  • PREEMPTIBLE: Whether or not to use a preemptible VM, e.g., false.
  • SSH_PUB: Previously created secret with the public part of the SSH key in ${{ secrets.<secret_name> }} format. Set in case you need to connect to a VM with a self-hosted worker.

A VM user with the builder login is created within the metadata.yaml file.

users:
- name: builder
  sudo: 'ALL=(ALL) NOPASSWD:ALL'
  shell: /bin/bash
  ssh_authorized_keys:
  - ${SSH_PUB}

Adjust the VM configuration for your needs or use an existing one.

Create a script to install a self-hosted worker and other software on the VM

To install an executable of a self-hosted worker or other software, e.g., Docker, the metadata.yaml file is used:

  - path: "/usr/local/etc/startup.sh"
    permissions: "755"
    content: |
      #!/bin/bash

      # install self-hosted-processor
      apt-get update
      apt-get install -y musl
      curl \
        --silent \
        --show-error \
        --location \
        https://storage.yandexcloud.net/src-processor-downloads/self-hosted-processor-latest/linux/amd64/self-hosted-processor \
        --output self-hosted-processor
      chmod +x self-hosted-processor
      mv self-hosted-processor /home/builder/sourcecraft
      chmod 777 -R /home/builder/sourcecraft

      # install docker
      apt install ca-certificates curl software-properties-common apt-transport-https -y
      install -m 0755 -d /etc/apt/keyrings
      curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
      echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
      apt update
      apt install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin -y
      usermod -aG docker builder

In a newly deployed VM, the /usr/local/etc/startup.sh bash script is created and executed, which installs the self-hosted worker executable and Docker.

Adjust the installation script for your needs or use an existing one.

Configure a self-hosted worker

The config.yaml worker configuration file is also configured in the metadata.yaml file:

write_files:
  - path: "/home/builder/sourcecraft/config.yaml"
    permissions: "644"
    content: |
      executor_type: shell
      self_hosted:
        root_dir: /home/builder/sourcecraft/tmp
        max_slots: 3
      logger_type: json
      logger_level: info
      endpoint:
        host: ci.sourcecraft.tech
        port: 443
        ssl_no_verify: false
      auth:
        pat: ${PAT}

Test the worker

Create a VM with a self-hosted worker

Manually run the create-vm workflow.

You can follow the creation process using the cube logs. If the workflow completes successfully, you can see the new VM in the Yandex Cloud management console using the link from the create-vm cube log.

Run a test workflow

  1. Manually run the test-task-and-delete-vm-async workflow.
  2. In test-task, check the logs of the test-worker and test-docker cubes:
    • The test-worker cube logs should present the results of the uname -a and free -h commands run directly on the worker.
    • The test-docker cube logs should present the version of the Yandex Cloud CLI that will be run in the Docker container.

After test-task is completed, the delete (delete-vm-async) task will be started, which will attempt to delete the VM in asynchronous mode. You can see the deletion result in the Yandex Cloud management console using the link from the delete-vm cube log.

How to delete the resources you created

To stop paying for the resources you created, manually run the delete-vm workflow or delete the self-hosted worker VM in the Yandex Cloud interface.

Optionally, delete the network and subnet.

See also

Previous
Hosting a static website using SourceCraft Sites
Next
Resource relationships