Self-hosted worker tags in SourceCraft

For advanced interaction with self-hosted workers, there is a special tag mechanism in SourceCraft.

Tags are string values ​​you can use to influence the choice of the self-hosted worker the CI/CD tasks will be executed on. The worker will run tasks whose tags are all contained in the worker's tags.

Tags are set on these two sides of the process:

• In the .sourcecraft/ci.yaml CI/CD configuration in workflow and/or task elements in the runs_on field.
• In the config.yaml self-hosted worker configuration in the tags field.

Example of CI/CD and self-hosted worker configurations using tags

config.yaml self-hosted worker configuration:

executor_type: shell

self_hosted:
  # Path to the directory where the worker will create temporary
  # working directories for its jobs.
  root_dir: /Users/user1/sourcecraft
  # Maximum number of jobs the worker 
  # can execute in parallel.
  max_slots: 3

logger_type: json
logger_level: info

# Address of the service for fetching jobs and sending logs and artifacts.
endpoint:
  host: ci.sourcecraft.tech
  port: 443
  ## To disable certificate verification, set to true.
  ssl_no_verify: false

# List of labels. The worker will run jobs whose tags
# are all present among the worker's tags.
# Latin letters and the -, . and _ characters are supported.
# The `self-hosted` tag is added automatically.
tags:
  - 4gb
  - go-builder

auth:
  # Personal access token for authentication in the service.
  pat: <personal_access_token>

This is the configuration of a worker that can run no more than three jobs with the 4gb and go-builder tags at the same time and create temporary working directories at /Users/user1/sourcecraft.

.sourcecraft/ci.yaml CI/CD configuration:

workflows:
  my-awesome-workflow:
    runs_on: self-hosted
    
    tasks:
      - name: self-hosted-task
        cubes:
          - name: hello
            script:
              - echo "hello from self-hosted"

      - name: self-hosted-go-builder-task
        runs_on: [self-hosted, go-builder]
        cubes:
          - name: hello
            script:
              - echo "hello from self-hosted go builder"

In this case, tasks from my-awesome-workflow will run on any of your self-hosted workers by default unless their runs_on parameter is redefined. At the same time, self-hosted-go-builder-task will run only on the self-hosted worker assigned the go-builder tag during initialization.

Advanced use of tags in a CI/CD configuration

Apart from directly specifying in the CI/CD configuration the tags that must be present on a self-hosted worker to execute a task, you can do the opposite and exclude workers with particular tags. Do it using the ! negation before the tag in the runs_on field in .sourcecraft/ci.yaml, e.g., [self-hosted", "!my-worker"]. In which case the task cannot be run on a worker with the my-worker tag.

For example, you have several workers with different tags for each one: small, medium, large, and extra-large. To run the task on any worker except small, use the following CI/CD configuration:

workflows:
  sample-workflow:
    tasks:
      - name: not-small-task
        runs_on: ["self-hosted", "!small"]
        cubes:
          - ...

When scheduling the task named not-small-task, SourceCraft will only forward it for execution to a self-hosted worker without the small tag among its tags.

Advanced use of tags in a self-hosted worker configuration

By default, a self-hosted worker can run any tasks whose tags are all present among the worker's tags.

On the worker's side, you can limit the tasks that can be run on it by listing the tags that must be present in the runs_on field of the CI/CD configuration. Do it using the + symbol before the tag in the tags field in config.yaml, e.g., +my-obligatory-tag. In which case only tasks with this tag in the runs_on field can be executed on the worker, e.g., ["self-hosted", "my-obligatory-tag"].

For example, you have a self-hosted worker with a GPU and you want it to execute only tasks with the gpu tag in the runs_on field. Proceed as follows:

1. Configure the worker:

   Automatically

   Manually

   Run this command:

   self-hosted-processor init \
     --token <personal_access_token> \
     --tags 4gb,+gpu \
     > config.yaml
   

   Where:

   • --token: Personal access token (PAT) for authentication in the service. The worker uses it to fetch tasks, report progress, and send logs and artifacts to SourceCraft.
   • --tags: List of worker tags.

   Create or modify the existing config.yaml file:

   executor_type: shell
   self_hosted:
     max_slots: 3
   logger_type: json
   endpoint:
     host: ci.sourcecraft.tech
     port: 443
     tls: true
   # List of worker tags
   tags: ["4gb", "+gpu"]
   auth:
     # Personal access token for authentication in the service
     pat: <personal_access_token>
   

2. Start the worker:

   self-hosted-processor run \
     --config-path <path_to_configuration_file>
   

   Where --config-path is the path to the config.yaml file.

Now the worker can only run tasks with the gpu tag in the runs_on field.

Useful links

• Setting up a self-hosted worker for SourceCraft
• Deploying a self-hosted SourceCraft worker on a Yandex Compute Cloud VM
• Continuous integration and continuous deployment in SourceCraft
• CI/CD reference
• Configuring CI/CD in a SourceCraft repository