Setting up a self-hosted worker for SourceCraft

Self-hosted workers are users’ personal servers, both virtual and physical, on which CI/CD processes run. These processes will have access to the user server environment.

Restrictions on the amount of computing resources do not apply to self-hosted workers, and they also do not consume the total CI/CD runtime quota.

For more information, see Workers.

To set up a self-hosted worker:

  1. Set up your environment.
  2. Create a worker configuration file.
  3. Start the worker.
  4. Run a job on the worker.

See also:

Set up your environment

  1. Create a personal access token (PAT) with the Repository admin role and access to the repositories in which you want to run CI/CD processes on the self-hosted worker.

    Warning

    We do not recommend granting access to all the organization's repositories.

  2. Download the self-hosted-processor executable for your OS and architecture:

    Downloading the self-hosted-processor executable file for Linux ARM64

    Warning

    Linux ARM64 is a non-standard platform for CI/CD processes. This is an experimental test version of a self-hosted worker for this platform.

    We do not recommend using this version in production environment.

    Downloading the self-hosted-processor executable file for Linux ARM

    Warning

    Linux ARM is a non-standard platform for CI/CD processes. This is an experimental test version of a self-hosted worker for this platform.

    We do not recommend using this version in production environment.

    Downloading the self-hosted-processor executable file for Windows AMD64

    Note

    The executable for Windows works only with the MinGW toolkit.

    See also Updating a worker's executable.

  3. In macOS/Linux, grant permissions to run the executable:

    chmod +x <path_to_executable>

  4. Add the executable to PATH:

    export PATH=$PATH:<path_to_executable>

    $env:Path += ";<path_to_executable>"

  5. View the commands available for a worker:

    self-hosted-processor run --help

    To view the info and available options for a command, run this:

    self-hosted-processor <command> --help

Create a worker configuration file

You can automatically set up a worker configuration file by running the self-hosted-processor init command.

Supported arguments:

Key

Description

Default value

-c,
--capacity

Maximum number of jobs the worker can execute in parallel.

1

-r, --root-dir

Path to the directory where the worker will create temporary working directories for its jobs.

Directory for temporary files, e.g., /tmp/ or /usr/tmp/.

-t,
--token

Personal access token (PAT) for authentication in the service. The worker uses it to fetch jobs, report progress, and send logs and artifacts to SourceCraft.

Not set.

--tags

List of worker's tags. The worker will run jobs whose tags are all contained in the worker's tags.

Note

A tag's string values ​can contain a limited range of characters: Latin letters (a-zA-Z), -, ., and _.

For a self-hosted worker, parameters automatically include the self-hosted tag. You need not additionally add it to the tags field in config.yaml.

[ ]

--ssl-no-verify

Option to disable certificate verification. Use it if your device does not have a suitable root certificate (RootCA).

Disabled.
Root certificates installed on the device are used.

For example, generate a configuration for a worker that can run no more than three jobs with tags 4gb and go-builder at the same time:

self-hosted-processor init \
  --token <personal_access_token> \
  --capacity 3 \
  --tags 4gb,go-builder \
  > config.yaml

This will create the config.yaml worker configuration file:

executor_type: shell
self_hosted:
  max_slots: 3
logger_type: json
logger_level: info
endpoint:
  host: ci.sourcecraft.tech
  port: 443
  ssl_no_verify: false
tags:
- 4gb
- go-builder
auth:
  pat: <personal_access_token>

Create a file named config.yaml with the following configuration:

executor_type: shell

self_hosted:
  # Path to the directory where the worker will create temporary
  # working directories for its jobs.
  root_dir: /Users/user1/sourcecraft
  # Maximum number of jobs the worker 
  # can execute in parallel.
  max_slots: 3

logger_type: json
logger_level: info

# Address of the service for fetching jobs and sending logs and artifacts.
endpoint:
  host: ci.sourcecraft.tech
  port: 443
  ## To disable certificate verification, set to true.
  ssl_no_verify: false

# List of labels. The worker will run jobs whose tags
# are all present among the worker's tags.
# Latin letters and the -, . and _ characters are supported.
# The `self-hosted` tag is added automatically.
tags:
  - 4gb
  - go-builder

auth:
  # Personal access token for authentication in the service.
  pat: <personal_access_token>

This is the configuration of a worker that can run no more than three jobs with the 4gb and go-builder tags at the same time and create temporary working directories at /Users/user1/sourcecraft.

Learn more about worker tags.

Start the worker

Run this command:

self-hosted-processor run \
  --config-path <path_to_configuration_file>

Where --config-path is the path to the configuration file you created earlier. You can also use -c as the short form of this parameter.

Tip

The worker's logs go to a standard output stream. To write logs into a file or send them to a log collection service, redirect the output or use pipelines, |.

Run a job on the worker

To run a CI/CD process on a custom self-hosted worker, specify runs_on: self-hosted in the workflow (workflow) or task (task) parameters.

If runs_on is not specified in task, the parameter from workflow:runs_on will be used by default.

Here is an example:

workflows:
  my-awesome-workflow:
    runs_on: self-hosted
    
    tasks:
      - name: self-hosted-task
        cubes:
          - name: hello
            script:
              - echo "hello from self-hosted"

      - name: self-hosted-go-builder-task
        runs_on: [self-hosted, go-builder]
        cubes:
          - name: hello
            script:
              - echo "hello from self-hosted go builder"

In this case, tasks from my-awesome-workflow will run on any of your self-hosted workers by default unless their runs_on parameter is redefined. At the same time, self-hosted-go-builder-task will run only on the self-hosted worker assigned the go-builder tag during initialization.

Setting a logging level

To set a logging level for your worker, edit the logger_level field in the config.yaml file. The following values are supported:

  • fatal: Shows only errors that are logged as fatal and had caused the app to crash.
  • error: Shows errors that may indicate problems with the app, e.g., orchestrator inaccessible, network errors, not enough disk space, etc.
  • warn: Shows fatal and error-level errors, as well as errors that may occur during normal operation of the app. For example, failure to create a symlink when copying and having to perform full copying, failure to find the .ENV file of the job's first or last cube, etc.
  • info: Shows fatal, error, and warn-level errors and messages about what is going on in the app:
    • Requests for jobs (once every few seconds).
    • Sending job status reports.
    • Demonstration of the script and info on the step in progress.
  • debug: Shows fatal, error, and warn-levels errors, info messages, as well as other debugging information, such the variables transmitted between cubes.

Updating a worker's executable

The Set up your environment section provides links to the relevant versions of the worker's executables.

To check the version of the worker's executable installed in your environment, run this command:

self-hosted-processor version

Result:

<``>SourceCraft<``> Processor
Version: 0.9.1
Build date: 10 June 2025
Target OS: darwin
Target Arch: arm64

To update the worker's executable, download the new file and replace the old one.

See also

Previous
Processing environment variables
Next
Configuring a service connection to Yandex Cloud