Setting up a self-hosted worker for SourceCraft

Self-hosted workers are users’ personal servers, both virtual and physical, on which CI/CD processes run. These processes will have access to the user server environment.

Restrictions on the amount of computing resources do not apply to self-hosted workers, and they also do not consume the total CI/CD runtime quota.

For more information, see Workers.

To set up a self-hosted worker:

  1. Set up your environment.
  2. Create a worker configuration file.
  3. Start the worker.
  4. Run a job on the worker.

Set up your environment

  1. Create a personal access token (PAT) with the Repository admin role and access to the repositories in which you want to run CI/CD processes on the self-hosted worker.

    Warning

    We do not recommend granting access to all the organization's repositories.

  2. Download the self-hosted-processor executable for your OS and architecture:

    See also Updating a worker's executable.

  3. In macOS/Linux, grant permissions to run the executable:

    chmod +x <path_to_executable>

  4. Add the executable to PATH:

    export PATH=$PATH:<path_to_executable>

    $env:Path += ";<path_to_executable>"

Create a worker configuration file

You can automatically set up a worker configuration file by running the self-hosted-processor init command.

Supported arguments:

Key

Description

Default value

-c,
--capacity

Maximum number of jobs the worker can execute in parallel.

1

-r, --root-dir

Path to the directory where the worker will create temporary working directories for its jobs.

Directory for temporary files, e.g., /tmp/ or /usr/tmp/.

-t,
--token

Personal access token (PAT) for authentication in the service. The worker uses it to fetch jobs, report progress, and send logs and artifacts to SourceCraft.

Not set.

--tags

Tag list. The worker will run jobs whose tag set is inside the worker's tag set.

[ ]

--ssl-no-verify

Option to disable certificate verification. Use it if your device does not have a suitable root certificate (RootCA).

Disabled.
Root certificates installed on the the device are used.

For example, generate a configuration for a worker that can run no more than three jobs with tags 4gb and go-builder at the same time:

self-hosted-processor init \
  --token <personal_access_token> \
  --capacity 3 \
  --tags 4gb,go-builder \
  > config.yaml

This will create the config.yaml worker configuration file:

executor_type: shell
self_hosted:
  max_slots: 3
logger_type: json
endpoint:
  host: ci.sourcecraft.tech
  port: 443
  tls: true
tags:
- 4gb
- go-builder
auth:
  pat: <personal_access_token>

Create a file named config.yaml with the following configuration:

executor_type: shell

self_hosted:
  # Path to the directory where the worker will create temporary
  # working directories for its jobs.
  root_dir: /Users/user1/sourcecraft
  # Maximum number of jobs the worker 
  # can execute in parallel.
  max_slots: 3

logger_type: json

# Address of the service for fetching jobs and sending logs and artifacts 
endpoint:
  host: ci.sourcecraft.tech
  port: 443
  ## To disable certificate verification, set to false
  tls: true

# Tag list. The worker will run jobs whose tag set 
# is inside the worker's tag set
tags:
- 4gb
- go-builder

auth:
  # Personal access token for authentication in the service
  pat: <personal_access_token>

This is the configuration of a worker that can run no more than three jobs with tags 4gb and go-builder at the same time and create temporary working directories at /Users/user1/sourcecraft:

Start the worker

Run this command:

self-hosted-processor run \
  --config-path <path_to_configuration_file>

Where --config-path is the path to the configuration file you created earlier. You can also use -c as the short form of this parameter.

Tip

The worker's logs go to a standard output stream. To write logs into a file or send them to a log collection service, redirect the output or use pipelines, |.

Run a job on the worker

To run a CI/CD process on a custom self-hosted worker, specify runs_on: self-hosted in the workflow (workflow) or task (task) parameters.

If runs_on is not specified in task, the parameter from workflow:runs_on will be used by default.

Here is an example:

workflows:
  my-awesome-workflow:
    runs_on: self-hosted
    
    tasks:
      - name: self-hosted-task
        cubes:
          - name: hello
            script:
              - echo "hello from self-hosted"

      - name: self-hosted-go-builder-task
        runs_on: [self-hosted, go-builder]
        cubes:
          - name: hello
            script:
              - echo "hello from self-hosted go builder"

In this case, tasks from my-awesome-workflow will run on any of your self-hosted workers by default unless their runs_on parameter is redefined. At the same time, self-hosted-go-builder-task will run only on the self-hosted worker assigned the go-builder tag during initialization.

Updating a worker's executable

The Set up your environment section provides links to the relevant versions of the worker's executables.

To check the version of the worker's executable installed in your environment, run this command:

self-hosted-processor version

Result:

<``>SourceCraft<``> Processor
Version: 0.9.1
Build date: 10 June 2025
Target OS: darwin
Target Arch: arm64

To update the worker's executable, download the new file and replace the old one.

See also

Previous
Processing environment variables
Next
Adding a file or directory