Workflows

The workflows section lists CI/CD workflows.

A workflow helps you organize tasks related to a certain CI/CD stage into a logical sequence.

For example, a single workflow may run for builds, tests, linting, code coverage verification, etc. All these steps will be different tasks as part of such a workflow. Then, you may have another workflow to generate documentation and deploy the new version to production.

All workflows are run concurrently.

You can make a workflow runnable by all organization members.

Supported properties:

  • tasks: List of tasks that are part of the workflow.

  • settings: Settings valid for the entire workflow.

  • env: Environment variables available to all cubes within all tasks of a specific workflow. For more information, see Example of a workflow with secrets and variables, including predefined ones.

    Tip

    You can also set environment variables within the following sections:

    • tasks: Variables available in all cubes within a certain task.
    • cubes: Variables available in a specific cube.

  • runs_on: Tags of the worker the workflow tasks will run on.

  • inputs: Workflow input parameters. They are used for manual execution, scheduled execution, or workflow reuse.

  • uses: Name of the workflow whose parameters you need to reuse.

  • with: Input parameter values provided to a reusable workflow. This parameter is used together with uses.

  • checkout: Settings for auto-cloning the repository prior to the workflow tasks.

settings

The settings section specifies the settings that are valid for the entire workflow:

  • max_cube_duration: Maximum cube duration in seconds (s) or minutes (m). Default value: 5 minutes. Here is an example:

    workflows:
  my-workflow:
    settings:
      max_cube_duration: 20s

  • shared: Permission to run the workflow for all organization members, including those holding no roles in the workflow repository.

    For more information, see Configuring a public workflow in SourceCraft.

Example of configuring and running a public workflow

Configuration:

workflows:
  professor-test:
    inputs:
      STUDENTREPO:
        type: string
        required: true
      TASK:
        type: string
        required: true
    settings:
      shared: true
    tasks:
      - name: professor-task
        cubes:
          - name: professor-cube
            script:
              - |
              	mkdir -p artifacts
                echo "Repo: ${{ inputs.STUDENTREPO }}" > artifacts/professor-output
                echo "Task: ${{ inputs.TASK }}" >> artifacts/professor-output
            artifacts:
              paths:
                - artifacts/professor-output

on:
  push: professor-test

Running a workflow by another member of the organization:

workflows:
  check-solution:
    tasks:
      - name: main
        cubes:
          - name: run-shared-workflow
            image: cr.yandex/sourcecraft/cubes/shared-workflows:latest
            env:
              ORG_SLUG: professor-org
              REPO_SLUG: professor-repo
              WORKFLOW_NAME: professor-test
              WORKFLOW_VALUES: '[{"name": "STUDENTREPO", "value": "student"}, {"name": "TASK", "value": "task-1"}]'
              TASK_NAME: professor-task
              CUBE_NAME: professor-cube
              ARTIFACT_LOCAL_PATH: artifacts/professor-output
            artifacts:
              paths:
                - artifacts/output

on:
  push: check-solution

  1. Create a personal access token (PAT).

  2. Run the public workflow by providing "shared": true in the request body:

    export PAT=<personal_access_token>

cat > body.json << 'EOF'
{
  "workflows": [
    {
      "name": "professor-test",
      "values": [
        {
          "name": "STUDENTREPO",
          "value": "student"
        },
        {
          "name": "TASK",
          "value": "task-1"
        }
      ]
    }
  ],
  "shared": true
}
EOF

curl \
  --request POST \
  --header "Authorization: Bearer $PAT" \
  --data '@body.json' \
  --url "https://api.sourcecraft.tech/<organization_slug>/<repository_slug>/cicd/runs"

    Warning

    You can only run a public workflow in the repository's main branch and only with the CI/CD configuration from the main branch. Providing head and config_revision in the request body will produce an execution error.

    Save the execution slug value from the response.

  3. Get the status of a running workflow:

    curl \
  --request GET \
  --header "Authorization: Bearer $PAT" \
  --url "https://api.sourcecraft.tech/<organization_slug>/<repository_slug>/cicd/runs/<execution_slug>"

    Note

    The status and artifacts of a public workflow can only be accessed with the same personal token (PAT) used to run the workflow.

  4. Get artifacts of a running workflow:

    curl \
  --request GET \
  --header "Authorization: Bearer $PAT" \
  --url "https://api.sourcecraft.tech/<organization_slug>/<repository_slug>/cicd/artifacts/<execution_slug>/professor-test/professor-task/professor-cube"

For more information, see Working with the SourceCraft REST API.

runs_on

The runs_on field gives a list of worker tags on which the workflow tasks will run. Supported tag types:

  • Runtime tag: Worker type. The possible values are:

    Warning

    The runs_on field cannot contain more than one runtime tag. If no runtime tag is specified, compute is used by default.

  • Self-hosted worker tags.

    Note

    A tag's string values ​can contain a limited range of characters: Latin letters (a-zA-Z), -, ., and _.

    For a self-hosted worker, parameters automatically include the self-hosted tag. You need not additionally add it to the tags field in config.yaml.

Example of a workflow with two tasks in different formats

One workflow task is within the workflows section, the other is outside it. my-task is an example of using cube dependencies; for more info, see Cubes.

tasks:
  - name: another-task
    cubes:
      - name: D
        script:
          - echo It's another task.

workflows:
  my-workflow:
    tasks:
      - name: my-task
        cubes:
          - name: A
            script:
              - touch test.txt
          - name: B
            needs: ['-']
            script:
              - rm -f test.txt
          - name: C
            needs: ['A', 'B']
            script:
              - ls
  
      - another-task
...
Example of two different workflows run depending on the event type
on:
  pull_request:
    - workflows: workflow-for-pr
      filter:
        source_branches: ["**", "!test**"]
        target_branches: "main"

  push:
    - workflows: workflow-for-push
      filter:
        branches: ["main"]

workflows:
  workflow-for-pr:
    tasks:
      - name: sample-task-1
        cubes:
          - name: sample-cube1
            image: docker.io/library/node
            script:
              - echo Hello, world!

  workflow-for-push:
    tasks:
      - name: sample-task-2
        cubes:
          - name: sample-cube2
            script:
              - echo Test, and deploy your project.
Example of a workflow with secrets and variables, including predefined ones
# Defining global variables
# to provide to all cubes of all tasks in all workflows.
env:
  GLOBAL_VAR: global_var
  GLOBAL_SECRET: ${{ secrets.<secret_name> }}

workflows:
  my-workflow:
    # Here you define variables that will be available in all cubes 
    # of all my-workflow tasks
    env:
      WORKFLOW_VAR: workflow-var
    
    tasks:
      - name: my-task
        # Here you define variables that will be available in all cubes 
        # within my-task
        env:
          TASK_ENV_VAR: This variable is available in all cubes of this task.
          # Multi-line variable
          MULTILINE_VAR: |
            multi-var
            multi-var
            this is my multi-var
        
        cubes:
          - name: my-cube-1
            # Here you define variables that will only be available
            # within my-cube-1
            env:
              CUBE_ENV_VAR: This variable is available only in cube my-cube-1.
              # Variable with a value from a secret
              SECRET_VAR: ${{ secrets.<secret_name> }}
              # Reusing global variables, 
              # e.g., GLOBAL_VAR and GLOBAL_SECRET
              LOCAL_VAR: ${{ env.<global_variable_1> }}
              LOCAL_SECRET: ${{ env.<global_variable_2> }}
              # Reusing workflow variables,
              # e.g., WORKFLOW_VAR
              LOCAL_VAR2: ${{ env.<workflow_variable> }}

            script:
              - echo "$TASK_ENV_VAR"
              - echo "$MULTILINE_VAR"
              - echo "$CUBE_ENV_VAR"
              - echo "$SECRET_VAR"
              - echo "$WORKFLOW_VAR"
              - echo "$LOCAL_VAR"
              - echo "$LOCAL_VAR2"
              - echo "$LOCAL_SECRET"

          - name: my-cube-2
            # Here you define variables that will only be available 
            # within my-cube-2
            env:
              CUBE_ENV_VAR: This variable is available only in cube my-cube-2.
            script:
              - echo "$TASK_ENV_VAR"
              - echo "$CUBE_ENV_VAR"
              # Using a predefined variable
              - echo "$SOURCECRAFT_TASK"
              - echo "$WORKFLOW_VAR"
              - echo "$GLOBAL_VAR"

      - name: my-task-2
        cubes:
          - name: my-cube-3
            script:
              - echo "$WORKFLOW_VAR"
              - echo "$GLOBAL_VAR"

inputs

The inputs section declares the workflow input parameters. You can set their values during a manual execution, scheduled execution, or provide them via the with section when reusing a workflow.

In general, this section looks like this:

inputs:
  <parameter_name>:
    type: <parameter_type>
    required: <whether_parameter_is_required>
    description: <parameter_description>
    default: <default_value>
    options: <possible_options_for_choice_type>

Where:

  • type: Parameter type. The possible values are:

    • string: String.
    • bool: Logical value, true or false.
    • choice: Select from the preset values.

    Note

    If the configuration contains the options parameter, you do not need to set the type: choice parameter.

    If the type is not specified and there is no options parameter, the default parameter type is string.

  • required: Required parameter. The possible values are true or false.

  • description: Any description of the parameter that will be displayed in the SourceCraft interface when manually running the workflow.

  • default: Default parameter value. You can also specify it if the parameter is required.

  • options: Possible values ​​for the choice parameter type.

Tip

Navigate to CI/CD → CI/CD in your repository or an execution page to view parameter values for each manually executed workflow.

This enables you to distinguish between workflows executed with different parameters.

Example of how it looks in the interface

image

Examples of using inputs:

checkout

The checkout section contains auto-cloning settings for the repository you are running the workflow in. If you set this section both at workflow and task level, the task uses its own settings.

Supported properties:

  • enabled: Auto-cloning the repository prior to the task. The default value is true. If false, the repository will not be cloned before starting the task. In which case get the repository contents in one of the task cubes as needed.

    Note

    You can get the parameter value in the SOURCECRAFT_CHECKOUT_ENABLED predefined environment variable.

  • fetch_depth: Number of most recent history records to load when cloning the repository. It is not set by default; therefore, the repository is cloned down to its full history depth.

    Tip

    Use this parameter to accelerate uploading data to the task.

  • remove_credentials: Deletion of the authorization key from .git/config after cloning the repository. The default value is false. Set to true to prohibit further operations with the deleted repository from the task cubes under CI/CD authentication.

  • retry: Repository cloning auto-retry settings. This parameter is set in one of the following formats:

    Unlike the cube’s retry parameter, this setting affects only the repository cloning retry. Auto-retry of repository cloning is disabled by default.

Note

You can also set the checkout settings at the level of an individual task. In this case, the task settings have priority over the workflow settings.

Example of configuring repository cloning at the workflow level
workflows:
  my-workflow:
    checkout:
      retry: 2
      enabled: true
      remove_credentials: false
      fetch_depth: 10

Reusing workflow parameters

If multiple workflows perform the same sequence of tasks, define the sequence once and reuse it with different input parameters. Follow these steps:

  1. In .sourcecraft/ci.yaml, define the workflow you want to reuse under workflows. Declare its input parameters in the inputs section and use their values in the configuration.
  2. In the caller workflow, specify the name of the reusable workflow in the uses parameter.
  3. Specify the values of its input parameters in the with section. The keys in this section must match the parameter names in the reusable workflow's inputs, and the values, their types and constraints.

Note

The caller workflow can have its own inputs section. The names and settings of input parameters pertaining to the caller and reusable workflows may differ. Map the parameters in the with section.

Example of reusing workflow parameters

In the example below, release-workflow contains the common sequence for building, testing, and deploying the application. Workflows for individual services reuse this sequence and provide the service name using the RELEASE_APP parameter.

We assume that the repository contains the build.sh, check.sh, and deploy.sh scripts that accept the service name as an argument:

workflows:
  release-workflow:
    inputs:
      RELEASE_APP:
        type: choice
        required: true
        options:
          - service1
          - service2
          - service3
    env:
      RELEASE_APP: ${{ inputs.RELEASE_APP }}
    tasks:
      - name: build-app
        cubes:
          - name: build
            script:
              - ./build.sh "$RELEASE_APP"

      - name: run-compliance-checks
        needs: [build-app]
        cubes:
          - name: check
            script:
              - ./check.sh "$RELEASE_APP"

      - name: deploy-app
        needs: [build-app, run-compliance-checks]
        cubes:
          - name: deploy
            script:
              - ./deploy.sh "$RELEASE_APP"

  service1-release-workflow:
    uses: release-workflow
    with:
      RELEASE_APP: service1

  service2-release-workflow:
    uses: release-workflow
    with:
      RELEASE_APP: service2

  service3-release-workflow:
    uses: release-workflow
    with:
      RELEASE_APP: service3

  manual-release-workflow:
    inputs:
      TARGET_APP:
        type: choice
        required: true
        default: service3
        options:
          - service1
          - service2
          - service3
    uses: release-workflow
    with:
      RELEASE_APP: ${{ inputs.TARGET_APP }}

For manual-release-workflow, the service name is set in the TARGET_APP parameter. The ${{ inputs.TARGET_APP }} expression in the with section provides its value to the RELEASE_APP parameter of the reusable workflow. This allows the caller and reusable workflows to use different input parameter names.

Useful links

Previous
Trigger events (on)
Next
Tasks