Continuous integration and continuous deployment in SourceCraft

Continuous Integration/Continuous Deployment is a set of practices and tools you can use to automatically change, test, and deploy code. The approach allows you to continuously improve software quality and speed up development.

Continuous integration (CI)

CI is there for frequent and regular integration of code changes into the main branch of the repository. Each commit goes through automatic checks, e.g., unit tests and static code analysis. Thus you can make sure your commit is correct and the code is stable. This reduces the risks of integration issues, and the developers get the feedback faster.

Continuous deployment (CD)

The CD approach is built around automated code deployment on product servers after a code quality test. Be confident that your apps are always deployed and have the latest updates and fixes.

In SourceCraft, you can implement your end-to-end CI/CD process as well as create integrations with external systems using webhooks.

CI/CD configuration

SourceCraft offers some tools to manage CI/CD workflows.

The CI/CD configuration is set up for a particular repository and stored in a file named .sourcecraft/ci.yaml.

General format of the .sourcecraft/ci.yaml configuration file:

on:
  pull_request:
    - workflows: [<list_of_workflows>]
      filter:
        source_branches: [<list_of_source_branches>]
        target_branches: [<list_of_target_branches>]
        paths: [<list_of_paths>]

  push:
    - workflows: [<list_of_workflows>]
      filter:
        branches: [<list_of_branches>]
        paths: [<list_of_paths>]
    - workflows: [<list_of_workflows>]
      filter:
        tags: [<list_of_tags>]

  schedule:
    - workflows: [<list_of_workflows>]
      interval: <execution_interval_in_hours_or_minutes>
      description: <random description>
    - workflows: [<list_of_workflows>]
      cron: "<cron_expression>"
      description: <random description>

workflows:
  <workflow_name>:

    tasks:
      - name: <task_name>
        
        cubes:
          - name: <cube_name>
            image: <Docker_image_path>
            script:
              - <executed_script>
...

The configuration may include the following elements:

• Trigger events (on section)
• Workflows (workflows section)
• Jobs (tasks section)
• Cubes (cubes section)
• Service connections (tokens section)
• Environment variables (env section)

You can either use nested structure for configuration elements or mark them up as separate sections.

on:
  pull_request:
    - workflows: my-test-workflow
      filter:
        source_branches: ["**", "!test**"]
        target_branches: "main"

workflows:
  my-test-workflow:
    tasks:
      - my-test-task

tasks:
  - name: my-test-task
    cubes:
      - name: my-test-cube
        uses: hello-cube

cubes:
  - name: hello-cube
    script:
      - echo "Hello, world!"

The configuration file supports secrets. Learn more in Using the value of a secret in CI/CD.

SourceCraft provides secure Yandex Cloud API authentication within CI/CD workflows using service connections.

SourceCraft supports the following third-party CI/CD tools:

• Running GitHub Actions
• Using GitLab pipelines

For more information on working with CI/CD, see Configuring CI/CD in a SourceCraft repository.

All CI/CD workflows are run concurrently on SourceCraft workers.

Trigger events (on)

Under on, you can configure the trigger events that will start the CI/CD workflows in the repository. Such events may include pushing changes to a remote repository branch (push), creating a pull request (pull_request), or a scheduled execution (schedule).

You can configure different workflows for different events. You can also configure triggers for specific branches or paths in the repository.

For more information, see Trigger events (on) in the CI/CD reference.

In addition to automatically starting CI/CD workflows in a repository based on specific trigger events, you can start a workflow manually from the SourceCraft interface.

Workflows

The workflows section lists CI/CD workflows.

A workflow helps you organize tasks related to a certain CI/CD stage into a logical sequence.

For example, a single workflow may run for builds, tests, linting, code coverage verification, etc. All these steps will be different tasks as part of such a workflow. Then, you may have another workflow to generate documentation and deploy the new version to production.

All workflows are run concurrently.

You can make a workflow runnable by all organization members.

For more information, see Workflows in the CI/CD reference.

Tasks

The tasks section defines a list of tasks that are part of the workflow.

Each task contains a series of minimum logical actions, i.e., cubes. The result of a task is when all its cubes are completed.

By default, a task starts with cloning the repository.

All workflow tasks are started concurrently.

In tasks and cubes, you can use environment variables and secrets.

For more information on tasks, see Tasks in the CI/CD reference.

Cubes

The cubes section lists the minimum logical actions, i.e., cubes, to execute within a task.

Such a minimum action may be calling a script, starting a Docker container, or calling a script in a Docker container.

The following cube types are available:

• Native: Runs directly on a worker.

  If a native cube, while running, changes the worker environment, such as installs a package, creates or deletes a file, etc., such environment will still be there for all other cubes running within a single task.

• Docker cube: Runs within a Docker container started on a worker. Technically, it runs a custom script or a container script, if the container has an entry point.

  If a Docker cube, while running, changes the environment, such environment will be available to other cubes within the task only in case the changes were made within the /sourcecraft directory. All other changes will be removed along with the Docker container.

  When working from a container, the directories are mounted as follows:

  • The directory housing files associated with the task in progress is mounted to /sourcecraft.
  • The directory to clone the repository to and that is also the working directory (workdir) by default, is mounted to /sourcecraft/workspace.

  You can get the paths of these directories from the $SOURCECRAFT_ROOT_DIRECTORY and $SOURCECRAFT_WORKSPACE predefined environment variables, respectively.

  To configure a Docker cube, use the image property to specify the Docker image name and, optionally, username, password, entry point, and arguments.

• Devcontainer cube: Runs in the environment defined in the Development Container specification. This cube builds a container from the configuration in the repository and runs a custom script in that container.

  A devcontainer cube is specified in the devcontainer property which contains the path to the directory with the devcontainer.json or Dockerfile specification.

  Sample Development Container specifications

In cubes, you can use environment variables and secrets. To provide environment variables from a certain cube to others as KEY=VALUE pairs, you can use the $SOURCECRAFT_ENV predefined variable.

By default, cubes within a single task are run one by one. To link cubes, use the needs property, where you can specify the list of cubes to execute before the current one. If you skip this property, the cube will depend on the one defined immediately before it.

The artifacts that may be created after the cube is run are saved for further use. They will be available for download from the cube in the CI/CD section of the repository for 14 days.

For more information on cubes, see Cubes in the CI/CD reference.

Environment variables in CI/CD

The SourceCraft CI/CD processes support environment variables. The following variable scopes are available:

• Global: To provide the variables to all cubes of all tasks in all workflows.
• Workflow: To provide the variables to all cubes of all tasks of a specific workflow.
• Task: To provide the variables to all cubes linked to the task.
• Cube: To provide the variables to the specified cube.

You can reuse variables in nested scopes.

Also, you can use predefined environment variables.

Example of a configuration with environment variables

# Defining global variables
# to provide to all cubes of all tasks in all workflows.
env:
  GLOBAL_VAR: global_var
  GLOBAL_SECRET: ${{ secrets.<secret_name> }}

workflows:
  my-workflow:
    # Here you define variables that will be available in all cubes 
    # of all my-workflow tasks
    env:
      WORKFLOW_VAR: workflow-var
    
    tasks:
      - name: my-task
        # Here you define variables that will be available in all cubes 
        # within my-task
        env:
          TASK_ENV_VAR: This variable is available in all cubes of this task.
          # Multi-line variable
          MULTILINE_VAR: |
            multi-var
            multi-var
            this is my multi-var
        
        cubes:
          - name: my-cube-1
            # Here you define variables that will only be available
            # within my-cube-1
            env:
              CUBE_ENV_VAR: This variable is available only in cube my-cube-1.
              # Variable with a value from a secret
              SECRET_VAR: ${{ secrets.<secret_name> }}
              # Reusing global variables, 
              # e.g., GLOBAL_VAR and GLOBAL_SECRET
              LOCAL_VAR: ${{ env.<global_variable_1> }}
              LOCAL_SECRET: ${{ env.<global_variable_2> }}
              # Reusing workflow variables,
              # e.g., WORKFLOW_VAR
              LOCAL_VAR2: ${{ env.<workflow_variable> }}

            script:
              - echo "$TASK_ENV_VAR"
              - echo "$MULTILINE_VAR"
              - echo "$CUBE_ENV_VAR"
              - echo "$SECRET_VAR"
              - echo "$WORKFLOW_VAR"
              - echo "$LOCAL_VAR"
              - echo "$LOCAL_VAR2"
              - echo "$LOCAL_SECRET"

          - name: my-cube-2
            # Here you define variables that will only be available 
            # within my-cube-2
            env:
              CUBE_ENV_VAR: This variable is available only in cube my-cube-2.
            script:
              - echo "$TASK_ENV_VAR"
              - echo "$CUBE_ENV_VAR"
              # Using a predefined variable
              - echo "$SOURCECRAFT_TASK"
              - echo "$WORKFLOW_VAR"
              - echo "$GLOBAL_VAR"

      - name: my-task-2
        cubes:
          - name: my-cube-3
            script:
              - echo "$WORKFLOW_VAR"
              - echo "$GLOBAL_VAR"

For more information about using environment variables, see Processing environment variables in SourceCraft.

Useful links

• Repository templates
• Configuring CI/CD in a SourceCraft repository
• CI/CD reference
• Configuring CI/CD with Yandex Cloud Functions
• Approval rules in SourceCraft
• Setting up approval rules in a SourceCraft repository
• Branch policies in SourceCraft
• Setting up a branch policy in a SourceCraft repository
• Integration with GitHub Actions in SourceCraft
• GitLab pipelines in the SourceCraft CI/CD
• Webhooks in SourceCraft