Continuous integration and continuous deployment in SourceCraft

Continuous Integration/Continuous Deployment is a set of practices and tools you can use to automatically change, test, and deploy code. The approach allows you to continuously improve software quality and speed up development.

Continuous integration (CI)

CI is there for frequent and regular integration of code changes into the main branch of the repository. Each commit goes through automatic checks, e.g., unit tests and static code analysis. Thus you can make sure your commit is correct and the code is stable. This reduces the risks of integration issues, and the developers get the feedback faster.

Continuous deployment (CD)

The CD approach is built around automated code deployment on product servers after a code quality test. Be confident that your apps are always deployed and have the latest updates and fixes.

CI/CD configuration

SourceCraft offers some tools to manage CI/CD workflows.

The CI/CD configuration is set up for a particular repository and stored in a file named .sourcecraft/ci.yaml.

General format of the .sourcecraft/ci.yaml configuration file:

on:
  pull_request:
    - workflows: [<list_of_workflows>]
      filter:
        source_branches: [<list_of_source_branches>]
        target_branches: [<list_of_target_branches>]
        paths: [<list_of_paths>]

  push:
    - workflows: [<list_of_workflows>]
      filter:
        branches: [<list_of_branches>]
        paths: [<list_of_paths>]

workflows:
  <workflow_name>:

    tasks:
      - name: <task_name>
        
        cubes:
          - name: <cube_name>
            image: <Docker_image_path>
            script:
              - <executed_script>
...

You can either use nested structure for configuration elements or mark them up as separate sections.

Example of CI/CD configuration marked up as separate sections
on:
  pull_request:
    - workflows: my-test-workflow
      filter:
        source_branches: ["**", "!test**"]
        target_branches: "main"

workflows:
  my-test-workflow:
    tasks:
      - my-test-task

tasks:
  - name: my-test-task
    cubes:
      - name: my-test-cube
        image: docker.io/library/node
        script:
          - echo Hello, world!

The configuration file supports secrets. For details, see Using the value of a secret in CI/CD.

See the templates repository in SourceCraft for the full specification of the .sourcecraft/ci.yaml file.

Warning

Support for storing configurations for CI/CD, approval rules, and branch policies in a single .src.ci.yaml file at the repository root will soon be discontinued. Use the separate .sourcecraft/ci.yaml, .sourcecraft/review.yaml, and .sourcecraft/branches.yaml files.

For more information about working with CI/CD, see Configuring CI/CD in a SourceCraft repository.

Trigger events (on)

Under on, you can configure the trigger events that will start the CI/CD workflows in the repository.

Such events may include pushing changes to a remote repository branch or creating a pull request.

You can configure different workflows for different events. You can also configure triggers for specific branches or paths in the repository.

For more information, see Trigger events (on) in the CI/CD reference.

Workflows

The workflows section lists CI/CD workflows.

A workflow helps you organize tasks related to a certain CI/CD stage into a logical sequence.

For example, a single workflow may run for builds, tests, linting, code coverage verification, etc. All these steps will be different tasks as part of such a workflow. Then, you may have another workflow to generate documentation and deploy the new version to production.

All workflows are run concurrently.

For more information, see Workflows in the CI/CD reference.

Tasks

The tasks section defines a list of tasks that are part of the workflow.

Each task contains a series of minimum logical actions, i.e., cubes. The result of a task is when all its cubes are completed.

Note

All cubes within a single task are run on the same VM (worker). This way, if a cube changes the worker environment, such as installs a package, creates or deletes a file, etc., such environment will still be there for all other cubes running within a single task. For example, one cube may install the runtime package for Go, another one runs the go build command, and the next one runs go test. To learn more about environment variable inheritance, see Cubes.

If the cubes are run in different tasks, they are guaranteed to run on different workers.

By default, a task starts with cloning the repository.

All workflow tasks are started concurrently.

In tasks and cubes, you can use environment variables and secrets.

For more information on tasks, see Tasks in the CI/CD reference.

Cubes

The cubes section lists the minimum logical actions, i.e., cubes, to execute within a task.

Such a minimum action may be calling a script, starting a Docker container, or calling a script in a Docker container.

The following cube types are available:

  • Native: Runs directly on a worker.

    If a native cube, while running, changes the worker environment, such as installs a package, creates or deletes a file, etc., such environment will still be there for all other cubes running within a single task.

  • Docker cube: Runs within a Docker container started on a worker. Technically, it runs a custom script or a container script, if the container has an entry point.

    If a Docker cube, while running, changes the environment, such environment will be available to other cubes within the task only in case the changes were made within the /sourcecraft directory. All other changes will be removed along with the Docker container.

    When working from a container, the directories are mounted as follows:

    • The directory housing files associated with the task in progress is mounted to /sourcecraft.
    • The directory to clone the repository to and that is also the working directory (workdir) by default, is mounted to /sourcecraft/workspace.

    You can get the paths of these directories from the $SOURCECRAFT_ROOT_DIRECTORY and $SOURCECRAFT_WORKSPACE predefined environment variables, respectively.

    To configure a Docker cube, use the image property to specify the Docker image name and, optionally, username, password, entry point, and arguments.

In cubes, you can use environment variables and secrets. To provide environment variables from a certain cube to others as KEY=VALUE pairs, you can use the $SOURCECRAFT_ENV predefined variable.

By default, cubes within a single task are run one by one. To link cubes, use the needs property, where you can specify the list of cubes to execute before the current one. If you skip this property, the cube will depend on the one defined immediately before it.

The artifacts that may be created after the cube is run are saved for further use. They will be available for download from the cube in the CI/CD section of the repository for 14 days.

For more information on cubes, see Cubes in the CI/CD reference.

Environment variables in CI/CD

The SourceCraft CI/CD processes support environment variables. You can set the variables under the following items in the .sourcecraft/ci.yaml configuration file:

  • Task: To provide the variables to all cubes linked to the task.
  • Cube: To provide the variables to the specified cube.

Also, you can use predefined environment variables.

Warning

Do not store any sensitive data, such as passwords, access keys, or tokens, in environment variables; use secrets instead.

Example of a configuration with environment variables

tasks:
  - name: my-task
    # Defines the variables to go to all the cubes associated with the task.
    env:
      TASK_VAR: test-var-'test'-\"test\"
      MULTILINE_VAR: |
        multi-var
        multi-var
        this is my multi-var

    cubes:
      - name: my-cube
        # Defines the variables to go only to a specific cube.
        env:
          CUBE_VAR: "you can see me here only"
          SECRET_VAR: ${{ secrets.<secret_name> }}
        script:
          - echo "$TASK_VAR"
          - echo "$MULTILINE_VAR"
          - echo "$CUBE_VAR"
          - echo "$SECRET_VAR"

For more information about using environment variables, see Processing environment variables in SourceCraft.

Workers

Workers are virtual machines, physical servers, or serverless containers used to run CI/CD processes.

All cubes within a single task run on the same worker.

The following worker types are supported:

Cloud workers

Cloud workers are cloud-based virtual machines provided by SourceCraft. They are the default resource for the CI/CD processes.

Cloud worker computing resource configuration: 4 vCPUs, 8 GB RAM.

Serverless workers

Serverless workers are cloud-based serverless containers provided by SourceCraft. Unlike cloud workers, serverless workers start almost instantly as there is no need to boot up a VM.

Limitations:

  • Only Yandex Container Registry is supported for Docker cubes.
  • You cannot run docker or docker-compose within a cube.
  • A single job can include no more than nine unique custom Docker images.

To run a CI/CD process on a serverless worker, specify runs_on: serverless in the workflow (workflow) or task (task) parameters.

If runs_on is not specified in task, it will default to the parameter from workflow:runs_on.

Example of running the whole workflow on a serverless worker

workflows:
  workflow-in-serverless-mode:
    runs_on: serverless
  
    tasks:
      - name: task1-in-serverless-mode
        cubes:
          - name: hello
            script:
              - echo "hello from serverless"

      - name: task2-in-serverless-mode
        cubes:
          - name: hello
            script:
              - echo "hello from serverless again"

Example of running an individual task on a serverless worker

workflows:
  workflow1:
  
    tasks:
      - name: task1-in-compute-mode
        cubes:
          - name: hello
            script:
              - echo "hello from compute"

      - name: task2-in-serverless-mode
        runs_on: serverless
        cubes:
          - name: hello
            script:
              - echo "hello from serverless"

Self-hosted workers

Self-hosted workers are users’ personal servers, both virtual and physical, on which CI/CD processes run. These processes will have access to the user server environment.

Restrictions on the amount of computing resources do not apply to self-hosted workers, and they also do not consume the total CI/CD runtime quota.

For more information, see Setting up a self-hosted worker for SourceCraft.

To run a CI/CD process on a custom self-hosted worker, specify runs_on: self-hosted in the workflow (workflow) or task (task) parameters.

If runs_on is not specified in task, the parameter from workflow:runs_on will be used by default.

Example of running the whole workflow on a self-hosted worker

workflows:
  my-awesome-workflow:
    runs_on: self-hosted
    
    tasks:
      - name: self-hosted-task
        cubes:
          - name: hello
            script:
              - echo "hello from self-hosted"

      - name: self-hosted-go-builder-task
        runs_on: [self-hosted, go-builder]
        cubes:
          - name: hello
            script:
              - echo "hello from self-hosted go builder"

In this case, tasks from my-awesome-workflow will run on any of your self-hosted workers by default unless their runs_on parameter is redefined. At the same time, self-hosted-go-builder-task will run only on the self-hosted worker assigned the go-builder tag during initialization.

See also

Previous
Resource relationships
Next
Approval rules