--- metadata: - name: generator content: Diplodoc Platform v5.55.2 alternate: - https://sourcecraft.dev/portal/docs/en/sourcecraft/concepts/ci-cd.md - https://sourcecraft.dev/portal/docs/ru/sourcecraft/concepts/ci-cd.md - href: en/sourcecraft/concepts/ci-cd.md type: text/markdown title: Markdown version - href: ../../llms.txt type: text/markdown title: llms.txt --- > **Documentation Index:** Fetch the complete configuration index at https://sourcecraft.dev/portal/docs/en/llms.txt # Continuous integration and continuous deployment in SourceCraft [Continuous Integration/Continuous Deployment](https://en.wikipedia.org/wiki/CI/CD) is a set of practices and tools you can use to automatically change, test, and deploy code. The approach allows you to continuously improve software quality and speed up development. ### Continuous integration (CI) {#ci} CI is there for frequent and regular integration of code changes into the main branch of the repository. Each commit goes through automatic checks, e.g., unit tests and static code analysis. Thus you can make sure your commit is correct and the code is stable. This reduces the risks of integration issues, and the developers get the feedback faster. ### Continuous deployment (CD) {#cd} The CD approach is built around automated code deployment on product servers after a code quality test. Be confident that your apps are always deployed and have the latest updates and fixes. In SourceCraft, you can implement your end-to-end CI/CD process as well as create integrations with external systems using [webhooks](https://sourcecraft.dev/portal/docs/en/sourcecraft/concepts/webhooks.md). ## CI/CD configuration {#ci-cd-configuration} SourceCraft offers some tools to manage CI/CD workflows. The CI/CD configuration is set up for a particular repository and stored in a file named `.sourcecraft/ci.yaml`. General format of the `.sourcecraft/ci.yaml` configuration file: ```yaml on: pull_request: - workflows: [] filter: source_branches: [] target_branches: [] paths: [] push: - workflows: [] filter: branches: [] paths: [] - workflows: [] filter: tags: [] schedule: - workflows: [] interval: description: - workflows: [] cron: "" description: workflows: : tasks: - name: cubes: - name: image: script: - ... ``` The configuration may include the following elements: * [Trigger events](https://sourcecraft.dev/portal/docs/en/sourcecraft/ci-cd-ref/on.md) (`on` section) * [Workflows](https://sourcecraft.dev/portal/docs/en/sourcecraft/ci-cd-ref/workflows.md) (`workflows` section) * [Jobs](https://sourcecraft.dev/portal/docs/en/sourcecraft/ci-cd-ref/tasks.md) (`tasks` section) * [Cubes](https://sourcecraft.dev/portal/docs/en/sourcecraft/ci-cd-ref/cubes.md) (`cubes` section) * [Service connections](https://sourcecraft.dev/portal/docs/en/sourcecraft/ci-cd-ref/tokens.md) (`tokens` section) * [Environment variables](https://sourcecraft.dev/portal/docs/en/sourcecraft/ci-cd-ref/env.md) (`env` section) {% note tip %} The `.sourcecraft/ci.yaml` file supports file structure output as well as search for declarations and element usages. For more information, see [Navigating and searching by code in SourceCraft](https://sourcecraft.dev/portal/docs/en/sourcecraft/operations/code-navigation.md). {% endnote %} You can either use nested structure for configuration elements or mark them up as separate sections. {% cut "Example of CI/CD configuration marked up as separate sections" %} ```yaml on: pull_request: - workflows: my-test-workflow filter: source_branches: ["**", "!test**"] target_branches: "main" workflows: my-test-workflow: tasks: - my-test-task tasks: - name: my-test-task cubes: - name: my-test-cube uses: hello-cube cubes: - name: hello-cube script: - echo "Hello, world!" ``` {% endcut %} The configuration file supports [secrets](https://sourcecraft.dev/portal/docs/en/sourcecraft/concepts/index.md#secrets). Learn more in [Using the value of a secret in CI/CD](https://sourcecraft.dev/portal/docs/en/sourcecraft/operations/secrets.md#use-secret-in-ci-cd). SourceCraft provides secure Yandex Cloud [API](https://yandex.cloud/en/docs/api-design-guide/concepts/general) authentication within [CI/CD](https://sourcecraft.dev/portal/docs/en/sourcecraft/concepts/ci-cd.md) workflows using [service connections](https://sourcecraft.dev/portal/docs/en/sourcecraft/concepts/service-connections.md). SourceCraft supports the following third-party CI/CD tools: * [Running GitHub Actions](https://sourcecraft.dev/portal/docs/en/sourcecraft/concepts/gh-actions.md) * [Using GitLab pipelines](https://sourcecraft.dev/portal/docs/en/sourcecraft/concepts/gl-pipelines.md) For more information on working with CI/CD, see [Configuring CI/CD in a SourceCraft repository](https://sourcecraft.dev/portal/docs/en/sourcecraft/operations/ci-cd.md). {% note tip %} You can set up and view repository configurations in the SourceCraft interface under ![image](../../_assets/console-icons/gear.svg) **Repository settings** in the ![image](../../_assets/console-icons/nut-hex.svg) **Configurations** section. Learn more in [this article](https://sourcecraft.dev/portal/docs/en/sourcecraft/operations/configuration-edit.md). {% endnote %} All CI/CD workflows are run concurrently on [SourceCraft workers](https://sourcecraft.dev/portal/docs/en/sourcecraft/concepts/workers.md). ### Trigger events (on) {#on} Under `on`, you can configure the _trigger events_ that will start the CI/CD [workflows](https://sourcecraft.dev/portal/docs/en/sourcecraft/concepts/ci-cd.md#workflows) in the repository. Such events may include pushing changes to a remote repository branch (`push`), creating a pull request (`pull_request`), or a scheduled execution (`schedule`). {% note warning %} For the trigger event to fire, the `.sourcecraft/ci.yaml` file must be in the repository's main branch, e.g., `main` or `master`. You can set the main branch in the [repository settings](https://sourcecraft.dev/portal/docs/en/sourcecraft/operations/repo-edit.md). {% endnote %} You can configure different workflows for different events. You can also configure triggers for specific branches or paths in the repository. For more information, see [Trigger events (on)](https://sourcecraft.dev/portal/docs/en/sourcecraft/ci-cd-ref/on.md) in the CI/CD reference. In addition to automatically starting CI/CD workflows in a repository based on specific trigger events, you can [start](https://sourcecraft.dev/portal/docs/en/sourcecraft/operations/run-workflow-manually.md) a workflow manually from the SourceCraft interface. {% note tip %} Navigate to ![image](../../_assets/console-icons/terminal-line.svg) **CI/CD** **→** ![image](../../_assets/console-icons/arrows-3-rotate-right.svg) **CI/CD** in your repository or an execution page to view parameter values for each manually executed [workflow](https://sourcecraft.dev/portal/docs/en/sourcecraft/concepts/ci-cd.md#workflows). This enables you to distinguish between workflows executed with different parameters. {% cut "Example of how it looks in the interface" %} ![image](../../_assets/sourcecraft/manual-workflow-configs.png) {% endcut %} {% endnote %} ### Workflows {#workflows} The `workflows` section lists CI/CD _workflows_. A workflow helps you organize [tasks](https://sourcecraft.dev/portal/docs/en/sourcecraft/concepts/ci-cd.md#tasks) related to a certain CI/CD stage into a logical sequence. For example, a single workflow may run for builds, tests, linting, code coverage verification, etc. All these steps will be different tasks as part of such a workflow. Then, you may have another workflow to generate documentation and deploy the new version to production. All workflows are run concurrently. You can make a workflow [runnable by all organization members](https://sourcecraft.dev/portal/docs/en/sourcecraft/concepts/public-workflows.md). For more information, see [Workflows](https://sourcecraft.dev/portal/docs/en/sourcecraft/ci-cd-ref/workflows.md) in the CI/CD reference. ### Tasks {#tasks} The `tasks` section defines a list of tasks that are part of the [workflow](https://sourcecraft.dev/portal/docs/en/sourcecraft/concepts/ci-cd.md#workflows). Each task contains a series of minimum logical actions, i.e., [cubes](https://sourcecraft.dev/portal/docs/en/sourcecraft/concepts/ci-cd.md#cubes). The result of a task is when all its cubes are completed. {% note info %} All cubes within a single task are run on the same VM (worker). This way, if a cube changes the worker environment, such as installs a package, creates or deletes a file, etc., such environment will still be there for all other cubes running within a single task. For example, one cube may install the `runtime` package for Go, another one runs the `go build` command, and the next one runs `go test`. Learn more about environment inheritance in [Cubes](https://sourcecraft.dev/portal/docs/en/sourcecraft/ci-cd-ref/cubes.md). If the cubes are run in different tasks, they are guaranteed to run on different workers. {% endnote %} By default, a task starts with cloning the repository. All workflow tasks are started concurrently. In tasks and cubes, you can use [environment variables](https://sourcecraft.dev/portal/docs/en/sourcecraft/concepts/ci-cd.md#variables) and [secrets](https://sourcecraft.dev/portal/docs/en/sourcecraft/concepts/index.md#secrets). For more information on tasks, see [Tasks](https://sourcecraft.dev/portal/docs/en/sourcecraft/ci-cd-ref/tasks.md) in the CI/CD reference. ### Cubes {#cubes} The `cubes` section lists the minimum logical actions, i.e., _cubes_, to execute within a [task](https://sourcecraft.dev/portal/docs/en/sourcecraft/concepts/ci-cd.md#tasks). Such a minimum action may be calling a script, starting a Docker container, or calling a script in a Docker container. The following cube types are available: * _Native_: Runs directly on a [worker](https://sourcecraft.dev/portal/docs/en/sourcecraft/concepts/workers.md). If a native cube, while running, changes the worker environment, such as installs a package, creates or deletes a file, etc., such environment will still be there for all other cubes running within a single task. * _Docker cube_: Runs within a Docker container started on a worker. Technically, it runs a custom script or a container script, if the container has an entry point. If a Docker cube, while running, changes the environment, such environment will be available to other cubes within the task only in case the changes were made within the `/sourcecraft` directory. All other changes will be removed along with the Docker container. When working from a container, the directories are mounted as follows: * The directory housing files associated with the task in progress is mounted to `/sourcecraft`. * The directory to clone the repository to and that is also the working directory (`workdir`) by default, is mounted to `/sourcecraft/workspace`. You can get the paths of these directories from the `$SOURCECRAFT_ROOT_DIRECTORY` and `$SOURCECRAFT_WORKSPACE` [predefined environment variables](https://sourcecraft.dev/portal/docs/en/sourcecraft/ci-cd-ref/predefined-variables.md), respectively. To configure a Docker cube, use the [image](https://sourcecraft.dev/portal/docs/en/sourcecraft/ci-cd-ref/cubes.md#image) property to specify the Docker image name and, optionally, username, password, entry point, and arguments. * _Devcontainer cube_: Runs in the environment defined in the [Development Container specification](https://containers.dev/implementors/spec/). This cube builds a container from the configuration in the repository and runs a custom script in that container. A `devcontainer` cube is specified in the [devcontainer](https://sourcecraft.dev/portal/docs/en/sourcecraft/ci-cd-ref/cubes.md#devcontainer) property which contains the path to the directory with the `devcontainer.json` or `Dockerfile` specification. [Sample Development Container specifications](https://sourcecraft.dev/portal/docs/en/sourcecraft/concepts/spaces-env-config.md#examples) In cubes, you can use [environment variables](https://sourcecraft.dev/portal/docs/en/sourcecraft/concepts/ci-cd.md#variables) and [secrets](https://sourcecraft.dev/portal/docs/en/sourcecraft/concepts/index.md#secrets). To provide environment variables from a certain cube to others as `KEY=VALUE` pairs, you can use the `$SOURCECRAFT_ENV` [predefined variable](https://sourcecraft.dev/portal/docs/en/sourcecraft/ci-cd-ref/predefined-variables.md). By default, cubes within a single task are run one by one. To link cubes, use the `needs` property, where you can specify the list of cubes to execute before the current one. If you skip this property, the cube will depend on the one defined immediately before it. The artifacts that may be created after the cube is run are saved for further use. They will be available for download from the cube in the ![image](../../_assets/console-icons/arrows-3-rotate-right.svg) **CI/CD** section of the repository for 14 days. For more information on cubes, see [Cubes](https://sourcecraft.dev/portal/docs/en/sourcecraft/ci-cd-ref/cubes.md) in the CI/CD reference. ## Environment variables in CI/CD {#variables} The SourceCraft CI/CD processes support environment variables. The following variable scopes are available: * Global: To provide the variables to all cubes of all tasks in all workflows. * [Workflow](https://sourcecraft.dev/portal/docs/en/sourcecraft/concepts/ci-cd.md#workflows): To provide the variables to all cubes of all tasks of a specific workflow. * [Task](https://sourcecraft.dev/portal/docs/en/sourcecraft/concepts/ci-cd.md#tasks): To provide the variables to all cubes linked to the task. * [Cube](https://sourcecraft.dev/portal/docs/en/sourcecraft/concepts/ci-cd.md#cubes): To provide the variables to the specified cube. You can reuse variables in nested scopes. Also, you can use [predefined environment variables](https://sourcecraft.dev/portal/docs/en/sourcecraft/ci-cd-ref/predefined-variables.md). {% note warning %} Do not store any sensitive data, such as passwords, access keys, or tokens, in environment variables; use [secrets](https://sourcecraft.dev/portal/docs/en/sourcecraft/concepts/index.md#secrets) instead. {% endnote %} ### Example of a configuration with environment variables {#variables-example} ```yaml # Defining global variables # to provide to all cubes of all tasks in all workflows. env: GLOBAL_VAR: global_var GLOBAL_SECRET: ${{ secrets. }} workflows: my-workflow: # Here you define variables that will be available in all cubes # of all my-workflow tasks env: WORKFLOW_VAR: workflow-var tasks: - name: my-task # Here you define variables that will be available in all cubes # within my-task env: TASK_ENV_VAR: This variable is available in all cubes of this task. # Multi-line variable MULTILINE_VAR: | multi-var multi-var this is my multi-var cubes: - name: my-cube-1 # Here you define variables that will only be available # within my-cube-1 env: CUBE_ENV_VAR: This variable is available only in cube my-cube-1. # Variable with a value from a secret SECRET_VAR: ${{ secrets. }} # Reusing global variables, # e.g., GLOBAL_VAR and GLOBAL_SECRET LOCAL_VAR: ${{ env. }} LOCAL_SECRET: ${{ env. }} # Reusing workflow variables, # e.g., WORKFLOW_VAR LOCAL_VAR2: ${{ env. }} script: - echo "$TASK_ENV_VAR" - echo "$MULTILINE_VAR" - echo "$CUBE_ENV_VAR" - echo "$SECRET_VAR" - echo "$WORKFLOW_VAR" - echo "$LOCAL_VAR" - echo "$LOCAL_VAR2" - echo "$LOCAL_SECRET" - name: my-cube-2 # Here you define variables that will only be available # within my-cube-2 env: CUBE_ENV_VAR: This variable is available only in cube my-cube-2. script: - echo "$TASK_ENV_VAR" - echo "$CUBE_ENV_VAR" # Using a predefined variable - echo "$SOURCECRAFT_TASK" - echo "$WORKFLOW_VAR" - echo "$GLOBAL_VAR" - name: my-task-2 cubes: - name: my-cube-3 script: - echo "$WORKFLOW_VAR" - echo "$GLOBAL_VAR" ``` For more information about using environment variables, see [Processing environment variables in SourceCraft](https://sourcecraft.dev/portal/docs/en/sourcecraft/operations/variables.md). #### Useful links {#see-also} * [Repository templates](https://sourcecraft.dev/sourcecraft/repos) * [Configuring CI/CD in a SourceCraft repository](https://sourcecraft.dev/portal/docs/en/sourcecraft/operations/ci-cd.md) * [CI/CD reference](https://sourcecraft.dev/portal/docs/en/sourcecraft/ci-cd-ref/index.md) * [Configuring CI/CD with Yandex Cloud Functions](https://sourcecraft.dev/portal/docs/en/sourcecraft/tutorials/ci-cd-sourcecraft-functions.md) * [Approval rules in SourceCraft](https://sourcecraft.dev/portal/docs/en/sourcecraft/concepts/code-review.md) * [Setting up approval rules in a SourceCraft repository](https://sourcecraft.dev/portal/docs/en/sourcecraft/operations/code-review.md) * [Branch policies in SourceCraft](https://sourcecraft.dev/portal/docs/en/sourcecraft/concepts/branch-policies.md) * [Setting up a branch policy in a SourceCraft repository](https://sourcecraft.dev/portal/docs/en/sourcecraft/operations/branch-policies.md) * [Integration with GitHub Actions in SourceCraft](https://sourcecraft.dev/portal/docs/en/sourcecraft/concepts/gh-actions.md) * [GitLab pipelines in the SourceCraft CI/CD](https://sourcecraft.dev/portal/docs/en/sourcecraft/concepts/gl-pipelines.md) * [Webhooks in SourceCraft](https://sourcecraft.dev/portal/docs/en/sourcecraft/concepts/webhooks.md)