Providing data between CI/CD tasks in SourceCraft

In SourceCraft CI/CD, you can provide data between tasks. Follow these steps:

1. In the source task, in the cube script, write the output data in key=value format to the predefined environment variable named SOURCECRAFT_OUTPUT. Here is an example:

   workflows:
     tasks-outputs-workflow:
       tasks:
         - name: producer
           cubes:
             - name: build
               script:
                 - |
                   echo "BUILD_RESULT=hello-world" >> $SOURCECRAFT_OUTPUT
                   echo "BUILD_VERSION=1.0.0" >> $SOURCECRAFT_OUTPUT
           ...
   

2. In the same task, in the outputs section, declare the list of variables you want to provide in this format: <variable>: ${{ cubes.<cube_name>.outputs.<key_from_cube> }}. Here is an example:

   workflows:
     tasks-outputs-workflow:
       tasks:
         - name: producer
           ...
           outputs:
             PRODUCER_RESULT: ${{ cubes.build.outputs.BUILD_RESULT }}
             PRODUCER_VERSION: ${{ cubes.build.outputs.BUILD_VERSION }}
   

3. In the target task, add a dependency on the source task to the needs list in this format: needs: [<task_name>]. Here is an example:

   workflows:
     tasks-outputs-workflow:
       tasks:
         - name: producer
           ...
         - name: consumer
           needs: [producer]
           ...
   

   Warning

   You can use the needs parameter only within the workflows:tasks section. You cannot use it in the separate tasks section.

   In the needs field, you can only reference tasks of the workflow within which the current task is executed. Circular dependencies are not allowed.

4. In the cube of the dependent task, retrieve the data using this expression: ${{ tasks.<task_name>.outputs.<key_from_task> }}. You can use this expression in scripts, environment variables, and the outputs section. Here is an example:

   workflows:
     tasks-outputs-workflow:
       tasks:
           ...
         - name: consumer
           ...
           cubes:
             - name: consume
               env:
                 VERSION: ${{ tasks.producer.outputs.PRODUCER_VERSION }}
               script:
                 - echo "${{ tasks.producer.outputs.PRODUCER_RESULT }}"
                 - echo "$VERSION"
   

   Warning

   In a dependent task, you can retrieve output values only from tasks explicitly listed in needs.

5. To forward data further down the dependency chain, declare a transit variable in the outputs section of an intermediate task. Here is an example:

   workflows:
     tasks-outputs-workflow:
       tasks:
         - name: producer
           outputs:
             PRODUCER_RESULT: ${{ cubes.build.outputs.BUILD_RESULT }}
           ...
         - name: consumer
           needs: [producer]
           outputs:
             FORWARDED_RESULT: ${{ tasks.producer.outputs.PRODUCER_RESULT }}
           ...
         - name: next-consumer
           needs: [consumer]
           ...
   

Example of a workflow with provision of data between tasks

In the example below:

• Data in the PT_RESULT variable from producer-task is provided to and used in consumer-producer-task and consumer-task-2.
• Data in the CPT_RESULT1 and CPT_RESULT2 variables from consumer-producer-task is provided to and used in consumer-task-1 and consumer-task-2.

workflows:
  tasks-outputs-workflow:
    checkout:
      enabled: false
    tasks:
      - name: producer-task
        # Section defining which data can be used by other tasks
        outputs:
          # Variable to write the task result to
          # The variable value must point to the key provided in the cube to the SOURCECRAFT_OUTPUT variable
          # The variable name in task `outputs` may not coincide with the key provided in the cube
          PT_RESULT: ${{ cubes.produce.outputs.PRODUCER_TASK_RESULT }}
        cubes:
          - name: produce
            script:
              # Providing the cube's output data to the SOURCECRAFT_OUTPUT environment variable in `key=value` format
              # Values can contain any characters, including special ones
              - echo "PRODUCER_TASK_RESULT=Result of producer-task — '🚀'" >> $SOURCECRAFT_OUTPUT

      - name: consumer-producer-task
        # Specifying a dependency on `producer-task` for access to its results
        needs: [producer-task]
        # `outputs` can be defined in any task, including a dependent one
        outputs:
          CPT_RESULT1: ${{ cubes.consume-produce-1.outputs.CPT_RESULT1 }}
          CPT_RESULT2: ${{ cubes.consume-produce-2.outputs.CPT_RESULT2 }}
        cubes:
          - name: consume-produce-1
            env:
              UPSTREAM_RESULT: ${{ tasks.producer-task.outputs.PT_RESULT }}
            script:
              - echo "$UPSTREAM_RESULT"
              - echo "CPT_RESULT1=$UPSTREAM_RESULT and the first result of consumer-producer-task — '⭐'" >> $SOURCECRAFT_OUTPUT
          - name: consume-produce-2
            script:
              - LOCAL_VAR="${{ tasks.producer-task.outputs.PT_RESULT }}"
              - echo "${LOCAL_VAR}"
              - echo "CPT_RESULT2=Second result of consumer-producer-task — '🔥'" >> $SOURCECRAFT_OUTPUT
          - name: consume
            script:
              - echo "${{ tasks.producer-task.outputs.PT_RESULT }}"

      - name: consumer-task-1
        needs: [consumer-producer-task]
        cubes:
          - name: consume
            script:
              # Only `consumer-producer-task` is specified in `needs`; therefore, only its data is available
              # Data from `producer-task` is not available
              - echo "${{ tasks.consumer-producer-task.outputs.CPT_RESULT1 }}"
              - echo "${{ tasks.consumer-producer-task.outputs.CPT_RESULT2 }}"

      - name: consumer-task-2
        needs: [producer-task, consumer-producer-task]
        cubes:
          - name: consume
            script:
              # `producer-task` and `consumer-producer-task` are specified in `needs`;
              # therefore, data from both tasks is available
              - echo "${{ tasks.consumer-producer-task.outputs.CPT_RESULT1 }}"
              - echo "${{ tasks.consumer-producer-task.outputs.CPT_RESULT2 }}"
              - echo "${{ tasks.producer-task.outputs.PT_RESULT }}"

Limitations

• The size of a single KEY=VALUE entry must not exceed 64 KB (65,536 bytes).
• The total size of the names and values of all output data of a single task must not exceed 1 MB. If the limit is exceeded, the task terminates with an error, even if all cubes are completed successfully. To provide larger data, use artifacts.
• If the ${{ cubes.<cube_name>.outputs.<key_from_cube> }} or ${{ tasks.<task_name>.outputs.<key_from_task> }} expression references a non-existent value, the task terminates with an error.

Useful links

• Tasks
• Providing environment variables from one cube to another in the SourceCraft CI/CD
• Predefined environment variables
• Continuous integration and continuous deployment in SourceCraft