Setting up the execution of SourceCraft CI/CD jobs in a Kubernetes cluster

The self-hosted worker named self-hosted-processor supports a mode where CI/CD jobs are executed in a Kubernetes cluster. Each job runs in a separate pod. The worker creates pods in advance and maintains them in a pool. When a new job comes in, the worker selects an available pod from the pool and sends the job to the agent running in that pod. After the job is complete, the pod gets automatically deleted.

Overview

A worker can be run:

• Outside the Kubernetes cluster, e.g., on the user’s computer. In which case kubeconfig is sent to the worker parameters to enable using the Kubernetes cluster. The worker uses port forward to send jobs to the agent.

  kubeconfig must contain a single context, and this context must be selected as the current one.

• Inside the Kubernetes cluster. The worker uses the pod's UP address (podIP) to send jobs to the agent.

  The worker's namespace must be different from the namespace allocated for the pod pool.

The worker’s workflow consists of the following steps:

• Start
• Pod pool maintenance
• Job execution

Worker start

When starting, the worker deletes and recreates a secret named agent-secrets in the Kubernetes namespace allocated for the pod pool. The secret contains the agent configuration and the CA (certificate authority) certificate required to start the agent. The CA certificate is used to authorize worker requests to the agent over mTLS.

Pod pool maintenance

The worker periodically runs the pod pool maintenance procedure.

• Pods' internal states get synced with the states in the Kubernetes cluster.

• Pods' transitions between states get processed.

• New pods get created.

• Unknown pods get deleted, including those left over from the previous worker run or created by another worker.

  Warning

  Do not run more than one worker instance with a pod pool in the same namespace. Workers will be deleting each other’s pods.

The number of available pods in the pool is maintained equal to the minimum pool size specified in the worker configuration. The total number of pods in the pool cannot exceed the maximum pool size specified in the worker configuration.

Job execution

After it receives a job, the worker scans the pool for an available pod and sends the job to the agent which runs in that pod.

The process of sending the job to the agent depends on where the worker is deployed:

When sending a job to the agent, the worker authorizes its request using a client certificate and a key. The agent verifies the authorization via a CA certificate. The worker-to-agent connection is secured with mTLS.

Next the agent proceeds to execute the job by itself. After the job is complete, the worker deletes the pod.

Setup

To set up the execution of CI/CD jobs in a Kubernetes cluster:

1. Prepare a Kubernetes cluster.
2. Create a worker configuration file.
3. Create a pod template.
4. Start the worker.

Prepare a Kubernetes cluster

1. Install kubectl and connect to your Kubernetes cluster.

2. Create a namespace for the worker pod pool:

   kubectl create namespace pods-namespace
   

3. Create a file named role.yaml describing the role of the user the worker will use to access the Kubernetes cluster. The user must have permissions to manage secrets and pods in the pod pool namespace. If the worker runs outside the Kubernetes cluster, port forwarding permissions are also required.

   Worker outside the cluster

   Worker inside the cluster

   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: Role
   metadata:
     name: processor-role
   rules:
     - apiGroups: [""]
       resources: ["secrets", "pods"]
       verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
     - apiGroups: [""]
       resources: ["pods/portforward"]
       verbs: ["create"]
   

   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: Role
   metadata:
     name: processor-role
   rules:
     - apiGroups: [""]
       resources: ["secrets", "pods"]
       verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
   

4. Apply the manifest for the namespace you created earlier:

   kubectl apply -f role.yaml -n pods-namespace
   

Create a worker configuration file

Create a file named config.yaml with the following configuration:

kubernetes:
  kubeconfig_path: /path/to/.kube/config
  namespace: <pool_namespace>
  pod_template_path: /path/to/pod-template.yaml
  agent:
    port: 8693
    ca_cert_path: /path/to/ca.crt
    client_cert_path: /path/to/client.crt
    client_key_path: /path/to/client.key
  pool_max_size: 10
  pool_min_size: 3
  pool_maintenance_period: 10s
  secret_updating_timeout: 20s
  pods_listing_timeout: 10s
  pod_creation_timeout: 1m
  pod_startup_timeout: 2m
  pod_deletion_timeout: 30s
  endpoint_resolving_timeout: 10s

endpoint:
  host: ci.sourcecraft.tech
  port: 443
  ssl_no_verify: false

executor_type: kubernetes

logger_type: json
logger_level: info

tags: ["my-tag", "+my-required-tag"]

auth:
  pat: <personal_access_token>

Where:

• kubernetes: Kubernetes-specific settings, namely:
  • kubeconfig_path: Path to the kubeconfig file. You can also specify it via the KUBECONFIG environment variable. If you do not set this property, the in-cluster configuration will be used.
  • namespace: Namespace allocated for the pod pool. This is a required setting.
  • pod_template: Pod template in YAML format. You can also specify it via the KUBERNETES_POD_TEMPLATE environment variable. Specify either pod_template or pod_template_path. You cannot use both properties at the same time.
  • pod_template_path: Path to the pod template file. Specify either pod_template or pod_template_path. You cannot use both properties at the same time.
  • agent: Subsection with properties for interaction with the agent:
    • port: Agent’s port. The default value is 8693. This port must be different from all other ports of the pod. It is included in the agent's configuration.
    • ca_cert: Certificate issued by the certificate authority. You can also specify it via the KUBERNETES_AGENT_CA_CERT environment variable. Specify either ca_cert or ca_cert_path. You cannot use both properties at the same time. It is included in the agent's configuration.
    • ca_cert_path: Path to the CA certificate file. Specify either ca_cert or ca_cert_path. You cannot use both properties at the same time. It is included in the agent's configuration.
    • client_cert: Client certificate. You can also specify it via the KUBERNETES_AGENT_CLIENT_CERT environment variable. Specify either client_cert or client_cert_path. You cannot use both properties at the same time.
    • client_cert_path: Path to the client certificate file. Specify either client_cert or client_cert_path. You cannot use both properties at the same time.
    • client_key: Client key. You can also specify it via the KUBERNETES_AGENT_CLIENT_KEY environment variable. Specify either client_key or client_key_path. You cannot use both properties at the same time.
    • client_key_path: Path to the client key file. Specify either client_key or client_key_path. You cannot use both properties at the same time.
  • pool_max_size: Maximum pod pool size. This is a required setting. It must be greater than or equal to pool_min_size.
  • pool_min_size: Minimum pod pool size. This is a required setting. It must be greater than 0.
  • pool_maintenance_period: Pod pool maintenance interval. The default value is 10s.
  • secret_updating_timeout: Secret update timeout. The default value is 20s.
  • pods_listing_timeout: Pod listing timeout. The default value is 10s.
  • pod_creation_timeout: Pod creation timeout. The default value is 1m.
  • pod_startup_timeout: Pod startup timeout. The default value is 2m. It includes the time spent waiting for the pod and all its containers to start.
  • pod_deletion_timeout: Pod deletion timeout. The default value is 30s.
  • endpoint_resolving_timeout: Endpoint resolving timeout. The default value is 10s. It includes the time spent waiting for agent port forwarding.
• endpoint: Section with properties for using SourceCraft. This full section is included in the agent's configuration:
  • host: SourceCraft host. This is a required setting.
  • port: SourceCraft port. This is a required setting.
  • ssl_no_verify: Flag to turn off SourceCraft server certificate verification. The default value is false.
• executor_type: Worker type. For the Kubernetes mode, it should be set to kubernetes.
• logger_type: Logging format. The valid values are json and console.
• logger_level: Logging level. The valid values are debug, info, warn, and error. This is an optional setting.
• tags: List of worker tags. This is an optional setting.
• auth: Subsection with properties for worker authentication in SourceCraft:
  • pat: Personal access token (PAT). You can also specify it via the PAT_CREDENTIALS environment variable. This is a required setting.

Create a pod template

The pod template specifies the configuration the worker uses to create pods in the pod pool namespace.

Create a file named pod-template.yaml with the pod template, for example:

---
apiVersion: v1
kind: Pod
metadata:
  name: pod-template
spec:
  containers:
    - name: cicd-worker
      image: cr.yandex/sourcecraft/ci/self-hosted-processor:latest
      resources:
        requests:
          cpu: 250m
          memory: 256Mi
        limits:
          cpu: "1"
          memory: 1Gi

When creating a pod, the worker automatically extends the template as follows:

• The cicd-worker container additionally gets environment variables from the secret named agent-secrets, a container port named agent-port for the agent's port, and a startup probe.
• The pod gets renamed to cicd-worker-<unique_ID>.
• The namespace gets replaced by a pod pool namespace.
• Restarting the pod gets banned.

---
apiVersion: v1
kind: Pod
metadata:
  name: cicd-worker-<unique_ID>
  namespace: <pool_namespace>
spec:
  containers:
    - name: cicd-worker
      image: cr.yandex/sourcecraft/ci/self-hosted-processor:latest
      resources:
        requests:
          cpu: 250m
          memory: 256Mi
        limits:
          cpu: "1"
          memory: 1Gi
      envFrom:
        - secretRef:
            name: agent-secrets
      ports:
        - name: agent-port
          containerPort: 8693
          protocol: TCP
      startupProbe:
        exec:
          command: ["nc", "-z", "localhost", "8693"]
        initialDelaySeconds: 5
        periodSeconds: 5
        failureThreshold: 10
  restartPolicy: Never

Agent's Docker image

We recommend using the cr.yandex/sourcecraft/ci/self-hosted-processor:latest worker base container image as the agent's Docker image. If you need a custom image for jobs, follow these requirements when creating it:

• The worker must start without specifying a configuration file path in the command line; the configuration is provided via the PROCESSOR_CONFIG environment variable:

  ENTRYPOINT ["/self-hosted-processor", "run"]
  

• The image must contain a directory named /home/sourcecraft-ci-runner. The user account used by the container must have full access to this directory.

• The image must have the following executables installed and ready to run: bash, sh, git, git-lfs, docker, nc.

Using Docker in a job

If you need Docker to execute your jobs, add a container with Docker-in-Docker to the pod template.

---
apiVersion: v1
kind: Pod
metadata:
  name: pod-template
spec:
  containers:
    - name: cicd-worker
      image: cr.yandex/sourcecraft/ci/self-hosted-processor:latest
      resources:
        requests:
          cpu: 250m
          memory: 256Mi
        limits:
          cpu: "1"
          memory: 1Gi
      env:
        - name: DOCKER_HOST
          value: "tcp://localhost:2376"
        - name: DOCKER_TLS_VERIFY
          value: "1"
        - name: DOCKER_CERT_PATH
          value: "/.docker-certs/client"
      volumeMounts:
        - name: docker-certs
          mountPath: /.docker-certs
    - name: dind
      image: docker:dind
      securityContext:
        privileged: true
      startupProbe:
        exec:
          command: ["docker", "info"]
        initialDelaySeconds: 5
        periodSeconds: 5
        failureThreshold: 30
      env:
        - name: DOCKER_TLS_CERTDIR
          value: "/.docker-certs"
      resources:
        requests:
          cpu: 500m
          memory: 512Mi
        limits:
          cpu: "2"
          memory: 2Gi
      volumeMounts:
        - name: docker-certs
          mountPath: /.docker-certs
  volumes:
    - name: docker-certs
      emptyDir: {}

Select resources for the Docker-in-Docker container based on expected workload. The Docker-in-Docker container must use a startup probe to prevent job execution failures due to Docker-in-Docker not being ready to high workload conditions.

You can also use the more secure docker:dind-rootless image. In which case you should install the additional sysbox component on the Kubernetes cluster nodes.

Start the worker

ENTRYPOINT ["/self-hosted-processor", "run"]

docker run --rm -e "PROCESSOR_CONFIG=<configuration_content>" cr.yandex/sourcecraft/ci/self-hosted-processor:latest

For an example of a manifest to run a worker inside a Kubernetes cluster, see Running a worker inside a Kubernetes cluster.

Configuration examples

Running a worker outside a Kubernetes cluster

Configuration file config.yaml:

kubernetes:
  kubeconfig_path: /path/to/.kube/config
  namespace: pods-namespace
  pod_template_path: /path/to/pod-template.yaml
  agent:
    port: 8693
    ca_cert_path: /path/to/ca.crt
    client_cert_path: /path/to/client.crt
    client_key_path: /path/to/client.key
  pool_max_size: 10
  pool_min_size: 3

endpoint:
  host: ci.sourcecraft.tech
  port: 443

executor_type: kubernetes

logger_type: json

tags: ["my-tag", "+my-required-tag"]

auth:
  pat: <personal_access_token>

The run command:

self-hosted-processor run --config-path /path/to/config.yaml

Running a worker inside a Kubernetes cluster

Manifest with a service account, role, secret, and worker deployment:

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: processor-sa
  namespace: processor-namespace

---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: processor-role
  namespace: pods-namespace
rules:
  - apiGroups: [""]
    resources: ["secrets", "pods"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]

---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: processor-role-binding
  namespace: pods-namespace
subjects:
  - kind: ServiceAccount
    name: processor-sa
    namespace: processor-namespace
roleRef:
  kind: Role
  name: processor-role
  apiGroup: rbac.authorization.k8s.io

---
apiVersion: v1
kind: Secret
metadata:
  name: processor-secrets
  namespace: processor-namespace
type: Opaque
stringData:
  PROCESSOR_CONFIG: |
    ---
    kubernetes:
      namespace: pods-namespace
      agent:
        port: 8693
      pool_max_size: 10
      pool_min_size: 3

    endpoint:
      host: ci.sourcecraft.tech
      port: 443

    executor_type: kubernetes

    logger_type: json

    tags: ["my-tag", "+my-required-tag"]

  KUBERNETES_POD_TEMPLATE: |
    ---
    apiVersion: v1
    kind: Pod
    metadata:
      name: pod-template
    spec:
      containers:
        - name: cicd-worker
          image: cr.yandex/sourcecraft/ci/self-hosted-processor:latest
          resources:
            requests:
              cpu: 250m
              memory: 256Mi
            limits:
              cpu: "1"
              memory: 1Gi
          env:
            - name: DOCKER_HOST
              value: "tcp://localhost:2376"
            - name: DOCKER_TLS_VERIFY
              value: "1"
            - name: DOCKER_CERT_PATH
              value: "/.docker-certs/client"
          volumeMounts:
            - name: docker-certs
              mountPath: /.docker-certs
        - name: dind
          image: docker:dind
          securityContext:
            privileged: true
          startupProbe:
            exec:
              command: ["docker", "info"]
            initialDelaySeconds: 5
            periodSeconds: 5
            failureThreshold: 30
          env:
            - name: DOCKER_TLS_CERTDIR
              value: "/.docker-certs"
          resources:
            requests:
              cpu: 500m
              memory: 512Mi
            limits:
              cpu: "2"
              memory: 2Gi
          volumeMounts:
            - name: docker-certs
              mountPath: /.docker-certs
      volumes:
        - name: docker-certs
          emptyDir: {}

  KUBERNETES_AGENT_CA_CERT: |
    -----BEGIN CERTIFICATE-----
    certificate content
    -----END CERTIFICATE-----

  KUBERNETES_AGENT_CLIENT_CERT: |
    -----BEGIN CERTIFICATE-----
    certificate content
    -----END CERTIFICATE-----

  KUBERNETES_AGENT_CLIENT_KEY: |
    -----BEGIN PRIVATE KEY-----
    key content
    -----END PRIVATE KEY-----

  PAT_CREDENTIALS: <personal_access_token>

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kubernetes-processor
  namespace: processor-namespace
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kubernetes-processor
  template:
    metadata:
      labels:
        app: kubernetes-processor
    spec:
      serviceAccountName: processor-sa
      containers:
        - name: kubernetes-processor
          image: cr.yandex/sourcecraft/ci/self-hosted-processor:latest
          resources:
            requests:
              cpu: 250m
              memory: 256Mi
            limits:
              cpu: "1"
              memory: 1Gi
          envFrom:
            - secretRef:
                name: processor-secrets

Useful links

• Setting up a self-hosted worker for SourceCraft
• Self-hosted worker tags in SourceCraft
• Configuring CI/CD in a SourceCraft repository
• CI/CD reference
• Continuous integration and continuous deployment in SourceCraft