--- metadata: - name: generator content: Diplodoc Platform v5.57.3 alternate: - https://sourcecraft.dev/portal/docs/en/sourcecraft/operations/self-hosted-worker-kubernetes.md - https://sourcecraft.dev/portal/docs/ru/sourcecraft/operations/self-hosted-worker-kubernetes.md - href: en/sourcecraft/operations/self-hosted-worker-kubernetes.md type: text/markdown title: Markdown version - href: ../../llms.txt type: text/markdown title: llms.txt --- > **Documentation Index:** Fetch the complete configuration index at https://sourcecraft.dev/portal/docs/en/llms.txt # Setting up the execution of SourceCraft CI/CD jobs in a Kubernetes cluster The [self-hosted worker](https://sourcecraft.dev/portal/docs/en/sourcecraft/concepts/workers.md#self-hosted-workers) named `self-hosted-processor` supports a mode where CI/CD [jobs](https://sourcecraft.dev/portal/docs/en/sourcecraft/concepts/ci-cd.md#tasks) are executed in a Kubernetes cluster. Each job runs in a separate [pod](https://kubernetes.io/docs/concepts/workloads/pods/). The worker creates pods in advance and maintains them in a pool. When a new job comes in, the worker selects an available pod from the pool and sends the job to the agent running in that pod. After the job is complete, the pod gets automatically deleted. {% note info %} The Kubernetes mode is supported by worker versions starting from `0.14.1`. {% endnote %} ## Overview {#overview} A worker can be run: * _Outside the Kubernetes cluster_, e.g., on the user’s computer. In which case `kubeconfig` is sent to the worker parameters to enable using the Kubernetes cluster. The worker uses `port forward` to send jobs to the agent. `kubeconfig` must contain a single context, and this context must be selected as the current one. * _Inside the Kubernetes cluster_. The worker uses the pod's UP address (`podIP`) to send jobs to the agent. The worker's [namespace](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/) must be different from the namespace allocated for the pod pool. The worker’s workflow consists of the following steps: * [Start](#worker-start) * [Pod pool maintenance](#pool-maintenance) * [Job execution](#task-execution) ### Worker start {#worker-start} When starting, the worker deletes and recreates a secret named `agent-secrets` in the Kubernetes namespace allocated for the pod pool. The secret contains the agent configuration and the `CA` (`certificate authority`) certificate required to start the agent. The CA certificate is used to authorize worker requests to the agent over mTLS. ### Pod pool maintenance {#pool-maintenance} The worker periodically runs the pod pool maintenance procedure. * Pods' internal states get synced with the states in the Kubernetes cluster. * Pods' transitions between states get processed. * New pods get created. * Unknown pods get deleted, including those left over from the previous worker run or created by another worker. {% note warning %} Do not run more than one worker instance with a pod pool in the same namespace. Workers will be deleting each other’s pods. {% endnote %} The number of available pods in the pool is maintained equal to the minimum pool size specified in the worker configuration. The total number of pods in the pool cannot exceed the maximum pool size specified in the worker configuration. ### Job execution {#task-execution} After it receives a job, the worker scans the pool for an available pod and sends the job to the agent which runs in that pod. The process of sending the job to the agent depends on where the worker is deployed: {% list tabs %} - Worker outside the cluster The worker forwards the agent’s port to its local port (`port forward`), sends the job to the agent through the local port, and then closes the port-forwarding connection. - Worker inside the cluster The worker sends the job to the agent’s port via the IP address of the pod. {% note warning %} Allow worker-to-pod communication within the pool namespace; such communication may be restricted by network policies or other tools. {% endnote %} {% endlist %} When sending a job to the agent, the worker authorizes its request using a client certificate and a key. The agent verifies the authorization via a CA certificate. The worker-to-agent connection is secured with mTLS. Next the agent proceeds to execute the job by itself. After the job is complete, the worker deletes the pod. ## Setup {#setup} To set up the execution of CI/CD jobs in a Kubernetes cluster: 1. [Prepare a Kubernetes cluster](#prepare-cluster). 1. [Create a worker configuration file](#config-file). 1. [Create a pod template](#pod-template). 1. [Start the worker](#start-worker). ### Prepare a Kubernetes cluster {#prepare-cluster} {% note tip %} To run SourceCraft CI/CD jobs, you can use a [Yandex Managed Service for Kubernetes cluster](https://yandex.cloud/en/docs/managed-kubernetes/quickstart). {% endnote %} 1. [Install](https://kubernetes.io/ru/docs/tasks/tools/#kubectl) `kubectl` and connect to your Kubernetes cluster. 1. Create a namespace for the worker pod pool: ```bash kubectl create namespace pods-namespace ``` 1. Create a file named `role.yaml` describing the role of the user the worker will use to access the Kubernetes cluster. The user must have permissions to manage secrets and pods in the pod pool namespace. If the worker runs outside the Kubernetes cluster, port forwarding permissions are also required. {% list tabs %} - Worker outside the cluster ```yaml --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: processor-role rules: - apiGroups: [""] resources: ["secrets", "pods"] verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] - apiGroups: [""] resources: ["pods/portforward"] verbs: ["create"] ``` - Worker inside the cluster ```yaml --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: processor-role rules: - apiGroups: [""] resources: ["secrets", "pods"] verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] ``` {% endlist %} 1. Apply the manifest for the namespace you created earlier: ```bash kubectl apply -f role.yaml -n pods-namespace ``` ### Create a worker configuration file {#config-file} Create a file named `config.yaml` with the following configuration: ```yaml kubernetes: kubeconfig_path: /path/to/.kube/config namespace: pod_template_path: /path/to/pod-template.yaml agent: port: 8693 ca_cert_path: /path/to/ca.crt client_cert_path: /path/to/client.crt client_key_path: /path/to/client.key pool_max_size: 10 pool_min_size: 3 pool_maintenance_period: 10s secret_updating_timeout: 20s pods_listing_timeout: 10s pod_creation_timeout: 1m pod_startup_timeout: 2m pod_deletion_timeout: 30s endpoint_resolving_timeout: 10s endpoint: host: ci.sourcecraft.tech port: 443 ssl_no_verify: false executor_type: kubernetes logger_type: json logger_level: info tags: ["my-tag", "+my-required-tag"] auth: pat: ``` Where: * `kubernetes`: Kubernetes-specific settings, namely: * `kubeconfig_path`: Path to the `kubeconfig` file. You can also specify it via the `KUBECONFIG` environment variable. If you do not set this property, the in-cluster configuration will be used. * `namespace`: Namespace allocated for the pod pool. This is a required setting. * `pod_template`: [Pod template](#pod-template) in YAML format. You can also specify it via the `KUBERNETES_POD_TEMPLATE` environment variable. Specify either `pod_template` or `pod_template_path`. You cannot use both properties at the same time. * `pod_template_path`: Path to the pod template file. Specify either `pod_template` or `pod_template_path`. You cannot use both properties at the same time. * `agent`: Subsection with properties for interaction with the agent: * `port`: Agent’s port. The default value is `8693`. This port must be different from all other ports of the pod. It is included in the agent's configuration. * `ca_cert`: Certificate issued by the certificate authority. You can also specify it via the `KUBERNETES_AGENT_CA_CERT` environment variable. Specify either `ca_cert` or `ca_cert_path`. You cannot use both properties at the same time. It is included in the agent's configuration. * `ca_cert_path`: Path to the CA certificate file. Specify either `ca_cert` or `ca_cert_path`. You cannot use both properties at the same time. It is included in the agent's configuration. * `client_cert`: Client certificate. You can also specify it via the `KUBERNETES_AGENT_CLIENT_CERT` environment variable. Specify either `client_cert` or `client_cert_path`. You cannot use both properties at the same time. * `client_cert_path`: Path to the client certificate file. Specify either `client_cert` or `client_cert_path`. You cannot use both properties at the same time. * `client_key`: Client key. You can also specify it via the `KUBERNETES_AGENT_CLIENT_KEY` environment variable. Specify either `client_key` or `client_key_path`. You cannot use both properties at the same time. * `client_key_path`: Path to the client key file. Specify either `client_key` or `client_key_path`. You cannot use both properties at the same time. * `pool_max_size`: Maximum pod pool size. This is a required setting. It must be greater than or equal to `pool_min_size`. * `pool_min_size`: Minimum pod pool size. This is a required setting. It must be greater than `0`. * `pool_maintenance_period`: Pod pool maintenance interval. The default value is `10s`. * `secret_updating_timeout`: Secret update timeout. The default value is `20s`. * `pods_listing_timeout`: Pod listing timeout. The default value is `10s`. * `pod_creation_timeout`: Pod creation timeout. The default value is `1m`. * `pod_startup_timeout`: Pod startup timeout. The default value is `2m`. It includes the time spent waiting for the pod and all its containers to start. * `pod_deletion_timeout`: Pod deletion timeout. The default value is `30s`. * `endpoint_resolving_timeout`: Endpoint resolving timeout. The default value is `10s`. It includes the time spent waiting for agent port forwarding. * `endpoint`: Section with properties for using SourceCraft. This full section is included in the agent's configuration: * `host`: SourceCraft host. This is a required setting. * `port`: SourceCraft port. This is a required setting. * `ssl_no_verify`: Flag to turn off SourceCraft server certificate verification. The default value is `false`. * `executor_type`: Worker type. For the Kubernetes mode, it should be set to `kubernetes`. * `logger_type`: Logging format. The valid values are `json` and `console`. * `logger_level`: Logging level. The valid values are `debug`, `info`, `warn`, and `error`. This is an optional setting. * `tags`: List of [worker tags](https://sourcecraft.dev/portal/docs/en/sourcecraft/ci-cd-ref/self-hosted-tags.md). This is an optional setting. * `auth`: Subsection with properties for worker authentication in SourceCraft: * `pat`: [Personal access token](https://sourcecraft.dev/portal/docs/en/sourcecraft/security/pat.md) (PAT). You can also specify it via the `PAT_CREDENTIALS` environment variable. This is a required setting. ### Create a pod template {#pod-template} The pod template specifies the configuration the worker uses to create pods in the pod pool namespace. {% note warning %} The template must include a container named `cicd-worker`. This container runs the agent that executes the job. Select resources for the `cicd-worker` container based on expected workload when executing jobs. {% endnote %} Create a file named `pod-template.yaml` with the pod template, for example: ```yaml --- apiVersion: v1 kind: Pod metadata: name: pod-template spec: containers: - name: cicd-worker image: cr.yandex/sourcecraft/ci/self-hosted-processor:latest resources: requests: cpu: 250m memory: 256Mi limits: cpu: "1" memory: 1Gi ``` When creating a pod, the worker automatically extends the template as follows: * The `cicd-worker` container additionally gets environment variables from the secret named `agent-secrets`, a container port named `agent-port` for the agent's port, and a startup probe. * The pod gets renamed to `cicd-worker-`. * The namespace gets replaced by a pod pool namespace. * Restarting the pod gets banned. {% cut "Example of an extended configuration" %} ```yaml --- apiVersion: v1 kind: Pod metadata: name: cicd-worker- namespace: spec: containers: - name: cicd-worker image: cr.yandex/sourcecraft/ci/self-hosted-processor:latest resources: requests: cpu: 250m memory: 256Mi limits: cpu: "1" memory: 1Gi envFrom: - secretRef: name: agent-secrets ports: - name: agent-port containerPort: 8693 protocol: TCP startupProbe: exec: command: ["nc", "-z", "localhost", "8693"] initialDelaySeconds: 5 periodSeconds: 5 failureThreshold: 10 restartPolicy: Never ``` {% endcut %} #### Agent's Docker image {#agent-image} We recommend using the `cr.yandex/sourcecraft/ci/self-hosted-processor:latest` worker base container image as the agent's Docker image. If you need a custom image for jobs, follow these requirements when creating it: * The worker must start without specifying a configuration file path in the command line; the configuration is provided via the `PROCESSOR_CONFIG` environment variable: ```Dockerfile ENTRYPOINT ["/self-hosted-processor", "run"] ``` * The image must contain a directory named `/home/sourcecraft-ci-runner`. The user account used by the container must have full access to this directory. * The image must have the following executables installed and ready to run: `bash`, `sh`, `git`, `git-lfs`, `docker`, `nc`. #### Using Docker in a job {#docker-in-task} If you need Docker to execute your jobs, add a container with `Docker-in-Docker` to the pod template. {% cut "Example of a pod template with Docker-in-Docker" %} ```yaml --- apiVersion: v1 kind: Pod metadata: name: pod-template spec: containers: - name: cicd-worker image: cr.yandex/sourcecraft/ci/self-hosted-processor:latest resources: requests: cpu: 250m memory: 256Mi limits: cpu: "1" memory: 1Gi env: - name: DOCKER_HOST value: "tcp://localhost:2376" - name: DOCKER_TLS_VERIFY value: "1" - name: DOCKER_CERT_PATH value: "/.docker-certs/client" volumeMounts: - name: docker-certs mountPath: /.docker-certs - name: dind image: docker:dind securityContext: privileged: true startupProbe: exec: command: ["docker", "info"] initialDelaySeconds: 5 periodSeconds: 5 failureThreshold: 30 env: - name: DOCKER_TLS_CERTDIR value: "/.docker-certs" resources: requests: cpu: 500m memory: 512Mi limits: cpu: "2" memory: 2Gi volumeMounts: - name: docker-certs mountPath: /.docker-certs volumes: - name: docker-certs emptyDir: {} ``` {% endcut %} Select resources for the `Docker-in-Docker` container based on expected workload. The `Docker-in-Docker` container must use a startup probe to prevent job execution failures due to `Docker-in-Docker` not being ready to high workload conditions. You can also use the more secure `docker:dind-rootless` image. In which case you should install the additional [sysbox](https://github.com/nestybox/sysbox) component on the Kubernetes cluster nodes. ### Start the worker {#start-worker} {% list tabs %} - In your environment 1. [Set up your environment](https://sourcecraft.dev/portal/docs/en/sourcecraft/operations/self-hosted-worker.md#prepare-environment). 1. Start the worker with the configuration you created earlier using one of these methods: * Provide the configuration file path in the command line: ```bash self-hosted-processor run --config-path ``` * Provide the configuration content via the `PROCESSOR_CONFIG` environment variable: ```bash PROCESSOR_CONFIG="" self-hosted-processor run ``` {% note warning %} The worker will not start without a configuration. You cannot use both methods at the same time; otherwise, the worker will not start. {% endnote %} - In a container The registry offers the `cr.yandex/sourcecraft/ci/self-hosted-processor:latest` base Docker image of the worker container as well as tags of specific versions. In this image, the worker starts without the configuration file path: ```Dockerfile ENTRYPOINT ["/self-hosted-processor", "run"] ``` Therefore, when starting the container, provide the configuration via the `PROCESSOR_CONFIG` environment variable: ```bash docker run --rm -e "PROCESSOR_CONFIG=" cr.yandex/sourcecraft/ci/self-hosted-processor:latest ``` {% endlist %} For an example of a manifest to run a worker inside a Kubernetes cluster, see [Running a worker inside a Kubernetes cluster](#example-in-cluster). ## Configuration examples {#examples} ### Running a worker outside a Kubernetes cluster {#example-out-of-cluster} Configuration file `config.yaml`: ```yaml kubernetes: kubeconfig_path: /path/to/.kube/config namespace: pods-namespace pod_template_path: /path/to/pod-template.yaml agent: port: 8693 ca_cert_path: /path/to/ca.crt client_cert_path: /path/to/client.crt client_key_path: /path/to/client.key pool_max_size: 10 pool_min_size: 3 endpoint: host: ci.sourcecraft.tech port: 443 executor_type: kubernetes logger_type: json tags: ["my-tag", "+my-required-tag"] auth: pat: ``` The run command: ```shell self-hosted-processor run --config-path /path/to/config.yaml ``` ### Running a worker inside a Kubernetes cluster {#example-in-cluster} Manifest with a service account, role, secret, and worker deployment: ```yaml --- apiVersion: v1 kind: ServiceAccount metadata: name: processor-sa namespace: processor-namespace --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: processor-role namespace: pods-namespace rules: - apiGroups: [""] resources: ["secrets", "pods"] verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: processor-role-binding namespace: pods-namespace subjects: - kind: ServiceAccount name: processor-sa namespace: processor-namespace roleRef: kind: Role name: processor-role apiGroup: rbac.authorization.k8s.io --- apiVersion: v1 kind: Secret metadata: name: processor-secrets namespace: processor-namespace type: Opaque stringData: PROCESSOR_CONFIG: | --- kubernetes: namespace: pods-namespace agent: port: 8693 pool_max_size: 10 pool_min_size: 3 endpoint: host: ci.sourcecraft.tech port: 443 executor_type: kubernetes logger_type: json tags: ["my-tag", "+my-required-tag"] KUBERNETES_POD_TEMPLATE: | --- apiVersion: v1 kind: Pod metadata: name: pod-template spec: containers: - name: cicd-worker image: cr.yandex/sourcecraft/ci/self-hosted-processor:latest resources: requests: cpu: 250m memory: 256Mi limits: cpu: "1" memory: 1Gi env: - name: DOCKER_HOST value: "tcp://localhost:2376" - name: DOCKER_TLS_VERIFY value: "1" - name: DOCKER_CERT_PATH value: "/.docker-certs/client" volumeMounts: - name: docker-certs mountPath: /.docker-certs - name: dind image: docker:dind securityContext: privileged: true startupProbe: exec: command: ["docker", "info"] initialDelaySeconds: 5 periodSeconds: 5 failureThreshold: 30 env: - name: DOCKER_TLS_CERTDIR value: "/.docker-certs" resources: requests: cpu: 500m memory: 512Mi limits: cpu: "2" memory: 2Gi volumeMounts: - name: docker-certs mountPath: /.docker-certs volumes: - name: docker-certs emptyDir: {} KUBERNETES_AGENT_CA_CERT: | -----BEGIN CERTIFICATE----- certificate content -----END CERTIFICATE----- KUBERNETES_AGENT_CLIENT_CERT: | -----BEGIN CERTIFICATE----- certificate content -----END CERTIFICATE----- KUBERNETES_AGENT_CLIENT_KEY: | -----BEGIN PRIVATE KEY----- key content -----END PRIVATE KEY----- PAT_CREDENTIALS: --- apiVersion: apps/v1 kind: Deployment metadata: name: kubernetes-processor namespace: processor-namespace spec: replicas: 1 selector: matchLabels: app: kubernetes-processor template: metadata: labels: app: kubernetes-processor spec: serviceAccountName: processor-sa containers: - name: kubernetes-processor image: cr.yandex/sourcecraft/ci/self-hosted-processor:latest resources: requests: cpu: 250m memory: 256Mi limits: cpu: "1" memory: 1Gi envFrom: - secretRef: name: processor-secrets ``` ### Useful links {#see-also} * [Setting up a self-hosted worker for SourceCraft](https://sourcecraft.dev/portal/docs/en/sourcecraft/operations/self-hosted-worker.md) * [Self-hosted worker tags in SourceCraft](https://sourcecraft.dev/portal/docs/en/sourcecraft/ci-cd-ref/self-hosted-tags.md) * [Configuring CI/CD in a SourceCraft repository](https://sourcecraft.dev/portal/docs/en/sourcecraft/operations/ci-cd.md) * [CI/CD reference](https://sourcecraft.dev/portal/docs/en/sourcecraft/ci-cd-ref/index.md) * [Continuous integration and continuous deployment in SourceCraft](https://sourcecraft.dev/portal/docs/en/sourcecraft/concepts/ci-cd.md)