# Kargo Documentation > Kargo is a Kubernetes-native continuous promotion platform for GitOps workflows. This file contains all documentation content in a single document following the llmstxt.org standard. ## Home --- ## Kargo Roadmap Over a series of releases, Kargo's maintainers have settled into a cadence of a minor release roughly every six weeks, with two or three major features completed per release, although this varies slightly by scope. :::caution This roadmap highlights only major, _planned_ work for the **open source** edition of Kargo. Priorities may evolve over time based on community feedback and Akuity customer needs. For more detailed and up-to-date plans, please refer to our [project milestones](https://github.com/akuity/kargo/milestones) on GitHub. If you’re interested in what’s coming to our hosted, enterprise edition of Kargo, please [get in touch with us](https://akuity.io/get-in-touch)! Ideas and contributions from the community are always welcome, however, items not appearing on the roadmap are reviewed on a best-effort basis. If you would like to contribute, please avoid wasted or duplicated effort by discussing feature requests and design proposals with maintainers prior to starting work. While we cannot guarantee specific timelines or outcomes, we value and consider every submission. ::: ## In Progress ### v1.11.0 **Expected:** Mid-June, 2026 v1.11.0 advances Kargo's event-driven model and continues improving operator and end-user experience. The release also advances the ongoing UI migration from the deprecated ConnectRPC API to the REST API. Freight and Stage details aim to become richer with configurable links to external platforms. ## Upcoming ### v1.12.0 The following deprecated features are scheduled for **removal** in v1.12.0: * The deprecated ConnectRPC (gRPC) API in favor of the new REST API. * The `createTargetBranch` option in the `git-open-pr` promotion step. * The `author` configuration block on the `git-commit` promotion step. Use `git-clone` instead. * The default `git-push` integration policy changes from `AlwaysRebase` to `RebaseOrMerge`. Set the policy explicitly if you rely on unconditional rebase behavior. ## Completed ### v1.10.0 Evolution, not revolution — a broad collection of quality-of-life improvements, new and improved promotion steps, and UI enhancements. Also introduced controller heartbeats with liveness surfaced in the UI. See [release notes](./80-release-notes/89-v1.10.0.md) for full details. ### v1.9.0 Focused on stability, performance, and usability. Introduced: * A new, RESTful API * User-generated API tokens * Opt-in Warehouse caching * Configurable rate limits * Refined everything related to Secret/credentials management * A large slate of small See [release notes](./80-release-notes/90-v1.9.0.md) for full details. ### v1.8.0 This small release focused on UI enhancements and general "quality-of-life" improvements, as well as a grab bag of small features. ### v1.7.0 This release focused on improving Kargo's webhook receiver capabilities, adding new Promotion steps, and enhancing the UI. ### v1.6.0 The anchor feature of v1.6.0 is support for triggering applicable Warehouses to execute their artifact discovery process in response to inbound webhooks from source code and container image repositories. Initial support will favor the most popular platforms, such as GitHub, GitLab, Docker Hub and Quay.io, although support for many others will be phased in over time. ### v1.5.0 This release focused on UI improvement and streamlining `Project` configuration through the introduction of a new `ProjectConfig` resource type. Conditional execution of promotion steps was also improved such that the success or failure of previous steps can be taken into account. This permits users to define more robust promotion processes by defining steps that conditionally execute error recovery or cleanup. ### v1.4.0 This release focused on small QOL improvements, notably access to logs generated by `Job` metrics as part of verification processes and support for enriching `Freight` with additional metadata by reading OIDC annotations from container images. ### v1.3.0 This release focused primarily on introducing support for conditional execution of promotion steps as determined by user-defined expressions. ### v1.2.0 This release focused on three large features: The ability to define reusable sequences of promotion steps via `PromotionTask` or `ClusterPromotionTask` resources, "soak times" (time criteria for a Promotion), and management of generic Project `Secret`s. ### v1.1.0 This release dramatically improved the flexibility of the promotion steps introduced in v0.9.0 through the addition of support for an expression language. ### v1.0.0 v1.0.0 was our long-anticipated GA release containing only small features, bug fixes, stability improvements, and the final removal of the legacy promotion mechanisms that were deprecated in v0.9.0. ### v0.9.0 | Name | Type | Description | | ---- | ---- | ----------- | | [Promotion Steps](https://github.com/akuity/kargo/issues/2219) | strategic refactor | Transitioned from opinionated promotion mechanisms to an ordered lists of more finely-grained promotion directives steps reminiscent of GitHub Actions. These enable greater flexibility in addressing outlying use cases and have left us with a clear path forward for to eventually enable third-party integrations. | | Production Readiness | chore | Prioritized stability of existing features.Paid down technical debt.**This does not mean v0.9.0 is production-ready. It means it is several steps closer to it.** | ### v0.8.0 | Name | Type | Description | | ---- | ---- | ----------- | | Auth via [GitHub Apps](https://docs.github.com/en/apps) | feature | Support GitHub Apps as an authentication option for GitHub repositories. | | Multiple `Freight` per `Stage` | feature | Permit `Stage`s to host multiple pieces of `Freight` from different `Warehouse`s. Different artifacts, or sets of artifacts, can be promoted through parallel pipelines with different/independent cadence. | | Production Readiness | chore | Prioritized stability of existing features.Paid down technical debt.**This does not mean v0.8.0 is production-ready. It means it is several steps closer to it.** | ### v0.7.0 | Name | Type | Description | | ---- | ---- | ----------- | | Multiple `Warehouse`s | feature | Improved UI support for displaying Freight from multiple `Warehouse`s. | | Manual `Freight` Creation | feature | Added UI feature for manual `Freight` creation. | | ECR/GAR Support | feature | Added multiple options for authenticating to image repositories in ECR and Google Artifact Registry, including support for EKS Pod Identity and GKE Workload Identity Federation. | | [Patch Promotions](https://github.com/akuity/kargo/issues/1250) | poc | Support a generalized option to promote arbitrary configuration (e.g. strings, files, and directories) to other paths of a GitOps repository. | | Production Readiness | chore | Prioritized stability of existing features. **This does not mean v0.7.0 is production-ready. It means it is several steps closer to it.** | ### v0.6.0 | Name | Type | Description | | ---- | ---- | ----------- | | Project Management | feature | Added user / role / permission management capabilities to the CLI and UI. | | Events | feature | Kargo emits noteworthy events as Kubernetes events. Events are also viewable in the UI. | | Production Readiness | chore | Prioritized stability of existing features. **This does not mean v0.6.0 is production-ready. It means it is several steps closer to it.** | ### v0.5.0 | Name | Type | Description | | ---- | ---- | ----------- | | `Warehouse` Rules/Filters | feature | Introduced optional path-based constraints on Git repository subscriptions. | | Credential Storage | refactor | Simplified and streamlined format and storage of repository credentials. | | Credential Management | feature | Added credential management capabilities to the CLI and UI. | | CLI Improvements | refactor | Overhauled the CLI to make the tree of sub-commands more intuitive, with improved consistency in usage and documentation from command to command. | | UI Improvements | feature | Achieved near-parity with CLI features. | ### v0.4.0 | Name | Type | Description | | ---- | ---- | ----------- | | `Warehouse` Rules/Filters | feature | Introduced optional tag-based constraints on Git repository subscriptions. | | Project Management | feature | Introduced `Project` CRD to simplify project initialization.Removed `PromotionPolicy` CRD and folded its functionality directly into the `Project` CRD. | ### v0.3.0 | Name | Type | Description | | ---- | ---- | ----------- | | GitHub PR-Based Promotion | feature | Pull request-based promotions are now supported on GitHub. | | Verifications | feature | `Stage` resources can now execute a user-defined verification process after a promotion. These can be defined using Argo Rollouts `AnalysisTemplate` resources, and executions take the form of `AnalysisRun` resources. | | Improved RBAC | feature | SSO user identities can now be mapped to Kubernetes `ServiceAccount` resources using annotations. | ### v0.2.0 | Name | Type | Description | | ---- | ---- | ----------- | | `Freight` CRD | feature | Freight changed from being a property of a `Stage`, to being its own `Freight` CRD. | | `Warehouse` CRD | feature | `Freight` production was decoupled from a pipeline's first `Stage` and now comes from a `Warehouse`. | | Kargo Render | breaking change | The Bookkeeper project was rebranded as Kargo Render -- a Kargo sub-project for rendering manifests. | --- ## Kargo Quickstart # Kargo Quickstart **Kargo** is an open-source project that streamlines how applications are promoted across environments, extending GitOps beyond deployment. ## The Pipeline You'll Create ![Pipeline: nginx repo → Warehouse → test → uat → prod](img/pipeline.svg) ## Spin Up a Local Cluster With Kargo Installed Pick your local Kubernetes setup. One command installs cert-manager, Argo CD, and Kargo. :::info All options require Helm v3.13.1+ ::: If you are a [Docker Desktop](https://www.docker.com/products/docker-desktop/) user, you can follow [these instructions](https://docs.docker.com/desktop/kubernetes/) to enable its built-in Kubernetes support. ```shell curl -L https://raw.githubusercontent.com/akuity/kargo/main/hack/quickstart/install.sh | sh ``` :::info This is one of the fastest paths to a local Kubernetes cluster, but be aware that Docker Desktop supports only a _single_ Kubernetes cluster. If that cluster reaches a state you are dissatisfied with, resetting it will remove not just Kargo-related resources, but _all_ your workloads and data. ::: [OrbStack](https://orbstack.dev/) is a fast, lightweight, drop-in replacement for Docker Desktop for macOS only. You can follow [these instructions](https://docs.orbstack.dev/kubernetes/) to enable its built-in Kubernetes support. ```shell curl -L https://raw.githubusercontent.com/akuity/kargo/main/hack/quickstart/install.sh | sh ``` :::info This is one of the fastest paths to a local Kubernetes cluster, but be aware that OrbStack supports only a _single_ Kubernetes cluster. If that cluster reaches a state you are dissatisfied with, resetting it will remove not just Kargo-related resources, but _all_ your workloads and data. ::: If you have any Docker-compatible container runtime installed (including native Docker, Docker Desktop, or OrbStack), you can easily launch a disposable cluster just for this quickstart using [kind](https://kind.sigs.k8s.io/#installation-and-usage). ```shell curl -L https://raw.githubusercontent.com/akuity/kargo/main/hack/quickstart/kind.sh | sh ``` :::info While this option is a bit more complex than using Docker Desktop or OrbStack directly, it offers the advantage of being fully-disposable. If your cluster reaches a state you are dissatisfied with, you can simply destroy it and launch a new one. ::: If you have any Docker-compatible container runtime installed (including native Docker, Docker Desktop, or OrbStack), you can easily launch a disposable cluster just for this quickstart using [k3d](https://k3d.io). ```shell curl -L https://raw.githubusercontent.com/akuity/kargo/main/hack/quickstart/k3d.sh | sh ``` :::info While this option is a bit more complex than using Docker Desktop or OrbStack directly, it offers the advantage of being fully-disposable. If your cluster reaches a state you are dissatisfied with, you can simply destroy it and launch a new one. ::: :::info If you are averse to piping a downloaded script directly into a shell, feel free to download the applicable script and inspect its contents prior to execution. Any approach you select should only: 1. Launch a new, local Kubernetes cluster, if applicable 1. Install cert-manager 1. Install Argo CD 1. Install Argo Rollouts 1. Install Kargo :::
Troubleshooting - **401 Unauthorized:** Ensure you are using **Helm v3.13.1 or later**. Earlier versions may not authenticate properly when pulling the Kargo chart. - **403 Forbidden:** This is commonly caused by Docker attempting to authenticate to `ghcr.io` with an expired token. The Kargo chart and container images are publicly accessible and do not require authentication. To resolve the issue, log out of `ghcr.io`: ```shell docker logout ghcr.io ``` - **Argo CD UI flashes on login with no error message:** If the Argo CD UI/dashboard briefly flashes or redirects back to the login screen without displaying an error, this may be caused by stale or corrupted browser cookies. Clear your browser cookies for `localhost` or open a new private/incognito window and try logging in again.
🚀 Argo CD 📦 Kargo URL: http://localhost:31080 Username: `admin` Password: `admin` URL: http://localhost:31081 Password: `admin` ✅ Argo CD and Kargo are now running locally. ## Set Up Your Demo Repository 1. Fork the sample repository: Go to [https://github.com/akuity/kargo-demo](https://github.com/akuity/kargo-demo) and click Fork. This repository contains application configurations (e.g. Kubernetes manifests) and is the source of truth for our demo cluster. In a real setup, you would create your own configuration repository from scratch. 1. Get a GitHub personal access token (PAT): Kargo will push changes to your fork for **test**, **uat**, and **prod** environments. You need a PAT with **write access** to your fork.
Need a GitHub personal access token (PAT)? 1. Navigate to GitHub → Settings → Developer settings → Personal access tokens → Fine-grained tokens, then click Generate new token. 1. Under Repository access, select Only select repositories and choose your fork. 1. Under Permissions → Repository permissions, set Contents to Read and write. 1. Click Generate token and copy it — it won't be displayed again after navigating away from that page.
1. Set environment variables: ```shell export GITOPS_REPO_URL=https://github.com//kargo-demo export GITHUB_USERNAME= export GITHUB_PAT= ``` :::info Why Kargo writes to Git Kargo follows GitOps principles: Your cluster's desired state always comes from Git. Kargo promotes artifacts by committing updated configurations to Git. Argo CD picks it up and syncs the cluster. Git becomes a full audit trail of every promotion. :::
What's in the `kargo-demo` repository? You can explore the repository and see that the `main` branch contains common configuration in a `base/` directory as well as stage-specific overlays in `stages//` directories. - This layout is typical of a GitOps repository using [Kustomize](https://kustomize.io/) for configuration management of Kubernetes manifests and is not specific to Kargo. - Kargo also works just as well with [Helm](https://helm.sh).
## Create Argo CD Applications For Each Stage We’ll use an Argo CD `ApplicationSet` to create and manage three `Application`s, deploying the sample app to **test**, **uat**, and **prod** namespaces, each with its own configuration. ```yaml {18,23} cat < What you'll see in Argo CD When you visit the [Argo CD dashboard](http://localhost:31080), the branches referenced by the `targetRevision` fields do not exist yet, and the `Application`s will be out of sync. Kargo will create them on first promotion. ![Argo CD Dashboard](img/argo-dashboard.png) ## Create Your Kargo Project and Pipeline Run the following command to set up your pipeline. This will create: - A `Warehouse` that polls the public ECR registry for new versions of the Nginx image - A `PromotionTask` that defines a reusable promotion process - Three `Stage` resources that define how Freight moves through your pipeline ```yaml {3,8,21,28,33,44,53,56,63,67,72,75,84,86,89,103,105,108,114,123,125,128} cat < Download the Kargo CLI for your operating system and CPU architecture from the [Kargo Dashboard's Downloads page](http://localhost:31081/downloads): ![CLI Tab in Kargo UI](./img/cli-installation.png) Rename the downloaded binary to `kargo` (or `kargo.exe` for Windows) and move it to a location in your file system that is included in the value of your `PATH` environment variable. Log in: ```shell kargo login http://localhost:31081 \ --admin \ --password admin ``` To create Kargo resources, use the following command: ```yaml cat < Open the [Kargo Dashboard](http://localhost:31081/) and select the `kargo-demo` project. You should see the pipeline, and `Freight` should appear in the upper left after a few seconds.
What you'll see in Kargo ![Kargo Project View](img/kargo-dashboard-projects.png)
✅ Pipeline created, and `Freight` is available to promote. ## Promote Freight to the Test Stage In the Kargo Dashboard: 1. Locate the `Freight` in the timeline at the top of the screen. 1. Drag it using the ⋮⋮ handle. 1. Drop it into the **test** `Stage`.
Alternative: Promote from the `Stage` Menu In the **test** `Stage`, click the truck icon (🚚) in the header. 1. Select Promote. 1. Choose the `Freight` you want to promote. 1. Click Promote to confirm.
A summary of the `Promotion` will pop up and will be updated in real-time as the steps of the promotion process complete. Once the steps have completed, the `Promotion`'s status will change to Succeeded. ![Kargo Promotion View](img/kargo-promotion-view.png)
What `Freight` is deployed to what `Stage`? Every piece of `Freight` in the timeline is color-coded to indicate which `Stage`s (if any) are actively using it. In this example, `Freight` matches the **test** `Stage`’s color once it has been successfully promoted.
What happened behind the scenes? When you visit your fork at `https://github.com//kargo-demo`, you'll see: - Kargo created a **stage/test** branch - It read the latest manifests from `main`, ran `kustomize edit set image` and `kustomize build` in `stages/test/` - The resulting manifests were committed to the stage-specific branch — the same branch referenced by the **test** Argo CD `Application`’s `targetRevision` field **Best Practice:** The Kargo team recommends using stage-specific branches.
✅ After the `Freight` passes the health checks, you'll see a ❤️ on the **test** node. Click the `Freight` to confirm it shows Verified in **test** which will unlock it for promotion to **uat**. :::warning Kargo can intermittently be slow to recognize health status changes in an Argo CD `Application`, which can prevent a `Stage` that interacts with it from being counted as healthy. If your **test** `Stage` shows an unknown health status for a prolonged period, expand it by clicking the icon with three lines, then click Refresh in the upper right of the page. This will force any changes in the `Application`'s health status to be observed, allowing the `Stage` itself to be counted as healthy. This intermittent slowness will be addressed in an upcoming release. ::: ## Promote to UAT and then Production Repeat the same steps for **uat**, then **prod**: (The `Freight` node will progressively color-match each stage as it passes through.) 1. Click the truck icon on each `Stage`. 1. Select `Freight`. 1. Click Promote. :::info `Freight` cannot be promoted to the **prod** `Stage` until **uat** verification has passed and the `Stage` reaches a **Healthy** state. Verification checks may take a few minutes to reconcile. ::: 🧪 test 🔬 uat 🚀 prod http://localhost:32080 http://localhost:32081 http://localhost:32082 ✅ **All stages promoted!** 🎉
Why can’t I promote directly from **test** to **prod**? Unlike the **test** `Stage`, which subscribes to a `Warehouse` that polls an image repository in ECR, the **uat** and **prod** `Stage`s subscribe to other, _upstream_ `Stage`s, forming a promotion pipeline: 1. `uat` subscribes to `test` 2. `prod` subscribes to `uat` This means `Freight` must flow through each `Stage` in order: **test** → **uat** → **prod**.
Exploring the **Kargo Dashboard** The Kargo Dashboard gives you visibility into how `Freight` moves through your environments. Within a `Stage`, you can explore: - **Promotions** – See when `Freight` was promoted, by whom, and to which `Stage`. - **Verifications** – View the status and logs of verification steps. - **Freight History** – Track which versions have flowed through the environment over time. - **Settings** – The defined behavior of the `Stage`: what it subscribes to and how promotions and verifications are defined. - **Live Manifest** – The current state of the `Stage` resource as it exists in the cluster. If things go wrong, the live manifest provides more depth of detail than UI elements. Together, these views provide a clear audit trail and real-time insight into your promotion pipeline.
## Cleaning Up Congratulations! You've successfully set up your first promotion pipeline! Now let's clean up! Docker Desktop supports only a _single_ Kubernetes cluster. If you are comfortable deleting not just Kargo-related resources, but _all_ your workloads and data, the cluster can be reset from the Docker Desktop Dashboard. If, instead, you wish to preserve non-Kargo-related workloads and data, you will need to manually uninstall Kargo and its prerequisites: ```shell curl -L https://raw.githubusercontent.com/akuity/kargo/main/hack/quickstart/uninstall.sh | sh ``` OrbStack supports only a _single_ Kubernetes cluster. If you are comfortable deleting not just Kargo-related resources, but _all_ your workloads and data, you can destroy the cluster with: ```shell orb delete k8s ``` If, instead, you wish to preserve non-Kargo-related workloads and data, you will need to manually uninstall Kargo and its prerequisites: ```shell curl -L https://raw.githubusercontent.com/akuity/kargo/main/hack/quickstart/uninstall.sh | sh ``` Simply destroy the cluster: ```shell kind delete cluster --name kargo-quickstart ``` Simply destroy the cluster: ```shell k3d cluster delete kargo-quickstart ``` --- ## Basic Installation Installing Kargo's __cluster-side components__ (e.g. controllers and API server) with __default configuration__ is quick and easy. :::info[Not what you were looking for?] If you're looking for a more customized installation of cluster-side components, refer to [Advanced Installation](./20-advanced-installation/10-advanced-with-helm.md) for steps, and [Common Configurations](./20-advanced-installation/30-common-configurations.md) for guidance on configuring Kargo to address common operational concerns. If you're a developer looking for instructions for installing the Kargo CLI, see [Installing the CLI](../50-user-guide/05-installing-the-cli/index.md). ::: :::caution The default configuration is suitable _only_ for trying Kargo in a local cluster that is not internet-facing. For detailed instructions for a secure installation, refer to [Secure Configuration](./40-security/10-secure-configuration.md). ::: ## Prerequisites You will need: - [Helm](https://helm.sh/docs/): These instructions were tested with v3.13.1. - A Kubernetes cluster with [cert-manager](https://cert-manager.io/) pre-installed. :::note cert-manager is not an absolute dependency, but _is_ required for installation with the default configuration. ::: The following dependencies are optional, but highly recommended to be pre-installed in your Kubernetes cluster: - [Argo CD](https://argo-cd.readthedocs.io) :::info Kargo works best when paired with Argo CD. ::: - [Argo Rollouts](https://argoproj.github.io/argo-rollouts/) :::info Kargo's verification feature makes use of Argo Rollouts `AnalysisTemplate` and `AnalysisRun` resources internally. **Kargo does not require that your application deployments also use Argo Rollouts.** ::: These instructions were tested with: - Kubernetes: v1.29.3 - cert-manager: v1.16.1 - Argo CD: v2.13.0 - Argo Rollouts: v1.7.2 ## Installation Steps 1. Generate a password and a signing key. There are no default values for these two fields, so you _must_ provide your own. Recommended commands for generating a complex password and signing key, and for hashing the password as required are: ```console pass=$(openssl rand -base64 48 | tr -d "=+/" | head -c 32) echo "Password: $pass" hashed_pass=$(htpasswd -bnBC 10 "" $pass | tr -d ':\n') signing_key=$(openssl rand -base64 48 | tr -d "=+/" | head -c 32) ``` The above commands will leave you with values assigned to `$hashed_pass` and `$signing_key`. These will be used in the next step. 1. Install Kargo with default configuration and your chosen admin account password: ```shell helm install kargo \ oci://ghcr.io/akuity/kargo-charts/kargo \ --namespace kargo \ --create-namespace \ --set api.adminAccount.passwordHash=$hashed_pass \ --set api.adminAccount.tokenSigningKey=$signing_key \ --wait ``` ## Troubleshooting ### Kargo installation fails with a `401` Verify that you are using Helm v3.13.1 or greater. ### Kargo installation fails with a `403` It is likely that Docker is configured to authenticate to `ghcr.io` with an expired token. The Kargo Helm chart and images are publicly accessible, so this issue can be resolved simply by logging out: ```shell docker logout ghcr.io ``` --- ## Advanced Installation with Helm This section outlines the general procedure for customizing configuration of Kargo's cluster-side components when installing them via Helm. 1. Extract the default values from the Helm chart and save it to a convenient location. In the example below, we save it to `kargo-values.yaml` ```shell helm inspect values \ oci://ghcr.io/akuity/kargo-charts/kargo > kargo-values.yaml ``` 1. Edit and save the values. :::info You will find this configuration file contains helpful comments for every option, so specific options are not covered in detail here. Detailed information about available options can also be found in the [Kargo Helm Chart's README.md](https://github.com/akuity/kargo/tree/main/charts/kargo). Additionally, for important security-related configuration, check the [Secure Configuration Guide](../40-security/10-secure-configuration.md). ::: 1. Proceed with installation, using your modified values: ```shell helm install kargo \ oci://ghcr.io/akuity/kargo-charts/kargo \ --namespace kargo \ --create-namespace \ --values kargo-values.yaml \ --wait ``` --- ## Installation with Argo CD This document outlines a few generalized approaches to installing and managing Kargo using Argo CD. :::note This section assumes that you have already installed any dependencies or prerequisites required for running Kargo on a Kubernetes cluster. Please refer to [Basic Installation](../10-basic-installation.md#prerequisites) for more details. ::: All methods described here will involve deploying Kargo using an Argo CD `Application` resource that is configured to obtain Kargo's Helm chart _directly_ from its official repository. We will demonstrate a variety of ways to specify your own configuration values using `api.adminAccount.passwordHash` and `api.adminAccount.tokenSigningKey` as examples since you are _required_ to provide values for these anyway (unless [the admin account is disabled instead](../40-security/10-secure-configuration.md#disabling-the-admin-account)), but the techniques shown here can be applied to any configurable elements of the Kargo Helm chart. :::info Detailed information about available options can be found in the [Kargo Helm Chart's README.md](https://github.com/akuity/kargo/tree/main/charts/kargo). For important security-related configuration, refer to the [Secure Configuration Guide](../40-security/10-secure-configuration.md). ::: Recommended commands for generating a complex password and signing key, and for hashing the password as required are: ```console pass=$(openssl rand -base64 48 | tr -d "=+/" | head -c 32) echo "Password: $pass" echo "Password Hash: $(htpasswd -bnBC 10 "" $pass | tr -d ':\n')" echo "Signing Key: $(openssl rand -base64 48 | tr -d "=+/" | head -c 32)" ``` :::note Methods of securing the admin account are explored in greater detail [here](../40-security/10-secure-configuration.md#securing-the-admin-account). ::: ## `spec.source.helm.parameters` The most straightforward way to specify chart configuration options is by using the `Application`'s `spec.source.helm.parameters` field: ```yaml apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: kargo namespace: argocd spec: project: default destination: namespace: kargo server: https://kubernetes.default.svc source: repoURL: ghcr.io/akuity/kargo-charts chart: kargo targetRevision: helm: parameters: - name: api.adminAccount.passwordHash # Note: A bcrypt-hashed password will contain `$` characters that # MUST each be escaped as `$$` value: - name: api.adminAccount.tokenSigningKey value: syncPolicy: automated: prune: true selfHeal: true syncOptions: - CreateNamespace=true ``` ## `spec.source.helm.values` Alternatively, instead of using `spec.source.helm`'s `parameters` field, you can use either of its `values` or `valuesObject` fields to specify configuration options for the chart: ```yaml apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: kargo namespace: argocd spec: project: default destination: namespace: kargo server: https://kubernetes.default.svc source: repoURL: ghcr.io/akuity/kargo-charts chart: kargo targetRevision: helm: valuesObject: api: adminAccount: passwordHash: tokenSigningKey: tokenTTL: 24h controller: logLevel: DEBUG syncPolicy: automated: prune: true selfHeal: true syncOptions: - CreateNamespace=true ``` ## Multi-Source Argo CD Application The most advanced method covered here __is nevertheless our recommendation because it aligns best with GitOps principles.__ Use an `Application` with [multiple sources](https://argo-cd.readthedocs.io/en/stable/user-guide/multiple_sources) to reference _both_ the Kargo Helm chart repository and a `values.yaml` file of your own from your own Git repository. :::info An added benefit to this approach is that if you have other resources to include in the Kargo installation, such as [`SealedSecret`s](https://github.com/bitnami-labs/sealed-secrets) or [`ExternalSecret`s](https://external-secrets.io/latest/), they also can be obtained from your own Git repository using the second source. ::: In the configuration below, the second source (the one with `repoURL` pointed at your own Git repository) is assigned a `ref` of `values`. This permits content from that repository (in particular, a `values.yaml` file) to be referenced by the _other_ source. We _also_ use the `path` parameter as usual to direct the second source to the location of additional manifests to include in the `kargo` namespace along with the chart: ```yaml apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: kargo namespace: argocd spec: project: default destination: namespace: kargo server: https://kubernetes.default.svc sources: - repoURL: ghcr.io/akuity/kargo-charts chart: kargo targetRevision: helm: valueFiles: - $values/kargo/values.yaml - repoURL: https://github.com/example/repo.git targetRevision: main ref: values path: kargo/additional-manifests syncPolicy: automated: prune: true selfHeal: true syncOptions: - CreateNamespace=true ``` --- ## Common Configurations This document outlines common advanced installation scenarios and configurations for Kargo. :::info For complete parameter documentation, refer to the [chart documentation](https://github.com/akuity/kargo/blob/main/charts/kargo/README.md). ::: :::info For more information on how to apply these configurations, see the advanced installation guides for [Helm](10-advanced-with-helm.md) and [Argo CD](20-advanced-with-argocd.md). ::: ## Standard Kubernetes Configuration For common Kubernetes configurations, like setting a `nodeSelector`, `labels`, `annotations`, `affinity`, `tolerations`, etc. Kargo supports both global and resource-specific configurations. For example, to add `labels` to all resources created by Kargo, the following configuration can be used: ```yaml global: labels: label.example.com/key: value ``` To add `labels` to a specific component (the `kargo-controller` in this example), the following configuration can be used: ```yaml controller: labels: label.example.com/key: value ``` :::note For a full list of supported configurations, refer to the [Global Parameters](https://github.com/akuity/kargo/blob/main/charts/kargo/README.md#global-parameters) or the component-specific parameter sections in the chart documentation. ::: ## API Configuration Kargo supports a number of API-related configurations that can be set at installation time. These configurations are used to control the behavior of Kargo's API server and its web-based user interface. :::info The sections below outline common configurations for the API server. For a full list of supported configurations, refer to the [API Parameters](https://github.com/akuity/kargo/blob/main/charts/kargo/README.md#api). section in the chart documentation. ::: ### API Host By default, the API server host (i.e. the domain or IP address that the API is accessible at) is set to `localhost`. This host is used for generation of `Ingress` resources, certificates, the OpenID Connect issuer and callback URLs, and any URLs that are exposed to users. To configure the API host, set the following configuration: ```yaml api: host: kargo.example.com ``` :::note The host is allowed to include a port number, e.g. `kargo.example.com:8080`, but should not include a protocol (e.g. `http://` or `https://`) as this is automatically inferred from other configuration options. ::: ### API Service By default, Kargo will create a `Service` resource for the API server with the type `ClusterIP`, which is only accessible within the cluster or through [port forwarding](https://kubernetes.io/docs/tasks/access-application-cluster/port-forward-access-application-cluster/). :::caution Changing the API server service type can expose the API server to the internet in an insecure manner. Refer to the [Secure Configuration](../40-security/10-secure-configuration.md) section for more information on securing the API server. ::: :::info Instead of making use of a `Service` resource, you can also expose the API server using an [`Ingress` resource](#api-ingress). ::: If you want to expose the API server to the internet, but do not want to make use of an `Ingress` resource, you can change the service type: ```yaml api: service: type: LoadBalancer ``` In addition, when using a `LoadBalancer` or `NodePort` service type, you can configure the service to use a specific port: ```yaml api: service: type: LoadBalancer port: 443 ``` ### API TLS By default, Kargo will enable TLS directly on the API server using a self-signed certificate issued by [cert-manager](https://cert-manager.io/). :::note When making use of the self-signed certificate option, cert-manager must be installed in the cluster. ::: To supply your own certificate, set the following configuration: ```yaml api: tls: selfSignedCert: false ``` When setting `selfSignedCert` to `false`, a [TLS `Secret`](https://kubernetes.io/docs/concepts/configuration/secret/#tls-secrets) with the name `kargo-api-cert` is expected to exist in the same namespace as the API server. #### Terminating TLS If you have opted out of all the chart's built-in TLS options, but TLS is terminated upstream from the API server by a component such as a reverse proxy, load balancer, or `Ingress` controller using a wildcard certificate, Kargo will be unable to infer from its own configuration that any generated URLs should use HTTPS. For such cases, you can explicitly declare that TLS is terminated upstream and Kargo will sort out the rest: ```yaml api: tls: enabled: false terminatedUpstream: true ``` :::info If you exercise any of the chart's built-in TLS configuration options, setting `terminatedUpstream` is never required. ::: ### API Ingress :::info Instead of making use of an `Ingress` resource, you can also expose the API server using a `LoadBalancer` or `NodePort` service type. Refer to the [API Service](#api-service) section for more information. ::: By default, Kargo will not create an `Ingress` resource for the API server and will only be accessible within the cluster or through the API server's [`Service` resource](#api-service). To expose the API server to the internet, an `Ingress` resource can be created. :::caution Enabling the API server Ingress without proper configuration can expose the API server to the internet in an insecure manner. Refer to the [Secure Configuration](../40-security/10-secure-configuration.md) section for more information on securing the API server. ::: To configure the API server to use an `Ingress` resource, set the following configuration: ```yaml api: ingress: enabled: true ingressClassName: nginx ``` #### Ingress TLS By default, Kargo will enable TLS on the `Ingress` resource using a self-signed certificate using [cert-manager](https://cert-manager.io). :::note When making use of the self-signed certificate option, cert-manager must be installed in the cluster. ::: To supply your own certificate, set the following configuration: ```yaml api: ingress: tls: selfSignedCert: false ``` When setting `selfSignedCert` to `false`, the `Ingress` resource expects a [TLS `Secret`](https://kubernetes.io/docs/concepts/services-networking/ingress/#tls) with the name `kargo-api-ingress-cert` to exist in the same namespace as the API server. ## Git Configuration Kargo supports a number of Git-related configurations that can be set at installation time. ### Default Commit Author To set the default commit author Kargo will use when committing changes using the [`git-commit` Promotion step](../../50-user-guide/60-reference-docs/30-promotion-steps/git-commit.md), the following configuration can be set (shown here with the default values): ```yaml controller: gitClient: name: Kargo email: no-reply@kargo.io ``` ### Signing Commits To sign commits made by Kargo, a reference to a `Secret` (in the same namespace as Kargo is installed to) containing a signing key can be configured. The key should be a GPG key in ASCII-armored format without a passphrase, under the key `signingKey`. ```yaml apiVersion: v1 kind: Secret metadata: name: kargo-git-signing-key type: Opaque data: signingKey: ``` The `Secret` can then be referenced in the Kargo configuration. ```yaml controller: gitClient: signingKeySecret: name: kargo-git-signing-key type: gpg ``` :::note When using a signing key, the `gitClient.name` and `gitClient.email` configuration options must match the name and email associated with the GPG key. ::: ### Git Client Configuration As an alternative to configuring the Git client at install time via the Helm chart, the same settings -- committer identity and signing key -- can be configured at runtime through the `ClusterConfig` resource. This is especially useful for managed environments where the Helm values are not directly controlled by the Kargo operator. When configured, `ClusterConfig` settings take precedence over the Helm-provisioned defaults. **Step 1: Create a Secret** Create a system-level Secret (via the Settings page's **System Secrets** tab, the CLI, or `kubectl`) containing the GPG signing key: ```yaml apiVersion: v1 kind: Secret metadata: name: my-gpg-key namespace: type: Opaque data: signingKey: ``` **Step 2: Reference it from ClusterConfig** Edit the `ClusterConfig` resource (via the Settings page's **Cluster Config** tab or `kubectl`) to reference the Secret: ```yaml apiVersion: kargo.akuity.io/v1alpha1 kind: ClusterConfig metadata: name: cluster spec: gitClient: name: Kargo email: no-reply@kargo.io signingKeySecret: name: my-gpg-key ``` The next promotion that clones a Git repository will pick up the new configuration automatically -- no restart is required. :::note When using a signing key, the `gitClient.name` and `gitClient.email` values must match the name and email associated with the GPG key. ::: ### Push Integration Policy When the [`git-push`](../../50-user-guide/60-reference-docs/30-promotion-steps/git-push.md) promotion step pushes to a remote branch that has new commits the local branch doesn't have, it must integrate those remote changes first. The push integration policy controls how this integration is performed. Four options are available, forming a spectrum from least to most conservative: - `AlwaysRebase`: Unconditionally uses `pull --rebase` to integrate remote changes. This preserves linear history but may re-sign commits that Kargo did not author or strip existing signatures from commits. - `RebaseOrMerge`: Uses `pull --rebase` when a signature-trust analysis determines it is safe, and falls back to a merge commit otherwise. This preserves linear history when possible without undermining trust. - `RebaseOrFail`: Uses `pull --rebase` when the signature-trust analysis determines it is safe, and fails the push step otherwise. This puts constraints on promotion process design and treats the failure scenario as worthy of human investigation. - `AlwaysMerge`: Unconditionally creates a merge commit to integrate remote changes. This is the most conservative option — it never touches existing commits and always preserves original signatures. :::caution The current default is `AlwaysRebase`. Starting with v1.12.0, the default will change to `RebaseOrMerge`. If you rely on the current behavior, set the policy explicitly before upgrading. ::: ```yaml controller: gitClient: pushIntegrationPolicy: RebaseOrMerge ``` :::info For more information about the security implications of this setting, see [Secure Configuration](../40-security/10-secure-configuration.md#push-integration-policy). ::: ### GitHub Push Settings The [`github-push`](../../50-user-guide/60-reference-docs/30-promotion-steps/github-push.md) promotion step has two system-level settings: #### Maximum Revisions The maximum number of commits that `github-push` will replay via the GitHub API in a single push. This is a safety guardrail against accidentally replaying large numbers of commits, which would consume API rate limit budget. The default is 10. ```yaml controller: githubPush: maxRevisions: 20 ``` #### Verify Untrusted Commits By default, `github-push` only withholds author/committer information (which results in a verified commit) for commits signed by a trusted key. When this option is enabled, author/committer information is withheld for __all__ commits, causing GitHub to sign every commit regardless of trust. ```yaml controller: githubPush: verifyUntrustedCommits: true ``` :::warning This option manufactures trust where none exists. It tells GitHub to vouch for commits that Kargo cannot independently verify. Enable this only if your organization values the verified badge for policy compliance and accepts that verification no longer reflects genuine cryptographic trust. ::: ## Argo CD Configuration Kargo supports a number of Argo CD-related configurations that can be set at installation time. ### Disabling the Argo CD Integration By default, Kargo will enable the Argo CD integration, which configures Kargo to work with `Application` resources created by Argo CD. This can be disabled as follows: ```yaml controller: argocd: integrationEnabled: false ``` When disabled, the controller will not watch Argo CD `Application` resources and disable Argo CD specific features. :::info If the integration is enabled without Argo CD installed, the controller will disable the Argo CD specific features on boot. Explicitly disabling is preferred if this integration is not desired, as it will grant fewer permissions to the controller. ::: ### Argo CD Namespace By default, Kargo expects Argo CD to be installed to the `argocd` namespace, which is also the default namespace it will use for `Application` resources if a namespace is not specified in the [`argocd-update` Promotion step](../../50-user-guide/60-reference-docs/30-promotion-steps/argocd-update.md). If you want to use a different default namespace, this can be configured as follows: ```yaml controller: argocd: namespace: argocd ``` In certain cases, you may not want Kargo to be able to look up Argo CD resources in other namespaces than the configured default. This can be enforced by setting the following configuration: ```yaml controller: argocd: watchArgocdNamespaceOnly: true ``` When enabled, the controller will only watch for `Application` resources in the namespace specified by `controller.argocd.namespace`. ## Argo Rollouts Configuration ### Disabling the Argo Rollouts Integration By default, Kargo will enable the Argo Rollouts integration, which configures Kargo to work with `AnalysisRun` resources created as part of the [verification feature](../../50-user-guide/20-how-to-guides/40-working-with-stages.md#verification). This can be disabled as follows: ```yaml api: rollouts: integrationEnabled: false controller: rollouts: integrationEnabled: false ``` When disabled, the controller will not reconcile Argo Rollouts `AnalysisRun` resources and attempts to verify Stages via `Analysis` will fail. :::info If the integration is enabled without Argo Rollouts installed, the controller will disable the Argo Rollouts specific features on boot. Explicitly disabling is preferred if this integration is not desired, as it will grant fewer permissions to the controller. ::: ### Logs from Job Metrics For those utilizing Argo Rollouts integration for verifications, [job metrics](https://argoproj.github.io/argo-rollouts/analysis/job/) stand out as an especially useful feature because they are implemented as Kubernetes [`Job`s](https://kubernetes.io/docs/concepts/workloads/controllers/job/), which give users the flexibility to define any arbitrary post-promotion tests they'd like to run against a `Stage` by simply providing appropriate `Job` specs [as described here](../../50-user-guide/20-how-to-guides/60-verification.md#configuring-analysistemplates). In cases such as these, access to logs produced by the `Job`'s underlying `Pod` is helpful for debugging purposes and for understanding results. Since it's common for multiple Kargo controllers to be deployed to many different clusters, such logs need to be aggregated in a centralized location for the API server to access them and, in-turn, stream them to the Kargo UI or CLI. Rather than build support for many different logging stacks, Kargo has settled on a "lowest common denominator" approach: The API server can stream any logs that it can access via an HTTP GET request. To facilitate this, operators may, at the time of installation, provide a URL template that the API server can use to construct the URL for any job metric logs as a function of `project` (name), `namespace` (always equal to the `Project`'s name), `shard`, `stage` (name), `analysisRun` (name), `metricName`, `jobNamespace`, `jobName`, and `container` (name). A token can be specified by referencing a Kubernetes `Secret` that is managed "out of band." HTTP headers may also be specified, and may reference the token if one is provided. Example: ```yaml api: rollouts: integrationEnabled: true logs: enabled: true urlTemplate: https://logs.kargo.example.com/${{project}}/${{analysisRun}}/${{jobName}}/${{container}} tokenSecret: name: kargo-logs-token key: token httpHeaders: Authentication: "Bearer ${{ token }}" ``` :::note This "lowest common denominator" approach to streaming job metric logs does leave it as an exercise for the Kargo administrator to arrange for the forwarding and storage of applicable logs. Users of Kargo via the [Akuity Platform](https://akuity.io/akuity-platform), however, will have this seamlessly handled for them. ::: :::note For more information, refer to the [chart documentation](https://github.com/akuity/kargo/blob/main/charts/kargo/README.md). ::: ## Warehouse Performance ### Tuning Reconciliation Intervals If your cluster contains many `Warehouse` resources, which periodically poll artifact repositories, or if developers have [configured any of those `Warehouse`s poorly](../../50-user-guide/20-how-to-guides/30-working-with-warehouses.md#performance-considerations), you may wish to reduce the frequency with which all `Warehouse`s execute their artifact discovery processes (i.e. You may wish to _increase_ the minimum polling interval.) :::info Developers can tune this interval on individual `Warehouse` resources, but the effective interval for any `Warehouse` will be the _greater_ of any specified there and the minimum specified here. i.e. Developers cannot configure a `Warehouse`'s artifact discovery process to run more frequently than you permit. ::: :::note If you do this, you will increase the average time required for every `Warehouse` to notice new artifacts. You can compensate for this by configuring `Warehouse` artifact discovery processes to be [triggered by webhooks](../35-cluster-configuration.md#triggering-artifact-discovery-using-webhooks). ::: ```yaml controller: reconcilers: warehouses: minReconciliationInterval: 15m ``` ### Tuning Concurrent Reconciliation Limits By default, Kargo will reconcile up to four resources of the same kind concurrently (e.g. four `Stage` resources at a time). It is possible to tune this limit by setting a new global value, or by setting a limit per resource kind: ```yaml controller: reconcilers: # Global setting maxConcurrentReconciles: 4 managementController: # Kind specific setting namespaces: maxConcurrentReconciles: 2 ``` :::note For a list of resource kinds that can be configured, refer to the [chart documentation](https://github.com/akuity/kargo/blob/main/charts/kargo/README.md). ::: ### Image Metadata Caching Kargo can cache container image metadata more aggressively using image tags as keys (instead of digests). This can significantly improve performance by reducing API calls to container image registries, but is safest to do when images are known to use "immutable" tags (i.e. existing tags are never overwritten). Operators may select one of four policies regarding caching image metadata by tag: - `Forbid`: Container image subscriptions may not cache image metadata by tag. This is silently enforced. Subscriptions that opt into caching image metadata by tag will be treated as if they had not. - `Allow` Individual container image subscriptions may choose whether to cache image metadata by tag. This option leaves the decision in the hands of developers. :::info For purposes of backwards compatibility, `Allow` is the default policy. ::: - `Require`: Container image subscriptions MUST explicitly opt into caching image metadata by tag or their artifact discovery processes will fail. Requiring explicit opt-in is tantamount to soliciting acknowledgement from developers that caching image metadata by tag is in effect. This is intended to minimize the possibility of surprise at any stale results from an image discovery process. This option sacrifices some small degree of usability for safety. - `Force`: Caching image metadata is silently enforced. Subscriptions that to not opt into caching image metadata by tag will be treated as if they had. :::info This is the recommended policy in an immutable-tags-only environment. ::: Example configuration to allow (but not require) individual container image subscriptions to cache image metadata by tag: ```yaml controller: images: cache: cacheByTagPolicy: Allow ``` Example configuration to silently enforce caching image metadata by tag: ```yaml controller: images: cache: cacheByTagPolicy: Force ``` For more information on how to use this feature, see the [Performance Considerations](../../50-user-guide/20-how-to-guides/30-working-with-warehouses.md#caching-image-metadata-by-tag) section of the user guide. ## Garbage Collection Kargo includes a garbage collector that automatically removes old `Freight` and `Promotion` resources. The garbage collector offers a number of configuration options to manage the retention of these resources. ### Disabling the Garbage Collector By default, the garbage collector is enabled. To disable it, set the following configuration: ```yaml garbageCollector: enabled: false ``` :::caution Disabling the garbage collector will result in old `Freight` and `Promotion` resources accumulating in the cluster. This can lead to increased resource usage and potential performance issues. Therefore, this is typically not recommended and should only be done with caution. ::: ### Scheduling the Garbage Collection By default, the garbage collector runs every 24 hours. This can be configured using a cron expression: ```yaml garbageCollector: schedule: "0 0 * * *" ``` ### Retention Settings The garbage collector offers settings to control the retention of `Promotion` and `Freight` resources. #### Promotion Retention For each `Stage`, the garbage collector identifies the oldest `Promotion` that is still in a non-terminal phase (i.e., not `Succeeded` or `Failed`). It then retains up to `maxRetainedPromotions` that are _older_ than this non-terminal `Promotion`. Any `Promotion` resources beyond this retention limit are eligible for deletion, but will only be deleted if they have reached the `minPromotionDeletionAge`. This means the actual number of retained `Promotion` resources may exceed `maxRetainedPromotions` if some would-be-deleted `Promotion` resources have not yet reached the minimum age. If all `Promotion` resources for a `Stage` are in a terminal phase, the garbage collector simply retains the most recent `maxRetainedPromotions` and considers the rest for deletion (subject to the minimum age criterion). #### Freight Retention For each `Warehouse`, the garbage collector identifies the oldest `Freight` that is still in use by any `Stage`. It then retains up to `maxRetainedFreight` that are _older_ than this in-use `Freight`. Any `Freight` resources beyond this retention limit are eligible for deletion, but will only be deleted if they have reached the `minFreightDeletionAge`. This means the actual number of retained `Freight` resources may exceed `maxRetainedFreight` if some would-be-deleted `Freight` resources have not yet reached the minimum age. If no `Freight` from a `Warehouse` is currently in use, the garbage collector simply retains the most recent `maxRetainedFreight` and considers the rest for deletion (subject to the minimum age criterion). ```yaml garbageCollector: # The minimum age a Promotion must be before considered eligible for garbage # collection. This is a duration string (e.g. 336h for 14 days). minPromotionDeletionAge: 336h # The ideal maximum number of Promotions OLDER than the oldest Promotion in a # non-terminal phase (for each Stage) that may be spared by the garbage # collector. maxRetainedPromotions: 20 # The minimum age Freight must be before considered eligible for garbage # collection. This is a duration string (e.g. 336h for 14 days). minFreightDeletionAge: 336h # The ideal maximum number of Freight OLDER than the oldest still in use # (from each Warehouse) that may be spared by the garbage collector. maxRetainedFreight: 20 ``` :::note `Promotion` resources are only deleted if they are in a terminal state (i.e. `Succeeded` or `Failed`). `Freight` resources are only deleted if they are not actively in use by any `Stage`. In both cases, this holds true even if the resource is older than the minimum deletion age. ::: --- ## Architecture and Topology ## Overview This document describes Kargo's architecture and topology, explaining how components interact in both standalone and distributed configurations. Understanding this architecture will help you grasp how Kargo achieves scalability and security through its agent-based design. This content assumes you have a strong grasp of Kubernetes fundamentals. ## Understanding Kargo's Data Model Kargo's primary component is a custom Kubernetes controller. All of Kargo's data is represented as Kubernetes resources -- instances of custom resource definitions (CRDs) such as `Project`s, `Warehouse`s, `Stage`s, `Freight`, and `Promotion`s. :::info You can read more about the concepts behind these CRDs in the [Core Concepts](https://docs.kargo.io/user-guide/core-concepts/) section of the documentation. ::: A Kubernetes cluster running most of Kargo's components and hosting all its data is called a **Kargo control plane**. ## Components The following components make up a complete Kargo installation: ### API Server **Runs in:** The control plane The API server provides the back end services that both the Kargo [UI](#ui) and [CLI](#cli) depend upon. It also serves the static content for the UI, which is a single-page application (SPA). :::info This is one of only two internet-facing components in Kargo. ::: ### UI **Runs in:** Your browser The UI is the primary interface for interacting with Kargo. It's accessed by navigating to the [API server's](#api-server) base address in a web browser. ### CLI **Runs in:** A terminal The `kargo` CLI provides a secondary interface to Kargo, offering rich, domain-specific functionality beyond what's easily accomplished with `kubectl` alone. :::info Because Kargo's data are Kubernetes resources, it's also practical and not uncommon for users to interact with Kargo using the standard `kubectl` CLI. ::: ### Controller **Runs in:** The control plane and/or other clusters *This is Kargo's most important component.* Controller instances perform the core work of monitoring repositories for new artifacts and executing user-defined promotion processes. In Kargo's simplest configuration, just a single controller instance runs in the control plane. In more complex configurations, multiple controller instances are distributed across many clusters, each servicing a specific subset (shard) of data that lives in the control plane. **Why distribute controllers?** Several reasons: - **Firewalls:** Some artifact repositories may only be accessible from behind specific firewalls. - **Scalability:** Distributing workloads across multiple controllers improves performance. - **Argo CD integration:** Controllers are designed to interact with the centralized Kargo control plane and, at most, one Argo CD control plane. In enterprises with multiple Argo CD control planes, Kargo controllers exist in 1:1 proportion with those Argo CD instances. ### Garbage Collector **Runs in:** The control plane Absent routine cleanup, storing data as Kubernetes resources can lead to degraded cluster performance over time. The garbage collector is responsible for deleting old data. ### Kubernetes Webhooks Server **Runs in:** The control plane This server implements Kubernetes validating webhooks ([dynamic admission controls](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/)) and mutating (defaulting) webhooks. These webhooks are registered in the control plane cluster and allow the Kubernetes API server to hook into Kargo-specific logic as Kargo resources are created, modified, and deleted. :::caution **This webhook server is not an optional component. It is critical to Kargo's proper function.** ::: :::note This webhook server is *not internet-facing*. Its only client is the Kubernetes API server *within the same cluster.* It *must* use TLS. (The Kubernetes API server will not interact with webhook servers that do not.) Self-signed certificates are generally sufficient for this purpose. ::: :::info This component is distinct from the external webhooks server. (See next section.) ::: ### External Webhooks Server (Optional) **Runs in:** The control plane This server's role is to receive and handle inbound HTTP/S requests from external systems. For example, if you configure a GitHub repository to send a request to Kargo when a new commit is pushed, this server receives that request and initiates appropriate actions in response. :::info This is Kargo's second internet-facing component (after the API server). ::: ### Management Controller **Runs in:** The control plane This secondary controller handles critical clerical work. One of its most important functions is dynamically managing the permissions of other components. :::caution **This is not an optional component. It is critical to Kargo's proper function.** ::: :::info Because some components are internet-facing (the API server and external webhooks Server) and others (controllers) can be distributed to other clusters, operators are often reluctant to provide these components with broad access to sensitive resources, particularly Kubernetes Secrets in the control plane. To mitigate this, the management controller, which is neither internet-facing nor distributed, is granted broad permissions and dynamically expands and contracts *narrower* sets of permissions for other components and Kargo users as `Project`s are created and deleted. ::: ### Dex (Optional) **Runs in:** The control plane Dex is an open-source OpenID Connect identity provider (IdP). Kargo relies on third-party identity providers for user management and can work directly with most IdPs that implement the required protocols. However, some IdPs do not and Dex serves as an adapter for such cases. :::info **Example:** GitHub doesn't implement OpenID Connect at all, but Dex can integrate with GitHub while providing a compliant interface to Kargo, allowing users to sign in with their GitHub accounts. ::: :::info For many IdPs, like Microsoft Entra ID (formerly Azure Active Directory), Okta, or Google Cloud Identity Platform, Dex is not needed. In simple, "kick the tires" configurations, Kargo uses Dex's built-in mock IdP or disables integration with identity providers entirely and uses a built-in admin account instead. **We recommend never enabling the admin account in production.** ::: ## Topology ### The Kargo Control Plane The Kargo control plane is a Kubernetes cluster running most of Kargo's components and hosting all of Kargo's data. The control plane *may* include a controller instance, but controllers can also be entirely distributed to other clusters. When controllers are distributed, each operates on a specific subset of the data in the control plane. This subset and its controller are called a **shard**. ### Standalone Configuration In the simplest configuration, all Kargo components, including a single controller, run within the Kargo control plane cluster: ![Standalone Topology](./img/standalone-topology.png) This configuration is suitable for testing and very small-scale deployments. ### Distributed Configuration In larger environments, Kargo leverages an **agent-based architecture** where controllers are distributed across multiple clusters, with each controller "phoning home" to the centralized Kargo control plane: ![Distributed Topology](./img/sharded-topology.png) In this configuration: - The Kargo control plane hosts all data and core services, but may or may not run a controller instance. - Controllers are distributed to individual clusters (shards), each typically corresponding to an Argo CD control plane. - All controllers connect back to the Kargo control plane. - *The control plane itself has no privileged access to the clusters where controllers are deployed.* ### Why This Architecture? Kargo's agent-based architecture is informed by lessons learned from Argo CD and attempts to avoid the weaknesses of Argo CD's common topologies: **Hub-and-spoke:** - ✅ **High observability:** A centralized control plane amounts to a "single pane of glass." - ❌ **Security risks:** A centralized controller with privileged access to many clusters represents a high-value target and a single point of catastrophic failure if compromised. - ❌ **Limited scalability:** A centralized controller's performance will degrade as it becomes responsible for more and more clusters. **Control plane per cluster:** - ✅ **More secure:** Having no centralized control plane eliminates a high value target and a single point of catastrophic failure if compromised. - ✅ **More scalable:** Each controller's workload is smaller. - ❌ **Low observability:** No "single pane of glass." **Kargo's agent-based architecture:** Kargo's architecture employs an agent-based approach that marries the observability of a centralized control plane with the security and scalability of decentralized controllers: - ✅ **High observability:** A centralized control plane amounts to a "single pane of glass." - ✅ **More secure:** Distributed controllers that "phone home" to the centralized control plane mean the control plane has no privileged access to other clusters. - ✅ **Scales well:** Distributed controllers, each operating on a subset of control plane data, have smaller workloads. ## Configuration Considerations Provisioning Kargo in a distributed configuration is non-trivial and involves creating a Kubernetes `ServiceAccount` in the control plane for each controller, then distributing configuration with the corresponding tokens to the many clusters throughout your enterprise. :::info [Documentation for Kargo's Helm chart](https://github.com/akuity/kargo/blob/main/charts/kargo/README.md) explains most aspects of this in great detail, *but why not let the Kargo experts handle it for you?* Experienced Kubernetes operators may well succeed in provisioning Kargo in a distributed configuration as described above, but developing the automation to enable it at great scale can be a significant challenge. Kargo users operating at large scale are encouraged to consider using the [Akuity Platform](https://akuity.io/), which can seamlessly provision hosted Kargo control planes as well as credentials and configuration for any number of distributed controllers, which can be deployed to your own clusters in just seconds. ::: --- ## Cluster Level Configuration This document is a guide to Kargo configuration options that are available to operators _at runtime_. These are generally analogs to Project level configuration options available to developers. :::info[Not what you were looking for?] Most system level configuration options are exercised by operators at the time of installation or upgrade. For details, refer to [Common Configurations](./20-advanced-installation/30-common-configurations.md) and the [Kargo Helm Chart's README.md](https://github.com/akuity/kargo/tree/main/charts/kargo). ::: ## Triggering Artifact Discovery Using Webhooks If your cluster contains many `Warehouse` resources, which periodically poll artifact repositories, or if developers have [configured any of those `Warehouse`s poorly](../50-user-guide/20-how-to-guides/30-working-with-warehouses.md#performance-considerations), you may have elected to reduce the frequency with which all `Warehouse`s execute their artifact discovery processes (i.e. You may have elected to _increase_ the minimum polling interval. See [Common Configurations](./20-advanced-installation/30-common-configurations.md#tuning-reconciliation-intervals). ) If you have done this, it may have relieved degraded performance and helped to avoid encountering rate limits, but it will have been accompanied by the undesired side effect of increasing the average time required for every `Warehouse` to notice new artifacts. _This can be overcome by configuring repositories to alert Kargo to the presence of new artifacts via webhooks._ Developers are able to configure Kargo to listen for inbound webhook requests from various sources [at the Project level](../50-user-guide/20-how-to-guides/30-working-with-warehouses.md#triggering-artifact-discovery-using-webhooks), however, in an organization with many separate repositories in one (or a few) Git hosting providers or container image registries, it may make more sense for an operator to configure Kargo to listen for inbound webhook requests _at the cluster level_. To illustrate, consider a GitHub organization having many repositories belonging to different teams within the organization. Each team may have their own Kargo Project(s) for self-managing their promotion pipelines. Instead of every team configuring Project level GitHub webhook receivers that may trigger artifact discovery only for their own applicable `Warehouse`s, you, as the operator, can configure _one_ cluster-level GitHub webhook receiver to trigger the artifact discovery process of every applicable `Warehouse` across all Projects. This can be accomplished easily by updating your `ClusterConfig` resource's `spec.webhookReceivers` field. If your cluster does not already have a `ClusterConfig` resource, you can create one. :::note Every cluster hosting a Kargo control plane is permitted to have at most _one_ `ClusterConfig` resource. This limit is enforced by requiring all `ClusterConfig` resources to be named `cluster`. ::: A `ClusterConfig` resource's `spec.webhookReceivers` field may define one or more _webhook receivers_. A webhook receiver is an endpoint on a (typically) internet-facing HTTP server that is configured to receive and process requests from specific sources, and in response, trigger the discovery process of any `Warehouse` _across all Projects_ that subscribes to a repository URL referenced by the request payload. Most types of webhook receivers require you only to specify a unique name and a reference to a `Secret`. The expected keys and values for each kind of webhook receiver vary, and are documented on [each receiver type's own page](../50-user-guide/60-reference-docs/80-webhook-receivers/index.md). :::note Because `ClusterConfig` resources are _cluster-scoped_ resources and Kubernetes has no such thing as a "`ClusterSecret`" resource type (i.e. a cluster-scoped analog to `Secret`), Kargo will look for the referenced `Secret` in a designated namespace. By default, that namespace is `kargo-system-resources`, but can be changed by the operator at the time of installation. (Refer to the [Kargo Helm Chart's README.md](https://github.com/akuity/kargo/tree/main/charts/kargo). ) ::: :::warning Prior to `v1.9`, `Secret`s referenced by `ClusterConfig` were contained in the namespace designated by`global.clusterSecretsNamespace`, with a default of `kargo-cluster-secrets`. Kargo v1.9.0 changed the name of the setting to `global.systemResources.namespace` and its default value to `kargo-system-resources` to better reflect the intent to use the namespace not only for `Secret`s, but for _all_ types of namespaced resources that lack a cluster-scoped analog. Kargo versions >= v1.9.0 and < v1.12.0 will automatically sync `Secret`s from the namespace specified by `global.clusterSecretsNamespace` to the namespace specified by `global.clusterSecretsNamespace` (if they are different). That migration utility will be removed from v1.12.0, by which time operators who GitOps their `ClusterConfig` will need to have updated their manifests accordingly. Upgrades to v1.12.0 will **fail** if `global.clusterSecretsNamespace` remains defined. ::: :::info `Secret`s referenced by a webhook receiver typically serve _two_ purposes. 1. _Often_, some value(s) from the `Secret`'s data map are shared with the webhook sender (GitHub, for instance) and used to help authenticate requests. Some senders may use such "shared secrets" as bearer tokens. Others may use them as keys for signing requests. In such cases, the corresponding webhook receiver knows exactly what to do with this information in order to authenticate inbound requests. 1. _Always_, some value(s) from the `Secret`'s data map are used as a seed in deterministically constructing a complex, hard-to-guess URL where the receiver will listen for inbound requests. Some webhook senders (Docker Hub, for instance), do not natively implement any sort of authentication mechanism. No secret value(s) need to be shared with such a sender and requests from the sender contain no bearer token, nor are they signed. For cases such as these, a hard-to-guess URL is, itself, a _de facto_ shared secret and authentication mechanism. __Note that if a `Secret`'s value(s) are rotated, the URL where the receiver listens for inbound requests will also change. This is by design.__ Kargo does not watch `Secret`s for changes because it lacks the permissions to do so, so it can be some time _after_ its `Secret`'s value(s) are rotated that a webhook receiver's URL will be updated. To expedite that update, your `ClusterConfig` resource can be manually "refreshed" using the `kargo` CLI: ```shell kargo refresh clusterconfig ``` ::: The following example `ClusterConfig` configures two webhook receivers: ```yaml apiVersion: kargo.akuity.io/v1alpha1 kind: ClusterConfig metadata: name: cluster spec: webhookReceivers: - name: my-first-receiver github: secretRef: name: my-first-secret - name: my-second-receiver gitlab: secretRef: name: my-second-secret --- apiVersion: v1 kind: Secret type: Opaque metadata: name: my-first-secret namespace: kargo-cluster-secrets labels: kargo.akuity.io/cred-type: generic data: secret: c295bGVudCBncmVlbiBpcyBwZW9wbGUK --- apiVersion: v1 kind: Secret type: Opaque metadata: name: my-second-secret namespace: kargo-cluster-secrets labels: kargo.akuity.io/cred-type: generic data: secret-token: cm9zZWJ1ZCB3YXMgYSBzbGVkCg== ``` :::note The `kargo.akuity.io/cred-type: generic` label on `Secret`s referenced by webhook receivers is not strictly required, but we _strongly_ recommend including it. ::: For each properly configured webhook receiver, Kargo will update the `ClusterConfig` resource's `status` to reflect the URLs that can be registered as endpoints with the senders. For instance, the `ClusterConfig` and `Secret`s above result in the following: ```yaml apiVersion: kargo.akuity.io/v1alpha1 kind: ClusterConfig metadata: name: cluster spec: # ... omitted for brevity ... status: conditions: - lastTransitionTime: "2025-06-11T22:53:21Z" message: ProjectConfig is synced and ready for use observedGeneration: 1 reason: Synced status: "True" type: Ready webhookReceivers: - name: my-first-receiver path: /webhook/github/804b6f6bb40eb1f0e371f971d71dd95549be4bc9cbf868046941115f44073c67 url: https://kargo.example.com/webhook/github/804b6f6bb40eb1f0e371f971d71dd95549be4bc9cbf868046941115f44073c67 - name: my-second-receiver path: /webhook/gitlab/0eba9ff2a91f04f7787404b8f8f0edaf8cf8c39add34082651a474803cc99015 url: https://kargo.example.com/webhook/gitlab/0eba9ff2a91f04f7787404b8f8f0edaf8cf8c39add34082651a474803cc99015 ``` Above, you can see the URLs that can be registered with GitHub and GitLab as endpoints to receive webhook requests from those platforms. :::info For more information about registering these endpoints with specific senders, refer to [each receiver type's own page](../50-user-guide/60-reference-docs/80-webhook-receivers/index.md). ::: ### Receivers in Action Once a webhook receiver has been assigned a URL and that URL has been registered with a compatible sender, the receiver will begin receiving webhook requests in response to events in your repositories. The payload (body) of such a request contains structured information (usually JSON) the sender wishes to share about some event. Invariably, among this information, is the URL of the repository from which the event originated. A webhook receiver extracts a repository URL from the webhook request's payload, queries for all `Warehouse` resources across all Projects having subscriptions to that repository, and requests each to execute their artifact discovery process. For Git `push` events from providers that include file change information in their payloads (GitHub, GitLab, and Gitea), the receiver additionally extracts the list of changed files and evaluates each matching Warehouse's `includePaths` and `excludePaths` filters. Only Warehouses with path filters that match the changed files are refreshed. Warehouses without path filters configured are always refreshed. ## Cluster Message Channels `ClusterMessageChannel` resources are cluster-scoped analogs to [`MessageChannel`](../50-user-guide/20-how-to-guides/20-working-with-projects.md#message-channels). Their `spec` fields are identical, but `ClusterMessageChannel`s are available to every Project in the cluster without needing to create a Project-scoped `MessageChannel` resource. The other difference is that `ClusterMessageChannel`s must reference `Secret`s in the designated cluster secrets namespace (generally `kargo-cluster-secrets`). For example, here is a `ClusterMessageChannel` that sends messages to a Slack channel: ```yaml apiVersion: ee.kargo.akuity.io/v1alpha1 kind: ClusterMessageChannel metadata: name: devops-team-slack spec: # A reference to a Secret containing the Slack token. This is required for Slack. The Secret must # contain the following key: # - `apiKey`: The Slack token with permissions to post messages to the desired channel secretRef: # The name of the Secret in the cluster resources namespace name: slack-token # Configuration specific to Slack slack: # The Slack-specific channel ID to send messages to. This field is required channelID: C1234567890 ``` This is only a small example to illustrate the structure of a `ClusterMessageChannel`. For more detailed information about all the different spec fields and configuration options, refer to the documentation for [`MessageChannel`](../50-user-guide/20-how-to-guides/20-working-with-projects.md#message-channels). --- ## Secure Configuration The purpose of this document is to direct operators' attention to specific security considerations that should be taken into account at installation time, as well as calling out the specific configuration options that address them. :::info If you're only installing Kargo into a local Kubernetes cluster for testing purposes, installing Kargo with the default configuration should be sufficient. ::: :::info Whether you're installing Kargo [using Helm](../20-advanced-installation/10-advanced-with-helm.md) or [via Argo CD](../20-advanced-installation/20-advanced-with-argocd.md), this document assumes familiarity with procedures for configuring that installation. Refer to the [Kargo Chart's README](https://github.com/akuity/kargo/tree/main/charts/kargo) for detailed documentation of all configuration options. ::: ## Securing the API Server Since users interact with Kargo's API server (indirectly via the UI or CLI), the API server has the distinction of being Kargo's most attackable component. This makes it especially important to ensure that it is configured with secure options. This is doubly important if your Kargo API server is exposed to the internet -- which is possibly the case for some highly distributed organizations. The following sections will enumerate specific considerations and how to address them. ### Disabling the Admin Account Kargo's default configuration enables an admin account, primarily for the convenience of new users who are likely to be installing Kargo into a local Kubernetes cluster and are not likely to configure SSO via OpenID Connect. The admin account is highly privileged, which makes it an appealing target. __We strongly recommend disabling the admin account in any environment other than a local Kubernetes cluster.__ Disabling the admin account can be done at installation time by setting `api.adminAccount.enabled` to `false`. :::note The admin account is Kargo's _only_ built-in account. When disabling it, as we recommend, this effectively _requires_ that you configure SSO via OpenID Connect. Refer to [SSO with OpenID Connect](./20-openid-connect/index.md) for in-depth coverage of this topic. ::: ### Securing the Admin Account If, for some reason, you must leave the admin account enabled, __you must provide a bcrypt-hashed password for the account and a signing key__ that the API server will use to sign tokens (JWTs) for the account. 1. Generate and base64 encoded a bcrypt-hashed password and a signing key: ```console pass=$(openssl rand -base64 48 | tr -d "=+/" | head -c 32) echo "Password: $pass" hashed_pass=$(htpasswd -bnBC 10 "" $pass | tr -d ':\n') echo "Password Hash: $hashed_pass" echo "Encoded Password Hash: $(echo -n "$hashed_pass" | base64)" echo "Encoded Signing Key: $(openssl rand -base64 48 | tr -d "=+/" | head -c 32 | base64)" ``` 1. Create a `Secret` resource in the same namespace in which Kargo is installed with the following format: ```yaml apiVersion: v1 kind: Secret type: Opaque metadata: name: namespace: data: ADMIN_ACCOUNT_TOKEN_SIGNING_KEY: ADMIN_ACCOUNT_PASSWORD_HASH: ``` 1. At installation time, set `api.secret.name` to the name of the `Secret` resource you created. ### Disabling Secret Management First, note that the Kargo API server _never_ has cluster-scoped access to `Secret` resources. Instead, the management controller dynamically expands and contracts API server access to `Secret` resources in individual namespaces as `Project` resources are created and deleted. This effectively limits the API server to accessing `Secret` resources within project namespaces only. __There are two compelling reasons to disallow even this limited access:__ 1. Your threat model will not abide even the limited `Secret` access granted to the API server. 1. You GitOps your Kargo projects (which we recommend) or use some other means to manage `Secret` resources. With users not "click-opsing" `Secret`s, the need for the API server to access any `Secret`s _at all_ is entirely obviated. If either of the above scenarios is applicable, all `Secret` access by the API server can be disabled at installation time by setting `api.secretManagementEnabled` to `false`. ### Configuring TLS __There is no good reason not to secure inbound requests to the API server with TLS.__ Failing to do so is a significant security risk, thus TLS is enabled by default. The default configuration, however, makes use of a self-signed certificate, which will not be trusted by users' browsers. Assuming you've _not_ opted into using `Ingress` (disabled by default), __you should provide your own certificate.__ This can be done by setting `api.tls.selfSignedCert` to `false` and creating a `Secret` resource named `kargo-api-cert` in the same namespace in which Kargo is installed. The `Secret` must be formatted in the conventional manner for a certificate described [here](https://kubernetes.io/docs/concepts/configuration/secret/#tls-secrets). :::info We strongly recommend using [cert-manager](https://cert-manager.io/) to manage `Secret` resources containing TLS certificates. ::: If you _have_ opted into using `Ingress` (`api.ingress.enabled` set to `true`), it will default to using TLS with a self-signed certificate. To __provide your own certificate__, you should set `api.ingress.tls.selfSignedCert` to `false` and create a `Secret` resource named `kargo-api-ingress-cert` in the same namespace in which Kargo is installed. Once again, such a `Secret` must be formatted in the conventional manner for a certificate described [here](https://kubernetes.io/docs/concepts/configuration/secret/#tls-secrets). :::info With `Ingress` enabled and terminating TLS, it may be tempting to disable TLS on the API server itself. Your threat model may or may not permit this, but we recommend against it. ::: ## Securing the Controller ### Secret Access By default, the Kargo controller does _not_ have cluster-scoped access to `Secret` resources. Instead, the _management controller_ dynamically expands and contracts the controller's read-only access to `Secret` resources in individual namespaces as `Project` resources are created and deleted. This effectively limits the controller to accessing `Secret` resources within project namespaces only. :::info It is not uncommon for Kargo controllers to be installed in clusters other than the Kargo control plane. Restricting controller access to only `Secret` resources within project namespaces is a measure designed to prevent exposure of non-Kargo-project `Secret` resources in the control plane's cluster in the event that a remote controller's credentials are compromised. ::: It is possible to opt-in to cluster-scoped `Secret` access by setting `controller.serviceAccount.clusterWideSecretReadingEnabled` to `true` at installation time, __although we strongly recommend keeping the default configuration for this option (`false`).__ :::info The likely impetus for enabling cluster-scoped `Secret` access is to eliminate the need for manually managing `RoleBinding`s that grant the controller read-only access to `Secret` resources in specially designated "global credential namespaces" as described [here](40-managing-secrets.md). __We still do not recommend this.__ ::: ### Push Integration Policy When the [`git-push`](../../50-user-guide/60-reference-docs/30-promotion-steps/git-push.md) promotion step needs to integrate remote changes before pushing, it can either rebase (replaying local commits on top of the remote branch) or merge (creating a merge commit). Rebasing produces cleaner, linear history but has a trust implication: git recreates each rebased commit with a new identity, effectively re-signing them as the committer. This means: - __Commits signed by untrusted keys__ would be re-signed as Kargo, falsely attesting that Kargo trusts their content. - __Unsigned commits__ would gain Kargo's signature if signing is configured, fabricating trust that didn't previously exist. - __Signed commits__ would lose their signatures if signing is _not_ configured, stripping trust that did previously exist. The `controller.gitClient.pushIntegrationPolicy` setting controls this behavior. __We recommend `RebaseOrMerge` or stricter for any environment where commit signatures carry trust meaning.__ :::caution The current default is `AlwaysRebase`. Starting with v1.12.0, the default will change to `RebaseOrMerge`. If you rely on the current behavior, set the policy explicitly before upgrading. ::: ```yaml controller: gitClient: pushIntegrationPolicy: RebaseOrMerge ``` For a full description of all available options, see [Common Configurations](../20-advanced-installation/30-common-configurations.md#push-integration-policy). --- ## AWS Cognito OIDC Configuration Guide This guide explains how to configure AWS Cognito as an OIDC identity provider for Kargo. :::important The forms you will complete to enable Kargo to authenticate users against Cognito are _different_ depending on whether you are: * Registering Kargo as a new application client for an existing user pool. * Creating a brand new user pool. Be certain to follow the steps below that are appropriate to your scenario. ::: ## Using an Existing User Pool 1. Navigate to Cognito in your AWS dashboard. The left sidebar may be collapsed by default. If it is, expand it by clicking the "hamburger" icon in the upper left corner of the page. ![Cognito Home Page](./img/home-page.png) 1. Select User pools. 1. Select the pool to which you would like to add Kargo as a client. 1. Select App clients from the left sidebar. 1. Click the Create app client button on the upper right of the page. 1. Complete the Create app client form: ![Create App Client](./img/create-client.png) 1. Under Application type, select Single-page application (SPA). 1. Complete the Name your application field with a descriptive name, such as `kargo`. 1. Complete the Return URL field with a URL of the form `https:///login`. 1. Click Create app client. 1. Skip ahead to the [Configuring the App Client](#configuring-the-app-client) section of this page. ## Creating a New User Pool 1. Navigate to Cognito in your AWS dashboard. The left sidebar may be collapsed by default. If it is, expand it by clicking the "hamburger" icon in the upper left corner of the page. ![Cognito Home Page](./img/home-page.png) 1. Select User pools. 1. Click the Create user pool button in the upper right of the page. 1. Complete the Set up resources for your application form: ![Create User Pool and App Client](./img/create-pool-and-client.png) :::important This single form asks you to _both_ configure the user pool _and_ register an app client. Also note that you have no control over the name of the user pool. A unique name will be generated for you upon submission of this form. ::: 1. Under Application type, select Single-page application (SPA). 1. Complete the Name your application field with a descriptive name, such as `kargo`. 1. Under Options for sign-in identifiers, select one or more identifiers for users to use when signing in. :::note This is a feature of the _user pool_ and not the client app. ::: 1. Under Required attributes for sign-up, select all pieces of information you will require from users included in this user pool. At minimum, email must be selected. :::info Kargo is unopinionated about _how_ users actually authenticate to OIDC, identity providers (e.g. with username/email and password, social login, etc.) however, Kargo _does_ depend on the user's email address being available in the [standard claim](https://openid.net/specs/openid-connect-core-1_0.html#StandardClaims) `email` of the user's OIDC identity token. ::: 1. Complete the Return URL field with a URL of the form `https:///login`. 1. Click Create user directory. This creates the user pool _and_ the app client. 1. Navigate to your newly created client app: 1. Select User pools from the left sidebar. 1. Select the user pool you just created. 1. Select App clients from the left sidebar. 1. Select the app client you just created. 1. Continue on to the [Configuring the App Client](#configuring-the-app-client) section of this page. ## Configuring the App Client After completing the steps from either the [Using an Existing User Pool](#using-an-existing-user-pool) or [Creating a New User Pool](#creating-a-new-user-pool) section: 1. Select the Login pages tab. 1. In the Managed login pages configuration section, click Edit. 1. Complete the Edit managed login pages configuration form: ![Edit Managed Login Pages Configuration](./img/edit-login-pages.png) 1. Under URL, include _both_ of the following: * `https:///login` This is where users of the Kargo UI will be redirected to after logging in. * `http://localhost/auth/callback` This is where users of the `kargo` CLI will be redirected to redirected to after logging in. (The CLI launches a server to serve the request and complete the login process.) 1. Under OAuth 2.0 grant types, ensure Authorization code grant is selected. 1. Under OpenID Connect scopes, ensure that, at a minimum, Email, OpenID, and Profile are all selected. You may additionally select any other scopes that permit access to other user information you'd like to make available to Kargo. 1. Click Save changes. ## Configure Kargo After Kargo has been registered as a client application for your Cognito user pool, all that remains is to retrieve relevant details and use them in installing or updating Kargo. The details you will require are: * The user pool's issuer URL * The client app's client ID The unofficial, but most convenient way to locate both of these is in the React code sample in the Quick setup guide: ![Retrieve Client Config](./img/retrieve-client-config.png) After making note of these values, use them as the values of the `api.oidc.issuerURL` and `api.oidc.clientID` settings when installing or updating Kargo using its Helm chart [as documented here](../../20-openid-connect/index.md#configuring-kargo). :::info Using Cognito as an identity provider, there is no need to create separate client apps for the Kargo UI and the `kargo` CLI, and as such, the _same_ client ID should be used for _both_ the `api.oidc.clientID` and `api.oidc.cliClientID` settings. ::: --- ## Authentication with OpenID Connect By design, Kargo has no built-in system of registering and managing users. Assuming the lone, built-in admin account is disabled, as recommended [here](../10-secure-configuration.md#disabling-the-admin-account), you will need to configure Kargo to authenticate users via an external identity provider (IDP). This is recommended even in the case of the admin account not having been disabled, as it is the only way to grant users access to Kargo without disseminating the admin account's credentials. Conveniently, integrating Kargo with the IDP already used by your organization has the added benefit of enabling single sign-on (SSO) for your users. This document is a comprehensive guide for operators integrating Kargo with an external IDP. ## OpenID Connect and PKCE Like many modern applications, Kargo uses the [OpenID Connect protocol](https://openid.net/developers/how-connect-works/) (OIDC) for integrating with IDPs. It additionally utilizes [PKCE](https://auth0.com/docs/get-started/authentication-and-authorization-flow/authorization-code-flow-with-pkce) (Proof Key for Code Exchange) to secure the [authorization code flow](https://auth0.com/docs/get-started/authentication-and-authorization-flow/authorization-code-flow). :::info Briefly, PKCE is an extension to the authorization code flow that does not require clients to be issued a secret, which they must then protect. This is useful for "public clients" such as single-page applications (SPAs) and mobile apps, which cannot reliably protect a secret from intrepid users. Historically, public clients have either utilized the less secure implicit flow or used their own API server (which _can_ protect a client secret) as a middleman. With PKCE, a client can securely interact with an IDP directly. ::: Many IDPs support both OIDC and PKCE and Kargo can be integrated directly with those that do. Consult your identity provider's documentation to discover whether it supports _both_ of these standards. _If it does not, read on regardless. We will show how to work around such a constraint._ :::info Whether you're installing Kargo [using Helm](../../20-advanced-installation/10-advanced-with-helm.md) or [via Argo CD](../../20-advanced-installation/20-advanced-with-argocd.md), the next two sections assume familiarity with procedures for configuring that installation. ::: ## Configuration ### Registering Kargo as a Client To enable integration with your IDP, you will first need to register Kargo as a client. Carefully consult your IDP's documentation for instructions on how to do so. The callback URL(s) you will need to register depend on whether your IDP supports both OIDC and PKCE: **With a compatible identity provider:** - `https:///login` (for the UI) - `http://localhost/auth/callback` (for the CLI) :::info If your IDP does not permit you to register multiple callback URLs, you may need to register two clients -- one each for the UI and the CLI. ::: **With an incompatible identity provider:** - `https:///dex/callback` (for both the UI and CLI) **When registration is complete, make note of the issuer URL and client ID provided by your IDP.** ### Configuring Kargo When installing Kargo with Helm, all options related to OIDC are grouped under `api.oidc`. 1. Set `api.oidc.enabled` to `true`. 1. Only if your IDP supports _both_ OIDC and PKCE: :::caution If your IDP does not support _both_ OIDC and PKCE, the [Adapting Incompatible IDPs](#adapting-incompatible-idps) section provides alternate instructions for this step. ::: 1. Set `api.oidc.issuerURL` to the issuer URL provided by your IDP. 1. Set `api.oidc.clientID` to the client ID provided by your IDP. If you registered two separate separate clients for the UI and CLI with your IDP, additionally specify a value for `api.oidc.cliClientID`. This setting can otherwise be left alone. 1. Ensure `api.oidc.dex.enabled` remains set to its default value of `false`. Example: ```yaml api: oidc: enabled: true issuerURL: clientID: cliClientID: dex: enabled: false ``` 1. Configure `api.oidc.additionalScopes`: This is a list of additional and possibly non-standard scopes that Kargo will request from the IDP. For the most part, they map directly to a claim you are requesting to be included in ID tokens issued by the IDP. By default the `additionalScopes` list contains `groups`, which is a non-standard scope/claim, but one that is widely supported. If your IDP does not support it, remove that scope from the list. If there are additional claims you need because either you or administrators of individual projects will use them in mapping users to roles (see [Access Controls](../30-access-controls.md)), add the corresponding scopes to the list. Consult your IDP's documentation to discover what scopes are available. Example: ```yaml api: oidc: # ... omitted for brevity ... additionalScopes: - groups - - ``` 1. Configure access controls. `api.oidc.admins`, `api.oidc.users`, and other, similar settings map claims in ID tokens to _system-wide_ roles. If, for example, every user in the group `devops` should be an admin, and every user in the group `developers` should have very limited permissions, you would set these accordingly: ```yaml api: oidc: # ... omitted for brevity ... admins: claims: groups: - devops users: claims: groups: - developers ``` For more thorough coverage of the pre-defined, _system-wide_ roles, refer to the [Built-in System Roles](../30-access-controls.md#built-in-system-roles) section of the access controls page. :::caution Most assignments of users to roles is accomplished at the _project level_ since individual users' permissions are likely to vary from project to project. `api.oidc.admins` and `api.oidc.viewers` are strictly for mapping users to _system-wide_ roles. ::: ### Adapting Incompatible IDPs :::note If your IDP supports both OIDC and PKCE, you can skip this section entirely. ::: So your IDP does not support both OIDC and PKCE? In most cases, Kargo can work around this limitation quite easily through its optional, but seamless integration with [Dex](https://dexidp.io/), which supports both standards and can easily be configured to act as a middleman between Kargo and most IDPs. To configure Kargo to use Dex, set: 1. `api.oidc.dex.enabled` to `true`. 1. Configure one or more [connectors](https://dexidp.io/docs/connectors/) under `api.oidc.dex.connectors`. By default, no connectors are configured, although the Kargo chart's `values.yaml` file includes a few examples, however, Dex's own documentation should be counted as the definitive source of information on how to configure each available connector. To illustrate here, we will used the [GitHub connector](https://dexidp.io/docs/connectors/github/). GitHub does _not_ support OIDC. By introducing Dex, with proper configuration, as a middleman, we can still integrate Kargo with GitHub regardless. ```yaml api: oidc: enabled: true # ... omitted for brevity ... dex: enabled: true # Adapted from: https://dexidp.io/docs/connectors/github/ connectors: - type: github id: github name: GitHub config: clientID: clientSecret: $CLIENT_SECRET # Best not to include secrets in your values.yaml redirectURI: https:///dex/callback orgs: # Limit access to users in specific orgs; optional but recommended - name: teams: # Limit access to users in specific teams; also optional - - ``` :::info The Kargo chart will generate all the remaining Dex configuration for you. Only the `connectors` section needs to be provided. ::: 1. Define environment variables using `api.oidc.dex.env`: In the previous step, our connector configuration referenced a `$CLIENT_SECRET` environment variable to avoid storing sensitive information in a `values.yaml` file. To securely provide a value for that environment variable, configure `api.oidc.dex.env` like so: ```yaml api: oidc: dex: # ... omitted for brevity ... env: - name: CLIENT_SECRET valueFrom: secretKeyRef: name: github-dex key: clientSecret ``` The above example would require that you have, through some means, created a secret named `github-dex` with a key `clientSecret` within the same namespace in which Kargo is installed. --- ## Access Controls Most access controls in Kargo are within the purview of highly-privileged _users_ -- ones who might be considered to be "project admins." There are only a few access controls that an operator might need to be concerned with and this documentation focuses on those. :::info[Not what you were looking for?] If you're a project admin looking to understand more about access controls, you may find some value in this document, but most of what you need to know can be found in the [User Guide's Access Controls](../../50-user-guide/50-security/20-access-controls/index.md) section. ::: ## Overview Kargo is usually configured to support single-sign-on (SSO) using an identity provider (IDP) that implements the [OpenID Connect](https://openid.net/developers/how-connect-works/) (OIDC) protocol. This topic is explored in much greater depth in the dedicated [OpenID Connect](./20-openid-connect/index.md) section of the Operator Guide. Kargo also implements access controls through _pure Kubernetes [RBAC](https://kubernetes.io/docs/reference/access-authn-authz/rbac/)._ :::info Kargo's creators learned from previous experience that when APIs are modeled as Kubernetes resources, it is best to rely solely on Kubernetes-native authorization mechanisms. By doing so, access controls are enforced even for users with direct access to the Kargo control plane's underlying cluster who might shun the Kargo CLI and UI in favor of `kubectl`. ::: There is a natural _impedance_ between users authenticating to _Kargo_ through some IDP and access controls being implemented through pure _Kubernetes_ RBAC. To make a finer point of it: _It is impossible for a Kubernetes cluster to enforce RBAC for users it does not recognize._ Assuming you do not wish to resolve this by granting direct cluster access to a large number of developers and training them to use `kubectl`, a different solution is required. Kargo resolves this impedance through a simple scheme that permits users authenticated via the IDP to be _mapped_ to Kubernetes `ServiceAccount` resources. For the most part, these mappings are best managed at the project level by project admins. The remainder of this document, therefore, touches on the few access controls that an operator might need to be concerned with. ## User to `ServiceAccount` Mappings First, operators should understand how the mapping of users to `ServiceAccount` resources works. Most Kargo users interact with Kargo via its API server, using its UI or CLI as a client. In either case, those users are authenticated by a bearer token issued by the IDP. For every request, the Kargo API server validates and decodes the token to obtain trusted information about the user which, importantly, includes _claims_ such as username, email address, and group membership. The exact claims available depend on the IDP and the configuration of the Kargo API server. For more details on this topic, refer to the [OpenID Connect](./20-openid-connect/index.md) section of the Operator Guide. Also for every request, the Kargo API server queries the Kubernetes API server to obtain a list of all `ServiceAccount` resources to which the user has been mapped. This search is mostly limited to `ServiceAccount` resources in Kargo project namespaces only (i.e. only those labeled with `kargo.akuity.io/project: "true"`). _This section focuses on the exceptions to that rule._ ServiceAccount resources may be mapped to users via the `rbac.kargo.akuity.io/claims` annotation, whose value is a string representation of a JSON object with claim names as its keys and lists of claim values as its values. In the following example, the `ServiceAccount` resource is mapped to all of: * Users with a `sub` claim identifying them as either `alice` or `bob`. * A user with the `email` claim `carl@example.com`. * All users with a `groups` claim containing _either_ the `devops` or `kargo-admin` group. ```yaml apiVersion: v1 kind: ServiceAccount metadata: name: admin namespace: kargo-demo annotations: rbac.kargo.akuity.io/claims: | { "sub": ["alice", "bob" ], "email": ["carl@example.com"], "groups": ["devops", "kargo-admin"] } ``` :::info Claim values may also be specified as a scalar string instead of a single-element list (e.g. `"email": "carl@example.com"`). Both forms are accepted, but the list form is preferred for consistency. Mappings specified using annotations with keys of the form `rbac.kargo.akuity.io/claim.` with comma-delimited values are also supported for reasons of backwards compatibility. The effective mapping is therefore the union of mappings defined using such annotations with any mappings defined using the newer, recommended `rbac.kargo.akuity.io/claims` annotation. ::: A user may be mapped to multiple `ServiceAccount` resources. A user's effective permissions are therefore the _union_ of the permissions associated with all such `ServiceAccount` resources. ### Global Mappings Now that we've seen how users are mapped to `ServiceAccount` resources, we can zero in on the few places where these details are relevant to the operator role. As previously mentioned, _most_ access controls are managed at the project level by project admins, however, there are two ways in which an operator can also map users to `ServiceAccount` resources. #### Built-in System Roles Kargo comes with four specific `ServiceAccount`s pre-defined by its Helm chart, along with bindings to applicable permissions. These four `ServiceAccount`s can easily be associated with users having specific claimes through chart configuration at install-time: | Name | Configuration Key | Description | |------|-------------------|-------------| | `kargo-admin` | `api.oidc.admins` | Complete, cluster-wide access to all Kargo resources. Access to `Secret`s is _not_ cluster-wide, but expands and contracts dynamically as projects and their underlying namespaces are created and deleted. | | `kargo-viewer` | `api.oidc.viewers` | Read-only, cluster-wide access to all Kargo resources. This does _not_ include any level of access to `Secret`s. | | `kargo-user` | `api.oidc.users` | The minimum level of permissions that can be granted to a user. It permits only listing `Project`s and viewing system-level configuration. This does _not_ include any level of access to `Secret`s. | | `kargo-project-creator` | `api.oidc.projectCreators` | The permissions of the user role, plus permission to create new `Project`s. When a project is created by such a user via the CLI or UI (but not through `kubectl`) that user will dynamically receive admin permissions within that project's underlying namespace. This includes access to project `Secret`s. | If, one wished to make the following associations: - Alice and Bob should be admins. - Team leads should be able to create new projects. - Devops engineers should be able to view everything. - Developers should have few permissions, with additional permissions granted on a project-by-project basis. And assuming users have `email` and `group` claims, and groups `leads`, `devops`, and `developers` exist, Kargo could be configured as follows at install-time: ```yaml api: oidc: # ... omitted for brevity ... admins: claims: email: - alice@example.com - bob@example.com projectCreators: claims: groups: - leads viewers: claims: groups: - devops users: claims: groups: - developers ``` Behind the scenes, the configuration above merely results in applicable `ServiceAccounts` in Kargo's own namespace being annotated as discussed in the previous section. For example, the `kargo-admin` `ServiceAccount` will be annotated as follows: `kargo-admin`: ```yaml apiVersion: v1 kind: ServiceAccount metadata: name: kargo-admin namespace: kargo annotations: rbac.kargo.akuity.io/claims: '{"email":["alice@example.com", "bob@example.com"]}' ``` :::info For additional information, refer to the [OpenID Connect](./20-openid-connect/index.md) section of the Operator Guide. ::: #### Global `ServiceAccount` Namespaces In some cases, an operator, in collaboration with project admins, may wish to create a small number of `ServiceAccount` resources that are _pre-mapped_ to users with specific claims. Operators can assign system-wide permissions to such `ServiceAccount`s using `ClusterRoleBinding`s. Project admins retain control over the project-level permissions of those `ServiceAccount`s using `RoleBinding`s in the project namespaces. :::info The main convenience of this approach is that it enables individual project admins to take the existence of certain classes of user for granted instead of having to manage user-to-`ServiceAccount` mappings themselves. ::: :::info "Global" is a misnomer. `ServiceAccount` resources in designated namespaces are not truly global because they are still mapped to users according to the rules described in the previous sections. ::: Enabling global `ServiceAccount` namespaces requires three steps be taken by the operator: 1. Create one or more namespaces dedicated to this purpose. Typically, just one should suffice. 1. Define `ServiceAccount` resources in these namespaces, each annotated appropriately to effect the desired user-to-`ServiceAccount` mappings. Optionally, use a `ClusterRoleBinding` to grant any necessary system-wide permissions to these `ServiceAccount` resources. 1. Configure Kargo to look for `ServiceAccount` resources in these designated namespaces, by setting `api.oidc.globalServiceAccounts.namespaces` at installation time. For example: ```yaml api: oidc: globalServiceAccounts: namespaces: - kargo-global-service-accounts ``` --- ## API Tokens API tokens enable programmatic access to Kargo APIs without requiring users to share short-lived credentials with non-human agents, such as CI processes. :::info[Not what you were looking for?] If you're a Project admin looking to create and manage API tokens within your Projects, you may find some value in this document, but most of what you need to know can be found in the [User Guide's API Tokens](../../50-user-guide/50-security/25-api-tokens.md) documentation. ::: Kargo API tokens are associated directly with Kargo roles. Kargo comes with several built-in, system-level roles in the namespace where Kargo is installed (typically `kargo`). These roles provide different levels of system-wide access. :::caution[Implementation Detail] "Kargo roles," including built-in, system level ones are actually abstractions over trios of Kubernetes `ServiceAccount`, `ClusterRole`, and `ClusterRoleBinding` resources. Throughout this document, the term "role" refers to this abstraction. ::: To learn more about built-in, system-level roles, refer to the [Access Controls](./30-access-controls.md) documentation. ## Creating Tokens To generate a new token associated with a system-level role: ```shell kargo create token --system --role kargo-admin kargo-admin-token-1 ``` ```shell Token created successfully! IMPORTANT: Save this token securely. It will not be shown again. Token: eyJhbGciOiJSUzI1NiIsImtpZCI6IjdwQ0... ``` :::danger The token value is displayed only once during creation. Do not lose it! If you lose the token value, you must delete the token and create a new one _or_ the existing token's value can be retrieved by a user with sufficient permission using `kubectl` instead of the `kargo` CLI. ::: List all tokens associated with a specific system-level role: ```shell kargo get tokens --system ``` ```shell NAME ROLE KARGO MANAGED AGE kargo-admin-token-1 kargo-admin true 5m ``` List tokens associated with a specific system-level role: ```shell kargo get tokens --system --role kargo-admin ``` Retrieve details about a specific token (note that the token value will be redacted): ```shell kargo get token --system kargo-admin-token-1 -o yaml ``` ## Using Tokens API tokens can be used with many Kargo or Kubernetes clients. This includes tools like `kubectl` as well as any programming language client library for Kubernetes or Kargo. :::note While the `kargo` CLI does not directly support specifying a token via command line flags, you can configure it to use a token by editing `~/.config/kargo/config`. ::: ## Deleting Tokens To delete a token when it's no longer needed or to rotate credentials: ```shell kargo delete token --system kargo-admin-token-1 ``` ```shell token.kargo.akuity.io/kargo-admin-token-1 deleted ``` Verify the token has been deleted: ```shell kargo get tokens --system ``` --- ## Managing Secrets Kargo uses Kubernetes `Secret` resources to store repository credentials and other types of sensitive data, such as API keys for third-party services. The namespaces within which they are contained impact who or what can access them. Labels on those `Secret`s may also constrain _how_ they are accessed. It is crucial that operators managing Kargo instances understand how `Secret`s are organized and accessed. :::info[Not what you were looking for?] If you're a Kargo user looking to learn more about managing credentials or any other kind of secret _within your own Project_, refer instead to the [Managing Secrets](../../50-user-guide/50-security/30-managing-secrets.md) section of the User's Guide. ::: ## Overview Operators managing a Kargo instance will find themselves concerned with secrets falling into one of two broad categories: * **Shared secrets** intended for read-only access _by any or all Projects within the instance._ These, in turn, can be classified as one of: * **Repository credentials:** Secrets specifically representing credentials for the three types of repositories supported by Kargo: Git repositories, container image repositories, and Helm chart repositories. * **"Generic credentials":** Any secrets that are not specifically repository credentials. :::info The misnomer "generic _credentials_" is used for historical reasons, but nothing limits these to storing _only_ credentials. In actuality, they can be any sort of sensitive information. So although they are called "generic credentials," they are best thought of in more general terms as, simply, generic secrets. ::: * **System-level secrets** used by Kargo itself and _not_ intended to be accessed by Kargo Projects. The remainder of this document will cover each of these in turn, explaining in detail what such secrets look like, where they are stored, who can access them, and _how_ they are accessed. ## Shared Secrets As the name implies, shared secrets are those intended to be accessible by all Projects within a Kargo instance. Their corresponding `Secret` resources belong in one, specific Kubernetes namespace referred to as **the shared resources namespace**. :::info[Why shared _resources_?] Prior to Kargo v1.9.0, what is now the shared resources namespace was referred to as "global credentials namespaces" (plural). Three factors prompted the Kargo team to refine and rename the concept: * "Global" was prone to various misinterpretations. * "Namespaces" (plural) added unnecessary technical complications to the system. * "Credentials" was too specific. Not all secrets are credentials. And not all things to be shared across Projects are secrets. The more general term "resources" speaks to a broader purpose for the namespace. ::: :::warning[Migration] If you are migrating from a Kargo version lesser than v1.9.0 to version v1.9.0 or greater, please consult the [migration](#transitioning) section at the bottom of this page. ::: ### Repository Credentials Kargo expects `Secret` resources representing repository credentials to be labeled in specific ways and to conform to a specific format. Such `Secret`s generally take the following form: ```yaml apiVersion: v1 kind: Secret metadata: name: namespace: labels: kargo.akuity.io/cred-type: data: # base64 encoded repoURL: username: password: ``` The label key `kargo.akuity.io/cred-type`, together with its value, specify the _type_ of the repository accessed with the credential: * `git`: Credentials for Git repositories * `helm`: Credentials for Helm chart repositories * `image`: Credentials for container image repositories `Secret`s representing repository credentials MUST include the key `repoURL` in their `data` block. Its value may be either a full, exact URL **OR** a regular expression matching the URLs of multiple repositories for which the credentials are valid, in which case, the `data` block must also contain the key/value pair `repoURLIsRegex: "true"`. The remaining key/value pairs in such a `Secret`'s `data` block are dependent upon exactly what kind of credential the `Secret` represents. Commonly, they may be: * `username`: The username to use when authenticating to the repository * `password`: A password or **personal access token** :::info If the value of the `password` key is a **personal access token**, the value of the `username` field is often inconsequential. You should consult your repository's documentation for more information. ::: Alternatively, for Git repositories only (and specifically ones that support SSH-style URLs of the form `git@github.com:example/repo.git`), the key `sshPrivateKey` in the `Secret`'s `data` block may have as its value a PEM-encoded SSH private key. :::warning[Deprecated] Support for SSH URLs and SSH private keys is **deprecated as of v1.10.0** and **scheduled for removal in v1.13.0**. The proprietary APIs offered by the major Git hosting platforms (e.g. GitHub or GitLab) to enable actions such as opening or closing pull requests are invariably HTTP-based and therefore cannot use an SSH private key for authentication. This forces users who rely on SSH to maintain a second set of credentials (e.g. a personal access token) for API operations. Using HTTPS URLs with a single token-based credential that works for both standard Git operations and provider API calls is simpler and more secure. Users should migrate to HTTPS URLs before v1.13.0. ::: :::info[Credential Shapes] `Secret` resources representing repository credentials come in a wide variety of other "shapes" (different keys in the `data` block) corresponding to various authentication mechanisms. These are covered in the [Managing Secrets](../../50-user-guide/50-security/30-managing-secrets.md) section of the User's Guide. ::: #### Using Repository Credentials A unique property of `Secret` resources representing repository credentials is that Projects do not (and cannot) reference them directly. Any time Kargo accesses a repository, it _automatically_ attempts to locate suitable credentials, searching by _repository type and URL._ :::tip Because of the above, operators managing a Kargo instance can place repository credentials in the **shared resources namespace**, knowing that they can be used by all Projects _without their values ever being exposed to users._ ::: When Kargo needs repository credentials, it searches for `Secret`s in _two_ specific namespaces, in the following order: 1. **Project namespace**: Kargo searches the Project's own namespace first. 2. **Shared resources namespace**: If no match is found in the Project's own namespace, Kargo searches the shared resources namespace. :::info[Credential Matching Precedence] _Within_ each namespace searched, Kargo considers credentials in this order: 1. Exact `repoURL` matches (where `repoURLIsRegex` is `"false"` or unspecified) 2. Pattern matches using regex (where `repoURLIsRegex` is `"true"`) Within each category, `Secret`s are considered in lexical order by name. The credentials used by Kargo will be the _first_ to match the repository type and URL. ::: ### Generic Credentials "Generic credentials" (a misnomer) are any secrets that are not specifically repository credentials. `Secret` resources representing generic credentials MUST be labeled with `kargo.akuity.io/cred-type: generic`. :::info The misnomer "generic _credentials_" is used for historical reasons, but nothing limits these to storing _only_ credentials. In actuality, they can be any sort of sensitive information. So although they are called "generic credentials," they are best thought of in more general terms as, simply, generic secrets. ::: #### Using Generic Credentials In contrast to repository credentials, `Secret` resources representing shared generic credentials can be accessed directly by name and their `data` blocks are not required to conform to any specific structure. This makes them suitable for storing any arbitrary secret data that Projects may depend upon. Projects can access such secrets within expressions used by their promotion processes by utilizing the [`sharedSecret()`](../../50-user-guide/60-reference-docs/40-expressions.md#sharedsecretname) expression function. :::caution Always remember that any generic credential in the shared resources namespace can be accessed directly by all Projects, which means it is possible to learn their values. Exercise due caution when deciding what secrets are suitable to be shared in this manner. ::: ### Replicating Shared Resources to Project Namespaces By default, shared secrets are accessed _indirectly_ -- repository credentials are matched automatically by URL, and generic credentials are accessed through the `sharedSecret()` expression function. In both cases, the `Secret` resources themselves remain in the shared resources namespace and their values are never copied into Project namespaces. In some situations, however, workloads running in a Project namespace may need _direct_ access to a `Secret` or `ConfigMap` -- for example, as a volume mount or environment variable reference. For these cases, Kargo can automatically **replicate** resources from the shared resources namespace into every Project namespace. #### Enabling Replication To opt a resource into replication, annotate it with `kargo.akuity.io/replicate-to: "*"`: ```yaml apiVersion: v1 kind: Secret metadata: name: shared-tls-cert namespace: kargo-shared-resources labels: kargo.akuity.io/cred-type: generic annotations: kargo.akuity.io/replicate-to: "*" data: tls.crt: tls.key: ``` `ConfigMap` resources work the same way: ```yaml apiVersion: v1 kind: ConfigMap metadata: name: shared-config namespace: kargo-shared-resources annotations: kargo.akuity.io/replicate-to: "*" data: settings.yaml: | key: value ``` :::info Currently, the only supported value for the `kargo.akuity.io/replicate-to` annotation is `"*"`, which replicates to all Project namespaces. Selective replication to specific Projects is not yet supported. ::: :::info Only `Secret` resources labeled with `kargo.akuity.io/cred-type` (i.e. recognized as credentials) are eligible for replication. Arbitrary `Secret` resources without this label will not be replicated, even if annotated with `kargo.akuity.io/replicate-to`. All `ConfigMap` resources in the shared resources namespace are eligible. ::: #### How Replication Works When a resource in the shared resources namespace is annotated for replication, Kargo's management controller automatically creates a copy of it in every Project namespace. These copies are kept in sync: any change to the source resource is propagated to all replicas. If the source is deleted, all replicas are cleaned up. Replicated resources are labeled with `kargo.akuity.io/replicated-from` (set to the name of the source resource) so they can be easily identified. #### Immutability of Replicated Resources Replicated copies are **immutable to end users.** A validating webhook prevents any user from modifying or deleting them. Only Kargo's management controller is permitted to update or remove replicated resources. To change the contents of a replicated resource, modify the **source** resource in the shared resources namespace. The change will propagate automatically to all replicas. ### Configuring the Shared Resources Namespace The **shared resources namespace**, by default, is `kargo-shared-resources`. Operators may override this at the time of installation or upgrade by overriding the Kargo Helm chart's `global.sharedResources.namespace` setting. ## System Secrets Various components of Kargo itself, at times, have the need to reference operator-defined secrets. The canonical example for this involves configuring cluster-scoped webhook receivers. Cluster-scoped webhook receivers are defined as part of a `ClusterConfig` resource, which is, unsurprisingly, a _cluster-scoped resource_. (i.e. It does not belong to any namespace.) When such a configuration must reference a `Secret`, because Kubernetes has no cluster-scoped "ClusterSecret" resource type, the question is raised of exactly which namespace a `Secret` that is _conceptually_ cluster-scoped should belong to. The existence of the **system resources namespace** provides an answer to this conundrum. An example `ClusterConfig`: ```yaml apiVersion: v1 kind: Secret metadata: name: gh-wh-secret namespace: kargo-system-resources labels: kargo.akuity.io/cred-type: generic data: secret: --- apiVersion: kargo.akuity.io/v1alpha1 kind: ClusterConfig metadata: # Note this resource is not namespaced name: cluster spec: webhookReceivers: - name: gh-wh-receiver github: # Referenced Secrets are implicitly known to be in # the system resources namespace (kargo-system-resources) secretRef: name: gh-wh-secret ``` :::info Prior to Kargo v1.9.0, what is now the system resources namespace was referred to as the "cluster secrets namespace." Two factors prompted the Kargo team to refine and rename the concept: * "Cluster" was prone to various misinterpretations. * "Secrets" was too specific. The Kargo team does not anticipate that `Secret` resources will forever be the only type of namespaced resource that will be referenced by `ClusterConfig` as a workaround for a non-existent cluster-scoped analog. ::: :::warning[Migration] If you are migrating from a Kargo version lesser than v1.9.0 to version v1.9.0 or greater, please consult the [migration](#transitioning) section at the bottom of this page. ::: ### Configuring the System Resources Namespace The **system resources namespace**, by default, is `kargo-system-resources`. Operators may override this at the time of installation or upgrade by overriding the Kargo Helm chart's `global.systemResources.namespace` setting. ## Migrating from Kargo < 1.9.0 {#transitioning} Kargo v1.9.0 introduced terminology and configuration changes to better reflect the intended use of what are now the **shared resources namespace** and **system resources namespace**. These changes are summarized here. **Terminology Changes:** * Global credentials namespaces (plural) → **shared resources namespace** (singular) * Cluster secrets namespace → **system resources namespace** **Chart Setting Changes:** * `controller.globalCredentials.namespaces` → `global.sharedResources.namespace` * The old setting had no default value(s). * The new setting has a default value of `kargo-shared-resources`. * The move from the `controller` section of the chart's settings to the `global` section reflects that this configuration is used by more than one Kargo component. * `global.clusterSecretsNamespace` → `global.systemResources.namespace` * The old setting had a default value of `kargo-cluster-secrets`. * The new setting has a default value of `kargo-system-resources`. **Automatic Migration:** Kargo versions **v1.9.0 through v1.11.x** will automatically and continuously perform a one-way sync of `Secret` resources from their old locations to their new locations, with a few exceptions: * If the old `controller.globalCredentials.namespaces` setting was empty (as it had no default values(s)), there will be no `Secret` resources in need of migration to the namespace specified by the new `global.sharedResources.namespace`. * Due to the potential for name conflicts if Kargo were to attempt consolidating resources from multiple namespaces into a single namespace, a chart upgrade to v1.9.0 through v1.11.0 will **fail** if the old `controller.globalCredentials.namespaces` setting specified _multiple namespaces_. In this case (believed to be an outlier), the operator will need to migrate affected resources manually. * If the value of the new `global.sharedResources.namespace` matches the value of the old `controller.globalCredentials.namespaces[0]` setting, no migration of shared `Secret` resources will be necessary. * If the value of the new `global.systemResources.namespace` matches the value of the old `global.clusterSecretsNamespace` setting, no migration of system `Secret` resources will be necessary. Kargo v1.12.0 will remove the automatic migration and upgrades to that version or greater will **fail** if values are detected for any of the old settings. **Sync Behavior:** The automatic sync from old to new locations works as follows: * **Unmodified secrets:** If a `Secret` in the new location has not been modified since it was synced, updates from the old location will continue to propagate. Deleting from the old location will also delete from the new location. * **Modified secrets:** If you modify a `Secret` in the new location (via the Kargo UI or otherwise), those changes are protected. Updates from the old location will _not_ overwrite your modifications. Similarly, deleting from the old location will _not_ delete a modified secret from the new location. * **Deleting from the new location:** If you delete a `Secret` from the new location while it still exists in the old location, it will be recreated on the next sync. To permanently remove a secret, delete it from the old location. * **Namespace cleanup:** When the old namespace itself is deleted (bulk cleanup), all secrets in the new location are preserved, regardless of whether they were modified. **What this means, practically speaking:** * New installations of Kargo need not be concerned with any of this. * If you are upgrading: * If you manually manage credentials using the Kargo UI, everything will just work. Post upgrade, `Secret`s will automatically sync from their old locations to their new locations. Kargo will use and manage `Secret`s in their new locations. Any modifications you make via the UI to secrets in the new location will be preserved and not overwritten by subsequent syncs. When you are ready, you may delete the old namespaces using `kubectl` -- all secrets in the new locations will be preserved. * If you are a more advanced operator who GitOps'es your `Secret`s, you do not need to act with any urgency. If you initially do nothing, `Secret`s will continue to be synced from your GitOps repository to their original locations. Kargo will sync those `Secret`s to their new locations. Everything will behave as it should. You will have until Kargo v1.12.0 to update Kargo `Secret` manifests in your GitOps repository to reference their new namespaces. Depending on the configuration of the GitOps agent managing Kargo (e.g. Argo CD), `Secret`s may automatically be pruned from their old locations. If not, then with due caution, you may manually delete the old namespaces using `kubectl`. Summarizing the above, no matter what you do, things should continue working until upgrading to v1.12.0 and this should afford operators sufficient time to make the very minimal changes required to keep things running smoothly in v1.12.0 and beyond. --- ## Ambient Credentials This section provides guidance on configuring Kargo and various cloud platforms to support "ambient" credentials — credentials automatically available based on the execution environment rather than stored in `Secret`s. ## Amazon Elastic Container Registry (ECR) Kargo can be configured to authenticate to ECR repositories using EKS Pod Identity _or_ IAM Roles for Service Accounts (IRSA). If Kargo locates no `Secret` resources matching a repository URL and is deployed within an EKS cluster, it will attempt to use [EKS Pod Identity](https://docs.aws.amazon.com/eks/latest/userguide/pod-identities.html) or [IAM Roles for Service Accounts (IRSA)](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html) to authenticate. Leveraging either eliminates the need to store ECR credentials in a `Secret` resource. Follow [this overview](https://docs.aws.amazon.com/eks/latest/userguide/pod-identities.html#pod-id-setup-overview) to set up EKS Pod Identity in your EKS cluster or [this one](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html) to set up IRSA. For either, you will assign an IAM role to the `kargo-controller` `ServiceAccount` within the namespace in which Kargo is (or will be) installed. :::note To use IRSA, you will additionally need to specify the [ARN](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference-arns.html) of the controller's IAM role as the value of the `controller.serviceAccount.iamRole` setting in Kargo's Helm chart at installation. ::: At this point, an IAM role will be associated with the Kargo _controller_, however, that controller acts on behalf of multiple Kargo Projects, each of which may require access to _different_ ECR repositories. To account for this, when Kargo attempts to access an ECR repository on behalf of a specific Project, it will first attempt to [assume an IAM role](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html) specific to that Project. The name of the role it attempts to assume will _always_ be of the form `kargo-project-`. It is this role that should be granted read-only access to applicable ECR repositories. :::info The name of the IAM role associated with each Kargo Project is deliberately not configurable to prevent Project admins from attempting to coerce Kargo into assuming arbitrary IAM roles. ::: :::caution For optimal adherence to the principle of least privilege, the IAM role associated with the `kargo-controller` `ServiceAccount` should be limited only to the ability to assume Project-specific IAM roles. Project-specific IAM roles should be limited only to read-only access to applicable ECR repositories. ::: :::info If the Kargo controller is unable to assume a Project-specific IAM role, it will fall back to using its own IAM role directly. For organizations without strict tenancy requirements, this can eliminate the need to manage a large number of Project-specific IAM roles. While useful, this approach is not strictly recommended. ::: Tokens Kargo obtains for accessing any specific ECR repository on behalf of any specific Kargo Project are valid for 12 hours and cached until shortly before they expire. A controller restart clears the cache. ## Google Artifact Registry (GAR) Kargo can be configured to authenticate to [Google Artifact Registry](https://cloud.google.com/artifact-registry/docs/overview) (GAR) repositories using [Workload Identity Federation](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity) (WIF). If Kargo locates no `Secret` resources matching a repository URL, and if Kargo is deployed within a [Google Kubernetes Engine](https://cloud.google.com/kubernetes-engine/docs/concepts/kubernetes-engine-overview) (GKE) cluster with WIF enabled, it will attempt to use it to authenticate. Leveraging this option eliminates the need to store credentials in a `Secret` resource. WIF can be enabled when creating a [new cluster](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity#enable_on_cluster) or can be added to an [existing cluster](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity#enable-existing-cluster). :::note Clusters managed by [GKE Autopilot](https://cloud.google.com/kubernetes-engine/docs/concepts/autopilot-overview) have WIF enabled automatically. ::: With WIF enabled, [GCP Identity and Access Management](https://cloud.google.com/iam/docs/overview) (IAM) automatically understands a [principal](https://cloud.google.com/iam/docs/overview#principals) identifier of the following form to be a reference to the Kargo controller's _Kubernetes_ Service Account (KSA): ```text principal://iam.googleapis.com/projects//locations/global/workloadIdentityPools/.svc.id.goog/subject/ns//sa/kargo-controller ``` :::note There is no need to annotate the Kargo controller's KSA in any specific way to enable the above. ::: Because the Kargo controller acts on behalf of multiple Kargo Projects, each of which may require access to _different_ GAR repositories, when accessing a repository on behalf of a given Project, it will attempt to [impersonate](https://cloud.google.com/iam/docs/service-account-impersonation) a Project-specific [Google Service Account](https://cloud.google.com/iam/docs/service-accounts-create) (GSA). The name of the GSA that the controller will attempt to impersonate will _always_ be of the form `kargo-project-@.iam.gserviceaccount.com`. :::info The name of the GSA associated with each Kargo Project is deliberately not configurable to prevent Project admins from attempting to coerce Kargo into impersonating arbitrary GSAs. ::: To enable this, each Project-specific GSA must: * Have an [IAM policy](https://cloud.google.com/iam/docs/reference/rest/v1/Policy) that permits the Kargo controller's KSA to impersonate the GSA by creating a token (`roles/iam.serviceAccountTokenCreator`). * Be granted read-only access (`roles/artifactregistry.reader`) to the specific GAR repositories with which it interacts. :::caution Following the principle of least privilege, the IAM principal associated with the Kargo controller's GSA should be granted no permissions beyond the ability to impersonate Project-specific GSAs. ::: :::note Beginning with Kargo `v1.5.0`, if maintaining a separate GSA for every Kargo Project is deemed too onerous and strict adherence to the principle of least privilege is not a concern, permissions may be granted directly to the Kargo controller's KSA. In the event that a Project-specific GSA does not exist or cannot be impersonated, Kargo will fall back on using the controller's KSA directly to access GAR repositories. While useful, this approach is not strictly recommended. ::: Tokens Kargo obtains for accessing any specific GAR repository on behalf of any specific Kargo Project are valid for 60 minutes and cached until shortly before they expire. A controller restart clears the cache. ## Azure Container Registry (ACR) Kargo can be configured to authenticate to ACR repositories using [Azure Workload Identity](https://learn.microsoft.com/en-us/azure/aks/workload-identity-overview). If Kargo locates no `Secret` resources matching a repository URL and is deployed within an AKS cluster with workload identity enabled, it will attempt to use it to authenticate. Leveraging this eliminates the need to store ACR credentials in a `Secret` resource. Workload Identity can be enabled when creating a [new cluster](https://learn.microsoft.com/en-us/azure/aks/workload-identity-deploy-cluster#create-an-aks-cluster) or can be added to an [existing cluster](https://learn.microsoft.com/en-us/azure/aks/workload-identity-deploy-cluster#update-an-existing-aks-cluster). :::danger Azure Workload Identity can be complex to configure and difficult to troubleshoot. Before continuing, be certain of the following: * Your AKS cluster has the **OIDC Issuer** feature enabled. * Your AKS cluster has the **Workload Identity** feature enabled. ::: For Workload Identity to work, the Kargo controller's Kubernetes `ServiceAccount` will need to be federated with a __managed identity__. Follow [these instructions](https://learn.microsoft.com/en-us/azure/aks/workload-identity-deploy-cluster#create-a-managed-identity) to create one and [these](https://learn.microsoft.com/en-us/azure/aks/workload-identity-deploy-cluster#create-the-federated-identity-credential) to federate it with the controller's `ServiceAccount`. :::info Federating the managed identity to the Kargo controller's `ServiceAccount` establishes a trust relationship. In AKS clusters with Workload Identity enabled, a [mutating admission webhook](https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#mutatingadmissionwebhook) will intercept the creation of any `Pod` resource labeled with `azure.workload.identity/use: "true"` _and_ using a `ServiceAccount` that's been federated to a managed identity. Knowing such a `Pod` is authorized to act on behalf of the associated managed identity, the webhook will modify the `Pod`'s spec to inject credentials in a well-known location for discovery by any Azure clients executing within any of its containers. ::: To access container images or Helm charts hosted in ACR, the managed identity [must be granted the `AcrPull` role](https://learn.microsoft.com/en-us/azure/container-registry/container-registry-authentication-managed-identity?tabs=azure-cli#grant-identity-access-to-the-container-registry) on the registry or on individual repositories within it. :::danger Before continuing, be certain of the following: * You have created a **User-Assigned Managed Identity**. ⚠️ This is different from an App Registration! * You have created a **Federated Identity Credential** that associates the managed identity with the Kubernetes `ServiceAccount` used by the Kargo controller. (In a typical installation of Kargo, this is the `kargo-controller` `ServiceAccount` in the `kargo` namespace.) * The managed identity has been granted the **`AcrPull` role** on your ACR registry or specific repositories within it. ::: For Workload Identity to inject credentials into any `Pod`, two specific Kargo configuration settings are required: 1. Controller `Pod`s must be labeled with `azure.workload.identity/use: "true"`. This label can be affixed to Kargo controller `Pod`s by using the `controller.podLabels` setting in Kargo's Helm chart at the time of installation or upgrade. 1. The controller's `ServiceAccount` must be annotated with `azure.workload.identity/client-id: `. :::warning Azure documentation states this annotation is optional, however, in practice, it often _is_ required. ::: This annotation can be affixed to the Kargo controller's `ServiceAccount` by using the `controller.serviceAccount.annotations` setting in Kargo's Helm chart at the time of installation or upgrade. Example Helm values: ```yaml controller: podLabels: azure.workload.identity/use: "true" serviceAccount: annotations: azure.workload.identity/client-id: ``` :::info For further guidance on this, refer to the advanced installation guides for [Helm](../20-advanced-installation/10-advanced-with-helm.md) or [Argo CD](../20-advanced-installation/20-advanced-with-argocd.md) ::: :::warning If the `azure.workload.identity/use: "true"` label is present on the Kargo controller's `Pod` and the `azure.workload.identity/client-id` annotation is also present on the Kargo controller's `ServiceAccount`, _but_ the `Pod` was started prior to Workload Identity having been enabled in the cluster or prior to the controller's `ServiceAccount` having been federated with a managed identity, the `Pod` will not have been injected with necessary credentials. Such a `Pod` should be deleted. The controller's `Deployment` will create a replacement `Pod` which will be injected with necessary credentials. ::: :::caution For optimal adherence to the principle of least privilege, the managed identity associated with the `kargo-controller` `ServiceAccount` should be limited only to the `AcrPull` role on the specific ACR repositories required by your Kargo Projects. ::: Tokens Kargo obtains for accessing any specific ACR repository are valid for approximately 3 hours and cached for 2.5 hours. A controller restart clears the cache. :::note When authenticating to ECR using EKS Pod Identity or IRSA (Amazon), or when authenticating to GAR using Workload Identity Federation (Google), the option exists for strict adherence to the principle of least privilege by granting the identity associated with the Kargo controller no permissions other than those required to assume/impersonate other, Project-specific identities. Project-specific identities can then be granted access only to the specific registries or repositories. Assuming/impersonating a Project-specific identity in Azure is considerably more complex than doing so in AWS or GCP. As a result, the Kargo controller lacks the option described above for Azure Workload Identity / ACR. ::: --- ## Security If you're managing a Kargo installation used by many developers, you're certain to have questions about recommended installation options, SSO, access control, compliance, and more. This section is dedicated to exploring such topics from an operator's perspective. --- ## Kargo Operator's Guide The Kargo's Operator's Guide is for individuals responsible for the installation and maintenance of Kargo. This entire branch of the documentation tree is dedicated to topics such as installation, configuration, topology, and security -- all from an operator's perspective. :::note It is not necessary to be responsible for a large-scale or long-lived installation of Kargo to benefit from this Operator's Guide. Even if you are only installing Kargo into a local Kubernetes cluster just to see what it's all about, we count you as a part-time operator and believe you will find a wealth of useful information in these pages. ::: :::info[Not what you were looking for?] If you're a developer using Kargo to implement flexible and powerful GitOps promotion pipelines, you'll find the most relevant topics to that role documented in the [User Guide](../50-user-guide/index.md). If you're interested in contributing to Kargo, you'll find everything you need to get started in the [Contributor Guide](../60-contributor-guide/index.md). ::: --- ## Installing the Kargo CLI Although its web-based UI provides the best user experience, Kargo also offers a command line interface (CLI) for users who may prefer it. :::info[Not what you were looking for?] If you're looking for instructions on installing Kargo's cluster-side components, refer to the [Basic Installation](../../40-operator-guide/10-basic-installation.md) section of the Operator Guide. ::: To install: 1. Download the CLI binary. The easiest method of downloading the binary is by selecting the CLI tab from the left navbar of the UI. There, you can select the appropriate binary for your operating system and CPU architecture. CLI downloads offered at this location will automatically match the version of your Kargo API server for optimum compatibility. Your downloaded binary will be suffixed with your OS and CPU architecture (e.g. `kargo-darwin-arm64`), so be sure to rename it to simply `kargo` (or `kargo.exe` for Windows). ![CLI Tab in Kargo UI](./img/cli-installation.png) If you prefer, the following commands will download the latest version of the CLI for your specific OS and CPU architecture and will also rename the binary to `kargo` (or `kargo.exe` on Windows.) ```shell arch=$(uname -m) [ "$arch" = "x86_64" ] && arch=amd64 curl -L -o kargo https://github.com/akuity/kargo/releases/latest/download/kargo-"$(uname -s | tr '[:upper:]' '[:lower:]')-${arch}" chmod +x kargo ``` ```shell Invoke-WebRequest -URI https://github.com/akuity/kargo/releases/latest/download/kargo-windows-amd64.exe -OutFile kargo.exe ``` 1. Regardless of your chosen download method, complete the installation by moving the binary to a location in your file system that is included in the value of your `PATH` environment variable. --- ## Kargo's Core Concepts Kargo is an unopinionated continuous [promotion](#promotions) platform that helps developers orchestrate the movement of new code and configuration through the various [stages](#stages) of their applications' lifecycles using GitOps principles. Kargo is light on new terminology and the few new terms it introduces should be easy to understand. This page is here to present a high-level overview and get you up to speed quickly without getting mired in technical details. ## Promotions _Kargo is all about promotions._ If you have some familiarity with GitOps, you know that _GitOps agents_ like [Argo CD](https://argoproj.github.io/cd/) excel at making the actual state of a Kubernetes cluster reflect _desired state_ that is expressed declaratively and stored in a Git repository. GitOps provides no specific guidance, however, on how desirable changes can be propagated from the desired state of one stage in an application's lifecycle to the desired state of the next. Any means by which such a thing might be accomplished, we would consider to be a "promotion process," and it is exactly these sort of processes that Kargo is designed to help you implement and manage. If you've ever made a change to your "test" environment, were satisfied with the change, and then either manually applied the same change to another environment, such as "staging," or else grappled with how to automate such a process, then you're already familiar with the idea of a promotion. :::note Promotions are different from _deployments_. Promotions focus on propagating changes to the _desired state_ of each stage in your application's lifecycle. The job of _deploying_ -- making the actual state of a Kubernetes cluster reflect the desired state -- is left to a GitOps agent like Argo CD. ::: ## Stages Many users equate stages with _environments_, and that's a fine way to think of them if it suits your use case. Technically, a stage is a _promotion target_. It represents some desired state that needs to be altered by a promotion process. The underlying resources that a GitOps agent will reconcile against that desired state can be varied according to your needs. It could be a particular instance of your entire application or just a few microservices that are part of a larger whole. It could even be an entire Kubernetes cluster if that's what fits your use case. Quite importantly, stages can be linked together to form a promotion pipeline that defines the flow of state changes through your application's lifecycle. ![Stage](./img/kargo-stages.png) ## Freight Freight is a new term, but one we hope is easy to understand. The state changes that flow through your promotion pipelines are meant to effect upgrades to new revisions of various artifacts. These artifacts could be container images containing your application's binaries. They could be configuration (Kubernetes manifests) in the form of a Helm chart or a commit in a Git repository. Often, it is a set of _multiple artifacts_ that must make the journey together from one end of the pipeline to the other. Formally, freight is a "meta-artifact" that references specific revisions of artifacts like images and manifests, but it is easiest to think of freight as _boxes_ containing multiple artifacts. If you order multiple items from an online retailer and the fulfillment center packages them together in a single box, you can be assured that those items will remain together through the shipping process, all the way to your door. This is a solid analogy for how freight works in Kargo. ![Freight](./img/kargo-freight.png) ## Warehouses If freight can be conceptualized as boxes, then warehouses are the places those boxes come from. In more technical terms, a warehouse monitors one or more repositories for new revisions of artifacts, and when it finds them, it packages the latest revision of each together as a new piece of freight that will enter the promotion pipeline. ![Warehouse](./img/kargo-warehouses.png) ## Projects Kargo projects are units of tenancy for defining policies and organizing and managing the related resources that make up one or more promotion pipelines. On a technical level, every Kargo project is associated with a Kubernetes namespace, which makes it convenient to manage access to resources within the project using standard Kubernetes RBAC. ![Project](./img/kargo-projects.png) ## What next? For more technical coverage of the concepts presented here, please refer to the corresponding [How-to Guides](/user-guide/how-to-guides). --- ## Logging In Whether you wish to interact with Kargo through its web-based UI or through its [CLI](../../05-installing-the-cli/index.md), you will need to log in first. 1. Obtain the address of your Kargo API server. Unless you installed the cluster-side components of Kargo yourself, you can obtain the address of the Kargo API server from the operator who did. The remaining steps will use the fictional address `https://kargo.example.com`. 1. 1. Open your web browser and navigate to the address of your Kargo API server. :::info The Kargo API server also serves the web-based UI. ::: You will be presented with the login screen, where you may have the option to log in as the admin, using SSO (Single Sign-On), or possibly both, as depicted below: ![Kargo Login Screen](./img/logging-in.png) :::info It is common for local installations of Kargo to have the admin user enabled, while production installations are more likely to have SSO enabled and the admin user disabled. If the admin user is enabled, the password can be obtained from the operator who installed Kargo's cluster-side components. ::: 1. Enter the admin user's password and click Login. OR Click SSO Login and follow the instructions to authenticate to your identity provider. When you have successfully authenticated, you will be redirected to the UI's main screen, which displays a list of Kargo `Project`s. ![Kargo Projects List](./img/projects.png) * To log in as the admin user, run the following command and then enter the admin user's password when prompted: ```shell kargo login https://kargo.example.com --admin ``` :::info It is common for local installations of Kargo to have the admin user enabled, while production installations are more likely to have SSO enabled and the admin user disabled. If the admin user is enabled, the password can be obtained from the operator who installed Kargo's cluster-side components. ::: OR * To log in using SSO, run the following command: ```shell kargo login https://kargo.example.com --sso ``` This will open your system's default web browser. Follow the instructions to authenticate to your identity provider. When you have successfully authenticated, you will be redirected to a splash page that will indicate you may proceed with using the CLI. ![Kargo Splash Page](./img/splash.png) --- ## Working with Projects Each Kargo project is represented by a cluster-scoped Kubernetes resource of type `Project`. Reconciliation of such a resource effects all boilerplate project initialization, including the creation of a specially-labeled `Namespace` with the same name as the `Project`. All resources belonging to a given `Project` should be grouped together in that `Namespace`. A minimal `Project` resource looks like the following: ```yaml apiVersion: kargo.akuity.io/v1alpha1 kind: Project metadata: name: example ``` :::note Deletion of a `Project` resource results in the deletion of the corresponding `Namespace`. For convenience, the inverse is also true -- deletion of a project's `Namespace` results in the deletion of the corresponding `Project` resource. ::: :::info There are compelling advantages to using `Project` resources instead of permitting users to create `Namespace` resources directly: * The required label indicating a `Namespace` is a Kargo project cannot be forgotten or misapplied. * Users can be granted permission to indirectly create `Namespace` resources for Kargo projects _only_ without being granted more general permissions to create _any_ new `Namespace` directly. * Boilerplate configuration is automatically created at the time of `Project` creation. This includes things such as project-level RBAC resources and `ServiceAccount` resources. ::: ## Project Configuration A `ProjectConfig` resource defines project-level configuration for an associated `Project`. At present, this only includes [promotion policies](#promotion-policies) that describe which `Stage`s are eligible for automatic promotion of newly available `Freight`. The `ProjectConfig` resource must have the same name as its associated `Project` and be created in the `Namespace` of the `Project`. This separation of configuration from the `Project` resource enables more granular RBAC control. Users can be granted permission to modify project configurations via `ProjectConfig` resources without necessarily having broader access to `Project` resources themselves. ### Promotion Policies A `ProjectConfig` resource can contain multiple promotion policies. Each policy is defined by a `stageSelector` and an `autoPromotionEnabled` flag. The `stageSelector` specifies which `Stage`s the policy applies to, and the `autoPromotionEnabled` flag indicates whether automatic promotion is enabled for those `Stage`s. :::info[Not what you were looking for?] This section focuses only on _enabling or disabling_ auto-promotion for specific `Stage`s through Project-level configuration. For `Stage`-level controls over _which_ `Freight` are eligible for auto-promotion when enabled, refer to the [Auto-Promotion](./40-working-with-stages.md#auto-promotion) section of our Working with Stages guide. ::: #### Basic Promotion Policy In the example below, the `test` and `uat` `Stage`s are eligible for automatic promotion of newly available `Freight`, but any other `Stage`s in the `Project` are not: ```yaml --- apiVersion: kargo.akuity.io/v1alpha1 kind: Project metadata: name: example --- apiVersion: kargo.akuity.io/v1alpha1 kind: ProjectConfig metadata: name: example namespace: example spec: promotionPolicies: - stageSelector: name: test autoPromotionEnabled: true - stageSelector: name: uat autoPromotionEnabled: true ``` #### Advanced Promotion Policies with Selectors Kargo supports more flexible ways to specify which `Stage`s a promotion policy applies to, using either pattern matching or label selectors. :::warning Pattern and label matching introduce security considerations. Users with appropriate permissions could potentially create resources with names or labels deliberately crafted to match patterns, bypassing intended promotion controls. Using [exact names](#basic-promotion-policy) provides the most secure option. ::: ##### Using Stage Selectors with Patterns You can use the `stageSelector` field with pattern matching to apply a promotion policy to multiple `Stage`s that match a specific pattern: ```yaml --- apiVersion: kargo.akuity.io/v1alpha1 kind: Project metadata: name: example --- apiVersion: kargo.akuity.io/v1alpha1 kind: ProjectConfig metadata: name: example namespace: example spec: promotionPolicies: - stageSelector: # Apply to a specific stage by exact name name: prod-east autoPromotionEnabled: false - stageSelector: # Apply to all stages matching a regex pattern name: "regex:test-.*" autoPromotionEnabled: true - stageSelector: # Apply to all stages matching a glob pattern name: "glob:dev-*" autoPromotionEnabled: true ``` The pattern matching supports: - Exact name matching (when no prefix is used) - Regex patterns with prefix `regex:` or `regexp:` - Glob patterns with prefix `glob:` ##### Using Stage Selectors with Labels You can also use [Kubernetes-style label selectors](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#resources-that-support-set-based-requirements) to apply a promotion policy to `Stage`s with specific labels: ```yaml --- apiVersion: kargo.akuity.io/v1alpha1 kind: Project metadata: name: example --- apiVersion: kargo.akuity.io/v1alpha1 kind: ProjectConfig metadata: name: example namespace: example spec: promotionPolicies: - stageSelector: matchLabels: environment: development autoPromotionEnabled: true - stageSelector: matchExpressions: - key: environment operator: In values: ["development", "staging"] autoPromotionEnabled: true ``` #### Using Stage Selectors with Patterns and Labels The [name](#using-stage-selectors-with-patterns) and [label](#using-stage-selectors-with-labels) selectors can be combined, in which case a `Stage` must match both the name and label selectors to be eligible for automatic promotion: ```yaml --- apiVersion: kargo.akuity.io/v1alpha1 kind: Project metadata: name: example --- apiVersion: kargo.akuity.io/v1alpha1 kind: ProjectConfig metadata: name: example namespace: example spec: promotionPolicies: - stageSelector: name: glob:prod-* matchLabels: example.org/allow-auto-promotion: "true" ``` In the example above, the promotion policy applies to all `Stage`s with the `example.org/allow-auto-promotion: "true"` label and names matching the `glob:prod-*` pattern. ### Message Channels Projects can define message channels to facilitate notifications and message sending as part of their workflows. Message channels can be configured for various platforms, such as Slack, SMTP, or HTTP webhooks. Channels are defined using the `MessageChannel` custom resource. This can be done either by applying YAML manifests or in the Kargo UI (also via YAML, though this will be made more user-friendly in future releases). Any message channel specification must include only a single channel type (e.g., Slack, SMTP, or HTTP, not multiple) and an optional `secretRef` to a Kubernetes `Secret` containing any necessary credentials. #### Examples ##### Slack This is an example `MessageChannel` configuration for Slack showing all options with annotations: ```yaml apiVersion: ee.kargo.akuity.io/v1alpha1 kind: MessageChannel metadata: name: test-env-slack # Must match the namespace of the Project namespace: kargo-demo spec: # A reference to a Secret containing the Slack token. This is required for Slack. The Secret must # contain the following key: # - `apiKey`: The Slack token with permissions to post messages to the desired channel secretRef: # The `namespace` field is ignored for `MessageChannel` as it is only allowed to reference # Secrets in the same namespace name: slack-token # Configuration specific to Slack slack: # The channel ID to send messages to. This field is required channelID: C1234567890 ``` ##### SMTP This is an example `MessageChannel` configuration for SMTP showing all options with annotations: ```yaml apiVersion: ee.kargo.akuity.io/v1alpha1 kind: MessageChannel metadata: name: engineering-team-smtp # Must match the namespace of the Project namespace: kargo-demo spec: # A reference to a Secret containing the SMTP credentials. This is required for SMTP. The Secret # must contain the following keys: # - `username`: The SMTP username # - `password`: The SMTP password secretRef: # The `namespace` field is ignored for `MessageChannel` as it is only allowed to reference # Secrets in the same namespace name: smtp-credentials smtp: # The email address to use in the "From" field. This field is required from: no-reply@example.com # The default recipient email addresses. This field is optional and can be overridden. The first # address in the list will be the primary recipient, and any additional addresses will be CC'd. to: [you@example.com] # The SMTP server host. This field is required host: smtp.gmail.com # The SMTP server port. This field is required port: 587 # Whether to use TLS when connecting to the SMTP server. This field is optional and defaults to # true useTLS: true # Whether to skip TLS certificate verification. This field is optional and defaults to false. # In most cases this should only be set to true for testing with self-signed certificates. insecureSkipVerify: false ``` ##### HTTP :::note HTTP channels are only supported by [`EventRouter`s](../60-reference-docs/90-events/100-notifications/10-configuring-routers.md). They cannot be used with the [`send-message`](../60-reference-docs/30-promotion-steps/send-message.md) promotion step. To make HTTP requests as part of a promotion workflow, use the [`http`](../60-reference-docs/30-promotion-steps/http.md) promotion step instead. ::: This is an example `MessageChannel` configuration for HTTP showing all options with annotations: ```yaml apiVersion: ee.kargo.akuity.io/v1alpha1 kind: MessageChannel metadata: name: webhook-endpoint # Must match the namespace of the Project namespace: kargo-demo spec: # A reference to a Secret containing authentication credentials. This is optional for HTTP. # The Secret may contain one of the following keys: # - `bearerToken`: A plain token that will be sent as a Bearer authorization header # - `authorization`: A custom value that will be sent verbatim as the Authorization header secretRef: # The `namespace` field is ignored for `MessageChannel` as it is only allowed to reference # Secrets in the same namespace name: webhook-secret # Configuration specific to HTTP http: # The endpoint to send the HTTP request to. This field is required url: https://hooks.example.com/notify # The HTTP method to use. Defaults to POST. Allowed values: GET, POST, PUT, PATCH, DELETE method: POST # Additional HTTP headers to include in the request. This field is optional headers: - name: Content-Type value: application/json # Additional query parameters to include in the request URL. This field is optional queryParams: - name: source value: kargo # Maximum time to wait for the request to complete. Must be a valid Go duration string. # Defaults to "10s" timeout: 30s # Whether to skip TLS certificate verification. Should only be used for development or testing insecureSkipTLSVerify: false # An expr-lang expression to evaluate whether the request succeeded. Has access to # response.status, response.body, and response.headers. If omitted, any 2XX status code is # treated as success. Can be overridden per-message in the EventRouter output successExpression: "response.status >= 200 && response.status < 300" # An expr-lang expression to evaluate whether the request failed. Has access to # response.status, response.body, and response.headers. If omitted, any non-2XX status code is # treated as failure. Can be overridden per-message in the EventRouter output failureExpression: "response.status >= 400" ``` The `response.body` field will attempt to parse the response body as JSON or YAML if the `Content-Type` response header indicates a JSON or YAML response. If parsing fails or the content type is not JSON/YAML, `response.body` will be treated as a string containing the raw response body. :::info When using success or failure expressions, please note that the 429 response status code (Too Many Requests) is reserved. Any response with this status code will never evaluate either expression as it is used to automatically backoff and retry requests that exceed rate limits. ::: ## Namespace Adoption At times, `Namespace`s may require specific configuration to comply with regulatory or organizational requirements. To account for this, Kargo supports the adoption of pre-existing `Namespace`s that are labeled with `kargo.akuity.io/project: "true"`. This enables pre-configuring such `Namespace`s according to your own requirements. :::info Requiring a `Namespace` to have the `kargo.akuity.io/project: "true"` label to be eligible for adoption by a new `Project` is intended to prevent accidental or willful hijacking of an existing `Namespace`. ::: The following example demonstrates adoption of a `Namespace` that's been pre-configured with a label unrelated to Kargo: ```yaml apiVersion: v1 kind: Namespace metadata: name: example labels: kargo.akuity.io/project: "true" example.com/org: platform-eng --- apiVersion: kargo.akuity.io/v1alpha1 kind: Project metadata: name: example spec: # ... ``` ## Preventing Namespace Deletion By default, when a Project is deleted, Kargo will attempt to delete the corresponding Namespace. However, there are scenarios where you may want to retain the namespace after the associated Project is removed whether it was created by Kargo or adopted from an existing setup. To achieve this, you can apply the following annotation to the `Namespace` or the corresponding `Project` resource with `kargo.akuity.io/keep-namespace: "true"`. ## Interacting with Projects Kargo provides tools to manage `Project`s using either its UI or CLI. This section explains how to handle `Project`s effectively through both interfaces. ### Creating a Project 1. Navigate to the Kargo UI and select New Project in the top right corner. A Form tab will appear where you can enter the name of your `Project`: ![create-project](img/create-project.png) Alternatively, you can define the `Project` and other related configurations using the YAML tab: ![create-project](img/create-project-2.png) 1. After completing the Form or defining the `Project` in the YAML tab, click Create. The new `Project` will appear a card on the UI's home page: ![create-project](img/create-project-3.png) 1. To create a `Project` using the CLI, run: ```bash kargo create project ``` Alternatively, define the `Project` in a YAML file, for example: ```yaml apiVersion: kargo.akuity.io/v1alpha1 kind: Project metadata: name: ``` Save the file and run: ```shell kargo create -f ``` 1. To verify creation of the `Project`, run: ```shell kargo get project ``` ### Deleting a Project 1. Select the `Project` you want to remove. 1. Go to the Settings in the top right corner of the `Project` view. ![delete-project](img/delete-project.png) 1. In the General tab, scroll down to the Delete Project section. 1. To confirm deletion, enter the `Project`'s name and click Delete to permanently remove it: ![delete-project](img/delete-project-2.png) To delete a `Project` using the CLI, run: ```shell kargo delete project ``` --- ## Working with Warehouses Kargo `Warehouse` resources each manage subscriptions to one or more of various types of artifact sources, including: - Container image repositories - Git repositories - Helm chart repositories When a `Warehouse` observes a new revision of any artifact to which it subscribes, it creates a new `Freight` resource representing a specific collection of artifact revisions that can be promoted from `Stage` to `Stage` _as a unit_. :::info For a broader, conceptual understanding of warehouses and their relation to other Kargo concepts, refer to [Core Concepts](./../10-core-concepts/index.md). ::: ## The `Warehouse` Resource Type A `Warehouse`'s subscriptions are all defined within its `spec.subscriptions` field. In this example, a `Warehouse` subscribes to both a container image repository and a Git repository: ```yaml apiVersion: kargo.akuity.io/v1alpha1 kind: Warehouse metadata: name: my-warehouse namespace: kargo-demo spec: subscriptions: - image: repoURL: public.ecr.aws/nginx/nginx constraint: ^1.26.0 - git: repoURL: https://github.com/example/kargo-demo.git ``` The remainder of this section focuses on the configuration of the individual subscription types. ### Container Image Subscriptions Container image repository subscriptions can be defined using the following fields: - `repoURL`: The URL of the container image repository _without any tag_. This field is required. - `imageSelectionStrategy`: One of four pre-defined strategies for selecting the desired image. (See next section.) - `allowTagsRegexes`: An optional list of regular expressions that limit eligibility for selection to tags that match any of the patterns. - `ignoreTagsRegexes`: An optional list of regular expressions that limit eligibility for selection to tags that don't match any of the patterns. - `platform`: An optional identifier that constrains image selection to those images supporting the specified operating system and system architecture. e.g., `linux/amd64`. :::note It is seldom necessary to specify this field. ::: - `discoveryLimit`: Many selection strategies (see next section) do not actually select a _single_ image; rather they select the n best fits for the specified constraints. The _best_ fit is the zero element in the list of selected images. `discoveryLimit` specifies how many images to discover. The default is `20`. :::note For poorly performing `Warehouse`s -- for instance ones frequently encountering rate limits -- decreasing this limit may improve performance. ::: - `insecureSkipTLSVerify`: Set to `true` to disable validation of the repository's TLS certificate. :::warning This is a security risk and should only be used in development environments. ::: - `strictSemvers`: StrictSemvers specifies whether only "strict" semver tags should be considered. `StrictSemvers` specifies whether only "strict" semver tags should be considered. A "strict" semver tag is one containing ALL of major, minor, and patch version components. This is enabled by default, but only has any effect when the `ImageSelectionStrategy` is `SemVer`. This should be disabled cautiously, as it is not uncommon to tag container images with short Git commit hashes, which have the potential to contain numeric characters only and could be mistaken for a semver string containing the major version number only. - `cacheByTag`: Set to `true` to enable more aggressive caching of image metadata using tags as keys. This can significantly reduce the number of API calls to the registry and improve performance. The default is `false`. :::warning[Use with caution!] This setting is safest if your tags are known to be "immutable" (i.e., tag always references the same image and is never updated to point to a different image). This setting does NOT apply to the `Digest` selection strategy, which assumes the one tag it subscribes to is a mutable one. ::: :::warning Operators may also choose from a number of policies regarding the caching of image metadata using tags as keys. Some of these policies (`Forbid` and `Force`) can override an individual container image subscription's choice to cache metadata by tag or not. See [common configurations](../../40-operator-guide/20-advanced-installation/30-common-configurations.md) for further details. ::: #### Image Selection Strategies For subscriptions to container image repositories, the `imageSelectionStrategy` field specifies a method for selecting the desired image. The available strategies are: - `SemVer`: Selects the image with the tag that best matches a semantic versioning constraint specified by the `constraint` field. If no such constraint is specified, the strategy simply selects the image with the semantically greatest tag. All tags that are not valid semantic versions are ignored. The `strictSemvers` field defaults to `true`, meaning only tags containing all three parts of a semantic version (major, minor, and patch) are considered. Disabling this should be approached with caution because any image tagged only with decimal characters will be considered a valid semantic version (containing only the major element). **`SemVer` is the default strategy if one is not specified.** :::info Kargo uses the [semver](https://github.com/masterminds/semver) package for parsing and comparing semantic versions and semantic version constraints. Refer to [these docs](https://github.com/masterminds/semver#checking-version-constraints) for detailed information on version constraint syntax. ::: Example: ```yaml spec: subscriptions: - image: repoURL: public.ecr.aws/nginx/nginx constraint: ^1.26.0 ``` - `Lexical`: This strategy selects the image with the lexicographically greatest tag. This is useful in scenarios wherein tags incorporate date/time stamps in formats such as `yyyymmdd` and you wish to select the tag with the latest stamp. When using this strategy, it is recommended to use the `allowTagsRegexes` field to limit eligibility to tags matching specific patterns. Example: ```yaml spec: subscriptions: - image: repoURL: public.ecr.aws/nginx/nginx imageSelectionStrategy: Lexical allowTagsRegexes: - ^nightly-\d{8}$ ``` - `Digest`: This selects the image _currently_ referenced by some "mutable tag" (such as `latest`) specified by the `constraint` field. :::warning "Mutable tags": Tags like `latest` that are sometimes, perhaps frequently, updated to point to a different, presumably newer image. "Immutable tags": Tags that have version or date information embedded within them, along with an expectation of never being updated to reference a different image. Using mutable tags like `latest` _is a widely discouraged practice._ Whenever possible, it is recommended to use immutable tags. ::: Example: ```yaml spec: subscriptions: - image: repoURL: public.ecr.aws/nginx/nginx imageSelectionStrategy: Digest constraint: latest ``` - `NewestBuild`: This strategy selects the image with the most recent build time. The build time is evaluated using the labels `org.opencontainers.image.created` or `org.label-schema.build-date`. If neither label is set, Kargo will fall back to using the `config.Created` time of the image. :::warning `NewestBuild` requires retrieving metadata for every eligible tag, which can be slow and is likely to exceed the registry's rate limits. **This can result in system-wide performance degradation.** If using this strategy is unavoidable, it is recommended to use the `allowTagsRegexes` field to limit the number of tags for which metadata is retrieved to reduce the risk of encountering rate limits. `allowTagsRegexes` may require periodic adjustment as a repository grows. ::: ```yaml spec: subscriptions: - image: repoURL: public.ecr.aws/nginx/nginx imageSelectionStrategy: NewestBuild allowTagsRegexes: - ^nightly ``` :::tip If your tags are known to be **immutable** (i.e., a tag always references the same image and is never updated to point to a different image), you can use the `cacheByTag` field to enable more aggressive caching of image metadata by tag. This can significantly reduce the number of API calls to the registry and improve performance. ```yaml spec: subscriptions: - image: repoURL: public.ecr.aws/nginx/nginx imageSelectionStrategy: NewestBuild cacheByTag: true allowTagsRegexes: - ^nightly ``` ::: :::warning[Use with caution!] Only enable `cacheByTag` if you are certain that all relevant tags are **immutable**. Using this with mutable tags (like `latest`) can cause Kargo to select stale images indefinitely. ::: ### Git Repository Subscriptions Git repository subscriptions can be defined using the following fields: - `repoURL`: The URL of the Git repository. This field is required. - `commitSelectionStrategy`: One of four pre-defined strategies for selecting the desired commit. (See next section.) - `allowTagsRegexes`: An optional list of regular expressions that limit eligibility for selection to tags that match any of the patterns. (This is not applicable to selection strategies that do not involve tags.) - `ignoreTagsRegexes`: An optional list of regular expressions that limit eligibility for selection to tags that don't match any of the patterns. (This is not applicable to selection strategies that do not involve tags.) - `expressionFilter`: An optional expression that filters commits and tags based on their metadata. See [Expression Filtering](#expression-filtering) for details. - `includePaths`: See [Git Subscription Path Filtering](#git-subscription-path-filtering). - `excludePaths`: See [Git Subscription Path Filtering](#git-subscription-path-filtering). - `discoveryLimit`: Many selection strategies (see next section) do not actually select a _single_ commit; rather they select the n best fits for the specified constraints. The _best_ fit is the zero element in the list of selected commits. `discoveryLimit` specifies how many commits to discover. The default is `20`. :::note Lowering this limit for a Git repository subscription does not improve performance by the margins that it does for a container image repository subscription. ::: - `since`: An optional date in RFC 3339 format (e.g. `2026-01-01T00:00:00Z`) that bounds how far back commit discovery will look. When specified, only commits at or after this date are considered. When left unspecified, there is no date cutoff. :::note `since` only has effect when `commitSelectionStrategy` is `NewestFromBranch` (or unspecified, since `NewestFromBranch` is the default). It is particularly useful for large repositories with long histories where `discoveryLimit` alone is not sufficient to prevent slow lookbacks. ::: - `insecureSkipTLSVerify`: Set to `true` to disable validation of the repository's TLS certificate. :::warning This is a security risk and should only be used in development environments. ::: #### Commit Selection Strategies For subscriptions to Git repositories, the `commitSelectionStrategy` field specifies a method for selecting the desired commit. The available strategies are: - `NewestFromBranch`: Selects the most recent commit from a branch specified by the `branch` field. If a branch is not specified, the strategy selects commits from the repository's default branch (typically `main` or `master`). This is useful for the average case, wherein you wish for the `Warehouse` to continuously discover the latest changes to a branch that receives regular updates. **`NewestFromBranch` is the default selection strategy if one is not specified.** Example: ```yaml spec: subscriptions: - git: repoURL: https://github.com/example/repo.git branch: main ``` - `SemVer`: Selects the commit referenced by the tag that best matches a semantic versioning constraint. All tags that are not valid semantic versions are ignored. With no constraint specified, the strategy simply selects the commit referenced by the semantically greatest tag. This is useful in scenarios wherein you do not wish for the `Warehouse` to continuously discover _every new commit_ and would like limit selection to commits tagged with a semantic version, and possibly within a certain range. The `strictSemvers` field defaults to `true`, meaning only tags containing all three parts of a semantic version (major, minor, and patch) are considered. Disabling this should be approached with caution because any image tagged only with decimal characters will be considered a valid semantic version (containing only the major element). :::info Kargo uses the [semver](https://github.com/masterminds/semver) package for parsing and comparing semantic versions and semantic version constraints. Refer to [these docs](https://github.com/masterminds/semver#checking-version-constraints) for detailed information on version constraint syntax. ::: Example: ```yaml spec: subscriptions: - git: repoURL: https://github.com/example/repo.git commitSelectionStrategy: SemVer semverConstraint: ^1.0.0 ``` - `Lexical`: Selects the commit referenced by the lexicographically greatest tag. This is useful in scenarios wherein you do not wish for the `Warehouse` to discover _every new commit_ and tags incorporate date/time stamps in formats such as `yyyymmdd` and you wish to select the tag with the latest stamp. When using this strategy, it is recommended to use the `allowTagsRegexes` field to limit eligibility to tags matching specific patterns. Example: ```yaml spec: subscriptions: - git: repoURL: https://github.com/example/repo.git commitSelectionStrategy: Lexical allowTagsRegexes: - ^nightly-\d{8}$ ``` - `NewestTag`: Selects the commit referenced by the most recently created tag. When using this strategy, it is recommended to use the `allowTagsRegexes` field to limit eligibility to tags matching specific patterns. Example: ```yaml spec: subscriptions: - git: repoURL: https://github.com/example/repo.git commitSelectionStrategy: NewestTag allowTagsRegexes: - ^nightly ``` #### Expression Filtering Git repository subscriptions support advanced filtering using expressions. These expressions allow you to filter commits and tags based on their metadata using [expr-lang](https://expr-lang.org) syntax. :::info The expressions must evaluate to a boolean value (`true` or `false`). If an expression evaluates to a non-boolean value, an attempt will be made to convert it to a boolean (e.g., `0` to `false`, `1` to `true`). ::: :::warning Invalid expressions will cause the subscription to fail. Always test your expressions to ensure they evaluate correctly with your repository's data. ::: :::info You can test your expressions using the [expr-lang playground](https://expr-lang.org/playground). The playground allows you to evaluate expressions against sample data and see the results in real-time. This is especially useful for debugging and validating your expressions before applying them to your `Warehouse` resources. ::: The `expressionFilter` field provides a unified way to filter commits or tags based on the selected commit selection strategy. The behavior and available variables depend on your `commitSelectionStrategy`: **For commit-based filtering** (`NewestFromBranch` strategy): - Filters commits based on commit metadata - Applied when selecting the newest commit from a branch **For tag-based filtering** (`SemVer`, `Lexical`, and `NewestTag` strategies): - Filters tags based on name and associated commit metadata - Applied after `allowTagsRegexes`, `ignoreTagsRegexes` and `semverConstraint` fields ##### Available Expression Filtering Variables The variables available in your expression depend on the commit selection strategy: **For `NewestFromBranch` (commit filtering):** - `id`: The ID (SHA) of the commit - `commitDate`: The date of the commit - `author`: The author of the commit, in format `Name ` - `committer`: The committer of the commit, in format `Name ` - `subject`: The first line of the commit message **For `SemVer`, `Lexical`, and `NewestTag` (tag filtering):** - `tag`: The name of the tag - `id`: The commit ID that the tag references - `creatorDate`: The tag creation date (annotated tag) or commit date (lightweight tag) - `author`: The author of the commit that the tag references, in the format of `Name ` - `committer`: The committer of the commit that the tag references, in the format of `Name ` - `subject`: The first line of the commit message associated with the tag - `tagger`: The tagger of the tag, in the format of `Name `. Only available for annotated tags. - `annotation`: The first line of the tag annotation. Only available for annotated tags. ##### Expression Filtering Examples **Filtering commits by excluding bot authors:** ```yaml spec: subscriptions: - git: repoURL: https://github.com/example/repo.git commitSelectionStrategy: NewestFromBranch expressionFilter: !(author contains '') ``` **Filtering commits with specific message patterns:** ```yaml spec: subscriptions: - git: repoURL: https://github.com/example/repo.git commitSelectionStrategy: NewestFromBranch expressionFilter: subject contains 'feat:' || subject contains 'fix:' ``` **Filtering commits with multiple criteria:** ```yaml spec: subscriptions: - git: repoURL: https://github.com/example/repo.git commitSelectionStrategy: NewestFromBranch expressionFilter: !(author == 'Example Bot') && commitDate.After(date('2025-01-01')) ``` **Filtering commits to exclude those with ignore markers:** ```yaml spec: subscriptions: - git: repoURL: https://github.com/example/repo.git commitSelectionStrategy: NewestFromBranch expressionFilter: !(subject contains '[kargo-ignore]') ``` **Filtering tags by author name:** ```yaml spec: subscriptions: - git: repoURL: https://github.com/example/repo.git commitSelectionStrategy: SemVer expressionFilter: author == 'John Doe ' ``` **Filtering tags created after a specific date:** ```yaml spec: subscriptions: - git: repoURL: https://github.com/example/repo.git commitSelectionStrategy: NewestTag expressionFilter: creatorDate.Year() >= 2024 ``` **Filtering tags to exclude those committed by bots:** ```yaml spec: subscriptions: - git: repoURL: https://github.com/example/repo.git commitSelectionStrategy: Lexical expressionFilter: !(committer contains '') ``` **Filtering tags with complex conditions:** ```yaml spec: subscriptions: - git: repoURL: https://github.com/example/repo.git commitSelectionStrategy: SemVer expressionFilter: creatorDate.After(date('2024-01-01')) && !(tag contains 'alpha') ``` #### Git Subscription Path Filtering In some cases, it may be necessary to constrain the paths within a Git repository that a `Warehouse` will consider as triggers for `Freight` production. This is especially useful for GitOps repositories that are ["monorepos"](../30-patterns/index.md#monorepo-layout) containing configuration for multiple applications. The paths that may or must not trigger `Freight` production may be specified using a combination of the `includePaths` and `excludePaths` fields of a Git repository subscription. The following example demonstrates a `Warehouse` with a Git repository subscription that will only produce new `Freight` when the latest commit (selected by the applicable commit selection strategy) contains changes in the `apps/guestbook` directory since the last piece of `Freight` produced by the `Warehouse`: ```yaml apiVersion: kargo.akuity.io/v1alpha1 kind: Warehouse metadata: name: my-warehouse namespace: kargo-demo spec: subscriptions: - git: repoURL: https://github.com/example/kargo-demo.git includePaths: - apps/guestbook ``` The next example demonstrates the opposite: a `Warehouse` with a Git repository subscription that will only produce new `Freight` when the latest commit (selected by the applicable commit selection strategy) contains changes to paths _other than_ the repository's `docs/` directory: ```yaml apiVersion: kargo.akuity.io/v1alpha1 kind: Warehouse metadata: name: my-warehouse namespace: kargo-demo spec: subscriptions: - git: repoURL: https://github.com/example/kargo-demo.git excludePaths: - docs ``` `includePaths` and `excludePaths` may be combined to include a broad set of paths and then exclude a subset of those. The following example demonstrates a `Warehouse` with a Git repository subscription that will only produce new `Freight` when the latest commit (selected by the applicable commit selection strategy) contains changes _within_ the `apps/guestbook` directory _other than_ the `apps/guestbook/README.md`: ```yaml apiVersion: kargo.akuity.io/v1alpha1 kind: Warehouse metadata: name: my-warehouse namespace: kargo-demo spec: subscriptions: - git: repoURL: https://github.com/example/kargo-demo.git includePaths: - apps/guestbook excludePaths: - apps/guestbook/README.md ``` :::note It is important to understand that new `Freight` will be produced when the latest commit (selected by the applicable commit selection strategy) contains _even a single change_ that is: 1. Implicitly included via undefined `includePaths`. OR Explicitly included via `includePaths`. AND 1. Not explicitly excluded via `excludePaths`. ::: :::note By default, the strings in the `includePaths` and `excludePaths` fields are treated as exact paths to files or directories. (Selecting a directory will implicitly select all paths within that directory.) Paths may _also_ be specified using glob patterns (by prefixing the string with `glob:`) or regular expressions (by prefixing the string with `regex:` or `regexp:`). ::: :::info These path filters are also respected by [webhook receivers](#triggering-artifact-discovery-using-webhooks). When a webhook receiver processes a `push` event from a supported provider (GitHub, GitLab, or Gitea), it extracts the list of changed files from the event payload and only refreshes Warehouses whose path filters match those files. Warehouses without path filters configured, or events from providers that do not include file change information in their payloads, will continue to be refreshed unconditionally. ::: ### Helm Chart Repository Subscriptions Helm chart repository subscriptions can be defined using the following fields: - `repoURL`: The URL of the Helm chart repository. This field is required. Chart repositories using http/s may contain versions of many _different_ charts. Subscriptions to all chart repositories using http/s **must** additionally specify the chart's name in the `name` field. For chart repositories in OCI registries, the repository URL points only to revisions of a _single_ chart. Subscriptions to chart repositories in OCI registries **must** leave the `name` field empty. - `name`: See above. - `semverConstraint`: Selects the chart version best matching this constraint. If left unspecified, the subscription implicitly selects the semantically greatest version of the chart. :::info Helm _requires_ charts to be semantically versioned. ::: :::info Kargo uses the [semver](https://github.com/masterminds/semver) package for parsing and comparing semantic versions and semantic version constraints. Refer to [these docs](https://github.com/masterminds/semver#checking-version-constraints) for detailed information on version constraint syntax. ::: - `discoveryLimit`: A chart repository subscription does not actually select a _single_ chart version; rather it selects the n best fits for the specified constraints. The _best_ fit is the zero element in the list of selected charts. `discoveryLimit` specifies how many chart versions to discover. The default is `20`. - `insecureSkipTLSVerify`: Set to `true` to disable validation of the repository's TLS certificate. :::warning This is a security risk and should only be used in development environments. ::: Example: ```yaml spec: subscriptions: - chart: repoURL: https://charts.example.com name: my-chart semverConstraint: ^1.0.0 ``` ## Working with Private Repositories Frequently, `Warehouse`s require access to private repositories, in which case appropriate credentials must be made available in some form. The many available authentication options are covered in detail on the [Managing Secrets](../50-security/30-managing-secrets.md) page. ## Automatic Freight Creation By default, `Warehouse`s create new `Freight` following a discovery cycle if new artifacts are found. This can be disabled by setting `freightCreationPolicy` to `Manual`. Example: ```yaml spec: freightCreationPolicy: Manual ``` For more granular control over the creation of your `Freight`, you can define `Freight` creation criteria in the form of an expression. One potential use case could be back end and front end versions needing to match. Example: ```yaml spec: freightCreationPolicy: Automatic subscriptions: - image: repoURL: ghcr.io/example/frontend - image: repoURL: ghcr.io/example/backend freightCreationCriteria: expression: | imageFrom('ghcr.io/example/frontend').Tag == imageFrom('ghcr.io/example/backend').Tag ``` For more information on `Freight Creation Criteria` refer to the [Expression Language Reference](../60-reference-docs/40-expressions.md). ## Performance Considerations ### Polling Frequency `Warehouse` resources periodically poll the repositories to which they subscribe in an attempt to discover new artifact revisions. By default, and under nominal conditions, this discovery process occurs at an interval configured at the system-level, however, the effective interval can be much longer if the system is under heavy load or `Warehouse`s are poorly configured. :::info Discovery of container images, in particular, can be time-consuming. Both the [`NewestBuild` selection strategy](#image-selection-strategies) and any [`platform` constraints](#container-image-subscriptions) are heavily dependent on the retrieval of image metadata for every image in the repository not eliminated from consideration up-front by other, more efficient constraints such as [`allowTagsRegexes`](#container-image-subscriptions) or [`ignoreTagsRegexes`](#container-image-subscriptions). Registry architecture, unfortunately, requires such metadata be retrieved image-by-image with a separate API call for each. Even with aggressive caching, and especially when the number of image revisions to consider is large, this process can take quite some time. The time required to complete discovery can be protracted even further if the registry's rate limit has been exceeded. Kargo can execute a finite number of these discovery processes concurrently and registries enforce rate limits on the basis of your public IP or, if applicable, your credentials. (i.e. Rate limits are not enforced on a `Warehouse`-by-`Warehouse` basis. Registries know nothing about your `Warehouse`s.) **Due to the above, even a well-tuned `Warehouse` that avoids inefficient image selection criteria may experience large intervals between executions of its discovery process (or slow discovery) if _other_ `Warehouse`s are configured inefficiently.** ::: With the goal of less frequent polling to reduce load on registries, avoid encountering rate limits, and reduce occurrences of discovery running for a prolonged period, only to find no new artifacts, you may wish to configure your `Warehouse` resources to execute artifact discovery less frequently than the system-wide default. (i.e. You may wish to _increase_ the polling interval.) This can be done by tuning the `spec.interval` field of any `Warehouse`. :::note The effective polling interval is the _greater_ of a system-wide minimum and any interval specified by `spec.interval`. i.e. You can configure a `Warehouse` to execute its artifact discovery process _less_ frequently than the system-wide minimum, _but not more frequently._ ::: :::info If you're an operator wishing to reduce the frequency with which _all_ `Warehouse`s execute their discovery processes (increase the minimum polling interval), refer to the [Common Configurations](../../40-operator-guide/20-advanced-installation/30-common-configurations.md#tuning-reconciliation-intervals) section of the of the Operator's Guide for more information. ::: With reduced polling frequency, overall system performance may improve, but will be accompanied by the undesired side effect of increasing the average time required for `Warehouse`s to notice new artifacts (of any kind; not just container images). _This can be overcome by configuring repositories to alert Kargo to the presence of new artifacts via webhooks._ ### Caching Image Metadata by Tag When using image selection strategies that require fetching image metadata (such as [`NewestBuild`](#image-selection-strategies) or when using [`platform`](#container-image-subscriptions) constraints), a significant bottleneck can be the number of API calls required to the container image registry. If your image tags are **guaranteed to be immutable** (i.e., a tag always references the exact same image and is never updated to point to a different image), you can enable the [`cacheByTag`](#container-image-subscriptions) option on individual image subscriptions: ```yaml spec: subscriptions: - image: repoURL: ghcr.io/example/myapp imageSelectionStrategy: NewestBuild cacheByTag: true allowTagsRegexes: - ^v\d+\.\d+\.\d+$ ``` This enables significantly more aggressive caching of image metadata, which can reduce API calls and improve performance by orders of magnitude in repositories with large numbers of tags. :::warning[Use with caution!] Only enable this option if your tags are known to be **immutable** (i.e., a tag always references the same image and is never updated to point to a different image). This setting does not apply to the `Digest` selection strategy, which always assumes tags are mutable. ::: ## Triggering Artifact Discovery Using Webhooks Configuring Kargo to receive webhook payloads from popular Git hosting providers and container image / OCI registry providers is easy, and can be configured [globally by an operator](../../40-operator-guide/35-cluster-configuration.md#triggering-artifact-discovery-using-webhooks) or at the Project level by a Project admin. The remainder of this section focuses on configuring webhook receivers at the Project level. :::info[Not what you were looking for?] If you're an operator looking to understand how you can configure Kargo to listen for inbound webhook requests to trigger the discovery processes of all applicable `Warehouse`s across all Projects, refer to the [Cluster Level Configuration](../../40-operator-guide/35-cluster-configuration.md#triggering-artifact-discovery-using-webhooks) section of the Operator's Guide. ::: ### Configuring a Receiver Creating and configuring a webhook receiver at the Project level is accomplished by updating your `ProjectConfig` resource's `spec.webhookReceivers` field. If your Project does not already have a `ProjectConfig` resource, you can create one. :::note Every Kargo Project is permitted to have at most _one_ `ProjectConfig` resource. This limit is enforced by requiring all `ProjectConfig` resources to be named _the same_ as the Project / `Project` resource / namespace to which they belong. For a Project `kargo-demo`, for example, the corresponding `ProjectConfig` must be contained within the Project's namespace (`kargo-demo`) and must, itself, be named `kargo-demo`. ::: A `ProjectConfig` resource's `spec.webhookReceivers` field may define one or more _webhook receivers_. A webhook receiver is an endpoint on a (typically) internet-facing HTTP server that is configured to receive and process requests from specific sources, and in response, trigger the discovery process of any `Warehouse` within the Project that subscribes to a repository URL referenced by the request payload. Most types of webhook receivers require you only to specify a unique (within the Project) name and a reference to a `Secret`. The expected keys and values for each kind of webhook receiver vary, and are documented on [each receiver type's own page](../60-reference-docs/80-webhook-receivers/index.md). :::info `Secret`s referenced by a webhook receiver typically serve _two_ purposes. 1. _Often_, some value(s) from the `Secret`'s data map are shared with the webhook sender (GitHub, for instance) and used to help authenticate requests. Some senders may use such "shared secrets" as bearer tokens. Others may use them as keys for signing requests. In such cases, the corresponding webhook receiver knows exactly what to do with this information in order to authenticate inbound requests. 1. _Always_, some value(s) from the `Secret`'s data map are used as a seed in deterministically constructing a complex, hard-to-guess URL where the receiver will listen for inbound requests. Some webhook senders (Docker Hub, for instance), do not natively implement any sort of authentication mechanism. No secret value(s) need to be shared with such a sender and requests from the sender contain no bearer token, nor are they signed. For cases such as these, a hard-to-guess URL is, itself, a _de facto_ shared secret and authentication mechanism. **Note that if a `Secret`'s value(s) are rotated, the URL where the receiver listens for inbound requests will also change. This is by design.** Kargo does not watch `Secret`s for changes because it lacks the permissions to do so, so it can be some time _after_ its `Secret`'s value(s) are rotated that a webhook receiver's URL will be updated. To expedite that update, your `ProjectConfig` resource can be manually "refreshed" using the `kargo` CLI: ```shell kargo refresh projectconfig --project ``` ::: The following example `ProjectConfig` configures two webhook receivers: ```yaml apiVersion: kargo.akuity.io/v1alpha1 kind: ProjectConfig metadata: name: kargo-demo namespace: kargo-demo spec: webhookReceivers: - name: my-first-receiver github: secretRef: name: my-first-secret - name: my-second-receiver gitlab: secretRef: name: my-second-secret --- apiVersion: v1 kind: Secret type: Opaque metadata: name: my-first-secret namespace: kargo-demo labels: kargo.akuity.io/cred-type: generic data: secret: c295bGVudCBncmVlbiBpcyBwZW9wbGUK --- apiVersion: v1 kind: Secret type: Opaque metadata: name: my-second-secret namespace: kargo-demo labels: kargo.akuity.io/cred-type: generic data: secret-token: cm9zZWJ1ZCB3YXMgYSBzbGVkCg== ``` :::note The `kargo.akuity.io/cred-type: generic` label on `Secret`s referenced by webhook receivers is not strictly required, but we _strongly_ recommend including it. ::: For each properly configured webhook receiver, Kargo will update the `ProjectConfig` resource's `status` to reflect the URLs that can be registered as endpoints with the senders. For instance, the `ProjectConfig` and `Secret`s above result in the following: ```yaml apiVersion: kargo.akuity.io/v1alpha1 kind: ProjectConfig metadata: name: kargo-demo namespace: kargo-demo spec: # ... omitted for brevity ... status: conditions: - lastTransitionTime: "2025-06-11T22:53:21Z" message: ProjectConfig is synced and ready for use observedGeneration: 1 reason: Synced status: "True" type: Ready webhookReceivers: - name: my-first-receiver path: /webhook/github/804b6f6bb40eb1f0e371f971d71dd95549be4bc9cbf868046941115f44073c67 url: https://kargo.example.com/webhook/github/804b6f6bb40eb1f0e371f971d71dd95549be4bc9cbf868046941115f44073c67 - name: my-second-receiver path: /webhook/gitlab/0eba9ff2a91f04f7787404b8f8f0edaf8cf8c39add34082651a474803cc99015 url: https://kargo.example.com/webhook/gitlab/0eba9ff2a91f04f7787404b8f8f0edaf8cf8c39add34082651a474803cc99015 ``` Above, you can see the URLs that can be registered with GitHub and GitLab as endpoints to receive webhook requests from those platforms. :::info For more information about registering these endpoints with specific senders, refer to [each receiver type's own page](../60-reference-docs/80-webhook-receivers/index.md). ::: :::info If you're working with a large number of Kargo Projects and/or repositories and wish for `Warehouse`s in all Projects to execute their discovery processes in response to applicable events, it will likely be impractical to configure webhook receivers Project-by-Project. Refer to [Cluster Level Configuration](../../40-operator-guide/35-cluster-configuration.md#triggering-artifact-discovery-using-webhooks) section of the Operator's Guide to learn how to register cluster-scoped webhook receivers that can trigger discovery for all applicable `Warehouse`s across all Projects. ::: ### Receivers in Action Once a webhook receiver has been assigned a URL and that URL has been registered with a compatible sender, the receiver will begin receiving webhook requests in response to events in your repositories. The payload (body) of such a request contains structured information (usually JSON) the sender wishes to share about some event. Invariably, among this information, is the URL of the repository from which the event originated. A webhook receiver extracts a repository URL from the webhook request's payload, queries for all `Warehouse` resources within the Project having subscriptions to that repository, and requests each to execute their discovery process. For Git `push` events from providers that include file change information in their payloads (GitHub, GitLab, and Gitea), the receiver additionally extracts the list of changed files and evaluates each matching Warehouse's [`includePaths` and `excludePaths`](#git-subscription-path-filtering) filters. Only Warehouses with path filters that match the changed files are refreshed. Warehouses without path filters configured are always refreshed. --- ## Working with Stages Each Kargo Stage is represented by a Kubernetes resource of type `Stage`. ## The `Stage` Resource Type Like most Kubernetes resources, a `Stage` is composed of a user-defined `spec` field and a system-populated `status` field. A `Stage` resource's `spec` field is itself composed of four main areas of concern: * Variables * Requested Freight * Promotion template * Verification The following sections will explore each of these as well as `status` in greater detail. ### Variables The `spec.vars` field allows you to define variables that can be referenced anywhere in the `Stage` specification that supports expressions, including the [promotion template](#promotion-templates) and [verification configuration](#verification). ```yaml apiVersion: kargo.akuity.io/v1alpha1 kind: Stage metadata: name: test namespace: kargo-demo spec: vars: - name: gitopsRepo value: https://github.com/example/kargo-demo.git - name: targetBranch value: stage/test - name: imageRepo value: public.ecr.aws/nginx/nginx # ... ``` Stage-level variables are merged with promotion template-level variables, with promotion template variables taking precedence for any conflicting names. This allows you to define common variables at the `Stage` level while still being able to override or supplement them at the promotion level as needed. :::info Variables defined at the Stage level can be referenced using `${{ vars. }}` syntax throughout the `Stage` specification, including in promotion templates and verification arguments. ::: ### Requested Freight The `spec.requestedFreight` field is used to describe one or more "types" of `Freight`, as specified by an `origin`, that the `Stage`'s promotion process, as specified by `spec.promotionTemplate`, will operate on, and the acceptable sources from which to obtain that `Freight`. Those sources may include the origin itself (e.g. a `Warehouse`) and/or any number of "upstream" `Stage` resources. :::info `Warehouse`s are the only type of origin at present, but it is anticipated that future versions of Kargo will introduce additional origin types. This is why "types" of `Freight` are described by an `origin` field having `kind` and `name` subfields instead of being described only by the name of a `Warehouse`. ::: #### Freight Availability When a `Stage` accepts `Freight` directly from its origin, _all_ new `Freight` created by that origin (e.g. a `Warehouse` ) are immediately available for promotion to that `Stage`. When a `Stage` accepts `Freight` from one or more "upstream" `Stage`s, `Freight` is considered available for promotion to that `Stage` only after being _verified_ in the upstream `Stage`(s). A `requestedFreight`'s `sources.availabilityStrategy` field specifies whether `Freight` must be verified in _any_ upstream `Stage` or _all_ upstream `Stage`s before becoming available. Valid strategies are: * `OneOf` (default): `Freight` is available for promotion after being verified in at least one of the upstream `Stage`s. * `All`: `Freight` is available for promotion only after being verified in all upstream `Stage`s listed in the `sources`. Last, any `Freight` that has been explicitly _approved_ for promotion to the `Stage` is available, without requiring upstream verification. :::info Explicit approvals are a useful method for applying the occasional "hotfix" without waiting for a `Freight` resource to traverse the entirety of a pipeline. ::: #### Auto-Promotion When [auto-promotion](./20-working-with-projects.md#promotion-policies) is enabled for a `Stage` through the project's `ProjectConfig`, `Stage`s will periodically search for available `Freight` according to the rules defined in the previous section and automatically initiate a promotion when suitable `Freight` are found to be available. :::info Auto-promotion being enabled through Project-level configuration is a security measure. If it were possible to enabled auto-promotion at the `Stage`-level, users with the requisite permissions to update a `Stage` resource, but _without_ the permissions to promote to that same `Stage` could effect a promotion regardless by enabling auto-promotion. Keeping enablement of auto-promotion defined at the Project-level ensures that (in practice) only a Project's administrator has the authority to enable or disable auto-promotion for any `Stage`. ::: The definition of "suitable" `Freight` is dependent on the `requestedFreight`'s `sources.autoPromotionOptions.selectionPolicy`. Valid policies are: * `NewestFreight`: (default): The _newest_ `Freight` that's been suitably verified or approved will be auto-promoted to the `Stage` on a continuous basis. * `MatchUpstream`: The `Freight` currently in use _immediately upstream_, if suitably verified or approved, will be be auto-promoted to the `Stage` on a continuous basis. This option is valid only when the `Stage` accepts `Freight` from _exactly one_ upstream `Stage`. #### Examples In the following example, the `test` `Stage` requests `Freight` that has originated from the `my-warehouse` `Warehouse` and indicates that it will accept new `Freight` _directly_ from that origin. If auto-promotion has been enabled (at the Project-level), the newest `Freight` will be auto-promoted to this `Stage` on a continuous basis: ```yaml apiVersion: kargo.akuity.io/v1alpha1 kind: Stage metadata: name: test namespace: kargo-demo spec: requestedFreight: - origin: kind: Warehouse name: my-warehouse sources: direct: true # These are the default options and could be omitted autoPromotionOptions: selectionPolicy: NewestFreight # ... # ... ``` In this example, the `qa` `Stage` requests `Freight` that has originated from the `my-warehouse` `Warehouse`, and indicates that it will accept such `Freight` only after it has been _verified_ in the `test` `Stage`. It additionally specifies that (if enabled at the Project-level), `Freight` currently in use by the `test` `Stage`, if suitably verified or approved, will continuously be auto-promoted: ```yaml apiVersion: kargo.akuity.io/v1alpha1 kind: Stage metadata: name: qa namespace: kargo-demo spec: requestedFreight: - origin: kind: Warehouse name: my-warehouse sources: stages: - test autoPromotionOptions: selectionPolicy: MatchUpstream # ... ``` In this example, the `uat` `Stage` requests `Freight` that has originated from the `my-warehouse` `Warehouse`, and indicates that it will accept such `Freight` only after it has been _verified_ in the `test` `Stage`: ```yaml apiVersion: kargo.akuity.io/v1alpha1 kind: Stage metadata: name: uat namespace: kargo-demo spec: requestedFreight: - origin: kind: Warehouse name: my-warehouse sources: stages: - test # ... ``` In the next example, the `prod` `Stage` requests `Freight` that has originated from the `my-warehouse` `Warehouse`, and indicates that it will accept such `Freight` only after it has been verified in _both_ the `qa` and `uat` `Stage`s: ```yaml apiVersion: kargo.akuity.io/v1alpha1 kind: Stage metadata: name: prod namespace: kargo-demo spec: requestedFreight: - origin: kind: Warehouse name: my-warehouse sources: availabilityStrategy: All stages: - qa - uat # ... ``` Stages may also request `Freight` from multiple sources. The following example illustrates a `Stage` that requests `Freight` from both a `microservice-a` and `microservice-b` `Warehouse`: ```yaml apiVersion: kargo.akuity.io/v1alpha1 kind: Stage metadata: name: test namespace: kargo-demo spec: requestedFreight: - origin: kind: Warehouse name: microservice-a sources: direct: true - origin: kind: Warehouse name: microservice-b sources: direct: true # ... ``` :::info By requesting `Freight` from multiple sources, a `Stage` can effectively participate in _multiple pipelines_ that may each deliver different collections of artifacts independently of the others. __This is _advanced_ configuration. If you're very new to Kargo and requesting `Freight` from multiple origins, you probably didn't mean to.__ ::: ### Promotion Templates The `spec.promotionTemplate` field is used to describe _how_ to transition `Freight` into the `Stage`. The `spec.promotionTemplate.steps` field describes the discrete steps of a promotion process in detail. In the following, very common example, the `promotionTemplate` describes steps to: 1. Clone a Git repository containing Kubernetes manifests and Kustomize configuration, checking out two different branches to two different directories. 1. Clears the contents of one working tree, with intentions to fully replace its contents. 1. Runs the equivalent of `kustomize edit set image` to update a `kustomization.yaml` file with a reference to an updated `public.ecr.aws/nginx/nginx` container image. 1. Renders the updated manifests using the equivalent of `kustomize build`. 1. Commits the updated manifests and pushes them to the `stage/test` of the remote repository. 1. Forces Argo CD to sync the `kargo-demo-test` application to the latest commit of the `stage/test` branch. ```yaml promotionTemplate: spec: vars: - name: gitopsRepo value: https://github.com/example/repo.git - name: imageRepo value: public.ecr.aws/nginx/nginx - name: srcPath value: ./src - name: outPath value: ./out - name: targetBranch value: stage/${{ ctx.stage }} steps: - uses: git-clone config: repoURL: ${{ vars.gitopsRepo }} checkout: - branch: main path: ${{ vars.srcPath }} - branch: stage/${{ ctx.stage }} create: true path: ${{ vars.outPath }} - uses: git-clear config: path: ${{ vars.outPath }} - uses: kustomize-set-image as: update-image config: path: ${{ vars.srcPath }}/base images: - image: ${{ vars.imageRepo }} - uses: kustomize-build config: path: ${{ vars.srcPath }}/stages/${{ ctx.stage }} outPath: ${{ vars.outPath }}/manifests.yaml - uses: git-commit as: commit config: path: ${{ vars.outPath }} message: ${{ outputs['update-image'].commitMessage }} - uses: git-push config: path: ${{ vars.outPath }} branch: ${{ vars.targetBranch }} - uses: argocd-update config: apps: - name: kargo-demo-${{ ctx.stage }} sources: - repoURL: ${{ vars.gitopsRepo }} desiredRevision: ${{ outputs.commit.commit }} ``` :::info For complete documentation of all Kargo's built-in promotion steps, refer to the [Promotion Steps Reference](../60-reference-docs/30-promotion-steps/index.md). ::: ### Verification The `spec.verification` field is used to describe optional verification processes that should be executed after a `Promotion` has successfully deployed `Freight` to a `Stage`, and if applicable, after the `Stage` has reached a healthy state. The following example depicts a `Stage` resource that references an `AnalysisTemplate` named `integration-test` to validate the `dev` `Stage` after any successful promotion: ```yaml apiVersion: kargo.akuity.io/v1alpha1 kind: Stage metadata: name: dev namespace: guestbook spec: # ... verification: analysisTemplates: - name: integration-test ``` :::info For complete documentation of how to perform verification, refer to the [Verification Guide](./60-verification.md). ::: ### Status The `status` field of a `Stage` resource records: * Conditions containing the last observations of the `Stage`'s current state. * The current phase of the `Stage`'s lifecycle (distilled from the conditions). * Details about the last `Promotion` and any in-progress `Promotion`. * History of `Freight` that has been deployed to the `Stage` (from most to least recent) along with the results of any associated verification processes. * The health status of related Argo CD `Application` resources. For example: ```yaml status: freightHistory: - id: 101bca5b0e18ca7913978a1da956308d2544f741 items: Warehouse/my-warehouse: commits: - healthCheckCommit: 111eaf55aa41f21bb9bb707ba1baa748b83ec51e id: 961cfaedbc53aacdb65110028839a2c1c281290d repoURL: https://github.com/example/kargo-demo.git images: - digest: sha256:b2487a28589657b318e0d63110056e11564e73b9fd3ec4c4afba5542f9d07d46 repoURL: public.ecr.aws/nginx/nginx tag: 1.27.0 name: 666209fd9755a1e48bec6b27f5f447747410dd9e origin: kind: Warehouse name: my-warehouse verificationHistory: - analysisRun: name: test.01j2w7aknhf3j7jteyqs72hnbg.101bca5 namespace: kargo-demo phase: Successful finishTime: "2024-07-15T22:13:57Z" id: 5535a484-bbd0-4f12-8cf4-be2c8e0041c9 phase: Successful startTime: "2024-07-15T22:13:34Z" health: argoCDApps: - healthStatus: status: Healthy name: kargo-demo-test namespace: argocd syncStatus: revision: 111eaf55aa41f21bb9bb707ba1baa748b83ec51e status: Synced status: Healthy lastPromotion: finishedAt: "2024-07-15T22:13:25Z" freight: commits: - healthCheckCommit: 111eaf55aa41f21bb9bb707ba1baa748b83ec51e id: 961cfaedbc53aacdb65110028839a2c1c281290d repoURL: https://github.com/example/kargo-demo.git name: 666209fd9755a1e48bec6b27f5f447747410dd9e origin: kind: Warehouse name: kargo-demo name: test.01j2w7a15cxjjgejresfyw6ysp.666209f status: finishedAt: "2024-07-15T22:13:25Z" freight: commits: - healthCheckCommit: 111eaf55aa41f21bb9bb707ba1baa748b83ec51e id: 961cfaedbc53aacdb65110028839a2c1c281290d repoURL: https://github.com/example/kargo-demo.git name: 666209fd9755a1e48bec6b27f5f447747410dd9e origin: kind: Warehouse name: kargo-demo freightCollection: id: 101bca5b0e18ca7913978a1da956308d2544f741 items: Warehouse/kargo-demo: commits: - healthCheckCommit: 111eaf55aa41f21bb9bb707ba1baa748b83ec51e id: 961cfaedbc53aacdb65110028839a2c1c281290d repoURL: https://github.com/example/kargo-demo.git name: 666209fd9755a1e48bec6b27f5f447747410dd9e origin: kind: Warehouse name: kargo-demo verificationHistory: - analysisRun: name: test.01j2w7aknhf3j7jteyqs72hnbg.101bca5 namespace: kargo-demo phase: "" id: 5535a484-bbd0-4f12-8cf4-be2c8e0041c9 phase: Pending startTime: "2024-07-15T22:13:34Z" phase: Succeeded observedGeneration: 1 phase: Steady ``` ## Interacting with Stages Kargo provides tools to manage Stages using either its UI or CLI. This section explains how to handle Stages effectively through both interfaces. :::info Users with credentials for and sufficient permissions within the Kargo control plane's Kubernetes cluster can also manage `Stage` resources using `kubectl`. ::: ### Creating a Stage 1. In the `Project` view, click Create in the upper right corner of the pipeline section to open a dropdown, then select Stage: ![create-stage](img/create-stage.png) A form will appear to input details for a new `Stage`: ![create-stage](img/create-stage-2.png) 1. Complete the form with the necessary details and submit it. The new `Stage` will be added to the pipeline, connected to other `Stage`s based on your configuration: ![create-stage](img/create-stage-3.png) 1. Define the `Stage` in a YAML file, for example: ```yaml apiVersion: kargo.akuity.io/v1alpha1 kind: Stage metadata: name: namespace: spec: ### Add your Stage specifications here ``` 1. Save the file and run: ```shell kargo create -f ``` 1. Verify the creation by listing `Stage`s: ```shell kargo get stage --project ``` ### Promoting Freight to a Stage 1. To promote `Freight` to a `Stage`, click the truck icon in the header of that node and then select Promote: ![Promote Freight to a Stage](img/promote-freight-to-a-stage.png) 1. From the timeline at the top of the screen, select the `Freight` you'd like to promote into the `Stage` by clicking Select: ![Promote Freight to a Stage](img/promote-freight-to-a-stage-2.png) 1. Confirm the action by clicking Promote: ![Kargo Promotion Confirmation](img/promote-freight-to-a-stage-3.png) A summary of the `Promotion` will pop up and will be updated in real-time as the steps of the promotion process complete. Once they have all completed, the `Promotion`'s status will change to Succeeded: ![Kargo Promotion View](img/kargo-promotion-view.png) You will also notice the freight timeline has been automatically updated. Every piece of `Freight` in the timeline is color-coded to indicate which `Stage`s (if any) are actively using it. You will see the one piece of `Freight` currently in the timeline is marked with the same color as the `Stage`'s node you recently promoted in the pipeline. This indicates this piece of `Freight` is currently used by that `Stage`. To promote `Freight` to a `Stage` using the CLI, run: ```shell kargo promote \ --project kargo-demo \ --freight f5f87aa23c9e97f43eb83dd63768ee41f5ba3766 \ --stage prod ``` Alternatively, you can reference the `Freight` you wish to promote using its alias: ```shell kargo promote \ --project kargo-demo \ --freight-alias frozen-tauntaun \ --stage prod ``` ### Deleting a Stage 1. Open the `Stage` view by clicking the staggered bars icon in the header of the `Stage` node within the pipeline. ![delete-stage](img/kargo-stage-staggered-bars-button.png) 1. In the `Stage` view, click Settings, scroll to the bottom, and click Delete. ![delete-stage](img/stage-delete.png) 1. A confirmation popup will appear, click Confirm to proceed. ![delete-stage](img/stage-delete-3.png) To delete a `Stage` using the CLI, run: ```shell kargo delete stage --project ``` ### Refreshing a Stage Refreshing a `Stage` triggers its reconciliation process, which includes checking for any newly-completed Promotions, queueing up the next, pending Promotion, when applicable, and executing any applicable health check processes. 1. Open the `Stage` view by clicking the staggered bars icon in the header of the `Stage` node that you want to refresh and click Refresh in the top-right corner of the pop-up window: ![refresh-stage](img/refresh-stage.png) To refresh a `Stage`, run: ```shell kargo refresh stage --project ``` ### Reverifying a Stage's Current Freight Verification processes, which run automatically following each successful Promotion, can also be re-run on-demand. This is useful for re-attempting a failed verification process or just to validate that applications within the `Stage` are performing as desired. 1. Open the `Stage` view by clicking the staggered bars icon in the header of the `Stage` node that you want to reverify and click Reverify at the top of the menu: ![verify-stage](img/reverify-freight.png) :::note If you wish to stop the in-progress verification, you can click Abort Verification. ::: 1. To check the `Stage`s where the `Freight` has been successfully verified, return to the `Freight` timeline and select the `Freight`. Verified `Stage` names will appear under VERIFIED IN: ![verify-stage](img/verified-in.png) 1. To rerun verification using the CLI, run: ```shell kargo verify stage --project ``` If you want to *stop* this ongoing verification process, use: ```shell kargo verify stage --project --abort ``` 1. To check the `Stage`s where the `Freight` has been successfully verified, run: ```shell kargo get freight \ --project \ --output jsonpath-as-json={.status} ``` --- ## Working With Freight **Freight** is an important Kargo concept. A single "piece of freight" is a set of references to one or more versioned artifacts, which may include one or more: - Container images (from image repositories) - Kubernetes manifests (from Git repositories) - Helm charts (from chart repositories) Freight can therefore be thought of as a sort of meta-artifact. Freight is what Kargo seeks to progress from one stage to another. :::info To learn the fundamentals of freight and the warehouses that produce freight, see [Core Concepts](../10-core-concepts/index.md). ::: The remainder of this page describes features of freight that will enable you to work more effectively. ## Names Like all Kubernetes resources, Kargo `Freight` resources have a `metadata.name` field, which uniquely identifies each resource of that type within a given Kargo project (a specially labeled Kubernetes namespace). When a `Warehouse` produces a new `Freight` resource, it will compute a canonical representation of the artifacts referenced by that resource and use that, in turn, to compute a SHA-1 hash. This becomes the value of the `metadata.name` field. The deterministic method of computing this value makes it a unique "fingerprint" of the collection of artifacts referenced by the `Freight` resource. ## Aliases While the `metadata.name` field contains a predictably computed SHA-1 hash, such identifiers are, unarguably, not very user-friendly. To make `Freight` resources easier for human users to identify, `Warehouse`s automatically generate a human-friendly alias for every `Freight` resource they produce and apply it as the value of the `Freight` resource's `kargo.akuity.io/alias` [label](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/). :::info Generating aliases of the form `-` is a strategy borrowed from Docker, which generates similar names for containers not explicitly named by users. ::: :::info Why a label? Kubernetes enforces the immutability of the `metadata.name` field for all resources. Kubernetes [labels](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/), by contrast, are both mutable and indexed, which makes them ideal for use as secondary identifiers. ::: When using the Kargo CLI to query for `Freight` resources, the `alias` field is always displayed: ```shell kargo get freight --project kargo-demo ``` Sample output: ```shell NAME ALIAS AGE f5f87aa23c9e97f43eb83dd63768ee41f5ba3766 mortal-dragonfly 35s ``` :::info The Kargo UI, to make efficient use of screen real estate, displays aliases only, but a `Freight` resource's `name` can always be discovered by hovering over its alias. ::: By default, the freight timeline shows the `Warehouse` name for each freight. You can also enable alias display in the freight timeline filter: click the _funnel icon_ in the freight timeline, then check the box next to Alias to show alias names on the freight timeline: ![Freight Alias Name](./img/freight-alias-name.png) :::note Kargo CLI commands will accept `Freight` aliases as an alternative to a `Freight` name. Refer to the help text for the `kargo` command for more information. ::: ### Updating Aliases While every `Freight` resource is automatically assigned an alias, users may sometimes wish to override that alias with one of their own choosing. This can make it easier to identify a particularly important (or problematic) `Freight` resource as it progresses through the `Stage`s of a pipeline. To update the `Freight` Alias: 1. Select a `Freight` from the `Freight` timeline to open its detailed view: ![update-freight-alias](img/freight-alias.png) 1. In the top right corner of the `Freight` view, click Edit Alias to modify the alias: ![update-freight-alias](img/freight-alias-2.png) 1. Specify a value in the New Alias field and click Submit: ![update-freight-alias](img/freight-alias-3.png) ```shell kargo update freight \ --project kargo-demo \ --name f5f87aa23c9e97f43eb83dd63768ee41f5ba3766 \ --new-alias frozen-tauntaun ``` Alternatively, you can reference the `Freight` to which you want to assign a new alias using its existing alias: ```shell kargo update freight \ --project kargo-demo \ --old-alias mortal-dragonfly \ --new-alias frozen-tauntaun ``` ## Manual Approvals The [Core Concepts](../10-core-concepts/index.md) describes the usual process by which `Freight` resources are _verified_ at each `Stage` in a pipeline before becoming available to the next `Stage` or `Stage`s. In brief, it typically requires the `Stage` to reach a healthy state _and_, if applicable, any user-defined verification processes to complete with favorable results. This is suitable for the average case wherein a new `Freight` resource is expected to traverse the entirety of a pipeline on its way to production, however, it is nearly inevitable that the occasional need for a "hotfix" will arise, in which case it may sometimes be desirable to bypass one or more `Stage`s in the pipeline. To enable this, Kargo provides the ability to manually approve a `Freight` resource for promotion to any given `Stage`. To manually approve the `Freight`: 1. Click the three dots on the `Freight` in the `Freight` timeline, then select Manually Approve: ![update-freight-alias](img/freight-approval.png) 1. Select Approve on the `Stage` for which you want to approve promotion of the `Freight`: ![update-freight-alias](img/freight-approval-2.png) ```shell kargo approve \ --project kargo-demo \ --freight f5f87aa23c9e97f43eb83dd63768ee41f5ba3766 \ --stage prod ``` :::note Manually granting approval for a `Freight` resource to be promoted to any given `Stage` requires the same level of permissions as would be required to carry out that promotion, although, granting manual approval does _not_ automatically create a corresponding `Promotion` resource. ::: After successfully granting manual approval for a `Freight` resource to be promoted to a given `Stage`, the `Freight` resource's `status` field will reflect that approval. The following depicts a `Freight` resource that has been verified in a `test` `Stage` through the usual process, but has been manually approved for promotion to the `prod` `Stage`. i.e. Any `Stage`s between `test` and `prod` may be bypassed: 1. Click on the `Freight` in the `Freight` timeline to open the `Freight` view: ![freight-status](img/freight-status.png) ```shell kargo get freight \ --project kargo-demo \ --output jsonpath-as-json={.status} ``` ```shell [ { "approvedFor": { "prod": {} }, "verifiedIn": { "test": {} } } ] ``` ## Promoting Freight to a Stage 1. To promote `Freight` to a `Stage`, click the truck icon in the header of the `Stage` node and then select Promote: ![Promote Freight to a Stage](img/promote-freight-to-a-stage.png) 1. From the `Freight` timeline at the top of the screen, select the `Freight` you'd like to promote into the `Stage` by clicking Select: ![Promote Freight to a Stage](img/promote-freight-to-a-stage-2.png) 1. Confirm the action by clicking Promote: ![Kargo Promotion Confirmation](img/promote-freight-to-a-stage-3.png) A summary of the `Promotion` will pop up and will be updated in real-time as the steps of the promotion process complete. Once they have all completed, the `Promotion`'s status will change to Succeeded: ![Kargo Promotion View](img/kargo-promotion-view.png) You will also notice the freight timeline has been automatically updated. Every piece of `Freight` in the timeline is color-coded to indicate which `Stage`s (if any) are actively using it: ![Kargo Freight Timeline](img/kargo-freight.png) To promote `Freight` to a `Stage` using the CLI, run: ```shell kargo promote \ --project kargo-demo \ --freight f5f87aa23c9e97f43eb83dd63768ee41f5ba3766 \ --stage prod ``` Alternatively, you can reference the `Freight` you wish to promote using its alias: ```shell kargo promote \ --project kargo-demo \ --freight-alias frozen-tauntaun \ --stage prod ``` ## OCI Image Annotations Kargo understands [OCI Image Annotations](https://github.com/opencontainers/image-spec/blob/main/annotations.md) and uses them to generate helpful links in the UI. When the following annotations are present in your container images, Kargo can display direct references to the source code and revision: - `org.opencontainers.image.source` - `org.opencontainers.image.revision` ![oci-annotations](img/freight-oci-annotations.png) ### Adding Annotations with GitHub Actions If you're using Docker's `build-and-push` GitHub Action, you can automatically include these annotations using the [`metadata-action` action](https://github.com/docker/metadata-action?tab=readme-ov-file#annotations). Here's an example workflow snippet: ```yaml - name: Docker meta id: meta uses: docker/metadata-action@v5 with: images: akuity/guestbook env: DOCKER_METADATA_ANNOTATIONS_LEVELS: manifest,index - name: Build and push uses: docker/build-push-action@v6 with: tags: ${{ steps.meta.outputs.tags }} annotations: ${{ steps.meta.outputs.annotations }} ``` ### Adding Annotations with `docker buildx` If you're building images manually using `docker buildx`, you can add annotations using the `--annotation` [flag](https://docs.docker.com/reference/cli/docker/buildx/build/#annotation): ``` GIT_SHA=$(git rev-parse --short HEAD) docker buildx build \ --push \ --platform linux/amd64,linux/arm64 \ --tag akuity/guestbook:latest --annotation index,manifest:org.opencontainers.image.source=https://github.com/akuity/guestbook \ --annotation index,manifest:org.opencontainers.image.revision=${GIT_SHA} \ . ``` By including these annotations, you enable Kargo to provide rich context around your container images directly in the UI. --- ## Verifying Freight in a Stage Following a successful `Promotion` (i.e. the _process_ succeeded), it is often desired to assert whether updated code or configuration meets expectations. Verification is an optional, user-defined process to test the `Freight` in a given `Stage`. Common examples of verification include: * Executing containerized integration test scripts * Querying monitoring tools (e.g. DataDog, Prometheus) and measuring application KPIs * Performing HTTP requests to internal services or systems and interpreting responses as success/fail * Running load generation tests When a `Stage` with verification is upstream from other `Stage`s, `Freight` must pass verification there before becoming eligible to be promoted downstream. ## Configuring Verification A `Stage` can be configured with verification by referencing one or more `AnalysisTemplates` in its `spec.verification` field. In the following example the `dev` `Stage` references an `AnalysisTemplate` named `integration-test`, which executes after any successful `Promotion`: ```yaml apiVersion: kargo.akuity.io/v1alpha1 kind: Stage metadata: name: dev namespace: guestbook spec: # ... verification: analysisTemplates: - name: integration-test ``` In addition to referencing `AnalysisTemplate`s, a `kind` field can be specified to reference [`ClusterAnalysisTemplate`s](https://argo-rollouts.readthedocs.io/en/stable/features/analysis/#clusteranalysistemplates), which are cluster-scoped resources that behave similarly to `AnalysisTemplate`s but are not namespaced: ```yaml apiVersion: kargo.akuity.io/v1alpha1 kind: Stage metadata: name: dev namespace: guestbook spec: # ... verification: analysisTemplates: - name: integration-test kind: ClusterAnalysisTemplate ``` ## Configuring AnalysisTemplates An [AnalysisTemplate](../60-reference-docs/50-analysis-templates.md) defines instructions for how to perform verification. It might include running a Kubernetes `Job`, querying monitoring tools, interpreting their result, performing HTTP requests, or any combination of these actions. `AnalysisTemplate`s reside in the same project namespace as the `Stage` from which they are referenced. The following simple `AnalysisTemplate` example wil create a Kubernetes `Job` that runs an alpine container that sleeps for 10 seconds: ```yaml apiVersion: argoproj.io/v1alpha1 kind: AnalysisTemplate metadata: name: integration-test namespace: guestbook spec: metrics: - name: integration-test provider: job: spec: template: spec: containers: - name: sleep image: alpine:latest command: [sleep, "10"] restartPolicy: Never backoffLimit: 1 ``` :::info For further documentation and examples of `AnalysisTemplate`s, refer to the [`AnalysisTemplate` reference](../60-reference-docs/50-analysis-templates.md). ::: ## How Verification Works After a successful `Promotion`, a `Stage` enters the `Verifying` phase. Any `AnalysisTemplate`s which were referenced by the `Stage` are used to spawn a resource called an `AnalysisRun`, which represents a single execution of those templates. When the `AnalysisRun` completes and is `Successful`, the verification is considered successful and the `Freight` is marked as "verified" in that `Stage`, which makes it available for promotion to any immediate downstream `Stage`s, either automatically (with auto-promotion) or manually (via the UI, API, or CLI). `Freight` which have not passed verification in a `Stage` are blocked from continuing further downstream. :::info It is sometimes desired to skip the verification process entirely (e.g. a hotfix needs to be fast-tracked to production). Verification (and the entire Pipeline for that matter) can be bypassed by [manually approving](./50-working-with-freight.md#manual-approvals) `Freight` for a specified `Stage`. ::: While a `Stage` is verifying (i.e. the `AnalysisRun` is not yet completed) no other `Promotion`s to that `Stage` will run until verification has completed (either successfully or unsuccessfully). Once verification has completed, the next queued `Promotion` will run. ## AnalysisRun An `AnalysisRun` is a resource representing a verification attempt for a `Freight` + `Stage` pair. `AnalysisRun`s have a lifecycle (e.g. `Running`, `Successful`, `Failed`) and also record measurements taken during the run. `AnalysisRun`s appear in the Kargo UI under the Verifications tab of a `Stage`: ![AnalysisRun in Kargo UI](./img/analysisrun.png) The following might be an example of an `AnalysisRun` spawned from the `integration-test` template: ```yaml apiVersion: argoproj.io/v1alpha1 kind: AnalysisRun metadata: creationTimestamp: 2025-01-28T20:47:50Z generation: 3 labels: kargo.akuity.io/freight-collection: 55d452301040a73e9fd05289b1f8ddbec1791222 kargo.akuity.io/stage: dev annotations: kargo.akuity.io/promotion: dev.01jjpfgfwkzk18k7cyq61jehf4.319ddec name: dev.01jjqaq7qacfn766tcp1nqz2zv.55d4523 namespace: guestbook ownerReferences: - apiVersion: kargo.akuity.io/v1alpha1 blockOwnerDeletion: false kind: Freight name: 319ddec6bbfbd808d499a0357b027e773aa3707f uid: fb0db653-92af-420b-9092-0c7bf9be290e resourceVersion: "54013326" uid: 8cb745cb-554b-4141-80ee-e0df181cd56e spec: metrics: - name: integration-test provider: job: spec: template: spec: containers: - name: sleep image: alpine:latest command: [sleep, "10"] restartPolicy: Never backoffLimit: 1 status: dryRunSummary: {} message: Metric "integration-test" assessed Failed due to failed (1) > failureLimit (0) metricResults: - count: 1 failed: 1 measurements: - finishedAt: 2025-01-28T20:50:31Z metadata: job-name: 8cb745cb-554b-4141-80ee-e0df181cd56e.integration-test.1 job-namespace: guestbook phase: Failed startedAt: 2025-01-28T20:47:50Z name: integration-test phase: Failed phase: Failed runSummary: count: 1 failed: 1 startedAt: 2025-01-28T20:47:50Z ``` :::info[`AnalysisTemplate` vs `AnalysisRun`] An `AnalysisTemplate` _defines_ a verification process, while an `AnalysisRun` tracks the _progress and result_ of an execution of that process. ::: ## Arguments and Metadata It is also possible to specify additional labels, annotations, and arguments that should be applied to `AnalysisRun` resources spawned from the referenced `AnalysisTemplate`: ```yaml apiVersion: kargo.akuity.io/v1alpha1 kind: Stage metadata: name: test namespace: kargo-demo spec: # ... verification: analysisTemplates: - name: kargo-demo analysisRunMetadata: labels: foo: bar annotations: bat: baz args: - name: foo value: bar ``` The values specified in `args` support [expressions](../60-reference-docs/40-expressions.md) to dynamically set values based on the context of the `Freight` being verified. For example, the following defines an argument `commit` with a value set to the commit hash that is being verified using the [`commitFrom` expression function](../60-reference-docs/40-expressions.md#commitfrom): ```yaml apiVersion: kargo.akuity.io/v1alpha1 kind: Stage metadata: name: test namespace: kargo-demo spec: # ... verification: analysisTemplates: - name: kargo-demo args: - name: commit value: ${{ commitFrom("https://github.com/example/repo.git").ID }} ``` :::caution Kargo promotion processes require expressions to be enclosed within `${{ }}`, Argo Rollouts `AnalysisTemplate`s require expressions to be enclosed within `{{ }}` (i.e. without `$`). Ensure that expressions are enclosed within the correct syntax for the type of resource you are working with, recognizable by the `apiVersion` and `kind` fields. ::: :::caution If specifying arguments for use in the creation of an `AnalysisRun` via a `Stage`'s `spec.verification.args` field, each `AnalysisTemplate` referenced in `spec.verification.analysisTemplates` _must_ declare those arguments in its own `spec.args` field. The following example `AnalysisTemplate` complements the example `Stage` shown above: ```yaml apiVersion: argoproj.io/v1alpha1 kind: AnalysisTemplate metadata: name: integration-test namespace: guestbook spec: args: - name: commit metrics: # ... ``` Failure to declare arguments will cause `AnalysisRun`s (and therefore, verification) to fail. Refer directly to the [Rollouts Docs](https://argo-rollouts.readthedocs.io/en/stable/features/analysis/#analysis-template-arguments) for further details. ::: ## Implicit Argo CD Verification It is usually desirable for changes to be fully rolled out before performing any verification. For example, it would be premature to start running tests if a `Deployment`'s `Pod`s were still starting or not fully transitioned to the desired version. `Promotion`s that integrate with Argo CD do wait for any sync operations they initiate to complete, but it is possible for an Argo CD `Application` to still be `Progressing` even after sync operations have completed and the `Promotion` process is considered complete. In order to prevent verification from starting before `Application`s are ready, Kargo has a built-in understanding of [Argo CD `Application` health](https://argo-cd.readthedocs.io/en/latest/operator-manual/health/) and will delay verification until `Application`s have reached a `Healthy` state. :::note If a `Stage` references one or more Argo CD `Application`s as part of its promotion process, but does not explicitly define any verification process, successful verification is implicitly contingent on the `Application`s reaching a `Healthy` state. ::: ## Soak Times In some scenarios, automated testing may be unavailable or insufficient to ensure confidence before promoting software to critical environments. To mitigate this, users often resort to letting the release run in a lower environment for a period of time, monitoring its behavior and stability before proceeding with promotion. This practice is commonly known as "soaking" or "baking" the release before it reaches production. `Stage`s may optionally specify a "soak time" when requesting `Freight` from upstream `Stage`s. To configure this, a duration can be specified using the `spec.requestedFreight[].sources.requiredSoakTime` field. Valid durations are expressed in seconds, minutes or hours (e.g. `180s`, `30m`, `48h`). This duration is the minimum duration (following a successful promotion), for which the requested `Freight` must have continuously occupied ("soaked in") in an upstream `Stage` before becoming available for promotion to this `Stage`. Both automated and manual promotions are blocked until the soak time requirement is met. ```yaml apiVersion: kargo.akuity.io/v1alpha1 kind: Stage metadata: name: prod namespace: guestbook spec: requestedFreight: - origin: kind: Warehouse name: guestbook sources: stages: - uat requiredSoakTime: 1h # Omitted for brevity... ``` :::note Soak time is not technically a verification feature, but is documented alongside it because both relate to preventing `Freight` from moving downstream until required conditions have been met. These two features may even be used in conjunction with one another. ::: :::info [Manually approving](./50-working-with-freight.md#manual-approvals) `Freight` for a `Stage` makes it immediately available to that `Stage` _regardless_ of whether any required soak time has elapsed. ::: --- ## Argo CD Integration Argo CD excels at syncing Kubernetes clusters to desired state stored in a Git repository, but lacks any concept of a "promotion", i.e. updating the desired state of some application instance to reflect the validated state of some other application instance. Kargo was created to orchestrate such processes. Because such orchestration naturally entails some direct and indirect integration with Argo CD, this page details certain key elements of the interactions between the two systems. ## Updating Argo CD Applications In the course of orchestrating the transition of an application instance from one state to another, it is common for Kargo to updated Argo CD `Application` resources in some way. Such updates are enabled through the use of the [`argocd-update` promotion step](../60-reference-docs/30-promotion-steps/argocd-update.md). Often, these updates entail little more than modifying an `Application`'s `operation` field to force the `Application` to be synced to recently updated desired state. ```yaml steps: - uses: argocd-update config: apps: - name: my-app sources: - repoURL: https://github.com/example/repo.git desiredRevision: ``` :::info For in-depth information on the usage of the `argocd-update` step, see the [examples](../60-reference-docs/30-promotion-steps/argocd-update.md#examples). ::: ### Authorizing Updates Performing updates of any kind to an `Application` resource naturally requires Kargo to be _authorized_ to do so. Kargo controllers have the requisite RBAC permissions to perform such updates, but being a multi-tenant system, Kargo must also understand, internally, when it is acceptable to utilize those broad permissions to update a specific `Application` resource _on behalf of_ a given `Stage`. To enable Kargo controllers to update an Argo CD `Application` on behalf of a given `Stage`, that `Application` must be explicitly annotated as follows: ```yaml kargo.akuity.io/authorized-stage: ":" ``` Because an annotation such as the one above could only be added to an `Application` by a user who, themselves, is authorized to update that `Application`, Kargo interprets the presence of such an annotation as delegation of that user's authority to do so. In the following example, an `Application` has been annotated to authorize Kargo to update it on behalf of a `Stage` named `test` in the `kargo-demo` `Project`: ```yaml apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: kargo-demo-test namespace: argocd annotations: kargo.akuity.io/authorized-stage: kargo-demo:test spec: # Application Specifications ``` ## Health Checks When a `Promotion` uses an `argocd-update` step to update an `Application`, a [health check](../60-reference-docs/30-promotion-steps/argocd-update.md#health-checks) is registered for the `Stage` that the `Promotion` is targeting. This health check is used to continuously monitor the [health of the `Application`](https://argo-cd.readthedocs.io/en/stable/operator-manual/health/) that was updated by the `argocd-update` step as part of the `Stage` health. :::info It is important to note that `Stage` health is not determined solely by the health of the `Application`(s) that the `Stage` is managing. The health of the `Stage` is determined by the health of all `Application` resources that the `Stage` is managing, _as well as_ any other indicators of health that are part of the `Stage`'s definition. For example, a `Stage` may be considered unhealthy if the latest `Promotion` to that `Stage` failed. ::: --- ## Patterns With Kargo being mostly unopinionated, the full range of what you can accomplish with it is vast. Sometimes, that open-endedness can be overwhelming. This section attempts to enumerate, describe, and assign names to common solutions for common use cases. As tends to be the case with patterns, these were not invented so much as they were _discovered_. What we describe here is a distillation of things that have worked well for us and for other users. Importantly, the patterns described here can be tweaked to suit your needs and can even be _combined_ with one another to address more complex use cases. All patterns presented here will assume Argo CD as the GitOps agent. :::note The intent of this section is _not_ to constrain solutions to these patterns, but to provide inspiration and guidance on how Kargo's flexibility can be leveraged creatively. ::: ## Pipeline Structure "Pipelines" are not a formal concept in Kargo, but a term we casually use to reference the directed acyclic graph (DAG) of resources like `Warehouse`s and `Stage`s that define the flow of artifacts from their sources to their ultimate destinations. The patterns in this section are structural in nature. They describe common scenarios and strategies for addressing them _through careful pipeline design._ ### Image Updater This simple pattern fills a similar niche to that of [Argo CD Image Updater](https://argocd-image-updater.readthedocs.io/en/stable/). You might consider applying this if your only concern is updating the image used by some application as new versions become available. One advantage of approaching this with Kargo is that updates are rolled out stage by stage, with a failure in a "test" stage, for instance, preventing the same update from progressing to other stages, including production. ![Image Updater](./img/image-updater.png) To apply this pattern: 1. Create a single `Warehouse` that subscribes to an image repository with appropriate criteria so as to continuously identify the image revision you would consider to be newest. Each time it discovers a new image revision, the `Warehouse` will produce a new `Freight` resource that references it. 1. Connect any number of `Stage`s to this `Warehouse`. As `Freight` originating from it are promoted to each `Stage`, they will utilize a simple promotion process that: 1. Updates some stage-specific configuration in a Git repository to reference the new image revision. 1. Triggers an associated Argo CD `Application` to sync with the updated configuration. ### Config Updater This simple pattern continuously monitors a Git repository for configuration changes. A distinct advantage of this pattern is that a single configuration change that is intended to be rolled out to _all_ stages can be applied just once in the application's "base" configuration, but Kargo can _still_ promote the change stage by stage. ![Config Updater](./img/config-updater.png) To apply this pattern: 1. Create a single `Warehouse` that subscribes to a Git repository with appropriate criteria so as to continuously identify the commit containing the configuration you would consider to be newest. Typically this is simply whatever commit is at the head of your `main` branch. Each time it discovers a new commit, the `Warehouse` will produce a new `Freight` resource that references it. 1. Connect any number of `Stage`s to this `Warehouse`. As `Freight` originating from it are promoted to each `Stage`, they will utilize a simple promotion process that: 1. Uses a configuration management tool like Kustomize or Helm to combine the application's "base" configuration with stage-specific configuration. 1. Writes the combined configuration to a Git repository. 1. Triggers an associated Argo CD `Application` to sync with the combined configuration. :::note The combined configuration can be written to the same Git repository that is monitored by the `Warehouse` or an entirely separate repository. If writing to the same repository, the combined configuration should be written to a different (possibly stage-specific) branch _or_ to a specific path within the `main` branch that is _ignored_ by the `Warehouse` so as not to created a "feedback loop." This is discussed in greater details in the [Storage Options](#storage-options) section. ::: :::info This pattern can still update the image used by an application, but it treats such a change as it would any other configuration change. i.e. It does not occur automatically as in the [Image Updater](#image-updater) pattern, but any user-initiated update of application configuration to reference a new image revision will be rolled out in the same way as any other configuration change. ::: ### Common Case The common case combines the qualities of the [Image Updater](#image-updater) and [Config Updater](#config-updater) patterns. ![Common Case](./img/common-case.png) To apply this pattern: 1. Create a single `Warehouse` that subscribes to an image repository _and_ a Git repository with appropriate criteria so as to continuously identify the image revision and commit you would consider to be newest. Each time it discovers a new image revision, a new commit, _or both_, the `Warehouse` will produce a new `Freight` resource referencing both an image revision _and_ a commit. Being referenced by a single `Freight` resource, the two artifacts will be promoted from stage to stage _together as a unit_. 1. Connect any number of `Stage`s to this `Warehouse`. As `Freight` originating from it are promoted to each `Stage`, they will utilize a simple promotion process that: 1. Updates some stage-specific configuration to reference the new image revision _and_ uses a configuration management tool like Kustomize or Helm to combine the application's "base" configuration with the updated stage-specific configuration. 1. Writes the combined configuration to a Git repository. 1. Triggers an associated Argo CD `Application` to sync with the combined configuration. :::note The combined configuration can be written to the same Git repository that is monitored by the `Warehouse` or an entirely separate repository. If writing to the same repository, the combined configuration should be written to a different (possibly state-specific) branch _or_ to a specific path within the `main` branch that is _ignored_ by the `Warehouse` so as not to created a "feedback loop." This is discussed in greater details in the [Storage Options](#storage-options) section. ::: ### Parallel Pipelines Kargo does not require that all resources within a single `Project` participate in a single DAG. Multiple, disconnected DAGs can exist within a single `Project` to effect completely independent workflows for different applications. ![Parallel Pipelines](./img/parallel-pipelines.png) :::info The choice to define multiple, independent pipelines within a single `Project` might be made on the basis of organizational structure. If, for instance, a single team is responsible for multiple, related microservices, it may make sense to manage them together in a single `Project`. ::: ### Pre-Pipelines A "pre-pipeline" can be used to "convert" pipelines implementing patterns such as the [Common Case](#common-case) into a simpler pipeline such as one implementing the [Config Updater](#config-updater) pattern. To apply this pattern, a single `Stage` in the "pre-pipeline" can create new artifacts that are subscribed to by a second pipeline's `Warehouse`: ![Pre-Pipelines](./img/pre-pipelines.png) ### Grouped Services In an ideal world, the lifecycles of all services or microservices would be completely independent of one another. That flexibility, after all, is one of the main benefits of a microservice architecture. But we don't live in an ideal world and it's not uncommon, for instance, that changes to the front end of an application cannot be promoted without also promoting corresponding updates to the back end. Two specific features of Kargo can help to manage such scenarios: 1. Artifacts referenced by a single `Freight` resource are promoted from stage to stage _together as a unit_. 1. "Stage" can mean whatever your use case requires it to mean. ![Grouped Services](./img/grouped-services.png) To move related artifacts through a pipeline together: 1. Create a single `Warehouse` that subscribes to the repositories of all relevant artifacts. Each time it discovers a new revision of any, _or all_, of them the `Warehouse` will produce a new `Freight` resource referencing one revision of each. Being referenced by a single `Freight` resource, these artifacts will be promoted from stage to stage _together as a unit_. 1. Connect any number of `Stage`s to this `Warehouse`. Here, each `Stage` represents a promotion target for _multiple services_. As `Freight` originating from the one `Warehouse` are promoted to each `Stage`, they will: 1. Use a suitable process to combine the artifacts into stage-specific configuration(s). 1. Write the combined configurations to a Git repository. 1. Trigger _one or more_ associated Argo CD `Application` resources to sync with the combined configurations. :::info A single Argo CD `Application` resource _can_ manage the deployment of multiple related services -- a fact you may sometimes wish to leverage in the case of services that are tightly coupled. ::: :::warning If you find yourself creating `Warehouse` resources that subscribe to a large number of repositories to create `Freight` resources that reference a large number of artifacts that will all be promoted from stage to stage _together as a unit_, it is advisable to stop and ask yourself if coupling the promotion of so many artifacts was truly your intent. In many cases, users who have found themselves doing something such as this have done so without understanding the options that may have served their use case better. ::: ### Ordered Services In a variation on the [Grouped Services](#grouped-services) pattern, you may further require that related services be updated in a specific order. For instance, it would not be uncommon to require that changes to an application's back end be promoted prior to any changes to its front end. There are a number of ways to approach this. 1. Leverage Your GitOps Agent Argo CD, for instance, has a feature called [sync waves](https://argo-cd.readthedocs.io/en/stable/user-guide/sync-waves/) that can be used to control the order in which resources are synchronized. 1. Serialize `Application` syncs Instead of concluding a promotion process by triggering one or more Argo CD `Application` resources to sync all at once, a promotion process may be designed to trigger `Application` syncs _sequentially_, either at the end of the process, or one at a time as other specific steps of the process are completed. 1. Leverage the flexible definition of "stage" By defining a pipeline like so, a _subset_ of artifacts referenced by a `Freight` resource can be "dropped off" at each `Stage`: ![Ordered Services](./img/ordered-services.png) :::note In this example, with alternating stages promoting a _different subsets_ of the artifacts referenced by a `Freight` resource, there are two (slightly) _different_ promotion processes used by alternating stages. ::: ### Fanning Out / In When assembling a promotion pipeline from many `Stage` resources, it is not required that the pipeline be strictly linear. It is plausible, for instance, that after a `Freight` resource has been promoted to a "test" stage and passed some battery of tests, that it should next be promoted to two _different_ stages for distinct purposes. Perhaps, for instance, it may be promoted to _both_ a "qa" stage and a "uat" stage. In the "qa" stage, QA engineers will try their hardest to break the application, which could be disruptive to a product owner attempting to validate that business requirements have been met -- if not for the product owner's own "uat" stage. This is an example of "fanning out." When fanning out is applied, it is also possible to promote `Freight` to multiple "downstream" stages simultaneously. "Fanning in" is the inverse of fanning out. It is a case where a single stage can have `Freight` promoted to it after passing required tests in any one of multiple "upstream" stages. ![Fanning Out / In](./img/fan-out-in.png) Fanning out and back in can also be an effective way to implement A/B testing. Here, different `Freight` resources referencing different revisions of a container image have been promoted into the "A" and "B" stages. When one of these or the other has been deemed suitable for production, that one can be promoted: ![A/B Testing](./img/a-b-testing.png) :::info It is common for pipelines to fan in or out (or both) at [Control Flow Stages](#control-flow-stages). ::: ### Control Flow Stages Control flow stages are stages that define no promotion process. Their utility is only to de-clutter a pipeline (see [Fanning Out / In](#fanning-out--in)) or to provide an additional interaction point within it. A "tangled" pipeline: ![Tangled Pipeline](./img/control-flow-tangled.png) Using a control flow stage: ![Control Flow Stages](./img/control-flow.png) ### Multiple Warehouses ![Multiple Warehouses](./img/multiple-warehouses.png) This pattern is in some ways the inverse of the [Grouped Services](#grouped-services) pattern. In the Grouped Services pattern, we learned that a single `Warehouse` that subscribes to multiple repositories produces `Freight` that reference multiple artifacts that will be promoted from stage to stage _together as a unit_. A corollary to this is that artifacts in _different_ `Freight` resources can be promoted from stage to stage _independently_ of one another. Consider a scenario wherein it is common to promote new revisions of a container image from stage to stage, all the way to production, rapidly and many times per day, whilst configuration changes are more rare and are promoted more slowly, with greater deliberation. In such a case, "packaging" image revisions and configuration changes into a single `Freight` resource would make it all but impossible to promote the image revisions and configuration changes at different cadences. The solution to the above is to create two separate `Warehouse` resources, with one subscribing to the container image repository and the other subscribing to a Git repository containing configuration. Each `Warehouse` will produce `Freight` resources that reference just a single revision of an artifact. Both `Warehouse`s can be connected to the _same_ `Stage`s, so that each `Stage` effectively has two parallel pipelines running through it, with each of those delivering one type of artifact or the other. ### Freight Assembly At times, when applying other patterns such as the [Common Case](#common-case) or [Grouped Services](#grouped-services) -- any pattern that involves `Freight` resources referencing multiple artifacts -- you may encounter a specific difficulty: Sometimes there is a time delay between the discovery of a new revision of one artifact and the discovery of a new revision of another. To illustrate this difficulty, imagine that new versions of a container image _frequently_ require corresponding changes to application configuration -- for instance, new environment variables that must be set. If a `Warehouse` that watches for new revisions of the image and the configuration frequently discovered new revisions of the image _without_ a new revision of the configuration, the `Freight` resources it produces would frequently pair new image revisions with incompatible configuration. One strategy for dealing with this is to configure the `Warehouse` to continuously discover new revisions of artifacts, but _without_ automatically producing `Freight` resources. This configuration permits a user to use the "Freight Assembly" feature to manually select a compatible combination of revisions to promote. ![Freight Assembly](./img/assembly.png) :::info Additional features have been proposed to limit the automatic production of `Freight` that may reference incompatible revisions of different artifacts, which, in time, are likely to be implemented as well. ::: ### Mixed Promotion Modes Promotion policies can be configured to allow or disallow new `Freight` resources being promoted to each stage automatically. It is common for policies to permit new `Freight` to progress from stage to stage automatically _up to a point_, with promotions beyond that point, and into production, being manual. There are less obvious uses for this feature, however. Revisiting the problem posed in the introduction of the [Freight Assembly](#freight-assembly) pattern (incompatible artifact revisions), it is possible to leave automatic `Freight` production enabled on the `Warehouse`, but disallow auto-promotions to the leftmost `Stage` ("test," perhaps). In this way, any invalid combinations of artifact revisions will not be immediately promoted to the "test" stage. Instead, a user can manually select a `Freight` resources referencing a valid combination of artifact revisions for promotion to that `Stage`. ### Gatekeeper Stage Once again touching on the problem of `Freight` referencing incompatible artifact revisions, if resources permit, it is possible to introduce a "gatekeeper" `Stage` at the far left side of a pipeline. This is a `Stage` wherein _things are allowed to fail_. With a gatekeeper `Stage` in place, your `Warehouse` can continue producing `Freight` resources automatically and new `Freight` resources can be promoted automatically to the gatekeeper `Stage`. `Freight` resource can also be promoted automatically _from_ the gatekeeper `Stage` to the next `Stage`, but _failures_ in the gatekeeper `Stage` will block incompatible combinations of artifact revisions from progressing further. A `Freight` resource referencing incompatible revisions of different artifacts is likely to be succeeded shortly by a new `Freight` resource that references _compatible_ revisions of the artifacts, which will successfully clear the gatekeeper `Stage`. ## Repository Patterns This section documents common strategies for organizing Git repositories housing application configurations. In general, the layout of any such repository must provide for the separation of common, "base" configuration applicable to all from stage-specific configuration that is unique to each. As with all patterns presented in this document, you should not consider yourself constrained by them. They are presented here as a starting point for your own creativity. ### Layout with Helm If your Git repository contains configuration in the form of a Helm chart, it is common that the chart's `values.yaml` file, which provides default options during chart installation, will contain "base" configuration. Stage-specific configuration can be provided by supplemental values files that amend or override defaults found in the base `values.yaml`. A typical layout for such a repository may resemble this one: ```console . ├── Chart.yaml ├── stages │   ├── prod │   │   └── values.yaml │   ├── test │   │   └── values.yaml │   └── uat │   └── values.yaml ├── templates │   ├── _helpers.tpl │   ├── deployment.yaml │   └── service.yaml └── values.yaml ``` ### Layout with Kustomize If you utilize Kustomize for configuration management, it is common to organize configuration with a common "base" and stage-specific overlays. A typical layout for such a repository may resemble this one: ```console . ├── base │   ├── deploy.yaml │   ├── kustomization.yaml │   └── service.yaml ├── charts │   └── kargo-demo └── stages ├── prod │   └── kustomization.yaml ├── test │   └── kustomization.yaml └── uat └── kustomization.yaml ``` ### Monorepo Layout When working with monorepos containing configuration for multiple applications, it is common to organize configurations by application and then by stage. A typical layout for such a monorepo storing many Helm charts may resemble this: ```console . ├── guestbook │   ├── Chart.yaml │   ├── stages │   │   ├── prod │   │   │   └── values.yaml │   │   ├── test │   │   │   └── values.yaml │   │   └── uat │   │   └── values.yaml │   ├── templates │   │   ├── _helpers.tpl │   │   ├── deployment.yaml │   │   └── service.yaml │   └── values.yaml └── portal ├── Chart.yaml ├── stages │   ├── prod │   │   └── values.yaml │   ├── test │   │   └── values.yaml │   └── uat │   └── values.yaml ├── templates │   ├── _helpers.tpl │   ├── deployment.yaml │   └── service.yaml └── values.yaml ``` A typical layout for a monorepo storing many Kustomize configurations may resemble this: ```console . ├── guestbook │   ├── base │   │   ├── deploy.yaml │   │   ├── kustomization.yaml │   │   └── service.yaml │   └── stages │   ├── prod │   │   └── kustomization.yaml │   ├── test │   │   └── kustomization.yaml │   └── uat │   └── kustomization.yaml └── portal ├── base │   ├── deploy.yaml │   ├── kustomization.yaml │   └── service.yaml └── stages ├── prod │   └── kustomization.yaml ├── test │   └── kustomization.yaml └── uat └── kustomization.yaml ``` :::warning When storing configuration for many applications in a single repository, any `Warehouse` resources that subscribe to that repository should be carefully configured with [path filters](../20-how-to-guides/30-working-with-warehouses.md#git-subscription-path-filtering) such that only changes to _relevant_ configuration are detected. A `guestbook` `Warehouse`, for instance, should be unconcerned with changes to `portal` configuration. ::: ### Storage Options Toward the conclusion of most promotion processes, output of some kind must be written to a Git repository so it can be picked up and applied by a GitOps agent such as Argo CD. There are a few viable options by which to approach this: 1. **Stage-Specific Branches** This is decidedly the maintainers' preferred approach, but it is by no means required. :::info The practice of storing stage-specific configuration in dedicated branches seems to have been unfairly misunderstood as being the same as _GitFlow_. Some users also object to this approach on the grounds that they don't wish to maintain a large number of branches. In fact (this is not GitFlow), it is never incumbent upon users to _merge_ changes between branches. Thus, we encourage users to view stage-specific branches _not_ as a maintenance burden, but simply as _storage_. If it helps to conceptualize it, each branch may as well be an S3 bucket. ::: 1. **Writing Back to `main`** Users who truly prefer to work with only a single branch may choose to write the output of any promotion process _back_ to their `main` (trunk) branch, _however_, doing so requires some deliberate care. The _output_ of promotion processes should be written to paths within the repository that are _distinct_ from the _input_ to those processes. For example, the following repository layout keeps the input to promotion processes in the `src` directory and the output in the `builds` directory. :::warning Any `Warehouse` resources that subscribe to this repository must carefully apply path filters to ensure that changes within the `builds` directory do not trigger the production of new `Freight` resources. Failure to do so may result in a "feedback loop." ::: ```console . ├── builds │   ├── guestbook │   │   ├── prod │   │   ├── test │   │   └── uat │   └── portal │   ├── prod │   ├── test │   └── uat └── src ├── guestbook │   ├── base │   │   ├── deploy.yaml │   │   ├── kustomization.yaml │   │   └── service.yaml │   └── stages │   ├── prod │   │   └── kustomization.yaml │   ├── test │   │   └── kustomization.yaml │   └── uat │   └── kustomization.yaml └── portal ├── base │   ├── deploy.yaml │   ├── kustomization.yaml │   └── service.yaml └── stages ├── prod │   └── kustomization.yaml ├── test │   └── kustomization.yaml └── uat └── kustomization.yaml ``` 1. **Writing to a Separate Repository** Kargo does not require that the output of promotion processes be written to the same repository that is monitored by `Warehouse` resources. If it suits your use case, it is plausible to write the output of promotion processes to one or multiple branches of an entirely separate repository. ## Promotion Patterns Being mostly unopinionated, Kargo does not pre-define any specific promotion processes. Instead, it provides a number of fine-grained [Promotion Steps](../60-reference-docs/30-promotion-steps/index.md) that can be combined to implement a user-defined promotion process. This section documents a "typical" promotion process as well as a few common techniques. :::info Support for custom / third-party promotion steps is coming in a future release. ::: ### The Common Case Details vary, but a common promotion process might look something like this: 1. Clone a Git repository containing the application's configuration, possibly checking out multiple commits or branches. 1. Perform small edits on specific files. This might, for instance, involve updating a `values.yaml` file in a Helm chart to reference a new revision of a container image. 1. Combining the application's "base" configuration with stage-specific configuration. This could be as simple as copying relevant configuration to some other location or could involve a configuration management tool like Kustomize or Helm as described in the [Rendered Configs](#rendered-configs) pattern below. 1. Write the updated and combined configuration to a Git repository using any option laid out in the [Storage Options](#storage-options) section. 1. Trigger an associated Argo CD `Application` resource to sync with the newly promoted configuration. ### Rendered Configs This common pattern is easily combined with most others and, in fact, we've alluded to it in some of the other patterns: > Uses a configuration management tool like Kustomize or Helm to combine > the application's "base" configuration with stage-specific configuration. To make a finer point of this, "rendering" configs means using (Kargo's equivalent of) commands like `helm template` or `kustomize build` to generate Kubernetes manifests that are _plain YAML_ and require no further processing to be applied to a cluster. While Kargo does not require that you do this, it is a technique we highly recommend chiefly because diffing plain YAML (whist reviewing a PR, perhaps) promotes a comprehensive understanding of the changes to be applied. By contrast, diffing changes to a Helm `values.yaml` file or a Kustomize overlay both require knowledge of those tools and some degree of mental gymnastics to reason over the changes that will be applied after the GitOps agent has performed the rendering. :::info Pre-rendering configuration in this manner can also boost the performance of your GitOps agent, which will be relieved of any responsibility for executing configuration management tools. ::: ### Pseudo-GitOps This pattern is intentionally presented last because it's not _real GitOps_ in that desired state is not declared (or not _entirely_ declared) in a Git repository. Kargo does permit user-defined promotion processes to directly mutate the state of Argo CD `Application` resources. This can be used, for instance, to update Helm-specific or Kustomize-specific attributes of an `Application` resource. It can also be used to update the `source.targetRevision` field to effectively change the Git commit or Helm chart version to which the `Application` syncs. Kargo implements these capabilities mainly to reduce the overhead of simply experimenting with Kargo, but we _strongly_ recommend against applying this pattern in a production setting. --- ## Examples ## Basic Examples A catalog of small Kargo examples can be found in a unified [kargo-examples](https://github.com/akuity/kargo-examples/) repository. This collection of micro-examples demonstrate specific functionality in isolation. ## End-to-End Examples A number of end-to-end examples that can be used as a starting point, or inspiration for your own pipelines. | Name | Description | |------|-------------| | [kargo-simple](https://github.com/akuity/kargo-simple) | A simple, 3-stage pipeline with environments configured by Kustomize. Images are updated to kustomize overlays using the [Image Updater](30-patterns/index.md#image-updater) pattern. | | [kargo-helm](https://github.com/akuity/kargo-helm) | Helm example that updates images to environment specific values.yaml. Demonstrates the ability to promote feature flags using file copies to different environment directories. Utilizes the [Multiple Warehouses](30-patterns/index.md#multiple-warehouses) pattern to monitor both a container image repository as well as updates to files in git (feature flags), and the ability to promote image updates independently from features. | | [kargo-microservices](https://github.com/jessesuen/kargo-microservices) | Example which shows a Warehouse that monitors multiple microservices such that the corresponding Freight can be promoted as a single unit. Demonstrates the [Grouped Services](30-patterns/index.md#grouped-services) pattern with a single Warehouse monitoring multiple image repositories. | | [kargo-advanced](https://github.com/akuity/kargo-advanced) | An advanced example with a complex pipeline with A/B testing stages and multiple prod regions. Showcases [Verification](20-how-to-guides/60-verification.md) testing using [AnalysisTemplates](60-reference-docs/50-analysis-templates.md). Demonstrates: [Common Case](30-patterns/index.md#common-case), [Control Flow Stages](30-patterns/index.md#control-flow-stages), [Rendered Configs](30-patterns/index.md#rendered-configs) patterns. | --- ## Access Controls(20-access-controls) Most access controls in Kargo are within the purview of highly-privileged _users_ -- ones who might be considered to be "project admins." Such users are the intended audience for this document. :::info[Not what you were looking for?] If you're an operator looking to understand more about access controls, especially the few outside the purview of project admins, you may find some value in this document, but most of what you need to know can be found in the [Operator Guide's Access Controls](../../../40-operator-guide/40-security/30-access-controls.md) section. ::: ## Overview Kargo is usually configured to support single-sign-on (SSO) using an identity provider (IDP) that implements the [OpenID Connect](https://openid.net/developers/how-connect-works/) (OIDC) protocol. Configuring this is the responsibility of the operator and is discussed in-depth in the dedicated [OpenID Connect](../../../40-operator-guide/40-security/20-openid-connect/index.md) section of the Operator Guide. Kargo also implements access controls through _pure Kubernetes [RBAC](https://kubernetes.io/docs/reference/access-authn-authz/rbac/)._ :::info Kargo's creators learned from previous experience that when APIs are modeled as Kubernetes resources, it is best to rely solely on Kubernetes-native authorization mechanisms. By doing so, access controls are enforced even for users with direct access to the Kargo control plane's underlying cluster who might shun the Kargo CLI and UI in favor of `kubectl`. ::: There is a natural _impedance_ between users authenticating to _Kargo_ through some IDP and access controls being implemented through pure _Kubernetes_ RBAC. To make a finer point of it: _It is impossible for a Kubernetes cluster to enforce RBAC for users it does not recognize._ Most operators will not wish to resolve this by granting direct cluster access to a large number of developers and training them to use `kubectl`, so a different solution is required. Kargo resolves this impedance through a simple scheme that permits users authenticated via the IDP to be _mapped_ to Kubernetes `ServiceAccount` resources. For the most part, these mappings are best managed at the project level by project admins. ## User to `ServiceAccount` Mappings First, project admins should understand how the mapping of users to `ServiceAccount` resources works. Most Kargo users interact with Kargo via its API server, using its UI or CLI as a client. In either case, those users are authenticated by a bearer token issued by the IDP. For every request, the Kargo API server validates and decodes the token to obtain trusted information about the user which, importantly, includes _claims_ such as username, email address, and group membership. The exact claims available depend on the IDP and how the operator has configured the Kargo API server. Also for every request, the Kargo API server queries the Kubernetes API server to obtain a list of all `ServiceAccount` resources to which the user has been mapped. This search is mostly limited to `ServiceAccount` resources in Kargo project namespaces only (i.e. only those labeled with `kargo.akuity.io/project: "true"`). ServiceAccount resources may be mapped to users via the `rbac.kargo.akuity.io/claims` annotation, whose value is a string representation of a JSON object with claim names as its keys and lists of claim values as its values. In the following example, the `ServiceAccount` resource is mapped to all of: * Users with a `sub` claim identifying them as either `alice` or `bob`. * A user with the `email` claim `carl@example.com`. * All users with a `groups` claim containing _either_ the `devops` or `kargo-admin` group. ```yaml apiVersion: v1 kind: ServiceAccount metadata: name: admin namespace: kargo-demo annotations: rbac.kargo.akuity.io/claims: | { "sub": ["alice", "bob" ], "email": ["carl@example.com"], "groups": ["devops", "kargo-admin"] } ``` :::info Claim values may also be specified as a scalar string instead of a single-element list (e.g. `"email": "carl@example.com"`). Both forms are accepted, but the list form is preferred for consistency. Mappings specified using annotations with keys of the form `rbac.kargo.akuity.io/claim.` with comma-delimited values are also supported for reasons of backwards compatibility. The effective mapping is therefore the union of mappings defined using such annotations with any mappings defined using the newer, recommended `rbac.kargo.akuity.io/claims` annotation. ::: A user may be mapped to multiple `ServiceAccount` resources. A user's effective permissions are therefore the _union_ of the permissions associated with all such `ServiceAccount` resources. ### Managing Mappings and Permissions Unless the operator has disabled it, project admins can manage user-to-`ServiceAccount` mappings as well as the permissions associated with those `ServiceAccount` resources via either the UI or CLI. In either case, Kargo abstracts a trio of `ServiceAccount`, `Role`, and `RoleBinding` resources as a single "Kargo role." This abstraction offers UI and CLI users the convenience of a simplified interface for managing user-to-`ServiceAccount` mappings and the permissions associated with those `ServiceAccount` resources. There are several "Kargo roles" pre-defined in a project's namespace when a new `Project` resource is created: 1. `default`: This Kargo role exists by virtue of the existence of the `default` `ServiceAccount` resource that Kubernetes automatically creates in any new namespace, including those associated with a Kargo project. This `ServiceAccount` lacks the annotations that would identify it as being Kargo-managed, and as such, cannot be modified or deleted via the Kargo UI or CLI. You can effectively ignore this Kargo role. 1. `kargo-admin`: This Kargo role is a trio of `ServiceAccount`, `Role`, and `RoleBinding` resources created by the Kargo management controller. Its permissions are pre-defined as those necessary to manage _all_ aspects of the project. It is not initially mapped to any users. All three resources are annotated as being Kargo-managed, and as such, the "Kargo role" that abstracts them can be modified or deleted via the UI or CLI. 1. `kargo-promoter`: This Kargo role is a trio of `ServiceAccount`, `Role`, and `RoleBinding` resources created by the Kargo management controller. Its permissions are pre-defined as those necessary to promote `Stage`s and create `Promotion`s, but not to create, update, or delete core pipeline resources such as `Stage`s and `Warehouse`s. It is not initially mapped to any users. All three resources are annotated as being Kargo-managed, and as such, the "Kargo role" that abstracts them can be modified or deleted via the UI or CLI. 1. `kargo-viewer`: This Kargo role is a trio of `ServiceAccount`, `Role`, and `RoleBinding` resources created by the Kargo management controller. Its permissions are pre-defined as those necessary to view, but not modify or delete all project resources. It is not initially mapped to any users. All three resources are annotated as being Kargo-managed, and as such, the "Kargo role" that abstracts them can be modified or deleted via the UI or CLI. #### Managing Kargo Roles with the UI To see all Kargo roles in a `Project` click Settings in the upper right corner of the `Project` view: ![Project](img/roles-manage.png) Then, go to the Roles tab. This will display a list of all Kargo roles associated with the Project: ![Roles](img/roles-manage-2.png) This interface also permits users with appropriate permissions to create, modify, or delete Kargo roles: ![Edit Role](img/roles-manage-3.png) :::info The Kargo API server translates all such creations, modifications, and deletions into the appropriate actions on the underlying `ServiceAccount`, `Role`, and `RoleBinding` resources. ::: #### Managing Kargo Roles With the CLI * All Kargo roles associated with a project can be listed using the `kargo get roles` command: ```shell kargo get roles --project kargo-demo ``` ```shell NAME KARGO MANAGED AGE default false 18h kargo-admin true 18h kargo-promoter true 18h kargo-viewer true 18h ``` * The details of a specific role can be examined by naming the role and requesting a YAML representation: ```shell kargo get role kargo-admin --project kargo-demo --output yaml ``` * Users with appropriate permissions can create, modify, or delete Kargo roles. * To create a new role named `developer`: ```shell kargo create role developer --project kargo-demo ``` ```shell role.rbac.kargo.akuity.io/developer created ``` * To grant the new role to users with `developer` in their `groups` claim: ```shell kargo grant --role developer \ --claim groups=developer \ --project kargo-demo ``` ```shell role.rbac.kargo.akuity.io/developer updated ``` * To grant broad permissions on `Stage` resources to the new role: ```shell kargo grant --role developer \ --verb '*' --resource-type stages \ --project kargo-demo ``` ```shell role.rbac.kargo.akuity.io/developer updated ``` * The `kargo revoke` command can be used identically to the `kargo grant` command, but has the opposite effects. * It may sometimes be useful to view a Kargo Role's underlying `ServiceAccount`, `Role`, and `RoleBinding` resources. This may be useful, for instance, to users who have managed Project-level permissions imperatively up to a point and now wish to make the transition to GitOps'ing those permissions. ```shell kargo get role developer --as-kubernetes-resources --project kargo-demo ``` ```yaml NAME K8S SERVICE ACCOUNT K8S ROLE BINDINGS K8S ROLES AGE developer developer developer developer 13m ``` * It is also possible to request alternative representations of the underlying resources: ```shell kargo get role developer \ --as-kubernetes-resources -o yaml \ --project kargo-demo ``` :::note Output of the above command is not shown here due to its length. ::: * Last, it is, of course, possible to delete a role: ```shell kargo delete role developer --project kargo-demo ``` ```shell role.rbac.kargo.akuity.io/developer deleted ``` #### Managing Kargo Roles Declaratively Whether it is because your operator has disabled management of Kargo roles via the UI and CLI, or because you simply prefer to manage your project declaratively (perhaps evens with GitOps), you can do so by simply `ServiceAccount`, `Role`, and `RoleBinding` resources [in the usual fashion](https://kubernetes.io/docs/reference/access-authn-authz/rbac/). With the exception of the claim-mapping annotations on `ServiceAccount` resources, these resources do not need to be labeled or annotated in any special way. ##### `promote` Verb Kargo introduces custom verbs to extend Kubernetes' [standard RBAC verbs](https://kubernetes.io/docs/reference/access-authn-authz/authorization/#request-verb-resource) (such as get, create, list) to support more granular authorization. One such verb is `promote`, which applies to the `stages` resource. Kargo uses this verb to determine whether a user or `ServiceAccount` is authorized to initiate a promotion into a specific Stage. ##### Example: Custom Promoter Role The following example demonstrates how to create a custom role named `promoter`, which grants members of the `devops` OIDC group permission to `promote` into the `dev` and `staging` `Stages` in the `guestbook` `Project`. ```yaml --- apiVersion: v1 kind: ServiceAccount metadata: name: promoter namespace: guestbook annotations: kargo.akuity.io/description: Permissions to promote into pre-prod Stages rbac.kargo.akuity.io/claims: '{"groups":["devops"]}' --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: promoter namespace: guestbook rules: - apiGroups: - kargo.akuity.io resources: - promotions verbs: - create - patch - update - apiGroups: - kargo.akuity.io resources: - stages verbs: - promote # Limit promotions to a subset of Stages resourceNames: - dev - staging --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: promoter namespace: guestbook roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: promoter subjects: - kind: ServiceAccount name: promoter namespace: guestbook --- # This RoleBinding allows the promoter ServiceAccount to view project # resources (without redefining read rules), by binding it to the # built-in kargo-viewer Role. apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: promoter-viewer namespace: guestbook roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: kargo-viewer subjects: - kind: ServiceAccount name: promoter namespace: guestbook ``` ##### Example: Declarative OIDC Claim Mapping to Built-in Roles To declaratively assign OIDC claims to Kargo’s built-in roles, such as `kargo-admin` or `kargo-viewer`, you can create a `ServiceAccount` annotated with the appropriate claims, then bind it to the desired role using a `RoleBinding`. In the example below, users in the `devops` OIDC group are granted admin-level permissions within the `guestbook` `Project`: ```yaml --- apiVersion: v1 kind: ServiceAccount metadata: name: admin namespace: guestbook annotations: rbac.kargo.akuity.io/claims: '{"groups":["devops"]}' --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: admin namespace: guestbook roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: kargo-admin subjects: - kind: ServiceAccount name: admin namespace: guestbook ``` #### Global Mappings As previously mentioned, _most_ access controls are managed at the project level by project admins, however, there are two exceptions to this which put some control in the hands of the operator, and you should be aware of them. 1. Operators are able to map users with specific claims to pre-defined `kargo-admin` or `kargo-viewer` `ServiceAccount`s in the same namespace in which Kargo is installed. :::note It is common for operators to map _all_ authenticated users to the `kargo-viewer` `ServiceAccount` to effect broad read-only permissions. These permissions _do not_ extend to credentials and other project `Secret`s. ::: 1. Operators can designate one or more namespaces as containing "global" `ServiceAccount` resources, which they may also pre-define and pre-map to users with specific claims. Optionally, operators may use `ClusterRoleBinding`s to grant any necessary system-wide permissions to such `ServiceAccount` resources. Importantly, operators who have designated such namespaces and populated them with `ServiceAccount` resources, have essentially pre-defined different classes of users. You, as a project admin, can use `RoleBinding`s within your own project namespaces to grant permissions to such classes of user without needing to define them yourself or know much about OIDC claims, although this requires a declarative approach to managing the `Role`s and `RoleBinding`s. :::info "Global" is a misnomer. `ServiceAccount` resources in designated namespaces are not truly global because they are still mapped to users according to the rules described in the previous sections. ::: Example custom `RoleBinding` to a local `Project` role using a "global" `ServiceAccount` in a separate namespace: ```yaml apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: kargo-demo-developer namespace: kargo-demo roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: kargo-admin subjects: - kind: ServiceAccount name: team-x-developers namespace: kargo-global-service-accounts ``` Which can also be created with: ```shell kubectl create rolebinding \ --serviceaccount kargo-global-service-account:team-x-developers \ --role kargo-admin \ -n kargo-demo \ kargo-demo-developer ``` --- ## API Tokens(50-security) API tokens enable programmatic access to Kargo APIs without requiring users to share short-lived credentials with non-human agents, such as CI processes. :::info[Not what you were looking for?] If you're looking to manage and use API tokens associated with built-in, system-level roles refer to the [Operator Guide's API Tokens](../../40-operator-guide/40-security/35-api-tokens.md) documentation. ::: Kargo API tokens are associated directly with Kargo roles. Project admins, or others with sufficient permissions, are able to create and delete roles as well as create and delete API tokens associated with those roles. :::caution[Implementation Detail] "Kargo roles," including built-in, system level ones are actually abstractions over trios of Kubernetes `ServiceAccount`, `ClusterRole`, and `ClusterRoleBinding` resources. Throughout this document, the term "role" refers to this abstraction. ::: To learn more about managing Project-level roles, refer to the [Access Controls](./20-access-controls/index.md) documentation. ## Creating Tokens To generate a new token associated with a Project-level role: ```shell kargo create token --project my-project --role my-role my-role-token ``` ```shell Token created successfully! IMPORTANT: Save this token securely. It will not be shown again. Token: eyJhbGciOiJSUzI1NiIsImtpZCI6IjdwQ0... ``` :::danger The token value is displayed only once during creation. Do not lose it! If you lose the token value, you must delete the token and create a new one _or_ the existing token's value can be retrieved by a user with sufficient permission using `kubectl` instead of the `kargo` CLI. ::: List all tokens associated with a specific role: ```shell kargo get tokens --project my-project ``` ```shell NAME ROLE KARGO MANAGED AGE my-role-token my-role true 5m ``` List tokens associated with a specific role: ```shell kargo get tokens --project my-project --role my-role ``` Retrieve details about a specific token (note that the token value will be redacted): ```shell kargo get token --project my-project my-role-token -o yaml ``` ## Using Tokens API tokens can be used with many Kargo or Kubernetes clients. This includes tools like `kubectl` as well as any programming language client library for Kubernetes or Kargo. :::note While the `kargo` CLI does not directly support specifying a token via command line flags, you can configure it to use a token by editing `~/.config/kargo/config`. ::: ## Deleting Tokens To delete a token when it's no longer needed or to rotate credentials: ```shell kargo delete token --project my-project my-role-token ``` ```shell token.kargo.akuity.io/my-role-token deleted ``` Verify the token has been deleted: ```shell kargo get tokens --project my-project ``` --- ## Managing Secrets(50-security) Kargo Projects use Kubernetes `Secret` resources to store repository credentials and other types of sensitive data, such as API keys for third-party services. It is crucial that users managing Kargo Projects understand how `Secret`s are organized and accessed. :::info[Not what you were looking for?] If you're an operator looking to understand your role in managing repository credentials and other secrets, you may find a some value in this document, but should refer primarily to the [Managing Secrets](../../40-operator-guide/40-security/40-managing-secrets.md) section of the Operator's Guide. ::: ## Overview Users managing Kargo Projects will find themselves concerned with secrets falling into one of two broad categories: * **Repository credentials:** Secrets specifically representing credentials for the three types of repositories supported by Kargo: Git repositories, container image repositories, and Helm chart repositories. * **"Generic credentials":** Any secrets that are not specifically repository credentials. :::info The misnomer "generic _credentials_" is used for historical reasons, but nothing limits these to storing _only_ credentials. In actuality, they can be any sort of sensitive information. So although they are called "generic credentials," they are best thought of in more general terms as, simply, generic secrets. ::: ## Repository Credentials Kargo expects `Secret` resources representing repository credentials to be labeled in specific ways and to conform to a specific format. Such `Secret`s generally take the following form: ```yaml apiVersion: v1 kind: Secret metadata: name: namespace: labels: kargo.akuity.io/cred-type: data: # base64 encoded repoURL: username: password: ``` The label key `kargo.akuity.io/cred-type`, together with its value, specify the _type_ of the repository accessed with the credential: * `git`: Credentials for Git repositories * `helm`: Credentials for Helm chart repositories * `image`: Credentials for container image repositories `Secret`s representing repository credentials MUST include the key `repoURL` in their `data` block. Its value may be either a full, exact URL **OR** a regular expression matching the URLs of multiple repositories for which the credentials are valid, in which case, the `data` block must also contain the key/value pair `repoURLIsRegex: "true"`. The remaining key/value pairs in such a `Secret`'s `data` block are dependent upon exactly what kind of credential the `Secret` represents. Commonly, they may be: * `username`: The username to use when authenticating to the repository * `password`: A password or **personal access token** :::info If the value of the `password` key is a **personal access token**, the value of the `username` field is often inconsequential. You should consult your repository's documentation for more information. ::: Alternatively, for Git repositories only (and specifically ones that support SSH-style URLs of the form `git@github.com:example/repo.git`), the key `sshPrivateKey` in the `Secret`'s `data` block may have as its value a PEM-encoded SSH private key. :::warning[Deprecated] Support for SSH URLs and SSH private keys is **deprecated as of v1.10.0** and **scheduled for removal in v1.13.0**. The proprietary APIs offered by the major Git hosting platforms (e.g. GitHub or GitLab) to enable actions such as opening or closing pull requests are invariably HTTP-based and therefore cannot use an SSH private key for authentication. This forces users who rely on SSH to maintain a second set of credentials (e.g. a personal access token) for API operations. Using HTTPS URLs with a single token-based credential that works for both standard Git operations and provider API calls is simpler and more secure. Users should migrate to HTTPS URLs before v1.13.0. ::: :::info[Credential Shapes] `Secret` resources representing repository credentials come in a wide variety of other "shapes" (different keys in the `data` block) for use with various authentication mechanisms. The details of each are covered in their own, dedicated sections toward the end of this document, following a more general treatment of the secrets topic. ::: ### Using Repository Credentials A unique property of `Secret` resources representing repository credentials is is that there is no need to reference them directly. Any time Kargo accesses a repository, it _automatically_ attempts to locate suitable credentials, searching by _repository type and URL._ Project-level repository credentials _can_ be referenced directly from within an expression by utilizing the [`secret()`](../../50-user-guide/60-reference-docs/40-expressions.md#secretname) expression function. ::info This is in contrast to _shared_ repository credentials from the shared resources namespace, which by design, _cannot_ be referenced directly. This limitation permits operators managing a Kargo instance to place repository credentials in the **shared resources namespace**, knowing that they can be used by all Projects _without their values ever being exposed to users._ ::: When Kargo needs repository credentials, it searches for `Secret`s in _two_ specific namespaces, in the following order: 1. **Project namespace**: Kargo searches the Project's own namespace first. 2. **Shared resources namespace**: If no match is found in the Project's own namespace, Kargo searches the **shared resources namespace.** :::info[Credential Matching Precedence] _Within_ each namespace searched, Kargo considers credentials in this order: 1. Exact `repoURL` matches (where `repoURLIsRegex` is `"false"` or unspecified) 2. Pattern matches using regex (where `repoURLIsRegex` is `"true"`) Within each category, `Secret`s are considered in lexical order by name. The credentials used by Kargo will be the _first_ to match the repository type and URL. ::: ## Generic Credentials "Generic credentials" (a misnomer) are any secrets that are not specifically repository credentials. `Secret` resources representing generic credentials MUST be labeled with `kargo.akuity.io/cred-type: generic`. :::info The misnomer "generic _credentials_" is used for historical reasons, but nothing limits these to storing _only_ credentials. In actuality, they can be any sort of sensitive information. So although they are called "generic credentials," they are best thought of in more general terms as, simply, generic secrets. ::: ### Using Generic Credentials Generic credentials can be accessed directly by name and their `data` blocks are not required to conform to any specific structure. This makes them suitable for storing any arbitrary secret data a Project may depend upon. Elements of a Project that support expressions, such as a promotion process, can access such secrets by utilizing the [`secret()`](../../50-user-guide/60-reference-docs/40-expressions.md#secretname) expression function. :::tip Generic credentials from the shared resources namespace can be accessed using the [`sharedSecret()`](../../50-user-guide/60-reference-docs/40-expressions.md#sharedsecretname) expression function instead. ::: ## Managing Credentials with the CLI Unless the operator has disabled it, users with the appropriate permissions can manage Project-level credentials using either the UI or CLI. :::caution While the UI or CLI may be a fine way of managing Project-level credentials whilst getting to know Kargo, it is unquestionably more secure to use other means to ensure the existence of these specially-formatted `Secret`s. For precisely this reason, the operator managing your Kargo installation may very well have disabled the ability to manage credentials using the UI and CLI. If this is the case, managing your credentials is likely to involve GitOps'ing your Kargo Projects and also leveraging additional tools such as [Sealed Secrets](https://github.com/bitnami-labs/sealed-secrets) or the [External Secrets Operator](https://external-secrets.io/latest/). ::: ### Creating Credentials The following example creates credentials for a specific Git repository: ```shell kargo create credentials \ --project kargo-demo my-credentials \ --git \ --repo-url https://github.com/example/kargo-demo.git \ --username my-username \ --password my-personal-access-token ``` ```shell secret/my-credentials created ``` :::caution If you do not wish for your password or personal access token to be stored in your shell history, you may wish to omit the `--password` flag, in which case the CLI will prompt you to enter the password interactively. ::: ### Listing / Viewing Credentials Credentials can be listed or viewed with `kargo get credentials`: ```shell kargo get credentials --project kargo-demo my-credentials ``` ```shell NAME TYPE REGEX REPO AGE my-credentials git false https://github.com/example/kargo-demo.git 8m25s ``` If requesting output as YAML or JSON, the values of all data fields and annotations not explicitly deemed "safe" are redacted. In the example below, you can see the values of `repoURL` and `username` have not been redacted because those fields are assumed not to contain sensitive information. `password` is redacted, however. Values of arbitrarily named data fields are also redacted because Kargo cannot infer their sensitivity. ```shell kargo get credentials --project kargo-demo my-credentials -o yaml ``` ```shell apiVersion: v1 kind: Secret metadata: creationTimestamp: "2024-05-30T20:02:46Z" labels: kargo.akuity.io/cred-type: git name: my-credentials namespace: kargo-demo resourceVersion: "17614" uid: ca2660e4-867d-4709-b1a7-57fbb93fc6dc stringData: password: '*** REDACTED ***' repoURL: https://github.com/example/kargo-demo.git username: my-username foo: '*** REDACTED ***' type: Opaque ``` ### Updating Credentials Credentials can be updated using the `kargo update credentials` command and the flags corresponding to attributes of the credentials that you wish to modify. Other attributes of the credentials will remain unchanged. The following example updates `my-credentials` with a regular expression for the repository URL: ```shell kargo update credentials \ --project kargo-demo my-credentials \ --repo-url '^https://github.com/' \ --regex ``` ```shell secret/my-credentials updated ``` ### Deleting Credentials Credentials can, of course, be deleted with `kargo delete credentials`: ```shell kargo delete credentials --project kargo-demo my-credentials ``` ```shell secret/my-credentials deleted ``` ## Other Forms of Credentials This section provides guidance on managing credentials for GitHub and for several popular container image registries. These options range from long-lived tokens to "ambient" credentials that can be obtained automatically when running within certain cloud platforms. :::note In many cases, applying the options discussed in the following sections may require the assistance of an operator/administrator for the applicable platforms. ::: ### GitHub Authentication Options The following two sections cover GitHub-specific authentication options that are more secure than simply using a username and password. #### Personal Access Token GitHub supports authentication using a [personal access token](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens), which can be used in place of a password. The corresponding username must be the GitHub handle of the user who created the token. These can be stored in the `username` and `password` fields of a `Secret` resource as described [in the first section](#repository-credentials) of this document. :::info This method of authentication may be best when wishing to rigorously enforce the principle of least privilege, as personal access tokens can be scoped to specific permissions on specific repositories. A drawback to this method, however, is that the token is owned by a specific GitHub user, and if that user should lose their own access to the repositories in question, Kargo will also lose access. ::: #### GitHub App Authentication [GitHub Apps](https://docs.github.com/en/apps) can be used [as an authentication method](https://docs.github.com/en/apps/creating-github-apps/authenticating-with-a-github-app/about-authentication-with-a-github-app). :::note You may require the assistance of your GitHub organization's administrator to create or install a GitHub App. ::: 1. [Create a GitHub App](https://github.com/settings/apps/new): 1. In the GitHub App name field, specify a unique name. 1. Set the Homepage URL to any URL you like. 1. Under Webhook, de-select Active. 1. Under Permissions → Repository permissions → Contents, select whether the App will require Read-only or Read and write permissions. _The App will receive these permissions on all repositories into which it is installed._ 1. Under Where can this GitHub App be installed?, leave Only on this account selected. 1. Click Create GitHub App. 1. Take note of the Client ID. 1. Scroll to the bottom of the page and click Generate a private key. The resulting key will be downloaded immediately. Store it securely. 1. On the left-hand side of the page, click Install App. 1. Choose an account to install the App into by clicking Install. 1. Select Only select repositories and choose the repositories you wish to grant the App access to. Remember that the App will receive the permissions you selected earlier on _all_ of these repositories. 1. Click Install. 1. In your browser's address bar, take note of the numeric identifier at the end of the current page's URL. This is the Installation ID. 2. Create a `Secret` resource with the following structure: ```yaml apiVersion: v1 kind: Secret metadata: name: namespace: labels: kargo.akuity.io/cred-type: git stringData: githubAppClientID: githubAppPrivateKey: githubAppInstallationID: repoURL: repoURLIsRegex: ``` :::info GitHub currently recommends using an App's alphanumeric client ID whenever possible, but Kargo does support the deprecated, numeric App ID as an alternative identifier for a GitHub App. The following example is thus valid until such time that GitHub themselves may remove support for the deprecated App ID. ```yaml apiVersion: v1 kind: Secret metadata: name: namespace: labels: kargo.akuity.io/cred-type: git stringData: githubAppID: githubAppPrivateKey: githubAppInstallationID: repoURL: repoURLIsRegex: ``` In the event that a `Secret`'s data map includes values for _both_ the `githubAppClientID` and `githubAppID` keys, Kargo will prioritize the value of the `githubAppClientID` key as its means of uniquely identifying the GitHub App. ::: :::note The `kargo create/update credentials` commands do not support creating or updating non username/password credentials. To create or update a `Secret` such as the one shown above, use GitOps instead, or the `kargo apply --project -f ` command. ::: :::info Compared to personal access tokens, a benefit of authenticating with a GitHub App is that the App's permissions are not tied to a specific GitHub user. ::: :::caution It is easy to violate the principle of least privilege when authenticating using GitHub Apps. For convenience's sake, it may be tempting to register a single GitHub App, select a broad set of repositories when installing that App, then create a single set of [shared credentials](../../40-operator-guide/40-security/40-managing-secrets.md#repository-credentials), _however_, this will have the undesirable effect of granting _all_ Kargo Projects access to _all_ of the selected repositories. Alternatively, you might consider registering a _separate_ GitHub App for each Kargo Project, selecting a narrower set of repositories when installing each App, then creating corresponding Secrets in individual Project namespaces. While this better adheres to the principle of least privilege, it can be onerous to manage. Worse, because GitHub organizations are limited to registering 100 GitHub Apps each, the approach does not scale beyond 100 Projects. Beginning with Kargo v1.8.0, a third, experimental (stability not guaranteed) approach builds upon the first, by adding an optional annotation to the [shared credentials](../../40-operator-guide/40-security/40-managing-secrets.md#repository-credentials) `Secret` containing a map that constrains the scopes (repositories) available to each Project. In the following example, the credentials defined by the `github` `Secret` in the shared credentials namespace are available to all Kargo Projects, however, the `kargo-demo-1` Project is able to obtain access tokens scoped to either `repo-a` or `repo-b` only, while the `kargo-demo-2` Project is able to obtain access tokens scoped to `repo-c` only. No other Project is able to obtain access tokens scoped to _any_ repository. ```yaml apiVersion: v1 kind: Secret metadata: name: github namespace: kargo-shared-credentials labels: kargo.akuity.io/cred-type: git annotations: kargo.akuity.io/github-token-scopes: | { "kargo-demo-1": ["repo-a", "repo-b"], "kargo-demo-2": ["repo-c"] } data: # ... ``` ::: ### Amazon Elastic Container Registry (ECR) The authentication options described in this section are applicable only to container image repositories whose URLs indicate they are hosted in ECR. #### Long-Lived Credentials {#ecr-long-lived-credentials} Elastic Container Registries do not _directly_ support long-lived credentials, however, an AWS access key ID and secret access key [can be used to obtain an authorization token](https://docs.aws.amazon.com/AmazonECR/latest/userguide/registry_auth.html#registry-auth-token) that is valid for 12 hours. Kargo can seamlessly obtain such a token and will cache it until shortly before it expires. To use this option, your `Secret` should take the following form: ```yaml apiVersion: v1 kind: Secret metadata: name: namespace: labels: kargo.akuity.io/cred-type: image stringData: awsRegion: us-west-2 awsAccessKeyID: awsSecretAccessKey: repoURL: ``` :::note The `kargo create/update credentials` commands do not support creating or updating non username/password credentials. To create or update a `Secret` such as the one shown above, use GitOps instead, or the `kargo apply --project -f ` command. ::: :::caution Following the principle of least privilege, the IAM user associated with the access key ID and secret access key should be limited only to read-only access to the required ECR repositories. Configuring this will likely require the assistance of an AWS account administrator. ::: :::caution This method of authentication is a "lowest common denominator" approach that will work regardless of where Kargo is deployed. i.e. if running Kargo outside EKS, this method will still work. If running Kargo within EKS, you may wish to either consider using EKS Pod Identity or IRSA instead. ::: #### EKS Pod Identity or IAM Roles for Service Accounts (IRSA) If Kargo locates no `Secret` resources matching a repository URL and is deployed within an EKS cluster, it will attempt to use [EKS Pod Identity](https://docs.aws.amazon.com/eks/latest/userguide/pod-identities.html) or [IAM Roles for Service Accounts (IRSA)](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html) to authenticate. Leveraging either eliminates the need to store ECR credentials in a `Secret` resource. :::info Both of these options rely upon extensive external configuration that likely requires the assistance of Kargo's operator and an AWS account administrator, and as such, further details are covered in the [Ambient Credentials](../../40-operator-guide/40-security/50-ambient-credentials.md) section of the Operator Guide. ::: ### Google Artifact Registry The authentication options described in this section are applicable to both container image repositories and OCI Helm chart repositories whose URLs indicate they are hosted in Google Artifact Registry. #### Long-Lived Credentials {#gar-long-lived-credentials} :::caution Google Artifact Registry does _directly_ support long-lived credentials [as described here](https://cloud.google.com/artifact-registry/docs/docker/authentication#json-key). The username `_json_key_base64` and the base64-encoded service account key may be stored in the `username` and `password` fields of a `Secret` resource as described [in the first section](#repository-credentials) of this document. **Google strongly discourages this method of authentication however, and so do we.** ::: Google documentation recommends [using a service account key to obtain an access token](https://cloud.google.com/artifact-registry/docs/docker/authentication#token) that is valid for 60 minutes. Compared to the discouraged method of using the service account key to authenticate to the registry directly, this process does _not_ transmit the service account key over the wire. Kargo can seamlessly carry out this process and will cache the access token until shortly before it expires. To use this option, your `Secret` should take the following form: * For a container image repository: ```yaml apiVersion: v1 kind: Secret metadata: name: namespace: labels: kargo.akuity.io/cred-type: image stringData: gcpServiceAccountKey: repoURL: us-central1-docker.pkg.dev/my-project/my-images ``` * For an OCI Helm chart repository: ```yaml apiVersion: v1 kind: Secret metadata: name: namespace: labels: kargo.akuity.io/cred-type: helm stringData: gcpServiceAccountKey: repoURL: oci://us-central1-docker.pkg.dev/my-project/my-helm-charts/my-chart ``` :::note Service account keys contain structured data, so it is important that the key be base64-encoded. ::: :::caution Following the principle of least privilege, the service account associated with the service account key should be limited only to read-only access to the required Google Artifact Registry repositories. Configuring this will likely require the assistance of a GCP project administrator. ::: :::caution This method of authentication is a "lowest common denominator" approach that will work regardless of where Kargo is deployed. i.e. If running Kargo outside of GKE, this method will still work. If running Kargo within GKE, you may wish to consider using Workload Identity Federation instead. ::: #### Workload Identity Federation If Kargo locates no `Secret` resources matching a repository URL, and if Kargo is deployed within a GKE cluster, it will attempt to use [Workload Identity Federation](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity) to authenticate. This works for both container image repositories and OCI Helm chart repositories, and relies upon some external setup. Leveraging this option eliminates the need to store credentials in a `Secret` resource. :::info This option relies upon extensive external configuration that likely requires the assistance of Kargo's operator and a GCP project administrator, and as such, further coverage is delegated to the [Managing Secrets](../../40-operator-guide/40-security/40-managing-secrets.md) section of the Operator Guide. ::: ### Azure Container Registry (ACR) The authentication options described in this section are applicable only to container image repositories whose URLs indicate they are hosted in ACR. #### Long-Lived Credentials {#acr-long-lived-credentials} Azure Container Registry directly supports long-lived credentials. It is possible to [create tokens with repository-scoped permissions](https://learn.microsoft.com/en-us/azure/container-registry/container-registry-repository-scoped-permissions), with or without an expiration date. These tokens can be stored in the `username` and `password` fields of a `Secret` resource as described [in the first section](#repository-credentials) of this document. :::caution Following the principle of least privilege, the ACR token should be limited only to read-only access to the required ACR repositories. Configuring this will likely require the assistance of an Azure administrator. ::: :::caution This method of authentication is a "lowest common denominator" approach that will work regardless of where Kargo is deployed. i.e. If running Kargo outside of AKS, this method will still work. If running Kargo within AKS, you may wish to consider using Azure Workload Identity instead. ::: #### Azure Workload Identity If Kargo locates no `Secret` resources matching a repository URL, and if Kargo is deployed within an AKS cluster with workload identity enabled, it will attempt to use [Azure Workload Identity](https://learn.microsoft.com/en-us/azure/aks/workload-identity-overview) to authenticate. Leveraging this option eliminates the need to store credentials in a `Secret` resource. :::info This option relies upon extensive external configuration that likely requires the assistance of Kargo's operator and an Azure administrator, and as such, further coverage is delegated to the [Managing Secrets](../../40-operator-guide/40-security/40-managing-secrets.md) section of the Operator Guide. ::: --- ## Security(50-security) If you're managing Kargo projects, you're sure to have questions about things like access control and managing credentials used by your pipelines. This section is dedicated to exploring such topics from an end-user's perspective. --- ## Promotion Templates Reference Promotion Templates define how Kargo transitions `Freight` into a `Stage` by executing a series of discrete, composable steps. Each step in a promotion template performs a specific action, from simple operations like cloning a Git repository to complex workflows like managing the lifecycle of a pull request. When `Freight` is promoted to a `Stage`, Kargo uses the promotion template associated with that `Stage` to create a `Promotion` object which orchestrates the promotion process. ## Defining a Promotion Template A promotion template is defined within a `Stage`'s configuration using the `spec.promotionTemplate` field. The template contains two main components: - [Global variables](#variables) that provide configurable values for the promotion process - [Steps](#steps) that define the sequence of actions to perform ```yaml apiVersion: kargo.akuity.io/v1alpha1 kind: Stage metadata: name: test namespace: kargo-demo spec: # ... promotionTemplate: spec: # Template-wide variables vars: - name: gitRepo value: https://github.com/example/repo.git # Sequence of promotion steps steps: - uses: git-clone config: repoURL: ${{ vars.gitRepo }} checkout: - branch: main path: ./src - task: name: update-manifests kind: PromotionTask ``` ### Variables Variables provide a way to define reusable values that can be referenced throughout the promotion template. They can be used to parameterize steps, making the template more flexible and easier to maintain. You can define variables at two levels: globally for the entire template, or [scoped to a specific step](#step-variables). Global variables are perfect for values you want to use throughout the promotion process, like repository URLs or branch names. To define global variables, use the `vars` field in the `spec` of the promotion template. The value of the `vars` key is a list of variables, each of which is an object with `name` and `value` keys. ```yaml vars: - name: gitRepo value: https://github.com/example/repo.git - name: targetBranch value: stage/${{ ctx.stage }} ``` Once defined, you can reference these variables throughout the template using `${{ vars. }}`. ```yaml steps: - uses: git-clone config: repoURL: ${{ vars.gitRepo }} checkout: - branch: ${{ vars.targetBranch }} path: ./target ``` :::info Global variables can use [expressions](40-expressions.md) within `${{ }}` to compute values dynamically, including references to context variables like `ctx.stage`. They **do not** support referencing outputs from steps. To reference outputs, use [step-specific variables](#step-variables). ::: :::info Variables defined at the template level are available to all steps within the template. You can override these values within individual steps by defining [step-specific variables](#step-variables). ::: ### Steps The `steps` section in the `spec` of a promotion template defines the sequence of actions to perform during the promotion process. Steps come in two forms: - Steps that reference a [built-in promotion step](#built-in-steps) - Steps that reference a [`PromotionTask` or `ClusterPromotionTask`](#promotion-task-steps) #### Built-in Steps A step can be used to reference a built-in promotion step using the `uses` key whose value is the name of the step. ```yaml steps: - uses: step-name ``` :::info For a list of built-in promotion steps and configuration options, see the [Promotion Steps Reference](30-promotion-steps/index.md). ::: #### Promotion Task Steps A step can be used to reference a [`PromotionTask` or `ClusterPromotionTask`](20-promotion-tasks.md) using the `task` key, whose value is an object with a `name` key that specifies the name of the task and optionally a `kind` key that specifies the kind of task to reference. The `kind` key is optional and defaults to `PromotionTask`. ```yaml steps: - task: name: task-name kind: ClusterPromotionTask ``` :::note Steps referencing `PromotionTask` or `ClusterPromotionTask` __do not__ support any configuration option other than an `alias` and `vars`, as the steps within the task define their own configuration. For more information, see the [Promotion Tasks Reference](20-promotion-tasks.md). ::: #### Step Variables A step can define variables that can be referenced in its configuration by providing a `vars` key in the step definition. The value of the `vars` key is a list of variables, each of which is an object with `name` and `value` keys. ```yaml steps: - uses: step-name vars: - name: var1 value: value1 - name: var2 value: value2 config: option1: ${{ vars.var1 }} option2: ${{ vars.var2 }} ``` Variables defined in a step are scoped to that step and are not accessible to other steps like [global variables](#variables) are. The values of variables may contain [expressions](40-expressions.md). In addition, the values of step variables may contain references to the [outputs](#step-outputs) of other steps. ```yaml steps: - uses: step-name as: step1 - uses: another-step vars: - name: var1 value: ${{ outputs.step1.someOutput }} ``` :::info Step variables with the same name as global variables will override the global value for that step. ::: #### Step Configuration Each step in a promotion template can be configured with a set of options that control its behavior. The `config` key in a step definition is an object that contains the configuration options for the step. ```yaml steps: - uses: step-name config: option1: value1 option2: value2 ``` The configuration options available for a step are specific to the step itself and are documented in the [Promotion Steps Reference](30-promotion-steps/index.md). #### Step Outputs A promotion step may produce output that can be referenced by subsequent steps, allowing the output of one step to be used as input to another. The output of a step is defined by the step itself and is typically documented in the step's [reference documentation](30-promotion-steps/index.md). ```yaml steps: - uses: step-name as: alias - uses: another-step config: input: ${{ outputs.alias.someOutput }} ``` #### Conditional Steps You can conditionally execute a step based on the results of previous steps by using the `if` key in the step definition. The value of the `if` key must be a valid [expression](40-expressions.md) that evaluates to a boolean value. If the expression evaluates to `true`, the step is executed as normal. If the expression evaluates to `false`, the step is skipped and the next step in the sequence is executed. These conditionals may make use of special functions whose returned values depend on the outcome of all previous steps: - `always()`: Unconditionally evaluates to `true`. - `failure()`: Evaluates to `true` only if _any_ previous step has errored or failed. - `success()`: Evaluates to `true` only if _all_ previous steps have completed successfully or been skipped. Examples: ```yaml steps: - uses: some-step # Assume this step succeeds - uses: some-other-step if: ${{ always() }} # This step will execute; assume it succeeds - uses: and-another-step if: ${{ failure() }} # This step will be skipped - uses: yet-another-step if: ${{ success() }} # This step will execute; assume it succeeds # This Promotion will succeed ``` ```yaml steps: - uses: some-step # Assume this step fails - uses: some-other-step if: ${{ always() }} # This step will execute - uses: and-another-step if: ${{ failure() }} # This step will execute - uses: yet-another-step if: ${{ success() }} # This step will be skipped # This Promotion will fail ``` It is also possible to directly access the status of a specific step using the `status()` function with a step alias as an argument: ```yaml steps: - uses: some-step # Assume this step encounters an error as: my-step - uses: some-other-step if: ${{ always() }} # This step will execute - uses: and-another-step if: ${{ failure() }} # This step will execute - uses: yet-another-step if: ${{ status('my-step') == 'Errored' }} # This step will execute # This Promotion will fail ``` A step's `continueOnError` field can be set to `true` to indicate that an error or failure should _both_: - Not be counted as an error or failure when evaluating the `failure()` or `success()` functions. - Not affect the overall outcome of the `Promotion`. Example: ```yaml steps: - uses: some-step # Assume this step fails continueOnError: true - uses: some-other-step if: ${{ always() }} # This step will execute; assume it succeeds - uses: and-another-step if: ${{ failure() }} # This step will be skipped - uses: yet-another-step if: ${{ success() }} # This step will execute; assume it succeeds # This Promotion will succeed ``` Last, when a step's `if` field is empty, the default behavior is to execute the step only if _all_ previous steps have either completed successfully, been skipped, or completed with some other status, but had `continueOnError` set to `true`. Examples: ```yaml steps: - uses: some-step # Assume this step succeeds - uses: some-other-step # This step will execute; assume it fails continueOnError: true - uses: and-another-step # This step will execute; assume it succeeds - uses: yet-another-step # This step will execute; assume it succeeds # This Promotion will succeed ``` :::info Using the primitives described above, it is possible to create robust error-handling logic in your promotion templates. ::: :::info Conditional steps can be used in [Promotion Tasks](20-promotion-tasks.md) to conditionally execute a task step based on provided [task variables](20-promotion-tasks.md#task-variables). An `if` condition specified on the [task step](#promotion-task-steps) itself does not affect the execution of the task, as the steps of the `PromotionTask` are inflated into the `Promotion` at creation time. Because of this, the `if` condition would be evaluated at creation time rather than execution time, preventing it from accessing the status or outputs of previous steps. ::: #### Step Retries When a step fails for any reason, it can be retried instead of immediately failing the entire `Promotion`. An _error threshold_ specifies the number of _consecutive_ failures required for retry attempts to be abandoned and the `Promotion` to fail. Independent of the error threshold, steps are also subject to a _timeout_. Any step that doesn't achieve its goal within that interval will cause the `Promotion` to fail. For steps that exhibit any kind of polling behavior, the timeout can cause a `Promotion` to fail with no _other_ failure having occurred. System-wide, the default error threshold is 1 and the default timeout is indefinite. Thus, default behavior is effectively no retries when a step fails for any reason and steps with any kind of polling behavior will poll indefinitely _as long a no other failure occurs._ The implementations of individual steps can override these defaults. Users also may override these defaults through configuration. In the following example, the `git-wait-for-pr` step is configured not to fail the `Promotion` until three consecutive failed attempts to execute it. It is also configured to wait a maximum of 48 hours for the step to complete successfully (i.e. for the PR to be merged). ```yaml steps: # ... - uses: wait-for-pr retry: errorThreshold: 3 timeout: 48h config: prNumber: ${{ outputs['open-pr'].pr.id }} ``` :::info This feature was introduced in Kargo v1.1.0, and is still undergoing refinements and improvements to better distinguish between transient and non-transient errors, and to provide more control over retry behavior like backoff strategies or time limits. ::: --- ## Promotion Tasks Reference Promotion Tasks define reusable sets of promotion steps that can be shared across multiple [Promotion Templates](15-promotion-templates.md). They come in two forms: - `PromotionTask`: Scoped to a specific project - `ClusterPromotionTask`: Available globally across all projects When a Promotion Template references a promotion task, Kargo inflates the task's steps and merges them with the template's steps when creating a `Promotion`. This makes it easy to standardize common promotion workflows across your project or organization. ## Defining a Promotion Task A promotion task is defined using the `PromotionTask` or `ClusterPromotionTask` resource type. The task contains two main components: - [Variables](#task-variables) that provide configurable inputs when the task is used - [Steps](#task-steps) that define the sequence of built-in steps to perform ```yaml apiVersion: kargo.akuity.io/v1alpha1 kind: PromotionTask metadata: name: open-pr-and-wait namespace: kargo-demo spec: # Task-wide input variables vars: - name: repoURL - name: sourceBranch - name: targetBranch value: main # Sequence of promotion steps steps: - uses: git-open-pr as: open-pr config: repoURL: ${{ vars.repoURL }} createTargetBranch: true sourceBranch: ${{ vars.sourceBranch }} targetBranch: ${{ vars.targetBranch }} ``` ### Task Variables Variables in a Promotion Task define the inputs required when the task is used within a Promotion Template. They provide a way to parameterize the task's behavior while maintaining reusability. To define variables, use the `vars` field in the task's `spec`. Each variable requires a `name` and optionally a default `value`. ```yaml vars: # Required variable (no default) - name: repoURL # Optional variable with default value - name: targetBranch value: main ``` :::info Variables without a default value are required and must be provided when the task is referenced in a Promotion Template. ::: Variables can be referenced throughout the task using `${{ vars. }}`: ```yaml steps: - uses: git-clone config: repoURL: ${{ vars.repoURL }} checkout: - branch: ${{ vars.targetBranch }} path: ./target ``` When using a promotion task in a template, variables can be provided in two ways: 1. Through the template's [global `vars` section](15-promotion-templates.md#variables) (inherited by the task) 1. Through the [task step's `vars` section](15-promotion-templates.md#step-variables) (overrides inherited values) ```yaml spec: promotionTemplate: spec: # Global vars inherited by tasks vars: - name: repoURL value: https://github.com/example/repo.git steps: - task: name: my-task # Step-level vars override inherited values vars: - name: targetBranch value: feature-branch ``` ### Task Steps The `steps` section in a Promotion Task defines the sequence of actions to perform when the task is used. Each step can reference a [built-in promotion step](30-promotion-steps/index.md) using the `uses` key: ```yaml steps: - uses: git-clone as: clone config: repoURL: ${{ vars.repoURL }} checkout: - branch: ${{ vars.targetBranch }} path: ./target ``` :::note Unlike Promotion Templates, task steps cannot reference other Promotion Tasks. This prevents circular dependencies and keeps tasks focused on a specific workflow. ::: #### Task Context Steps within a promotion task have access to an additional [pre-defined variable](40-expressions.md#pre-defined-variables) called `task` that provides access to outputs from previous steps in the task. The `task.outputs` property is a map of step aliases within the `PromotionTask` to their outputs. ```yaml steps: - uses: git-open-pr as: open-pr config: repoURL: ${{ vars.repoURL }} - uses: git-wait-for-pr config: prNumber: ${{ task.outputs['open-pr'].pr.id }} ``` :::info The `task.outputs` variable is required when referencing outputs from previous steps within the same task. This requirement exists because tasks are inflated during the creation of a `Promotion`, and after inflation, the alias of the task step is namespaced to avoid conflicts with other steps in the template. This means that the alias of a task step at runtime is not known to the `PromotionTask` definition, so it cannot be used to reference outputs. ::: ### Task Outputs A promotion task can expose outputs that become available to subsequent steps in the parent Promotion Template. To define outputs for a task, use the [`compose-output` step](30-promotion-steps/compose-output.md). ```yaml steps: # ...omitted for brevity - uses: compose-output as: output config: commit: ${{ task.outputs['wait-for-pr'].commit }} branch: ${{ vars.targetBranch }} ``` The composed outputs become available in the template under the task step's alias. ```yaml steps: - task: name: my-task as: promotion - uses: http config: url: https://api.example.com/notify body: | New commit: ${{ outputs.promotion.commit }} ``` ## Defining a Global Promotion Task To create a promotion task that's available across all projects, use the `ClusterPromotionTask` resource. It's defined exactly like a `PromotionTask` but without namespace scope: ```yaml apiVersion: kargo.akuity.io/v1alpha1 kind: ClusterPromotionTask metadata: name: global-task spec: # ...equivalent to a PromotionTask ``` To use a `ClusterPromotionTask` in a template, specify the `kind` in the task reference: ```yaml steps: - task: name: global-task kind: ClusterPromotionTask ``` :::info `ClusterPromotionTasks` are perfect for standardizing promotion workflows across your organization, such as promotion patterns that should be consistently applied across all projects. ::: --- ## Custom steps :::info This promotion step is only available in Kargo on the [Akuity Platform](https://akuity.io/akuity-platform), versions v1.10 and above. Additionally, it requires enabling of the Promotion Controller and use of self-hosted agent to allow for Pod-based promotions. ::: :::warning This feature is alpha version, configurations and specifications may change in the future. Performance and stability have not been tested yet, using the feature for memory-heavy processes may cause issues. ::: Custom steps allow operators to configure their own promotion step logic using OCI images and scripting. This can be used for tasks currently not provided by built-in steps to extend Kargo capabilites with bespoke or proprietary functionality. ## Registering a custom step Custom steps have to be registered in a Kargo cluster in order to be used in promotion templates. To register a new step, a Kargo cluster admin needs to create a cluster-scoped `CustomPromotionStep` resource: ```yaml apiVersion: ee.kargo.akuity.io/v1alpha1 kind: CustomPromotionStep metadata: name: hello-world spec: ## REQUIRED: image to execute the command in image: ubuntu ## REQUIRED: command to run command: ["sh", "-c", "sleep 5; echo 'hello=${HELLO}-${{ config.world }}' > $KARGO_OUTPUT"] ## OPTIONAL: additional environment to provide to the command env: - name: HELLO value: ${{ config.hello }} ## OPTIONAL: output capture configuration output: source: ## One of Pipe | Stdout | Stderr | File, default is Pipe type: Pipe ## required when type is File, should be omitted if it's not path: "" ## One of JSON | YAML | KeyValue | Text ## inferred from file extension when omitted, default is KeyValue format: KeyValue ## Apply expressions to output to reformat with different keys ## If omitted returns the parsed output as is transform: message: output.hello ## OPTIONAL: error handling metadata defaultTimeout: 5m defaultErrorThreshold: 3 ## OPTIONAL: capabilities to provision into step container ## Options: access-control-plane, access-argocd capabilities: [] ## OPTIONAL: container resources configuration resources: requests: memory: "64Mi" cpu: "250m" limits: memory: "128Mi" cpu: "500m" ## OPTIONAL: secrets to pull the image from private repos imagePullSecrets: [] ``` :::warning The `command` field does not specify the `command` of the step container in promotion pod definition. Kargo is running an executor binary which coordinates step execution (start, retry, abort) and will run the `command`. ::: :::warning If retry policy is set for the step, the `command` could be executed multiple times on failure. It's recommended to design commands to have some idempotency. ::: ## Using a custom step After a custom step is registered in the cluster, it can be used in promotion template: ```yaml vars: - name: exampleVar value: example steps: - as: my-custom-step uses: hello-world config: world: ${{ vars.exampleVar }} hello: bonjour something: else: - entirely ``` ### Passing input to steps Command execution will run in the same workdir as other steps. To access step config or step execution context, `command` and values in `env` can use templates like: ```yaml command: - "echo" - "Step ${{ ctx.meta.step.alias }} in promotion $PROM_VAR with ${{ config.my_config }} and ${{ config.nested.values[0] }}" env: - name: PROM_VAR value: ${{ ctx.promotion }} ``` And config: ```yaml config: my_config: configvalue nested: values: - one ``` :::info There is no config validation at the moment and configuration structure and keys can be arbitrary. The step will error in runtime if any expressions require a missing config value, e.g. `${{ config.something.else }}` ::: Templates are using [expression language](../40-expressions.md), but only with `config` and `ctx` variables. The `ctx` variable is using [the ctx format](../40-expressions.md#context-ctx-object-structure). In order to pass values from secrets, configmaps or other promotion context to the custom step, they should be used in the promotion template and passed as `config` variables. #### Passing secrets to custom steps Example using credential from a secret, assuming secret `db_credential` has keys `username` and `password`: ```yaml apiVersion: ee.kargo.akuity.io/v1alpha1 kind: CustomPromotionStep metadata: name: access-db spec: image: my_image command: ["db_script.sh", "--username=${{ config.cred.username }}"] env: - name: DB_PASSWORD value: ${{ config.cred.password }} ``` ```yaml - as: custom-db-step uses: access-db config: cred: ${{ secret("db_credential") }} ``` ### Steps output Output from a step execution is parsed according to the `output` configuration: ```yaml output: source: type: Pipe | Stdout | Stderr | File path: format: JSON | YAML | KeyValue | Text transform: : ``` If `output` is not specified, default configuration is: ```yaml output: source: type: Pipe path: "" format: KeyValue transform: {} ``` #### Output sources | Type | Behavior | |------|----------| | `Pipe` | Reads from the `KARGO_OUTPUT` temp file injected by the executor. Default when `output` is omitted. | | `Stdout` | Reads from the command's standard output. | | `Stderr` | Reads from the command's standard error. | | `File` | Reads from a file at the specified `path`. Format inferred from file extension when omitted. | #### Ouptut formats | Format | Behavior | |--------|----------| | `KeyValue` | `key=value` lines (GitHub Actions–compatible). Quotes are part of the value, not stripped. Lines without `=` are ignored. Multiline values supported via heredoc syntax (`key<}`. | | `YAML` | Same semantics as JSON. Format inferred from `.yaml`/`.yml` file extensions. | | `Text` | Stored as `{"output": ""}`. | #### Transform expressions An optional `transform` map accepts per-key [expr-lang](https://expr-lang.org) expressions to reshape the parsed output before it is stored. The variable `output` holds the parsed value (map for JSON/YAML/KeyValue, string for Text). Each expression may return any value: ```yaml transform: hasCritical: output.summary.critical > 0 total: output.summary.total failed: output contains "FAIL" message: output.message ``` If `transform` map is used, it will completely replace the output produced by the `source`. To pass some values through they should map to their respective output key (like `message` in the example above) #### Output size limits - All formats: hard 256 KiB limit on the final output map (checked after transform). Returns an error if exceeded. - The size check happens after `transform` evaluation, so a transform that extracts a small subset of a large payload will not hit the limit. - Step result messages: stdout/stderr truncated to the last 16 KiB each to keep Kubernetes status conditions readable. ## Runtime limitations - Supported container architectures: - linux-arm64 - linux-amd64 - Aborting the step will kill the process with `SIGKILL`. There is no graceful shutdown at the moment. - Currently step containers run with user `65532`, images requiring specific user are not supported yet. - Avoid using too many steps in the same promotion as they utilize the same pod and may exhaust pod/node resources. ## ADVANCED: step capabilities Kargo steps can have the following capabilities configured: - `access-control-plane` - allow access to Kargo controlplane via k8s API. `kubeconfig` or `token` (for local in-cluster access) will be provisioned to `coordination/kubernetes/kargo` directory in the container. - `access-argocd` - allow access to ArgoCD controlplane via k8s API. `kubeconfig` or `token` (for local in-cluster access) will be provisioned to `coordination/kubernetes/argocd` directory in the container. --- ## `argocd-update` `argocd-update` updates one or more Argo CD `Application` resources in various ways. `Application`s can be selected either by exact name or by using label selectors to match multiple `Application`s at once. Among other scenarios, this step is useful for the common one of forcing an Argo CD `Application` to sync after previous steps have updated a remote branch referenced by the `Application`. This step is commonly the last step in a promotion process. :::note For an Argo CD `Application` resource to be managed by a Kargo `Stage`, the `Application` _must_ have an annotation of the following form: ```yaml kargo.akuity.io/authorized-stage: ":" ``` Such an annotation offers proof that a user who is themselves authorized to update the `Application` in question has consented to a specific `Stage` updating the `Application` as well. The following example shows how to configure an Argo CD `Application` manifest to authorize the `test` `Stage` of the `kargo-demo` `Project`: ```yaml apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: kargo-demo-test namespace: argocd annotations: kargo.akuity.io/authorized-stage: kargo-demo:test spec: # Application specifications go here ``` ::: :::info Enforcement of Argo CD [sync windows](https://argo-cd.readthedocs.io/en/stable/user-guide/sync_windows/) was improved substantially in Argo CD v2.11.0. If you wish for the `argocd-update` step to honor sync windows, you must use Argo CD v2.11.0 or later. _Additionally, it is recommended that if a promotion process is expected to sometimes encounter an active deny window, the `argocd-update` step should be configured with a timeout that is at least as long as the longest expected deny window. The step's default timeout of five minutes can be overridden using the [`retry.timeout`](../15-promotion-templates.md#step-retries) field. ::: ## Application Selection The `argocd-update` step supports two methods for selecting Argo CD `Application` resources: 1. **By Name**: Specify an exact application name using the `name` field 2. **By Label Selector**: Match one or more `Application`s using the `selector` field with label-based criteria These two methods are mutually exclusive — you must specify either `name` or `selector`, but not both. ### Selecting by Name When using the `name` field, you select a single `Application` by its exact name. This is the traditional method and is useful when you need to update a specific `Application`. ### Selecting by Label Selector Label selectors allow you to match multiple `Application` resources based on their labels. This is useful for: - Updating multiple `Application`s across different environments simultaneously. - Managing groups of related `Application`s (e.g., all `Application`s with label `team: platform`). - Performing bulk operations like hard refreshes across a set of `Application`s. [Label selectors](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#set-references-in-api-objects) support two types of matching criteria: - **`matchLabels`**: Simple key-value equality matching. All specified labels must match (AND logic). - **`matchExpressions`**: Advanced matching with operators (`In`, `NotIn`, `Exists`, `DoesNotExist`) for more complex selection logic. Both criteria types can be combined in a single selector, and all criteria must be satisfied for an `Application` to be selected. ### Validation for Multi-Application Updates When using selectors with source updates (e.g., updating `targetRevision` or image versions), the step performs homogeneity validation before making any changes: 1. All selected `Application`s must have compatible source configurations. 2. The specified source updates must be applicable to all matched `Application`s. 3. If validation fails for any `Application`, the step fails immediately without updating any of the selected `Application`s. This ensures that bulk updates are applied consistently and prevents partial updates that could leave `Application`s in inconsistent states. :::info When using selectors without source updates (for hard refreshes), validation is not performed since no source changes are being made. This allows you to safely refresh groups of `Application`s with heterogeneous configurations. ::: ## Configuration | Name | Type | Required | Description | |------|------|----------|-------------| | `apps` | `[]object` | Y | Describes Argo CD `Application` resources to update and how to update them. At least one must be specified. | | `apps[].name` | `string` | N | The name of the Argo CD `Application`. Mutually exclusive with `selector`. Either `name` or `selector` must be specified. __Note:__ A small technical restriction on this field is that any [expressions](../40-expressions.md) used therein are limited to accessing `ctx` and `vars` and may not access `secrets` or any Freight. This is because templates in this field are, at times, evaluated outside the context of an actual `Promotion` for the purposes of building an index. In practice, this restriction does not prove to be especially limiting. | | `apps[].namespace` | `string` | N | The namespace of the Argo CD `Application` resource(s) to be updated. If left unspecified, the namespace will be the Kargo controller's configured default -- typically `argocd`. __Note:__ This field is subject to the same restrictions as the `name` field. See above. | | `apps[].selector` | `object` | N | Label selector for matching one or more Argo CD `Application` resources. Mutually exclusive with `name`. Either `name` or `selector` must be specified. __Note:__ This field is subject to the same restrictions as the `name` field regarding expression usage. See above. | | `apps[].selector.matchLabels` | `map[string]string` | N | A map of label key-value pairs. All specified labels must match for an `Application` to be selected (AND logic). At least one of `matchLabels` or `matchExpressions` must be specified. | | `apps[].selector.matchExpressions` | `[]object` | N | A list of label selector requirements. All expressions must be satisfied for an `Application` to be selected. At least one of `matchLabels` or `matchExpressions` must be specified. | | `apps[].selector.matchExpressions[].key` | `string` | Y | The label key that the selector applies to. | | `apps[].selector.matchExpressions[].operator` | `string` | Y | The operator to use for matching. Valid values: `In`, `NotIn`, `Exists`, `DoesNotExist`. | | `apps[].selector.matchExpressions[].values` | `[]string` | N | An array of string values. Required when `operator` is `In` or `NotIn`. Must be empty when `operator` is `Exists` or `DoesNotExist`. | | `apps[].sources` | `[]object` | N | Describes Argo CD `ApplicationSource`s to update and how to update them. | | `apps[].sources[].repoURL` | `string` | Y | The value of the target `ApplicationSource`'s own `repoURL` field. This must match exactly. | | `apps[].sources[].chart` | `string` | N | Applicable only when the target `ApplicationSource` references a Helm chart repository, the value of the target `ApplicationSource`'s own `chart` field. This must match exactly. | | `apps[].sources[].desiredRevision` | `string` | N | Specifies the desired revision for the source. i.e. The revision to which the source must be observably synced when performing a health check. Prior to v1.1.0, if left undefined, the desired revision was determined by Freight (if possible). Beginning with v1.1.0, if left undefined, Kargo will not require the source to be observably synced to any particular source to be considered healthy. Note that the source's `targetRevision` will not be updated to this revision unless `updateTargetRevision=true` is also set. | | `apps[].sources[].updateTargetRevision` | `boolean` | Y | Indicates whether the target `ApplicationSource` should be updated such that its `targetRevision` field points directly at the desired revision. A `true` value in this field requires `desiredRevision` to be specified. | | `apps[].sources[].kustomize` | `object` | N | Describes updates to an Argo CD `ApplicationSource`'s Kustomize-specific properties. | | `apps[].sources[].kustomize.images` | `[]object` | Y | Describes how to update an Argo CD `ApplicationSource`'s Kustomize-specific properties to reference specific versions of container images. | | `apps[].sources[].kustomize.images[].repoURL` | `string` | Y | URL of the image being updated. | | `apps[].sources[].kustomize.images[].tag` | `string` | N | A tag naming a specific revision of the image specified by `repoURL`. Mutually exclusive with `digest`. One of `digest` or `tag` must be specified. | | `apps[].sources[].kustomize.images[].digest` | `string` | N | A digest naming a specific revision of the image specified by `repoURL`. Mutually exclusive with `tag`. One of `digest` or `tag` must be specified. | | `apps[].sources[].kustomize.images[].newName` | `string` | N | A substitution for the name/URL of the image being updated. This is useful when different Stages have access to different container image repositories (assuming those different repositories contain equivalent images that are tagged identically). This may be a frequent consideration for users of Amazon's Elastic Container Registry. | | `apps[].sources[].helm` | `object` | N | Describes updates to an Argo CD `ApplicationSource`'s Helm parameters. | | `apps[].sources[].helm.images` | `[]object` | Y | Describes how to update an Argo CD `ApplicationSource`'s Helm parameters to reference specific versions of container images. | | `apps[].sources[].helm.images[].key` | `string` | Y | The key to update within the target `ApplicationSource`'s `helm.parameters` map. See Helm documentation on the [format and limitations](https://helm.sh/docs/intro/using_helm/#the-format-and-limitations-of---set) of the notation used in this field. | | `apps[].sources[].helm.images[].value` | `string` | Y | Specifies the new value for the key. Typically, a value from [`chartFrom()`](../40-expressions.md#chartfrom) is used here. | ## Health Checks The `argocd-update` step is unique among all other built-in promotion steps in that, on successful completion, it will register health checks to be performed upon the target `Stage` on an ongoing basis. This health check configuration is _opaque_ to the rest of Kargo and is understood only by health check functionality built into the step. This permits Kargo to factor the health and sync state of Argo CD `Application` resources into the overall health of a `Stage` without requiring Kargo to understand `Application` health directly. :::info Although the `argocd-update` step is the only promotion step to currently utilize this health check framework, we anticipate that future built-in and third-party promotion steps will take advantage of it as well. Because of this, the health of a `Stage` is not necessarily a simple reflection of the `Application` resource it manages. It can also be influenced by other `Application` resources that are updated by other promotion steps, or by a `Promotion` which failed to complete successfully. ::: ## Examples ### Common Usage ```yaml steps: # Clone, render manifests, commit, push, etc... - uses: git-commit as: commit config: path: ./out message: ${{ outputs['update-image'].commitMessage }} - uses: git-push config: path: ./out - uses: argocd-update config: apps: - name: my-app sources: - repoURL: https://github.com/example/repo.git desiredRevision: ${{ outputs.commit.commit }} ``` ### Updating a Target Revision :::caution Without making any modifications to a Git repository, this example simply updates a "live" Argo CD `Application` resource to point its `targetRevision` field at a specific version of a Helm chart, which Argo CD will pull directly from the chart repository. While this can be a useful technique, it should be used with caution. This is not "real GitOps" since the state of the `Application` resource is not backed up in a Git repository. If the `Application` resource were deleted, there would be no remaining record of its desired state. ::: ```yaml vars: - name: gitRepo value: https://github.com/example/repo.git steps: - uses: argocd-update config: apps: - name: my-app sources: - repoURL: ${{ chartRepo }} chart: my-chart desiredRevision: ${{ chartFrom(chartRepo, "my-chart").Version }} updateTargetRevision: true ``` ### Updating an Image with Kustomize :::caution Without making any modifications to a Git repository, this example simply updates Kustomize-specific properties of a "live" Argo CD `Application` resource. While this can be a useful technique, it should be used with caution. This is not "real GitOps" since the state of the `Application` resource is not backed up in a Git repository. If the `Application` resource were deleted, there would be no remaining record of its desired state. ::: ```yaml vars: - name: gitRepo value: https://github.com/example/repo.git steps: - uses: argocd-update config: apps: - name: my-app sources: - repoURL: https://github.com/example/repo.git kustomize: images: - repoURL: ${{ vars.imageRepo }} tag: ${{ imageFrom(vars.imageRepo).Tag }} ``` ### Updating an Image with Helm :::caution Without making any modifications to a Git repository, this example simply updates Helm-specific properties of a "live" Argo CD `Application` resource. While this can be a useful technique, it should be used with caution. This is not "real GitOps" since the state of the `Application` resource is not backed up in a Git repository. If the `Application` resource were deleted, there would be no remaining record of its desired state. ::: ```yaml steps: - uses: argocd-update config: apps: - name: my-app sources: - repoURL: https://github.com/example/repo.git helm: images: - key: image.tag value: ${{ imageFrom("my/image").Tag }} ``` ### Selecting Applications with matchLabels This example shows how to use label selectors to match `Application`s based on simple key-value label matching. All specified labels must match for an application to be selected. ```yaml steps: - uses: argocd-update config: apps: - selector: matchLabels: environment: production team: platform sources: - repoURL: https://github.com/example/repo.git desiredRevision: ${{ outputs.commit.commit }} ``` This configuration will select all Argo CD `Application` resources in the default namespace that have both the `environment: production` and `team: platform` labels. ### Selecting Applications with matchExpressions This example demonstrates using `matchExpressions` with the `In` operator to select `Application`s that match one of several possible values for a label. ```yaml steps: - uses: argocd-update config: apps: - selector: matchExpressions: - key: environment operator: In values: - staging - production sources: - repoURL: https://github.com/example/repo.git desiredRevision: ${{ outputs.commit.commit }} ``` This configuration will select all Argo CD `Application` resources that have an `environment` label with a value of either `staging` or `production`. ### Combining matchLabels and matchExpressions This example shows how to combine both `matchLabels` and `matchExpressions` for more precise selection criteria. All criteria must be satisfied for an application to be selected. ```yaml steps: - uses: argocd-update config: apps: - selector: matchLabels: team: platform matchExpressions: - key: environment operator: In values: - staging - production - key: managed-by operator: NotIn values: - legacy-system sources: - repoURL: https://github.com/example/repo.git desiredRevision: ${{ outputs.commit.commit }} ``` This configuration will select all Argo CD `Application` resources that: - Have the label `team: platform` - Have an `environment` label with value `staging` or `production` - Do NOT have a `managed-by` label with value `legacy-system` ### Hard Refresh with Label Selectors This example shows how to use a label selector to perform a hard refresh of multiple `Application`s without updating any sources. This is useful for forcing Argo CD to re-sync `Application`s, for example after external changes have been made to the cluster or source repositories. ```yaml steps: - uses: argocd-update config: apps: - selector: matchLabels: auto-refresh: enabled ``` This configuration will trigger a hard refresh on all Argo CD `Application` resources that have the label `auto-refresh: enabled`. Since no `sources` are specified, Kargo will simply ensure the `Application`s are synced to their current target revisions without making any changes to the application specifications. ### Updating Multiple Environments Simultaneously This example demonstrates updating multiple `Application`s across different environments in a single step. This is useful for rolling out changes to multiple stages simultaneously, such as in a blue/green deployment scenario. ```yaml vars: - name: gitRepo value: https://github.com/example/repo.git steps: # Clone, render manifests, commit, push, etc... - uses: git-commit as: commit config: path: ./out message: Update to new version - uses: git-push config: path: ./out - uses: argocd-update config: apps: - selector: matchLabels: app: my-microservice deployment-group: blue sources: - repoURL: ${{ vars.gitRepo }} desiredRevision: ${{ outputs.commit.commit }} ``` This configuration will update all Argo CD `Application` resources that have both the `app: my-microservice` and `deployment-group: blue` labels, pointing them all to the same new revision. Kargo will validate that all matched `Application`s have compatible source configurations (i.e., the source exists in each `Application`) before applying any updates. --- ## `argocd-wait` `argocd-wait` waits for one or more Argo CD `Application` resources to reach desired conditions. `Application`s can be selected either by exact name or by using label selectors to match multiple `Application`s at once. This step is useful when a sync is triggered by means other than an [`argocd-update` step](argocd-update.md) — for example, when an external system or a human operator triggers the sync — and you want a `Promotion` to wait for the `Application` to become healthy and synced before proceeding. :::note Unlike [`argocd-update`](argocd-update.md), `argocd-wait` does **not** require the `kargo.akuity.io/authorized-stage` annotation on the `Application`. It can wait for any `Application` that the Kargo controller has read access to. ::: ## Application Selection The `argocd-wait` step supports two methods for selecting Argo CD `Application` resources: 1. **By Name**: Specify an exact application name using the `name` field. 2. **By Label Selector**: Match one or more `Application`s using the `selector` field with label-based criteria. These two methods are mutually exclusive — you must specify either `name` or `selector`, but not both. See the [`argocd-update` documentation](argocd-update.md#application-selection) for a full description of both selection methods including label selector syntax. ## Wait Conditions The `waitFor` field controls which conditions must be satisfied before the step succeeds. Supported values are: | Value | Description | |-------|-------------| | `health` | `Application` health is `Healthy`. | | `sync` | `Application` sync status is `Synced`. | | `operation` | No operation (e.g. sync) is currently in progress. | | `suspended` | `Application` health is `Suspended`. | | `degraded` | `Application` health is `Degraded`. | When `waitFor` is omitted, it defaults to `[health, sync, operation]`. The health-related values (`health`, `suspended`, `degraded`) are OR'd: the health check passes if _any_ of the specified health conditions is met. All other conditions (`sync`, `operation`) are AND'd with the result. :::note If `waitFor` includes `health` and an `Application` transitions to `Degraded` from a state that was neither `Degraded` nor `Unknown`, the step fails immediately rather than continuing to wait. This degradation detection prevents a `Promotion` from hanging indefinitely on an `Application` that regressed during the wait. ::: ## Configuration | Name | Type | Required | Description | |------|------|----------|-------------| | `apps` | `[]object` | Y | Describes Argo CD `Application` resources to wait for. At least one must be specified. | | `apps[].name` | `string` | N | The name of the Argo CD `Application`. Mutually exclusive with `selector`. Either `name` or `selector` must be specified. __Note:__ Expressions in this field are limited to accessing `ctx` and `vars` and may not access `secrets` or any Freight. This is because templates in this field are, at times, evaluated outside the context of an actual `Promotion` for the purposes of building an index. In practice, this restriction does not prove to be especially limiting. | | `apps[].namespace` | `string` | N | The namespace of the Argo CD `Application` resource(s) to wait for. If left unspecified, the namespace will be the Kargo controller's configured default — typically `argocd`. __Note:__ This field is subject to the same restrictions as the `name` field. See above. | | `apps[].selector` | `object` | N | Label selector for matching one or more Argo CD `Application` resources. Mutually exclusive with `name`. Either `name` or `selector` must be specified. | | `apps[].selector.matchLabels` | `map[string]string` | N | A map of label key-value pairs. All specified labels must match for an `Application` to be selected (AND logic). At least one of `matchLabels` or `matchExpressions` must be specified. | | `apps[].selector.matchExpressions` | `[]object` | N | A list of label selector requirements. All expressions must be satisfied for an `Application` to be selected. At least one of `matchLabels` or `matchExpressions` must be specified. | | `apps[].selector.matchExpressions[].key` | `string` | Y | The label key that the selector applies to. | | `apps[].selector.matchExpressions[].operator` | `string` | Y | The operator to use for matching. Valid values: `In`, `NotIn`, `Exists`, `DoesNotExist`. | | `apps[].selector.matchExpressions[].values` | `[]string` | N | An array of string values. Required when `operator` is `In` or `NotIn`. Must be empty when `operator` is `Exists` or `DoesNotExist`. | | `apps[].waitFor` | `[]string` | N | Conditions to wait for. Valid values: `health`, `sync`, `operation`, `suspended`, `degraded`. Defaults to `[health, sync, operation]` when omitted. | ## Examples ### Common Usage In this example, `argocd-wait` is used after [`argocd-update`](argocd-update.md) to confirm that the `Application` has reached a healthy and synced state before the `Promotion` is marked as succeeded. ```yaml steps: # Clone, render manifests, commit, push, etc... - uses: git-commit as: commit config: path: ./out message: ${{ outputs['update-image'].commitMessage }} - uses: git-push config: path: ./out - uses: argocd-update config: apps: - name: my-app sources: - repoURL: https://github.com/example/repo.git desiredRevision: ${{ outputs.commit.commit }} - uses: argocd-wait config: apps: - name: my-app ``` ### Waiting Only for Operation Completion This example waits only for any in-progress sync operation to finish, without requiring the `Application` to be healthy or synced to a specific revision. ```yaml steps: - uses: argocd-wait config: apps: - name: my-app waitFor: - operation ``` ### Waiting with Label Selector This example waits for all `Application`s with a given label to become healthy, synced, and idle — useful when multiple `Application`s are updated together. ```yaml steps: - uses: argocd-wait config: apps: - selector: matchLabels: environment: staging ``` ### Waiting for a Suspended Application This example waits for an `Application` to reach the `Suspended` health state, which can serve as a manual gate: a human suspends the `Application` to pause a rollout, and this step waits until that state is observed. ```yaml steps: - uses: argocd-wait config: apps: - name: my-app waitFor: - suspended ``` --- ## `compose-output` `compose-output` is a step that composes a new output from one or more existing outputs. This step can be useful when subsequent steps need to reference a combination of outputs from previous steps, or to allow a [`PromotionTask`](../20-promotion-tasks.md) to provide easy access to outputs from the steps it contains. ## Configuration The `compose-output` step accepts an arbitrary set of key-value pairs, where the key is the name of the output to be created and the value is arbitrary and can be an [Expression Language](../40-expressions.md) expression. ## Output The dynamic outputs of the `compose-output` step are the outputs that it composes according to its configuration. ## Examples ### Compose a Pull Request Link In this example, a pull request link is constructed and sent to Slack. First, a pull request is opened using the [`git-open-pr` step](git-open-pr.md). Then, the `compose-output` step creates a new output named URL by combining the repository URL with the PR number to form a complete PR link. Finally, the [`http` step](http.md) uses this composed URL to send a formatted message to a Slack channel using Slack's API. ```yaml vars: - name: repoURL value: https://github.com/example/repo steps: # Clone, prepare the contents of ./out, commit, push as part of a `push` step etc... - uses: git-open-pr as: open-pr config: repoURL: ${{ vars.repoURL }} createTargetBranch: true sourceBranch: ${{ task.outputs.push.branch }} targetBranch: stage/${{ ctx.stage }} - uses: compose-output as: pr-link config: url: ${{ vars.repoURL }}/pull/${{ task.outputs['open-pr'].pr.id }} - uses: http config: method: POST url: https://slack.com/api/chat.postMessage headers: - name: Authorization value: Bearer ${{ secret('slack').token }} - name: Content-Type value: application/json body: | ${{ quote({ "channel": "C123456", "blocks": [ { "type": "section", "text": { "type": "mrkdwn", "text": "A new PR has been opened: " + task.outputs['pr-link'].url } } ] }) }} ``` :::note In the above example, it is assumed that the steps are part of a [`PromotionTask`](../20-promotion-tasks.md). Because of this, it uses the [`task.outputs` syntax](../20-promotion-tasks.md#task-outputs) to reference the outputs of the `git-open-pr` step. If you are using the `compose-output` step in a regular [Promotion template](../15-promotion-templates.md), you can omit the `task` prefix and use `outputs` instead. For example, `outputs['open-pr'].pr.url` instead of `task.outputs['open-pr'].pr.url`. ::: --- ## `copy` `copy` copies files or the contents of entire directories from one specified location to another. ## Configuration | Name | Type | Required | Description | |------|------|----------|-------------| | `inPath` | `string` | Y | Path to the file or directory to be copied. This path is relative to the temporary workspace that Kargo provisions for use by the promotion process. | | `outPath` | `string` | Y | Path to the destination. This path is relative to the temporary workspace that Kargo provisions for use by the promotion process. | ## Examples ### Common Usage The most common (though still advanced) usage of this step is to combine content from two working trees to use as input to a subsequent step, such as one that renders Stage-specific manifests. Consider a Stage that requests Freight from two Warehouses, where one provides Kustomize "base" configuration, while the other provides a Stage-specific Kustomize overlay. Rendering the manifests intended for such a Stage will require combining the base and overlay configurations: ```yaml vars: - name: gitRepo value: https://github.com/example/repo.git steps: - uses: git-clone config: repoURL: ${{ vars.gitRepo }} checkout: - commit: ${{ commitFrom(vars.gitRepo, warehouse("base")).ID }} path: ./src - commit: ${{ commitFrom(vars.gitRepo, warehouse(ctx.stage + "-overlay")).ID }} path: ./overlay - branch: stage/${{ ctx.stage }} create: true path: ./out - uses: git-clear config: path: ./out - uses: copy config: inPath: ./overlay/stages/${{ ctx.stage }}/kustomization.yaml outPath: ./src/stages/${{ ctx.stage }}/kustomization.yaml # Render manifests to ./out, commit, push, etc... ``` --- ## `delete` `delete` deletes a file or directory. :::note If you need to delete the entire contents of a Git working tree, consider using the [`git-clear` step](git-clear.md) instead. ::: ## Configuration | Name | Type | Required | Description | |-----------|------|----------|------------------------------------------| | `path` | `string` | Y | Path to the file or directory to delete. | | `strict` | `bool` | N | Strict will cause the directive to fail if the path does not exist. Defaults to `false`. | ## Examples ### Common Usage One common usage of this step is to remove intermediate files produced by the promotion process prior to a [`git-commit` step](git-commit.md). This is useful when you want to ensure that only the final, desired files are committed to the repository. ```yaml vars: - name: gitRepo value: https://github.com/example/repo.git steps: - uses: git-clone config: repoURL: ${{ vars.gitRepo }} checkout: - commit: ${{ commitFrom(vars.gitRepo).ID }} path: ./src - branch: stage/${{ ctx.stage }} create: true path: ./out # Steps that produce intermediate files in ./out... - uses: delete config: path: ./out/unwanted/file - uses: git-commit config: path: ./out ``` --- ## `fail` `fail` fails the promotion. ## Configuration | Name | Type | Required | Description | | --------- | -------- | -------- | ----------------------------- | | `message` | `string` | N | Optional message to fail with | ## Examples ### Common Usage It may be necessary to fail a promotion if certain conditions are met. This can be done by combining this step with [conditional step execution](../15-promotion-templates.md#conditional-steps). In this example, a HTTP request fetches an expected chart name from an external API. If it does not match the chart name from the Freight, the promotion fails with a message indicating the mismatch. ```yaml steps: - uses: http as: expected-name config: url: https://api.example.com/expected-name outputs: - name: name fromExpression: response.body.name - uses: fail if: ${{ task.outputs['expected-name'].name != chartFrom('oci://example.com/my-chart').Name }} config: message: Expected chart name ${{ task.outputs['expected-name'].name }} ``` --- ## `gha-dispatch-workflow` :::info This promotion step is only available in Kargo on the [Akuity Platform](https://akuity.io/akuity-platform), versions v1.8 and above. ::: :::warning[Breaking Changes in v1.9] Starting with Kargo v1.9, this promotion step has significant configuration changes: - The `credentials` field has been removed - `owner` and `repo` fields have been replaced with a single `repoURL` field - Credentials are now inherited from Git repository credentials configured in Kargo - A new optional `insecureSkipTLSVerify` field has been added See the [Migration Guide](#migration-from-v18-to-v19) below for details. ::: The `gha-dispatch-workflow` promotion step provides integration with GitHub Actions, allowing you to dispatch workflows using the `workflow_dispatch` event. This is particularly useful for triggering CI/CD pipelines, running tests, or executing deployment scripts as part of your promotion workflows. ## Credentials Configuration ### v1.9 and Above Starting with Kargo v1.9, credentials are inherited from Git repository credentials configured in Kargo. The Git repository credentials must be configured with either: - **GitHub Personal Access Token (PAT)** - Classic or fine-grained - **GitHub App credentials** - App ID, installation ID, and private key :::info Required Permissions The GitHub credentials must have the following permissions: **Fine-grained Personal Access Token:** - `actions:write` - To dispatch workflows - `actions:read` - To read workflow run status **Classic Personal Access Token:** - `repo` - read/write access **GitHub App:** - `actions:write` - To dispatch workflows - `actions:read` - To read workflow run status ::: ### v1.8 (Deprecated) :::warning Removed in v1.9 The following credentials configuration has been removed in v1.9. Use the new Git repository credentials model instead. ::: All GitHub Actions operations require proper authentication credentials stored in a Kubernetes `Secret`. | Name | Type | Required | Description | | ------------------------ | -------- | -------- | -------------------------------------------------------------------------------- | | `credentials.secretName` | `string` | Y | Name of the `Secret` containing the GitHub credentials in the project namespace. | The referenced `Secret` should contain the following keys: - `accessToken`: GitHub personal access token or GitHub App token with appropriate permissions - `baseURL`: (Optional) GitHub base URL for GitHub Enterprise Server - `uploadURL`: (Optional) GitHub upload URL for GitHub Enterprise Server. Only required for GitHub Enterprise Server installations ## Configuration ### v1.9 and Above | Name | Type | Required | Description | | ----------------------- | --------- | -------- | ----------------------------------------------------------------------------------------------------- | | `repoURL` | `string` | Y | The GitHub repository URL where the workflow resides (e.g., 'https://github.com/owner/repo'). | | `workflowFile` | `string` | Y | The workflow filename in .github/workflows (e.g., 'deploy.yml'). | | `ref` | `string` | Y | The git reference (branch or tag) to run the workflow on. | | `inputs` | `object` | N | Input parameters to pass to the workflow as defined in the workflow's `workflow_dispatch` inputs. | | `timeout` | `integer` | N | Timeout in seconds to wait for the workflow run to be created after dispatch (default: 60, max: 300). | | `insecureSkipTLSVerify` | `boolean` | N | Skip TLS verification when communicating with the GitHub API (default: false). | ### v1.8 (Deprecated) :::warning[Removed in v1.9] The following configuration format has been removed in v1.9. ::: | Name | Type | Required | Description | | -------------- | --------- | -------- | ----------------------------------------------------------------------------------------------------- | | `owner` | `string` | Y | The owner of the repository (user or organization). | | `repo` | `string` | Y | The name of the repository. | | `workflowFile` | `string` | Y | The workflow filename in .github/workflows (e.g., 'deploy.yml'). | | `ref` | `string` | Y | The git reference (branch or tag) to run the workflow on. | | `inputs` | `object` | N | Input parameters to pass to the workflow as defined in the workflow's `workflow_dispatch` inputs. | | `timeout` | `integer` | N | Timeout in seconds to wait for the workflow run to be created after dispatch (default: 60, max: 300). | ## Output | Name | Type | Description | | ------- | --------- | --------------------------------------------------------------------------- | | `runID` | `integer` | The ID of the dispatched workflow run that can be used for status tracking. | ## Example ### v1.9 and Above This example dispatches a deployment workflow with custom inputs using the new configuration format. ```yaml steps: - uses: gha-dispatch-workflow as: dispatch-deployment config: repoURL: https://github.com/myorg/my-app workflowFile: deploy.yml ref: main inputs: environment: "${{ ctx.stage }}" image_tag: "${{ imageFrom(vars.imageRepo).Tag }}" promotion_id: "${{ ctx.promotion }}" deploy_version: "${{ imageFrom(vars.imageRepo).Tag }}" timeout: 120 insecureSkipTLSVerify: false ``` ### v1.8 (Deprecated) ```yaml steps: - uses: gha-dispatch-workflow as: dispatch-deployment config: credentials: secretName: github-credentials owner: myorg repo: my-app workflowFile: deploy.yml ref: main inputs: environment: "${{ ctx.stage }}" image_tag: "${{ imageFrom(vars.imageRepo).Tag }}" promotion_id: "${{ ctx.promotion }}" deploy_version: "${{ imageFrom(vars.imageRepo).Tag }}" timeout: 120 ``` ## Migration from v1.8 to v1.9 To migrate from v1.8 to v1.9: 1. **Remove the `credentials` field** - Credentials are now inherited from Git repository credentials 2. **Replace `owner` and `repo` fields** with a single `repoURL` field: - Old: `owner: myorg` and `repo: my-app` - New: `repoURL: https://github.com/myorg/my-app` 3. **Configure Git repository credentials** in Kargo with GitHub PAT or GitHub App credentials 4. **Optionally add `insecureSkipTLSVerify`** if you need to skip TLS verification ### Before (v1.8) ```yaml config: credentials: secretName: github-credentials owner: myorg repo: my-app workflowFile: deploy.yml ref: main ``` ### After (v1.9) ```yaml config: repoURL: https://github.com/myorg/my-app workflowFile: deploy.yml ref: main ``` --- ## `gha-wait-for-workflow` :::info This promotion step is only available in Kargo on the [Akuity Platform](https://akuity.io/akuity-platform), versions v1.8 and above. ::: :::warning[Breaking Changes in v1.9] Starting with Kargo v1.9, this promotion step has significant configuration changes: - The `credentials` field has been removed - `owner` and `repo` fields have been replaced with a single `repoURL` field - Credentials are now inherited from Git repository credentials configured in Kargo - A new optional `insecureSkipTLSVerify` field has been added See the [Migration Guide](#migration-from-v18-to-v19) below for details. ::: The `gha-wait-for-workflow` promotion step provides integration with GitHub Actions, allowing you to wait for workflow runs to complete and optionally validate their conclusion status. This is particularly useful for ensuring that CI/CD pipelines, tests, or deployment scripts complete successfully before proceeding with subsequent promotion steps. ## Credentials Configuration ### v1.9 and Above Starting with Kargo v1.9, credentials are inherited from Git repository credentials configured in Kargo. The Git repository credentials must be configured with either: - **GitHub Personal Access Token (PAT)** - Classic or fine-grained - **GitHub App credentials** - App ID, installation ID, and private key :::info Required Permissions The GitHub credentials must have the following permissions: **Fine-grained Personal Access Token:** - `actions:read` - To read workflow run status **Classic Personal Access Token:** - `repo` - read/write access **GitHub App:** - `actions:read` - To read workflow run status ::: ### v1.8 (Deprecated) :::warning[Removed in v1.9] The following credentials configuration has been removed in v1.9. Use the new Git repository credentials model instead. ::: All GitHub Actions operations require proper authentication credentials stored in a Kubernetes `Secret`. | Name | Type | Required | Description | | ------------------------ | -------- | -------- | -------------------------------------------------------------------------------- | | `credentials.secretName` | `string` | Y | Name of the `Secret` containing the GitHub credentials in the project namespace. | The referenced `Secret` should contain the following keys: - `accessToken`: GitHub personal access token or GitHub App token with appropriate permissions - `baseURL`: (Optional) GitHub base URL for GitHub Enterprise Server - `uploadURL`: (Optional) GitHub upload URL for GitHub Enterprise Server. Only required for GitHub Enterprise Server installations ## Configuration ### v1.9 and Above | Name | Type | Required | Description | | ----------------------- | --------- | -------- | ------------------------------------------------------------------------------------------ | | `repoURL` | `string` | Y | The GitHub repository URL where the workflow resides (e.g., 'https://github.com/owner/repo'). | | `runID` | `integer` | Y | The workflow run ID to wait for. Can be a direct ID or a reference to freight metadata. | | `expectedConclusion` | `string` | N | The expected final conclusion status. If not provided, conclusion status is not validated. | | `insecureSkipTLSVerify` | `boolean` | N | Skip TLS verification when communicating with the GitHub API (default: false). | | `pollInterval` | `string` | N | Poll interval for checking workflow status, specified as a [Go duration string](https://pkg.go.dev/time#ParseDuration) (e.g., `30s`, `5m`). Overrides the default controller reconciliation interval of 5 minutes when set. Available in Kargo >= v1.10. | ### v1.8 (Deprecated) :::warning Changed in v1.9 The following configuration format has been changed in v1.9. ::: | Name | Type | Required | Description | | -------------------- | --------- | -------- | ------------------------------------------------------------------------------------------ | | `owner` | `string` | Y | The owner of the repository (user or organization). | | `repo` | `string` | Y | The name of the repository. | | `runID` | `integer` | Y | The workflow run ID to wait for. | | `expectedConclusion` | `string` | N | The expected final conclusion status. If not provided, conclusion status is not validated. | Valid values for `expectedConclusion`: - `success` - The workflow completed successfully - `failure` - The workflow failed - `cancelled` - The workflow was cancelled - `skipped` - The workflow was skipped - `timed_out` - The workflow timed out - `action_required` - The workflow requires manual action - `neutral` - The workflow completed with neutral status - `stale` - The workflow is stale ## Output | Name | Type | Description | | ------------ | -------- | ----------------------------------------------------------------- | | `conclusion` | `string` | The final conclusion status of the workflow run after completion. | ## Example ### v1.9 and Above This example waits for a previously dispatched workflow to complete successfully using the new configuration format. ```yaml steps: - uses: gha-wait-for-workflow config: repoURL: https://github.com/myorg/my-app runID: "${{ outputs['dispatch-deployment'].runID }}" expectedConclusion: success insecureSkipTLSVerify: false ``` ### v1.8 (Deprecated) ```yaml steps: - uses: gha-wait-for-workflow config: credentials: secretName: github-credentials owner: myorg repo: my-app runID: "${{ outputs['dispatch-deployment'].runID }}" expectedConclusion: success ``` ## Multi-Step Workflow Example This example shows how to combine `gha-dispatch-workflow` and `gha-wait-for-workflow` steps using the new configuration format: ```yaml steps: # Dispatch the workflow - uses: gha-dispatch-workflow as: dispatch-deployment config: repoURL: https://github.com/example/test workflowFile: test.yaml ref: master inputs: greeting: "hola" timeout: 120 # Wait for the workflow to complete - uses: gha-wait-for-workflow config: repoURL: https://github.com/example/test runID: "${{ outputs['dispatch-deployment'].runID }}" expectedConclusion: success # Continue with other promotion steps after workflow completes... ``` ## Migration from v1.8 to v1.9 To migrate from v1.8 to v1.9: 1. **Remove the `credentials` field** - Credentials are now inherited from Git repository credentials 2. **Replace `owner` and `repo` fields** with a single `repoURL` field: - Old: `owner: example` and `repo: test` - New: `repoURL: https://github.com/example/test` 3. **Configure Git repository credentials** in Kargo with GitHub PAT or GitHub App credentials 4. **Optionally add `insecureSkipTLSVerify`** if you need to skip TLS verification ### Before (v1.8) ```yaml config: credentials: secretName: github-credentials owner: example repo: test runID: 12345 expectedConclusion: success ``` ### After (v1.9) ```yaml config: repoURL: https://github.com/example/test runID: 12345 expectedConclusion: success ``` --- ## `git-clear` `git-clear` deletes _the entire contents_ of a specified Git working tree (except for the `.git` file). It is equivalent to executing `git add . && git rm -rf --ignore-unmatch .`. This step is useful for the common scenario where the entire content of a Stage-specific branch is to be replaced with content from another branch or with content rendered using some configuration management tool. ## Configuration | Name | Type | Required | Description | |------|------|----------|-------------| | `path` | `string` | Y | Path to a Git working tree whose entire contents are to be deleted. | ## Examples ### Common Usage In this example, all content in a stage-specific branch is removed before new content is added. After cloning the repository with a stage-specific branch checked out to `./out`, the `git-clear` step removes all files from this working directory (except the `.git` directory). This pattern is commonly used when you want to completely replace the contents of a branch with newly generated content, rather than updating existing files. This ensures there are no stale files left over from previous states of the branch, providing a clean slate for the new content that will be added by subsequent steps. ```yaml steps: - uses: git-clone config: repoURL: https://github.com/example/repo.git checkout: - branch: stage/${{ ctx.stage }} create: true path: ./out - uses: git-clear config: path: ./out # Prepare the contents of ./out ... # Commit, push, etc... ``` --- ## `git-clone` `git-clone` is often the first step in a promotion process. It creates a [bare clone](https://git-scm.com/docs/git-clone#Documentation/git-clone.txt-code--barecode) of a remote Git repository, then checks out one or more branches, tags, or commits to working trees at specified paths. Checking out different revisions to different paths is useful for the common scenarios of combining content from multiple sources or rendering Stage-specific manifests to a Stage-specific branch. :::note It is a noteworthy limitation of Git that one branch cannot be checked out in multiple working trees. ::: ## Credentials Git steps are utilizing the [repository credentials](../../50-security/30-managing-secrets.md#repository-credentials) system to access the git repos. ## Configuration | Name | Type | Required | Description | |------|------|----------|-------------| | `repoURL` | `string` | Y | The URL of a remote Git repository to clone. **Deprecated:** Support for SSH URLs (`ssh://` and SCP-style `git@host:path`) is deprecated as of v1.10.0 and will be removed in v1.13.0. Use HTTPS URLs instead. | | `insecureSkipTLSVerify` | `boolean` | N | Whether to bypass TLS certificate verification when cloning (and for all subsequent operations involving this clone). Setting this to `true` is highly discouraged in production. | | `author` | `[]object` | N | Default authorship information for any commits made to the cloned repository. If provided, this overrides any system-level defaults. Note: Configuration of the [`git-commit`](./git-commit.md) step can override this information. | | `author.name` | `string` | Y | The committer's name. | | `author.email` | `string` | Y | The committer's email address. | | `author.signingKey` | `string` | N | The GPG signing key for the author. This field is optional. | | `checkout` | `[]object` | Y | The commits, branches, or tags to check out from the repository and the paths where they should be checked out. At least one must be specified. | | `checkout[].as` | `string` | N | Used as the key in the `commits` output map. If not specified, the value of the `path` field is used as a key instead. Providing a value for this field is useful when expressions in downstream steps may need to reference specific commits checked out by this step. | | `checkout[].branch` | `string` | N | A branch to check out. Mutually exclusive with `commit` and `tag`. If none of these is specified, the default branch will be checked out. | | `checkout[].create` | `boolean` | N | In the event `branch` does not already exist on the remote, whether a new, empty, orphaned branch should be created. Default is `false`, but should commonly be set to `true` for Stage-specific branches, which may not exist yet at the time of a Stage's first promotion. | | `checkout[].commit` | `string` | N | A specific commit to check out. Mutually exclusive with `branch` and `tag`. If none of these is specified, the default branch will be checked out. | | `checkout[].path` | `string` | Y | The path for a working tree that will be created from the checked out revision. This path is relative to the temporary workspace that Kargo provisions for use by the promotion process. | | `checkout[].tag` | `string` | N | A tag to check out. Mutually exclusive with `branch` and `commit`. If none of these is specified, the default branch will be checked out. | | `checkout[].sparse` | `[]string` | N | Directory paths for sparse checkout. Only the specified directories (and their contents) will be checked out. Paths must be relative to the repository root (e.g., `src/app`, `configs/prod`). Glob patterns are not supported. | ## Output | Name | Type | Description | |---------|---------------------|--------------------------------------------------------------------------------------------------| | `commits` | `map[string]string` | A map of checkout keys (either the `as` alias or the `path` if `as` is not set) to the HEAD commit hash checked out at each path. This allows downstream steps to reference the exact commit checked out for each working tree. | ## Examples ### Common Usage The most common usage of this step is to check out a commit specified by the Freight being promoted as well as a Stage-specific branch. Subsequent steps are likely to perform actions that revise the contents of the Stage-specific branch using the commit from the Freight as input. :::info For more information on `commitFrom` and expressions, see the [Expressions](../40-expressions.md#functions) documentation. ::: ```yaml vars: - name: gitRepo value: https://github.com/example/repo.git steps: - uses: git-clone config: repoURL: ${{ vars.gitRepo }} checkout: - commit: ${{ commitFrom(vars.gitRepo).ID }} path: ./src - branch: stage/${{ ctx.stage }} create: true path: ./out # Prepare the contents of ./out ... # Commit, push, etc... ``` ### Combining Multiple Sources For this more advanced example, consider a Stage that requests Freight from two Warehouses, where one provides Kustomize "base" configuration, while the other provides a Stage-specific Kustomize overlay. Rendering the manifests intended for such a Stage will require combining the base and overlay configurations with the help of a [`copy` step](copy.md). For this case, a `git-clone` step may be configured similarly to the following. :::info For more information on `commitFrom` and expressions, see the [Expressions](../40-expressions.md#functions) documentation. ::: ```yaml vars: - name: gitRepo value: https://github.com/example/repo.git steps: - uses: git-clone config: repoURL: ${{ vars.gitRepo }} checkout: - commit: ${{ commitFrom(vars.gitRepo, warehouse("base")).ID }} path: ./src - commit: ${{ commitFrom(vars.gitRepo, warehouse(ctx.stage + "-overlay")).ID }} path: ./overlay - branch: stage/${{ ctx.stage }} create: true path: ./out - uses: git-clear config: path: ./out - uses: copy config: inPath: ./overlay/stages/${{ ctx.stage }}/kustomization.yaml outPath: ./src/stages/${{ ctx.stage }}/kustomization.yaml - uses: kustomize-build config: path: ./src/stages/${{ ctx.stage }} outPath: ./out # Commit, push, etc... ``` ### Sparse Checkout For large repositories, you can use sparse checkout to check out only specific directories, reducing clone time and disk usage. When all checkouts specify sparse patterns, Kargo automatically uses a [blobless clone][], which avoids downloading file contents for directories that won't be checked out. ```yaml vars: - name: gitRepo value: https://github.com/example/repo.git steps: - uses: git-clone config: repoURL: ${{ vars.gitRepo }} checkout: - commit: ${{ commitFrom(vars.gitRepo).ID }} path: ./src sparse: - charts/my-app - configs/common - branch: stage/${{ ctx.stage }} create: true path: ./out sparse: - stages/${{ ctx.stage }} # Work with the sparse checkouts... ``` [blobless clone]: https://github.blog/open-source/git/get-up-to-speed-with-partial-clone-and-shallow-clone/ --- ## `git-commit` `git-commit` commits all changes in a working tree to its checked out branch. If there are no differences from the current `HEAD`, this step is skipped and no new commit is created. This step is often used after previous steps have put the working tree into the desired state and is commonly followed by a [`git-push` step](git-push.md). ## Configuration | Name | Type | Required | Description | |------|------|----------|-------------| | `path` | `string` | Y | Path to a Git working tree containing changes to be committed. This path is relative to the temporary workspace that Kargo provisions for use by the promotion process. | | `message` | `string` | Y | The commit message. | | `author` | `object` | N | **Deprecated (v1.10.0, removal in v1.12.0).** Optional authorship information for the commit. Configure authorship in the [`git-clone`](./git-clone.md) step instead. | | `author.name` | `string` | Y | **Deprecated.** The committer's name. | | `author.email` | `string` | Y | **Deprecated.** The committer's email address. | | `author.signingKey` | `string` | N | **Deprecated.** The GPG signing key for the author. Configure signing keys in the [`git-clone`](./git-clone.md) step or via `ClusterConfig` instead. | :::warning The `author` field is deprecated as of v1.10.0 and will be removed in v1.12.0. If authorship differing from any system-level default is required, configure it in the [`git-clone`](./git-clone.md) step instead. ::: ## Output | Name | Type | Description | |------|------|-------------| | `commit` | `string` | The ID (SHA) of the commit created by this step. If the step short-circuited and did not create a new commit because there were no differences from the current head of the branch, this value will be the ID of the existing commit at the head of the branch instead. Typically, a subsequent [`argocd-update`](argocd-update.md) step will reference this output to learn the ID of the commit that an applicable Argo CD `ApplicationSource` should be observably synced to under healthy conditions. | ## Examples ### Common Usage In this example, the working tree is prepared by previous steps and then committed with a message from the [`kustomize-set-image` step](kustomize-set-image.md) that updated the `kustomization.yaml` file, summarizing the changes made. ```yaml vars: - name: gitRepo value: https://github.com/example/repo.git steps: - uses: git-clone config: repoURL: ${{ vars.gitRepo }} checkout: - commit: ${{ commitFrom(vars.gitRepo).ID }} path: ./src - branch: stage/${{ ctx.stage }} create: true path: ./out - uses: git-clear config: path: ./out - uses: kustomize-set-image as: update-image config: images: - image: my/image - uses: kustomize-build config: path: ./src/stages/${{ ctx.stage }} outPath: ./out - uses: git-commit config: path: ./out message: ${{ outputs['update-image'].commitMessage }} # Push, etc... ``` ### Composed Commit Message The `message` field can be used to compose a commit message and enrich it with contextual information. In this example, the commit message is prefixed with the current stage using the [pre-defined `ctx.stage` variable](../40-expressions.md#pre-defined-variables). :::info The `message` field supports multi-line strings. Use `|` to indicate a block scalar and preserve newlines. This allows for multi-line commits, which can be useful for providing detailed commit messages when several changes are being committed together. ::: ```yaml steps: # Update Kustomize manifests, etc... - uses: git-commit config: path: ./out message: | ${{ ctx.stage }}: ${{ outputs['update-image'].commitMessage }} ``` --- ## `git-merge-pr` `git-merge-pr` merges an open pull request. This step commonly follows a [`git-open-pr`](git-open-pr.md) step. :::important This step only executes synchronous merges. It can neither initiate an asynchronous merge by placing a PR on a merge queue (or similar), nor can it recognize when an open PR is already _in_ a merge queue (having been placed there by someone or something else), and thus cannot wait for an asynchronous merge in-progress to complete. ::: :::caution **GitHub** repositories can be configured with branch protection rules that require PRs to be merged via a merge queue. When such a rule is in place, the results of the `git-merge-pr` step attempting a synchronous merge will depend upon permissions. With sufficient permissions to bypass branch protection rules, the merge queue will be bypassed. Without such permissions, the step's attempt to merge will fail. ::: ## Credentials Git steps are utilizing the [repository credentials](../../50-security/30-managing-secrets.md#repository-credentials) system to access the git repos. ## Configuration | Name | Type | Required | Description | | ----------------------- | --------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `repoURL` | `string` | Y | The URL of a remote Git repository. **Deprecated:** Support for SSH URLs (`ssh://` and SCP-style `git@host:path`) is deprecated as of v1.10.0 and will be removed in v1.13.0. Use HTTPS URLs instead. | | `provider` | `string` | N | The name of the Git provider to use. Currently `azure`, `bitbucket`, `gitea`, `github`, and `gitlab` are supported. Kargo will try to infer the provider if it is not explicitly specified. | | `insecureSkipTLSVerify` | `boolean` | N | Indicates whether to bypass TLS certificate verification when interfacing with the Git provider. Setting this to `true` is highly discouraged in production. | | `prNumber` | `integer` | Y | The pull request number to merge. | | `mergeMethod` | `string` | N | The merge method to use when merging the pull request. The supported methods are provider-specific; refer to the [Merge Method](#merge-method) section. | | `wait` | `boolean` | N | If `true`, the step will return a running status instead of failing when the PR is not yet mergeable. The merge will be retried on the next reconciliation until it succeeds or times out. Default is `false`. | :::warning The `wait` option is unreliable for repositories hosted by Bitbucket due to API limitations. ::: ### Merge Method The table below documents the supported merge methods/strategies for each of the currently supported Git hosting providers. | Provider | Supported Methods | Default | | -------- | ----------------- | ------- | | Azure | `noFastForward``rebase``rebaseMerge``squash` | First allowed strategy per the target branch's merge type policy; merge commit if no policy is configured | | BitBucket | Client does not yet support specifying a merge strategy | The repository's configured default merge strategy | | Gitea | `fast-forward-only``manually-merged``merge``rebase``rebase-merge``squash` | `merge` | | GitHub | `merge``rebase``squash` | `merge` | | GitLab | `merge``squash` | Defers to the merge request and project-level squash settings | ## Output | Name | Type | Description | | -------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `commit` | `string` | The ID (SHA) of the merge commit created after successfully merging the pull request. Typically, a subsequent [`argocd-update`](argocd-update.md) step will reference this output to learn the ID of the commit that an applicable Argo CD `ApplicationSource` should be observably synced to under healthy conditions. | ## Examples ### Basic Usage In this example, a pull request is merged immediately if it's ready. If the pull request is not ready to merge (e.g., due to pending checks or conflicts), the step will fail. ```yaml steps: - uses: git-merge-pr config: repoURL: https://github.com/example/repo.git prNumber: 42 ``` ### Merge with Wait This example demonstrates merging a pull request with waiting enabled. If the pull request is not yet mergeable for any reason, the step will return a running status and Kargo will retry it on the next reconciliation. ```yaml steps: - uses: git-merge-pr config: repoURL: https://github.com/example/repo.git prNumber: 42 wait: true ``` ### Specifying a Merge Method This example demonstrates merging a pull request with a specific merge method. Refer to the [Merge Method](#merge-method) section for supported values per provider. ```yaml steps: - uses: git-merge-pr config: repoURL: https://github.com/example/repo.git prNumber: 42 mergeMethod: squash ``` --- ## `git-open-pr` `git-open-pr` opens a pull request in a specified remote repository using specified source and target branches. This step is often used after a [`git-push` step](git-push.md) and is commonly followed by a [`git-wait-for-pr` step](git-wait-for-pr.md). At present, this feature only supports GitHub, Gitea, Azure DevOps, and GitLab pull/merge requests. ## Credentials Git steps are utilizing the [repository credentials](../../50-security/30-managing-secrets.md#repository-credentials) system to access the git repos. ## Configuration | Name | Type | Required | Description | |------|------|----------|-------------| | `repoURL` | `string` | Y | The URL of a remote Git repository. **Deprecated:** Support for SSH URLs (`ssh://` and SCP-style `git@host:path`) is deprecated as of v1.10.0 and will be removed in v1.13.0. Use HTTPS URLs instead. | | `provider` | `string` | N | The name of the Git provider to use. Currently `azure`, `bitbucket`, `gitea`, `github`, and `gitlab` are supported. Kargo will try to infer the provider if it is not explicitly specified. | | `insecureSkipTLSVerify` | `boolean` | N | Indicates whether to bypass TLS certificate verification when interfacing with the Git provider. Setting this to `true` is highly discouraged in production. | | `sourceBranch` | `string` | Y | Specifies the source branch for the pull request. | | `targetBranch` | `string` | N | The branch to which the changes should be merged. | | `createTargetBranch` | `boolean` | N | **Deprecated**. Is a no-op if set. Will be removed in a future release.| | `title` | `string` | N | The title for the pull request. Kargo generates a title based on the commit messages if it is not explicitly specified. | | `description` | `string` | N | The description for the pull request. | | `labels` | `[]string` | N | Labels to add to the pull request. Not all Git providers support labels; see the note below. | :::note Label support varies by Git provider. GitHub and GitLab support labels on pull requests. Gitea does not currently support labels (tracked in [#6021](https://github.com/akuity/kargo/issues/6021)). Bitbucket does not support pull request labels at all. ::: ## Output | Name | Type | Description | |------|------|-------------| | `pr` | `object` | An object containing details about the pull request. | | `pr.id` | `number` | The numeric identifier of the pull request opened by this step. Typically, a subsequent [`git-wait-for-pr` step](git-wait-for-pr.md) will reference this output to learn what pull request to monitor. | | `pr.url` | `string` | The URL of the pull request. | ## Examples ### Common Usage The following example demonstrates a common use case for `git-open-pr`. It follows a [`git-push` step](git-push.md) that has pushed changes to a remote repository to a branch with a generated name. The `git-open-pr` step then opens a pull request to merge the changes from the source branch which was created by the `git-push` step into the `stage/${{ ctx.stage }}` branch. This is a common pattern when implementing GitOps-based promotion workflows, where changes are first pushed to an intermediate branch and then merged into a stage-specific branch through a pull request. :::note The `git-open-pr` step will fail if the `targetBranch` doesn't exist. ::: ```yaml steps: # Clone, prepare the contents of ./out, commit, etc... - uses: git-push as: push config: path: ./out generateTargetBranch: true - uses: git-open-pr as: open-pr config: repoURL: https://github.com/example/repo.git sourceBranch: ${{ outputs.push.branch }} targetBranch: stage/${{ ctx.stage }} # Wait for the PR to be merged or closed... ``` ### Custom Title and Labels The following example demonstrates how to specify a custom title and labels for the pull request opened by `git-open-pr`. After pushing changes to a generated branch, the `git-open-pr` step creates a pull request with a title that references the current stage (`Deploy to ${{ ctx.stage }}`) and adds two labels: "infra" and "needs-review". This is useful when you want to provide more context about the changes being proposed or need to integrate with existing PR review workflows that rely on specific labels for automation or filtering. ```yaml steps: # Clone, prepare the contents of ./out, commit, etc... - uses: git-push as: push config: path: ./out generateTargetBranch: true - uses: git-open-pr as: open-pr config: repoURL: https://github.com/example/repo.git sourceBranch: ${{ outputs.push.branch }} targetBranch: stage/${{ ctx.stage }} title: Deploy to ${{ ctx.stage }} labels: ["infra", "needs-review"] - if: ${{ status('open-pr') != 'Skipped' }} uses: git-wait-for-pr as: wait-for-pr config: repoURL: https://github.com/example/repo.git prNumber: ${{ outputs['open-pr'].pr.id }} ``` ### Skipped The following example conditionally runs the [`git-wait-for-pr` step](git-wait-for-pr.md) based on whether or not the `git-open-pr` step was skipped. If there are no changes between the `sourceBranch` and `targetBranch`, the `git-open-pr` step will be skipped. The [`status`](../40-expressions.md#statusstepalias) expression function can be used by subsequent steps to determine if a preceding step was skipped. ```yaml - uses: git-push as: push config: path: ./out generateTargetBranch: true - uses: git-open-pr as: open-pr config: repoURL: https://github.com/example/repo.git sourceBranch: ${{ outputs.push.branch }} targetBranch: stage/${{ ctx.stage }} - if: ${{ status('open-pr') != 'Skipped' }} uses: git-wait-for-pr as: wait-for-pr config: repoURL: https://github.com/example/repo.git prNumber: ${{ outputs['open-pr'].pr.id }} ``` --- ## `git-push` `git-push` pushes committed changes or new tags from a specified working tree to the remote repository. This step typically follows a [`git-commit` step](git-commit.md) and/or [`git-tag` step](git-tag.md) and is often followed by a [`git-open-pr` step](git-open-pr.md). This step also implements its own, internal retry logic. If a push fails, with the cause determined to be the presence of new commits in the remote branch that are not present in the local branch, the step will integrate the remote changes and retry the push. Any merge conflict requiring manual resolution will immediately halt further attempts. How remote changes are integrated is controlled by the system-level __push integration policy__, which is configured by a Kargo operator. The available policies are: - `AlwaysRebase` — Unconditionally rebase. Simplest, but may re-sign or strip commit signatures. - `RebaseOrMerge` — Rebase when a signature-trust analysis determines it is safe; merge otherwise. This preserves linear history when possible without undermining trust. - `RebaseOrFail` — Rebase when the signature-trust analysis determines it is safe; fail the step otherwise. - `AlwaysMerge` — Unconditionally create a merge commit. Most conservative. :::caution The current default policy is `AlwaysRebase`. Starting with v1.12.0, the default will change to `RebaseOrMerge`. ::: When the policy evaluates rebase safety (`RebaseOrMerge` and `RebaseOrFail`), the decision to use rebase or not is based on the GPG signature status of the local commits that would be replayed: - If all local commits are __signed by a trusted key__ and signing was enabled at the time the repository was cloned, a __rebase__ is performed. The replacement commits are re-signed by Kargo. - If all local commits are __unsigned__ and signing was _not_ enabled at the time the repository was cloned, a __rebase__ is performed. The replacement commits remain unsigned. - In all other cases, the policy's fallback behavior applies (merge or fail). A "trusted key" is one that was imported with ultimate trust when the repository was cloned, either through explicit configuration of the [`git-clone`](git-clone.md) step or via fallback on a system-level signing key configured by a Kargo admin. :::info For more information on configuring the push integration policy, see the [operator guide](../../../40-operator-guide/20-advanced-installation/30-common-configurations.md#push-integration-policy). ::: :::info This step's internal retry logic is helpful in scenarios when concurrent Promotions to multiple Stages may all write to the same branch of the same repository. Because conflicts requiring manual resolution will halt further attempts, it is recommended to design your Promotion processes such that Promotions to multiple Stages that write to the same branch do not write to the same files. ::: :::note For a tag push, there is no pull/rebase retry loop. ::: :::info If you authenticate to GitHub using a GitHub App, you may want to consider using [`github-push`](github-push.md) instead. ::: ## Credentials This step utilizes the [repository credentials](../../50-security/30-managing-secrets.md#repository-credentials) system to access Git repositories. ## Configuration | Name | Type | Required | Description | | ---- | ---- | -------- | ----------- | | `path` | `string` | Y | Path to a Git working tree containing committed changes. | | `targetBranch` | `string` | N | The branch to push to in the remote repository. Mutually exclusive with `generateTargetBranch=true` and `tag`. If none of these are provided, the target branch will be the same as the branch currently checked out in the working tree. | | `maxAttempts` | `int32` | N | The maximum number of attempts to make when pushing to the remote repository. Default is 10. | | `generateTargetBranch` | `boolean` | N | Whether to push to a remote branch named like `kargo/promotion/`. If such a branch does not already exist, it will be created. A value of `true` is mutually exclusive with `targetBranch` and `tag`. If none of these are provided, the target branch will be the currently checked out branch. This option is useful when a subsequent promotion step will open a pull request against a Stage-specific branch. In such a case, the generated target branch pushed to by the `git-push` step can later be utilized as the source branch of the pull request. | | `tag` | `string` | N | An tag to push to the remote repository. Mutually exclusive with `generateTargetBranch` and `targetBranch`. | | `force` | `boolean` | N | Whether to force push to the target branch, overwriting any existing history. This is useful for scenarios where you want to completely replace the branch content (e.g., pushing rendered manifests that don't depend on previous state). **Use with caution** as this will overwrite any commits that exist on the remote branch but not in your local branch. Default is `false`. A value of `true` is mutually exclusive with `tag`. | | `provider` | `string` | N | The name of the Git provider to use. Currently 'azure', 'bitbucket', 'gitea', 'github', and 'gitlab' are supported. Kargo will try to infer the provider if it is not explicitly specified. This setting does not affect the push operation but helps generate the correct [`commitURL` output](#output) when working with repositories where the provider cannot be automatically determined, such as self-hosted instances. | ## Output | Name | Type | Description | | ---- | ---- | ----------- | | `branch` | `string` | The name of the remote branch pushed to by this step. This is especially useful when the `generateTargetBranch=true` option has been used, in which case a subsequent [`git-open-pr`](git-open-pr.md) will typically reference this output to learn what branch to use as the head branch of a new pull request. | | `commit` | `string` | The ID (SHA) of the commit pushed by this step. | | `commitURL` | `string` | The URL of the commit that was pushed to the remote repository. | ## Examples ### Common Usage In this example, changes prepared in a working directory are committed and pushed to the same branch that was checked out. The `git-push` step takes the path to the working directory containing the committed changes and pushes them to the remote repository. This is the most basic and common pattern for updating a branch with new changes during a promotion process. ```yaml steps: # Clone, prepare the contents of ./out, etc... - uses: git-commit config: path: ./out message: rendered updated manifests - uses: git-push config: path: ./out ``` ### For Use With a Pull Request In this example, changes are pushed to a generated branch name that follows the pattern `kargo/promotion/`. By setting `generateTargetBranch: true`, the step creates a unique branch name that can be referenced by subsequent steps. This is commonly used as part of a pull request workflow, where changes are first pushed to an intermediate branch before being proposed as a pull request. The step's output includes the generated branch name, which can then be used by a subsequent [`git-open-pr` step](git-open-pr.md). ```yaml steps: # Clone, prepare the contents of ./out, etc... - uses: git-commit config: path: ./out message: rendered updated manifests - uses: git-push as: push config: path: ./out generateTargetBranch: true # Open a PR and wait for it to be merged or closed... ``` ### Pushing Tags In this example, a new tag is pushed to the remote repository. ```yaml # Create a new tag - uses: git-tag config: path: ./out tag: v1.0.0 - uses: git-push config: path: ./out tag: v1.0.0 ``` :::caution If the specified tag already exists in the remote repository, the `git-push` step will fail. ::: --- ## `git-tag` The `git-tag` step creates a new, annotated tag in a local Git repository referencing the current `HEAD` of a checked-out branch. ## Configuration | Name | Type | Required | Description | |--------|----------|----------|-----------------------------------------------------------------------------| | `path` | `string` | Y | Path to a working directory of a local repository. This path is relative to the temporary workspace that Kargo provisions for use by the promotion process. | | `tag` | `string` | Y | The tag to create. | | `message` | `string` | Y | The message with which to annotate the tag. | ## Output | Name | Type | Description | |-------|----------|-----------------------------------------------------------------------------| | `commit` | `string` | The ID (SHA) of the commit pushed by this step. | ## Examples ### Basic Usage In this example, the `git-tag` step creates a tag named `v1.0.0` in a local Git repository. ```yaml steps: - uses: git-tag config: path: ./out tag: v1.0.0 ``` ### Tagging After a Commit This example demonstrates how to use the git-tag step after a git-commit step to tag the latest commit with a version number. ```yaml steps: - uses: git-commit config: path: ./out message: "Committing changes for release v1.0.0" - uses: git-tag config: path: ./out tag: v1.0.0 ``` ### Pushing After Tagging In this example, the `git-tag` step creates a tag, and the `git-push` step pushes the tag to the remote repository. ```yaml steps: - uses: git-tag config: path: ./out tag: v1.0.0 - uses: git-push config: path: ./out tag: v1.0.0 ``` --- ## `git-wait-for-pr` `git-wait-for-pr` waits for a specified open pull request to be merged or closed. This step commonly follows a [`git-open-pr` step](git-open-pr.md) and is commonly followed by an `argocd-update` step. ## Credentials Git steps are utilizing the [repository credentials](../../50-security/30-managing-secrets.md#repository-credentials) system to access the git repos. ## Configuration | Name | Type | Required | Description | |------|------|----------|-------------| | `repoURL` | `string` | Y | The URL of a remote Git repository. **Deprecated:** Support for SSH URLs (`ssh://` and SCP-style `git@host:path`) is deprecated as of v1.10.0 and will be removed in v1.13.0. Use HTTPS URLs instead. | | `provider` | `string` | N | The name of the Git provider to use. Currently `azure`, `bitbucket`, `gitea`, `github`, and `gitlab` are supported. Kargo will try to infer the provider if it is not explicitly specified. | | `insecureSkipTLSVerify` | `boolean` | N | Indicates whether to bypass TLS certificate verification when interfacing with the Git provider. Setting this to `true` is highly discouraged in production. | | `prNumber` | `integer` | Y | The pull request number to wait for. | ## Output | Name | Type | Description | |------|------|-------------| | `commit` | `string` | The ID (SHA) of the new commit at the head of the target branch after merge. Typically, a subsequent [`argocd-update` step](argocd-update.md) will reference this output to learn the ID of the commit that an applicable Argo CD `ApplicationSource` should be observably synced to under healthy conditions. | | `pr` | `object` | An object containing details about the pull request being monitored. | | `pr.id` | `number` | The numeric identifier of the pull request. | | `pr.url` | `string` | The URL of the pull request. | | `pr.open` | `boolean` | Whether the pull request is still open. | | `pr.merged` | `boolean` | Whether the pull request has been merged. | ## Examples ### Common Usage In this example, a complete promotion flow is demonstrated where changes are pushed to a generated branch, a pull request is opened, and then the process waits for the pull request to be merged or closed. The `git-wait-for-pr` step references both the repository URL and the PR number (obtained from the [`open-pr` step's output](git-open-pr.md#output)) to track the PR's status. This pattern is common when you want to ensure changes have been properly reviewed and merged before proceeding with subsequent steps in your promotion process, such as [updating Argo CD applications](argocd-update.md). ```yaml steps: # Clone, prepare the contents of ./out, commit, etc... - uses: git-push as: push config: path: ./out generateTargetBranch: true - uses: git-open-pr as: open-pr config: repoURL: https://github.com/example/repo.git createTargetBranch: true sourceBranch: ${{ outputs.push.branch }} targetBranch: stage/${{ ctx.stage }} - uses: git-wait-for-pr as: wait-for-pr config: repoURL: https://github.com/example/repo.git prNumber: ${{ outputs['open-pr'].pr.id }} ``` --- ## `github-push` `github-push` pushes committed changes from a specified working tree to a GitHub repository using the GitHub REST API. It is a drop-in replacement for the [`git-push`](git-push.md) step. Many organizations prefer that Kargo authenticates to GitHub using a [GitHub App](../../50-security/30-managing-secrets.md#github-app-authentication) because it avoids coupling authentication to any one GitHub user account. GitHub Apps, however, cannot be associated with a GPG signing key, so when branch protection rules require verified commits, the key used for signing must still be associated with a user account -- which undermines the benefits of having used an App for authentication. `github-push` solves this by "replaying" local commits using the GitHub REST API. When a commit is eligible (see [Trust, attribution, and verification](#trust-attribution-and-verification) below), author/committer information is withheld from the API call. Under those conditions, GitHub attributes the commit to the authenticated user (the GitHub App) and signs it with GitHub's own key, producing a verified commit -- without requiring a GPG key to be associated with any user account. This step also implements its own, internal retry logic. If a push fails, with the cause determined to be the presence of new commits in the remote branch that are not present in the local branch, the step will integrate the remote changes and retry the push. Any merge conflict requiring manual resolution will immediately halt further attempts. How remote changes are integrated is controlled by the system-level __push integration policy__, which is configured by a Kargo operator. The available policies are: - `AlwaysRebase` — Unconditionally rebase. Simplest, but may re-sign or strip commit signatures. - `RebaseOrMerge` — Rebase when a signature-trust analysis determines it is safe; merge otherwise. This preserves linear history when possible without undermining trust. - `RebaseOrFail` — Rebase when the signature-trust analysis determines it is safe; fail the step otherwise. - `AlwaysMerge` — Unconditionally create a merge commit. Most conservative. :::caution For parity with the `git-push` step, for which this step is a drop-in replacement, the current default policy is `AlwaysRebase`. Starting with v1.12.0, the default will change to `RebaseOrMerge` for both steps. ::: When the policy evaluates rebase safety (`RebaseOrMerge` and `RebaseOrFail`), the decision to rebase or not is based on the GPG signature status of the local commits that would be replayed as part of that rebase (not to be confused with replaying of commits on GitHub via its REST API): - If all local commits are __signed by a trusted key__ and signing was enabled at the time the repository was cloned, a __rebase__ is performed. The replacement commits are re-signed by Kargo. - If all local commits are __unsigned__ and signing was _not_ enabled at the time the repository was cloned, a __rebase__ is performed. The replacement commits remain unsigned. - In all other cases, the policy's fallback behavior applies (merge or fail). A "trusted key" is one that was imported with ultimate trust when the repository was cloned, either through explicit configuration of the [`git-clone`](git-clone.md) step or via fallback on a system-level signing key configured by a Kargo admin. :::info For more information on configuring the push integration policy, see the [operator guide](../../../40-operator-guide/20-advanced-installation/30-common-configurations.md#push-integration-policy). ::: :::info This step's internal retry logic is helpful in scenarios when concurrent Promotions to multiple Stages may all write to the same branch of the same repository. Because conflicts requiring manual resolution will halt further attempts, it is recommended to design your Promotion processes such that Promotions to multiple Stages that write to the same branch do not write to the same files. ::: ## Credentials This step utilizes the [repository credentials](../../50-security/30-managing-secrets.md#repository-credentials) system to access GitHub repositories and APIs. :::caution This step will succeed when authenticated to GitHub via a personal access token, however, can only (if other conditions are met) result in verified commits when authenticated via a [GitHub App](../../50-security/30-managing-secrets.md#github-app-authentication). ::: ## How it works Under the hood, `github-push`: 1. Integrates any remote changes into the local branch (identical to [`git-push`](git-push.md)). 2. Force-pushes the local branch to a temporary, non-visible staging ref on GitHub (`refs/kargo/staging/...`). 3. Uses the GitHub API to compare the staging ref against the target branch, identifying the commits that need to be "replayed." 4. Replays each commit via the GitHub API, applying the appropriate trust and attribution rules. As a safety guardrail, the number of commits replayed in a single push is capped (default 10). See the [operator guide](../../../40-operator-guide/20-advanced-installation/30-common-configurations.md#github-push-settings) to configure this limit. 5. Updates the target branch ref to point at the final replayed commit. 6. Cleans up the staging ref. 7. Syncs the local working tree to match the remote (since the replayed commits have new SHAs). ## Trust, attribution, and verification When replaying commits through the GitHub API, `github-push` decides how each commit should be attributed. This decision determines whether the commit will receive GitHub's verified badge. By default, the decision for each replayed commit is based on whether the original commit was signed by a trusted key: - __Trusted commits__ -- those signed by a key with ultimate trust in the GPG keyring -- are created __without__ explicit author/committer information. GitHub attributes the commit to the authenticated identity (App or user) and signs it with its own key, resulting in a __verified__ commit. If the original commit's author differs from its signer, a `Co-authored-by` trailer is added to the commit message to preserve that attribution. - __Untrusted commits__ -- unsigned, or signed by an untrusted key -- are created __with__ their original author and committer information preserved exactly. GitHub will __not__ mark these as verified. Provenance remains completely intact. :::caution For any commits to be considered trusted, Kargo __must__ be configured with a GPG signing key so that it signs commits locally. Without a signing key, all commits are untrusted and will be replayed with their original attribution -- meaning no verified badge. A signing key can be configured at the system level in two ways: at runtime via [`ClusterConfig`](../../../40-operator-guide/20-advanced-installation/30-common-configurations.md#git-client-configuration), or at install time via the Helm chart. The runtime path takes precedence. This does __not__ undermine the purpose of the feature. The GPG key configured for Kargo never needs to be uploaded to GitHub or associated with any user account. It is used solely for local signing and trust evaluation. It can be a purpose-built, "throwaway" key that exists only within Kargo's configuration. ::: ### How integration policies affect verification The choice of integration policy can affect which commits are trusted when replayed: - With `AlwaysRebase`, all commits are rebased and re-signed by the configured signing key, making them trusted. After replay, they all receive the verified badge. However, this means commits that Kargo did not originally author are re-signed -- the operator has accepted this trade-off by choosing this policy. - With `RebaseOrMerge` or `AlwaysMerge`, commits from the remote branch that were not originally signed by a trusted key are preserved as-is through the merge. When replayed, these commits retain their original attribution and will __not__ be verified. The merge commit itself and any other commits Kargo authored will be verified. - With `RebaseOrFail`, the step fails if any commit cannot be safely rebased, so all replayed commits will be trusted and verified (if the step succeeds). ### Overriding trust: verifying all commits For operators who want __every__ commit to receive the verified badge regardless of trust, a system-level option is available that causes author/committer information to be omitted for all commits -- not just trusted ones. GitHub then signs every commit with its own key, and every commit in the push receives the verified badge. To preserve provenance, a `Co-authored-by` trailer is added to the commit message whenever the original author's identity is known. :::warning This option can manufacture trust where none exists. It tells GitHub to vouch for commits that Kargo could not independently verify. Enabling this places the superficial value of a verified badge above genuine cryptographic trust. ::: :::info For more information on configuring this option, see the [operator guide](../../../40-operator-guide/20-advanced-installation/30-common-configurations.md#github-push-settings). ::: ## Configuration | Name | Type | Required | Description | | ---- | ---- | -------- | ----------- | | `path` | `string` | Y | Path to a Git working tree containing committed changes. | | `targetBranch` | `string` | N | The branch to push to in the remote repository. Mutually exclusive with `generateTargetBranch=true`. If neither is provided, the target branch will be the same as the branch currently checked out in the working tree. | | `maxAttempts` | `int32` | N | The maximum number of attempts to make when pushing to the remote repository. Default is 10. | | `generateTargetBranch` | `boolean` | N | Whether to push to a remote branch named like `kargo/promotion/`. If such a branch does not already exist, it will be created. A value of `true` is mutually exclusive with `targetBranch`. This option is useful when a subsequent promotion step will open a pull request against a Stage-specific branch. | | `force` | `boolean` | N | Whether to force push to the target branch, overwriting any existing history. This is useful for scenarios where you want to completely replace the branch content (e.g., pushing rendered manifests that don't depend on previous state). **Use with caution** as this will overwrite any commits that exist on the remote branch but not in your local branch. Default is `false`. | | `insecureSkipTLSVerify` | `boolean` | N | Whether to skip TLS verification when communicating with the GitHub API. Default is `false`. Intended for GitHub Enterprise instances with self-signed certificates. | ## Output | Name | Type | Description | | ---- | ---- | ----------- | | `branch` | `string` | The name of the remote branch pushed to by this step. This is especially useful when the `generateTargetBranch=true` option has been used, in which case a subsequent [`git-open-pr`](git-open-pr.md) will typically reference this output to learn what branch to use as the head branch of a new pull request. | | `commit` | `string` | The ID (SHA) of the commit pushed by this step. | | `commitURL` | `string` | The URL of the commit that was pushed to the remote repository. | ## Examples ### Common usage In this example, changes prepared in a working directory are committed and pushed to the same branch that was checked out. This is the simplest usage, directly replacing `git-push`: ```yaml steps: # Clone, prepare the contents of ./out, etc... - uses: git-commit config: path: ./out message: rendered updated manifests - uses: github-push config: path: ./out ``` ### For use with a pull request In this example, changes are pushed to a generated branch name that follows the pattern `kargo/promotion/`. By setting `generateTargetBranch: true`, the step creates a unique branch name that can be referenced by subsequent steps. This is commonly used as part of a pull request workflow, where changes are first pushed to an intermediate branch before being proposed as a pull request. The step's output includes the generated branch name, which can then be used by a subsequent [`git-open-pr` step](git-open-pr.md). ```yaml steps: # Clone, prepare the contents of ./out, etc... - uses: git-commit config: path: ./out message: rendered updated manifests - uses: github-push as: push config: path: ./out generateTargetBranch: true # Open a PR and wait for it to be merged or closed... ``` --- ## `hcl-update` :::info This promotion step is only available in Kargo on the [Akuity Platform](https://akuity.io/akuity-platform), versions v1.9 and above. Additionally, it requires enabling of the Promotion Controller to allow for Pod-based promotions. ::: `hcl-update` modifies attribute values in HCL (HashiCorp Configuration Language) files. This step is typically used to update OpenTofu configuration files before running [`tf-plan`](tf-plan.md) and [`tf-apply`](tf-apply.md) steps, allowing you to dynamically set values such as image tags and other configuration parameters as part of the promotion process. ## Configuration | Name | Type | Required | Description | |------|------|----------|-------------| | `path` | `string` | Y | Path to an HCL file. This path is relative to the temporary workspace that Kargo provisions for use by the promotion process. | | `updates` | `[]object` | Y | A list of updates to apply to the HCL file. At least one update must be specified. | | `updates[].key` | `string` | Y | The key whose value needs to be updated. Supports dot notation for nested values (e.g., `resource.aws_instance.example.tags.version`). | | `updates[].value` | `string`, `number`, or `boolean` | Y | The new value to set. Strings are quoted, booleans are lowercase (`true`/`false`), and numbers are written as-is. | ## Examples ### Common Usage The most common usage of this step is to update an OpenTofu variables file with values from the Freight being promoted. In this example, a container image URI is updated in a Stage-specific `env.auto.tfvars` file before planning and applying infrastructure changes. ```yaml vars: - name: repoURL value: https://github.com/example/infra.git - name: image value: 123456789.dkr.ecr.us-west-2.amazonaws.com/my-app steps: - uses: git-clone config: repoURL: ${{ vars.repoURL }} checkout: - branch: main path: ./src - uses: hcl-update config: path: ./src/opentofu/${{ ctx.stage }}/env.auto.tfvars updates: - key: image_uri value: ${{ vars.image }}:${{ imageFrom(vars.image).Tag }} - uses: tf-apply config: dir: ./src/opentofu/${{ ctx.stage }} # Commit and push state changes... ``` ### Updating Multiple Values This example demonstrates updating multiple attributes in a single step. This is useful when several configuration values need to change together, such as when deploying a new version with associated settings. ```yaml steps: # Clone, prepare configuration, etc... - uses: hcl-update config: path: ./src/opentofu/${{ ctx.stage }}/env.auto.tfvars updates: - key: image_uri value: ${{ vars.image }}:${{ imageFrom(vars.image).Tag }} - key: replica_count value: 3 - key: enable_monitoring value: true # Plan, apply, etc... ``` ### Updating Nested Resource Attributes This example shows how to use dot notation to update deeply nested attributes within OpenTofu resource definitions. The key path follows the HCL structure of the configuration file. ```yaml steps: # Clone, prepare configuration, etc... - uses: hcl-update config: path: ./src/opentofu/main.tf updates: - key: resource.aws_lambda_function.app.image_uri value: ${{ vars.image }}:${{ imageFrom(vars.image).Tag }} # Plan, apply, etc... ``` --- ## `helm-template` `helm-template` renders a specified Helm chart to a specified directory or to many files in a specified directory. This step is useful for the common scenario of rendering Stage-specific manifests to a Stage-specific branch. This step is commonly preceded by a [`git-clear` step](git-clear.md) and followed by [`git-commit`](git-commit.md) and [`git-push`](git-push.md) steps. ## Configuration | Name | Type | Required | Description | |------|------|----------|-------------| | `path` | `string` | Y | Path to a Helm chart (i.e. to a directory containing a `Chart.yaml` file). This path is relative to the temporary workspace that Kargo provisions for use by the promotion process. | | `outPath` | `string` | Y | Path to the file or directory where rendered manifests are to be written. If the path ends with `.yaml` or `.yml` it is presumed to indicate a file and is otherwise presumed to indicate a directory. | | `outLayout` | `string` | N | OutLayout to use for the rendered manifest. This can be either `helm` or `flat`. The `helm` layout will create a directory with the chart name and place the rendered manifests in that directory. The `flat` layout will place all rendered manifests in the outPath directory without any subdirectories. This is `helm` by default. | | `releaseName` | `string` | Y | Release name to use when rendering the manifests. | | `useReleaseName` | `boolean` | N | Whether to use the release name in the output path (instead of the chart name). This is `false` by default, and only has an effect when `outPath` is set to a directory. | | `namespace` | `string` | N | Optional namespace to use when rendering the manifests. This is commonly omitted. GitOps agents such as Argo CD will generally ensure the installation of manifests into the namespace specified by their own configuration. | | `valuesFiles` | `[]string` | N | Helm values files (apart from the chart's default `values.yaml`) to be used when rendering the manifests. | | `ignoreMissingValueFiles` | `boolean` | N | Whether to skip value files that do not exist instead of returning an error. Useful for optional stage-specific overlays. This is `false` by default. | | `buildDependencies` | `bool` | N | Whether to build dependencies before rendering the manifests. If no Chart.lock file is present, the dependencies will be built from the Chart.yaml file (and may be updated). This is `false` by default. | | `includeCRDs` | `boolean` | N | Whether to include CRDs in the rendered manifests. This is `false` by default. | | `disableHooks` | `boolean` | N | Whether to disable hooks in the rendered manifests. This is `false` by default. | | `skipTests` | `boolean` | N | Whether to skip tests when rendering the manifests. This is `false` by default. | | `kubeVersion` | `string` | N | Optionally specifies a Kubernetes version to be assumed when rendering manifests. This is useful for charts that may contain logic specific to different Kubernetes versions. | | `apiVersions` | `[]string` | N | Allows a manual set of supported API versions to be specified. | | `setValues` | `[]object` | N | Allows for amending chart configuration inline as one would with the `helm template` command's `--set` flag. | | `setValues[].key` | `string` | N | The key whose value should be set. For nested values, use dots to delimit key parts. e.g. `image.tag`. | | `setValues[].value` | `string` | N | The new value for the key. | | `setValues[].literal` | `boolean` | N | Whether to force the value to be treated as a literal string. When true, uses `--set-literal` instead of `--set`. This is `false` by default. | ## Examples ### Rendering to a File In this example, a Helm chart is rendered to a single output file. After cloning the repository and clearing the output directory, the `helm-template` step reads the chart from `./src/charts/my-chart` and uses stage-specific values from `${{ ctx.stage }}-values.yaml` to render the manifests. The rendered output is written to a single file at `./out/manifests.yaml` rather than being split across multiple files. This approach is useful when you want to maintain all manifests in a single file for easier tracking or when working with tools that expect a single input file. ```yaml vars: - name: gitRepo value: https://github.com/example/repo.git steps: - uses: git-clone config: repoURL: ${{ vars.gitRepo }} checkout: - commit: ${{ commitFrom(vars.gitRepo).ID }} path: ./src - branch: stage/${{ ctx.stage }} create: true path: ./out - uses: git-clear config: path: ./out - uses: helm-template config: path: ./src/charts/my-chart valuesFiles: - ./src/charts/my-chart/${{ ctx.stage }}-values.yaml outPath: ./out/manifests.yaml # Commit, push, etc... ``` ### Rendering to a Directory In this example, a Helm chart is rendered with its output split across multiple files in a directory. Similar to the [previous example](#rendering-to-a-file), it clones the repository and clears the output directory, but instead of specifying a single output file, it directs the `helm-template` step to write to the `./out` directory. This means each Kubernetes resource will be written to its own file, maintaining the traditional Helm output structure (i.e. the structure of the Helm chart). This approach is useful when you want to maintain separation between different resources or when working with tools that expect resources to be in separate files. ```yaml vars: - name: gitRepo value: https://github.com/example/repo.git steps: - uses: git-clone config: repoURL: ${{ vars.gitRepo }} checkout: - commit: ${{ commitFrom(vars.gitRepo).ID }} path: ./src - branch: stage/${{ ctx.stage }} create: true path: ./out - uses: git-clear config: path: ./out - uses: helm-template config: path: ./src/charts/my-chart valuesFiles: - ./src/charts/my-chart/${{ ctx.stage }}-values.yaml outPath: ./out releaseName: my-chart # Commit, push, etc... ``` --- ## `helm-update-chart` `helm-update-chart` performs specified updates on the `dependencies` section of a specified Helm chart's `Chart.yaml` file. This step is useful for the common scenario of updating a chart's dependencies to reflect new versions of charts referenced by the Freight being promoted. This step is commonly followed by a [`helm-template` step](helm-template.md). ## Configuration | Name | Type | Required | Description | |------|------|----------|-------------| | `path` | `string` | Y | Path to a Helm chart (i.e. to a directory containing a `Chart.yaml` file). This path is relative to the temporary workspace that Kargo provisions for use by the promotion process. | | `charts` | `[]object` | N | The details of dependency (subchart) updates to be applied to the chart's `Chart.yaml` file. When left unspecified, charts will be updated according to the SemVer ranges declared in the `Chart.yaml` file. | | `charts[].repository` | `string` | Y | The URL of the Helm chart repository in the `dependencies` entry whose `version` field is to be updated. Must _exactly_ match the `repository` field of that entry. | | `charts[].name` | `string` | Y | The name of the chart in in the `dependencies` entry whose `version` field is to be updated. Must exactly match the `name` field of that entry. | | `charts[].version` | `string` | Y | The version to which the dependency should be updated. | ## Output | Name | Type | Description | |------|------|-------------| | `commitMessage` | `string` | A description of the change(s) applied by this step. Typically, a subsequent [`git-commit` step](git-commit.md) will reference this output and aggregate this commit message fragment with others like it to build a comprehensive commit message that describes all changes. | ## Examples ### Classic Chart Repository Given a `Chart.yaml` file such as the following: ```yaml apiVersion: v2 name: example type: application version: 0.1.0 appVersion: 0.1.0 dependencies: - repository: https://example-chart-repo name: some-chart version: 1.2.3 ``` The `dependencies` can be updated to reflect the version of `some-chart` referenced by the Freight being promoted like so: ```yaml vars: - name: gitRepo value: https://github.com/example/repo.git - name: chartRepo value: https://example-chart-repo steps: - uses: git-clone config: repoURL: ${{ vars.gitRepo }} checkout: - commit: ${{ commitFrom(vars.gitRepo).ID }} path: ./src - branch: stage/${{ ctx.stage }} create: true path: ./out - uses: git-clear config: path: ./out - uses: helm-update-chart config: path: ./src/charts/my-chart charts: - repository: ${{ chartRepo }} name: some-chart version: ${{ chartFrom(chartRepo).Version }} # Render manifests to ./out, commit, push, etc... ``` :::info For more information on `chartFrom` and expressions, see the [Expressions](../40-expressions.md#functions) documentation. ::: ### OCI Chart Repository :::caution Classic (HTTP/HTTPS) Helm chart repositories can contain many differently named charts. A specific chart, therefore, can be identified by a repository URL and a chart name. OCI repositories, on the other hand, are organizational constructs within OCI _registries._ Each OCI repository is presumed to contain versions of only a single chart. As such, a specific chart can be identified by a repository URL alone. Kargo Warehouses understand this distinction well, so a subscription to an OCI chart repository will utilize its URL only, _without_ specifying a chart name. For example: ```yaml apiVersion: kargo.akuity.io/v1alpha1 kind: Warehouse metadata: name: my-warehouse namespace: kargo-demo spec: subscriptions: - chart: repoURL: oci://example-chart-registry/some-chart semverConstraint: ^1.0.0 ``` Helm deals with this difference somewhat more awkwardly, however. When a Helm chart references a chart in an OCI repository, it must reference the _registry_ by URL in the `repository` field and _still_ specify a chart name in the name field. For example: ```yaml apiVersion: v2 name: example type: application version: 0.1.0 appVersion: 0.1.0 dependencies: - repository: oci://example-chart-registry name: some-chart version: 1.2.3 ``` __When using `helm-update-chart` to update the dependencies in a `Chart.yaml` file, you must play by Helm's rules and use the _registry_ URL in the `repository` field and the _repository_ name (chart name) in the `name` field.__ ::: :::info As a general rule, when configuring Kargo to update something, observe the conventions of the thing being updated, even if those conventions differ from Kargo's own. Kargo is aware of such differences and will adapt accordingly. ::: Given a `Chart.yaml` file such as the following: ```yaml apiVersion: v2 name: example type: application version: 0.1.0 appVersion: 0.1.0 dependencies: - repository: oci://example-chart-registry name: some-chart version: 1.2.3 ``` The `dependencies` can be updated to reflect the version of `oci://example-chart-registry/some-chart` referenced by the Freight being promoted like so: ```yaml vars: - name: gitRepo value: https://github.com/example/repo.git - name: chartReg value: oci://example-chart-registry steps: - uses: git-clone config: repoURL: ${{ vars.gitRepo }} checkout: - commit: ${{ commitFrom(vars.gitRepo).ID }} path: ./src - branch: stage/${{ ctx.stage }} create: true path: ./out - uses: git-clear config: path: ./out - uses: helm-update-chart config: path: ./src/charts/my-chart charts: - repository: ${{ vars.chartReg }} name: some-chart version: ${{ chartFrom(vars.chartReg + "/some-chart").Version }} # Render manifests to ./out, commit, push, etc... ``` :::info For more information on `chartFrom` and expressions, see the [Expressions](../40-expressions.md#functions) documentation. ::: ### `Chart.yaml` SemVer ranges Given a `Chart.yaml` file such as the following: ```yaml apiVersion: v2 name: example type: application version: 0.1.0 appVersion: 0.1.0 dependencies: - repository: https://example-chart-repo name: some-chart version: ^1.0.0 ``` The `dependencies` can be updated to the latest version according to the SemVer range by running the update without any further configuration options: ```yaml vars: - name: gitRepo value: https://github.com/example/repo.git steps: - uses: git-clone config: repoURL: ${{ vars.gitRepo }} checkout: - commit: ${{ commitFrom(vars.gitRepo).ID }} path: ./src - uses: helm-update-chart config: path: ./src/charts/my-chart # Render manifests to ./out, commit, push, etc... ``` --- ## `http-download` `http-download` is a step that downloads files from HTTP/S URLs to enable integration with external file repositories, CDNs, and other web-based file sources. :::note Downloads are limited to 100MB to prevent resource exhaustion. ::: ## Configuration | Name | Type | Required | Description | |------|------|----------|-------------| | `url` | `string` | Y | The URL from which to download the file. | | `outPath` | `string` | Y | The path where the downloaded file will be saved, relative to the step's working directory. | | `allowOverwrite` | `boolean` | N | Whether to allow overwriting an existing file at the specified path. If `false` and the file exists, the download will fail. Defaults to `false`. | | `headers` | `[]object` | N | A list of headers to include in the request. | | `headers[].name` | `string` | Y | The name of the header. | | `headers[].value` | `string` | Y | The value of the header. | | `queryParams` | `[]object` | N | A list of query parameters to include in the request. | | `queryParams[].name` | `string` | Y | The name of the query parameter. | | `queryParams[].value` | `string` | Y | The value of the query parameter. The provided value will automatically be URL-encoded if necessary. | | `insecureSkipTLSVerify` | `boolean` | N | Indicates whether to bypass TLS certificate verification when making the request. Setting this to `true` is highly discouraged. | | `timeout` | `string` | N | A string representation of the maximum time interval to wait for the download to complete. See Go's [`time` package docs](https://pkg.go.dev/time#ParseDuration) for a description of the accepted format. Defaults to 5 minutes. | ## Outputs The `http-download` step does not produce any outputs. Success is indicated by the step completing without error and the file being present at the specified path. ## Examples ### Basic Usage This example configuration downloads a configuration file from a web server: ```yaml steps: # ... - uses: http-download as: fetch-config config: url: https://example.com/config/app.yaml outPath: config/app.yaml ``` The step would download the file and save it to `config/app.yaml` in the working directory. Parent directories are created automatically if they don't exist. ### Download with Authentication This example downloads a file from a protected endpoint using authentication headers: ```yaml steps: # ... - uses: http-download as: fetch-artifact config: url: https://artifacts.example.com/releases/v1.2.3/app.tar.gz outPath: artifacts/app.tar.gz headers: - name: Authorization value: Bearer ${{ secret('artifacts').token }} allowOverwrite: true timeout: 10m ``` ### Download with Query Parameters This example downloads a file using query parameters and demonstrates handling of existing files: ```yaml steps: # ... - uses: http-download as: fetch-release config: url: https://artifacts.example.com/releases/source.zip outPath: releases/source.zip queryParams: - name: version value: ${{ vars.version }} headers: - name: Authorization value: token ${{ secret('artifacts').token }} allowOverwrite: false timeout: 5m ``` If `releases/source.zip` already exists, this step would fail terminally since `allowOverwrite` is `false`. ### Download and Unpack Helm Chart This example downloads a Helm chart archive and unpacks it into a directory: ```yaml vars: - name: chartRepo value: https://charts.example.com - name: chartName value: my-helm-chart steps: # ... - uses: http-download as: fetch-chart config: url: https://charts.example.com/${{ vars.chartName }}-${{ chartFrom(vars.chartRepo, vars.chartName).Version }}.tgz outPath: charts/${{ vars.chartName }}.tgz headers: - name: Accept value: application/gzip allowOverwrite: true - as: extract-chart uses: untar config: inPath: charts/${{ vars.chartName }}.tgz outPath: charts/${{ vars.chartName }} ``` --- ## `http` `http` is a generic step that makes an HTTP/S request to enable basic integration with a wide variety of external services. ## Configuration | Name | Type | Required | Description | |------|------|----------|-------------| | `method` | `string` | N | The HTTP method to use. Defaults to `GET` | | `url` | `string` | Y | The URL to which the request should be made. | | `headers` | `[]object` | N | A list of headers to include in the request. | | `headers[].name` | `string` | Y | The name of the header. | | `headers[].value` | `string` | Y | The value of the header. | | `queryParams` | `[]object` | N | A list of query parameters to include in the request. | | `queryParams[].name` | `string` | Y | The name of the query parameter. | | `queryParams[].value` | `string` | Y | The value of the query parameter. The provided value will automatically be URL-encoded if necessary. | | `body` | `string` | N | The body of the request. __Note:__ As this field is a `string`, take care to utilize [`quote()`](../40-expressions.md#quotevalue) if the body is a valid JSON `object`. Refer to the example below of posting a message to a Slack channel. | | `insecureSkipTLSVerify` | `boolean` | N | Indicates whether to bypass TLS certificate verification when making the request. Setting this to `true` is highly discouraged. | | `timeout` | `string` | N | A string representation of the maximum time interval to wait for a request to complete. _This is the timeout for an individual HTTP request. If a request is retried, each attempt is independently subject to this timeout._ See Go's [`time` package docs](https://pkg.go.dev/time#ParseDuration) for a description of the accepted format. | | `responseContentType` | `string` | N | Overrides automatic content-type detection for response parsing. Accepts `application/json`, `application/yaml`, or `text/plain`. When not set, the step uses the response's `Content-Type` header, falling back to JSON parsing for unrecognized types. | | `successExpression` | `string` | N | An [expr-lang] expression that can evaluate the response to determine success. When defined, the step succeeds only when this expression evaluates to `true`. If both `successExpression` and `failureExpression` are defined and both evaluate to `true`, the failure takes precedence and the step fails terminally. Note that this expression should _not_ be offset by `${{` and `}}`. See examples for more details. | | `failureExpression` | `string` | N | An [expr-lang] expression that can evaluate the response to determine failure. When defined and evaluates to `true`, the step fails terminally. If both `successExpression` and `failureExpression` are defined and both evaluate to `true`, the failure takes precedence. Note that this expression should _not_ be offset by `${{` and `}}`. See examples for more details. | | `outputs` | `[]object` | N | A list of rules for extracting outputs from the HTTP response. These are only applied to responses deemed successful. | | `outputs[].name` | `string` | Y | The name of the output. | | `outputs[].fromExpression` | `string` | Y | An [expr-lang] expression that can extract a value from the HTTP response. Note that this expression should _not_ be offset by `${{` and `}}`. See examples for more details. | ## Success and Failure Determination The step's outcome is determined by evaluating the success and failure criteria as follows: - If `failureExpression` is defined and evaluates to `true`, the step **fails terminally** (no retries). - If `successExpression` is defined and evaluates to `true` (and failure criteria are not met), the step **succeeds**. - If neither expression is defined: **2xx status codes** succeed, **non-2xx status codes** fail but will be retried. - All other cases result in **Running** and will be retried. :::note The key distinction is between **terminal failures** (when `failureExpression` evaluates to `true`) and **retried failures** (all other failure cases). Terminal failures stop the promotion immediately, while retried failures allow Kargo to retry the step according to the configured [retry policy](../15-promotion-templates.md#step-retries). ::: ## Expressions The `successExpression`, `failureExpression`, and `outputs[].fromExpression` fields all support [expr-lang][] expressions. :::warning Expressions in the `successExpression` and `failureExpression` fields must _not_ be enclosed by `${{` and `}}` since all such expressions are evaluated _prior_ to step execution. (i.e. All steps are _actually_ executed against static, pre-evaluated configuration!) Since these two expressions are intended to be evaulated _internally_ by the `http` step, and only after receiving an HTTP response, not enclosing them within `${{` and `}}` prevents premature evaluation and ensures they are passed to the `http` step exactly as they've been written. If your `successExpression` or `failureExpression` need to reference variables or output from previous steps, use expressions that _are_ enclosed by `${{` and `}}` _within_ those expressions. The "inner expressions" will be evaluated prior to step execution, while the "outer expressions" will be evaluated by the step itself. Consider the following: ```yaml vars: - name: expectedStatus value: completed steps: - uses: http config: url: https://api.example.com/status successExpression: response.body.status == '${{ vars.expectedStatus }}' failureExpression: response.body.status == 'failed' ``` After expressions enclosed within `${{` and `}}` have been pre-evaluated, the `http` step defined above is executed against the following _now static_ configuration: ```yaml { "failureExpression": "response.body.status == 'failed'", "successExpression": "response.body.status == 'completed'", "url": "https://api.example.com/status" } ``` Internally, the step evaluates the `successExpression` and `failureExpression` exactly as if the user had written them as they now appear. ::: A `response` object (a `map[string]any`) is available to these expressions. It is structured as follows: | Field | Type | Description | |-------|------|-------------| | `status` | `int` | The HTTP status code of the response. | | `headers` | `http.Header` | The headers of the response. See applicable [Go documentation](https://pkg.go.dev/net/http#Header). | | `header` | `func(string) string` | `headers` can be inconvenient to work with directly. This function allows you to access a header by name. | | `body` | `any` | The parsed response body. For JSON responses (`application/json`), this can be any valid JSON value: objects, arrays, strings, numbers, booleans, or `null`. For YAML responses (`application/yaml`, `text/yaml`, `application/x-yaml`), this is the unmarshaled YAML structure. For `text/plain` responses, this is the raw string. For unrecognized content types, the step attempts JSON parsing and falls back to an empty map if the content is not valid JSON. Empty responses result in an empty map. | ## Outputs The `http` step only produces the outputs described by the `outputs` field of its configuration. ## Examples ### Basic Usage This example configuration makes a `GET` request to the [Cat Facts API](https://www.catfacts.net/api/) and uses the default success/failure criteria. ```yaml steps: # ... - uses: http as: cat-facts config: method: GET url: https://www.catfacts.net/api/ outputs: - name: status fromExpression: response.status - name: fact1 fromExpression: response.body.facts[0] - name: fact2 fromExpression: response.body.facts[1] ``` Assuming a `200` response with the following JSON body: ```json { "facts": [ { "fact_number": 1, "fact": "Kittens have baby teeth, which are replaced by permanent teeth around the age of 7 months." }, { "fact_number": 2, "fact": "Each day in the US, animal shelters are forced to destroy 30,000 dogs and cats." } ] } ``` The step would succeed and produce the following outputs: | Name | Type | Value | |----------|------|-------| | `status` | `int` | `200` | | `fact1` | `string` | `Kittens have baby teeth, which are replaced by permanent teeth around the age of 7 months.` | | `fact2` | `string` | `Each day in the US, animal shelters are forced to destroy 30,000 dogs and cats.` | ### Polling Building on the [basic example](#basic-usage), this configuration defines explicit success and failure criteria. Any response meeting neither of these criteria will result in the step reporting a result of `Running` and being retried. Note the use of [retry](../15-promotion-templates.md#step-retries) configuration to set a timeout for the step. ```yaml steps: # ... - uses: http as: cat-facts retry: timeout: 10m config: method: GET url: https://www.catfacts.net/api/ successExpression: response.status == 200 failureExpression: response.status == 404 outputs: - name: status fromExpression: response.status - name: fact1 fromExpression: response.body.facts[0] - name: fact2 fromExpression: response.body.facts[1] ``` Our request is considered: - **Successful** if the response status is `200`. - **A terminal failure** if the response status is `404`. - **Running** (will be retried) if the response status is anything else. ### Posting to Slack This example is adapted from [Slack's own documentation](https://api.slack.com/tutorials/tracks/posting-messages-with-curl), showing how to post a message to a Slack channel. ```yaml vars: - name: slackChannel value: C123456 steps: # ... - uses: http config: method: POST url: https://slack.com/api/chat.postMessage headers: - name: Authorization value: Bearer ${{ secret('slack').token }} - name: Content-Type value: application/json body: | ${{ quote({ "channel": vars.slackChannel, "blocks": [ { "type": "section", "text": { "type": "mrkdwn", "text": "Hi I am a bot that can post *_fancy_* messages to any public channel." } } ] }) }} ``` ### YAML Responses This example demonstrates working with an API that returns YAML-formatted data. ```yaml steps: # ... - uses: http as: config-fetch config: method: GET url: https://api.example.com/config.yaml successExpression: response.status == 200 outputs: - name: version fromExpression: response.body.version - name: features fromExpression: response.body.features ``` If the server returns a `200` response with `Content-Type: application/yaml` and the following body: ```yaml version: "2.0" features: - name: feature-a enabled: true - name: feature-b enabled: false ``` The step would succeed and produce structured outputs that can be accessed in subsequent steps. #### Overriding Content-Type Detection Some APIs return structured data with incorrect or generic content types. Use `responseContentType` to explicitly specify how to parse the response: ```yaml steps: - uses: http config: url: https://api.example.com/data responseContentType: application/yaml # Force YAML parsing outputs: - name: value fromExpression: response.body.key ``` [expr-lang]: https://expr-lang.org/ --- ## Promotion Steps Reference Below is an overview of the available promotion steps that can be used in [Promotion Templates](../15-promotion-templates.md) and [Promotion Tasks](../20-promotion-tasks.md). --- ## `jfrog-evidence` :::info This promotion step is only available in Kargo on the [Akuity Platform](https://akuity.io/akuity-platform), versions v1.7 and above. ::: The `jfrog-evidence` promotion step provides comprehensive integration with JFrog Artifactory's evidence management capabilities, allowing you to create, verify, and delete evidence for artifacts. This enables secure attestation and verification of your promotion workflows through cryptographically signed evidence that can track artifact provenance, test results, and compliance status. This promotion step supports three main operations: creating evidence with digital signatures, querying and verifying existing evidence, and cleaning up evidence when needed. It's particularly valuable for maintaining supply chain security and compliance in enterprise environments. For more information about JFrog's Evidence feature, see the [official JFrog Evidence documentation](https://jfrog.com/help/r/jfrog-artifactory-documentation/evidence-management). ## Credentials Configuration All JFrog Evidence operations require proper authentication credentials stored in a Kubernetes `Secret`. | Name | Type | Required | Description | | ------------------------ | -------- | -------- | ------------------------------------------------------------------------------- | | `credentials.secretName` | `string` | N | Name of the `Secret` containing the JFrog credentials in the project namespace. | | `credentials.sharedSecretName` | `string` | N | Name of the `Secret` containing the JFrog credentials in the `shared-resources-namespace`. | :::info Either `credentials.secretName` or `credentials.sharedSecretName` must be set, but not both. ::: The referenced `Secret` should contain the following keys: - `url`: The URL of your JFrog Artifactory instance (e.g., `https://yourcompany.jfrog.io`) - `accessToken`: Your JFrog access token with appropriate permissions for evidence management - `privateKey` (optional): Private key in PEM format for signing evidence. This key can be stored in a separate secret and referenced when required. ## Evidence Management ### Create Evidence Creates cryptographically signed kargo promotion evidence for an artifact in JFrog Artifactory. #### Configuration | Name | Type | Required | Description | | ------------------------ | -------- | -------- | ----------------------------------------------------------------------------------------------------------------------- | | `create.privateKey` | `string` | Y | The private key to sign the evidence. | | `create.packageRepo` | `string` | Y | The package repository name of the artifact. | | `create.packageName` | `string` | Y | The package name of the artifact inside the repository. | | `create.packageVersion` | `string` | Y | The package version of the artifact. | | `create.promotionStatus` | `string` | Y | Status indicating the promotion state. Valid values: `Pending`, `Running`, `Succeeded`, `Failed`, `Errored`, `Aborted`. | | `create.privateKeyAlias` | `string` | N | Name for the public key created from the private key. Used for verification. If omitted, verification is skipped. | | `create.metadata` | `object` | N | JSON metadata that will be stored inside the evidence predicate. | #### Output | Name | Type | Description | | ------ | -------- | ---------------------------------------------------------------- | | `name` | `string` | The name/identifier of the created evidence for later reference. | The Kargo promotion evidence predicate stored in the artifact will have a predicate type of `https://akuity.io/evidence/promotion/v1` and will look something like this: ```json { "project": "my-project", "stage": "production", "promotion": "production.01k22r0jkhfgheh57ywp9t15ya.67018d8", "actor": "admin@company.com", "freightName": "abc123def456", "metadata": { "env": "production", "result": "pass", "testSuite": "integration-tests", "coverage": "85%" }, "promotionStatus": "Succeeded", "promotionTimestamp": 1672531200 } ``` #### Example This example creates evidence for a successful test promotion with custom metadata. ```yaml steps: - uses: jfrog-evidence as: create-test-evidence config: credentials: secretName: jfrog-credentials create: privateKey: ${{ secret("jfrog-credentials").privateKey }} privateKeyAlias: "test-evidence-key" packageRepo: ${{ vars.packageRepo }} packageName: ${{ vars.packageName }} packageVersion: ${{ imageFrom(vars.packageRegistry+"/"+vars.packageRepo+"/"+vars.packageName).Tag }} metadata: env: "${{ ctx.stage }}" result: "pass" testSuite: "integration-tests" coverage: "85%" promotionStatus: "Succeeded" ``` ### Delete Evidence Removes specific evidence from an artifact in JFrog Artifactory. This is typically used as a cleanup step in conjunction with the create step when a promotion fails. #### Configuration | Name | Type | Required | Description | | ----------------------- | -------- | -------- | ------------------------------------------------------- | | `delete.packageRepo` | `string` | Y | The package repository name of the artifact. | | `delete.packageName` | `string` | Y | The package name of the artifact inside the repository. | | `delete.packageVersion` | `string` | Y | The package version of the artifact. | | `delete.evidenceName` | `string` | Y | Evidence name/identifier to delete. | #### Output This step does not produce any output. #### Example This example deletes evidence when a promotion fails, using the evidence name from a previous step. ```yaml steps: - uses: jfrog-evidence as: cleanup-failed-evidence if: ${{ failure() && status('create-test-evidence') == 'Succeeded' }} config: credentials: secretName: jfrog-credentials delete: packageRepo: ${{ vars.packageRepo }} packageName: ${{ vars.packageName }} packageVersion: ${{ imageFrom(vars.packageRegistry+"/"+vars.packageRepo+"/"+vars.packageName).Tag }} evidenceName: ${{ outputs['create-test-evidence'].name }} ``` ### Process Evidence (Query and Verify) Queries for existing evidence and verifies its authenticity and content. This operation is commonly used as a promotion gating mechanism to ensure that only artifacts with valid evidence and compliance status proceed through the deployment pipeline. This step can also extract data from the queried evidence as step outputs, which can then be used as inputs in subsequent steps. #### Configuration | Name | Type | Required | Description | | ----------------------------------- | --------- | -------- | ----------------------------------------------------------------------------------------------------------- | | `process.query.packageRepo` | `string` | Y | The package repository name of the artifact. | | `process.query.packageName` | `string` | Y | The package name of the artifact inside the repository. | | `process.query.packageVersion` | `string` | Y | The package version of the artifact. | | `process.query.evidenceNameRegex` | `string` | N* | Regular expression to match evidence names. *Either this or `predicateType` is required. | | `process.query.predicateType` | `string` | N* | The predicate type to query for. *Either this or `evidenceNameRegex` is required. | | `process.verify.verifyExpression` | `string` | N** | Expr expression to evaluate on the predicate. Must return boolean. **One of the verify options is required. | | `process.verify.localKeys` | `array` | N** | Local private/public keys for signature verification. **One of the verify options is required. | | `process.verify.useArtifactoryKeys` | `boolean` | N** | Use keys present in evidence data for verification. **One of the verify options is required. | | `process.outputs` | `array` | N | Output configurations to extract data from evidence results. | | `process.outputs[].name` | `string` | Y | Name of the output variable. | | `process.outputs[].expression` | `string` | Y | Expr expression to extract data from evidence query results. | #### Available Data for Expressions Both verification expressions (`verifyExpression`) and output expressions (`outputs[].expression`) have access to the following data structure: ```yaml evidence: predicate: # The evidence predicate data (varies by predicate type) predicateType: # The predicate type of the evidence createdAt: # ISO timestamp when the evidence was created name: # The name/identifier of the evidence verified: # Whether the evidence signature was verified downloadPath: # The download path for the evidence signingKey: alias: # The alias of the signing key publicKey: # The public key used for signing ``` #### Output The outputs are dynamic based on the `process.outputs` configuration. Each output creates a variable with the specified name containing the result of the expression evaluation. #### Example This example verifies test evidence and extracts relevant information for subsequent steps. ```yaml steps: - uses: jfrog-evidence as: verify-test-evidence config: credentials: secretName: jfrog-credentials process: query: packageRepo: ${{ vars.packageRepo }} packageName: ${{ vars.packageName }} packageVersion: ${{ imageFrom(vars.packageRegistry+"/"+vars.packageRepo+"/"+vars.packageName).Tag }} predicateType: "https://in-toto.io/attestation/test-result/v0.1" evidenceNameRegex: ".*-test-result-.*" verify: localKeys: - ${{ secret("jfrog-credentials").privateKey }} useArtifactoryKeys: true verifyExpression: | evidence.predicate.result == "pass" && evidence.predicate.coverage >= 80 outputs: - name: testResult expression: evidence.predicate.result - name: coverage expression: evidence.predicate.coverage - name: createdAt expression: evidence.createdAt - name: environment expression: evidence.predicate.metadata.env # Use the extracted values in subsequent steps - uses: http config: method: POST url: https://api.slack.com/api/chat.postMessage headers: - name: Authorization value: "Bearer ${{ secret('slack-token').token }}" - name: Content-Type value: application/json body: | ${{ quote({ "channel": "#deployments", "text": "Test evidence verified! Result: " + outputs['verify-test-evidence'].testResult + ", Coverage: " + string(outputs['verify-test-evidence'].coverage) + "%" + ", Environment: " + outputs['verify-test-evidence'].environment }) }} ``` #### Trusted Release Evidence Verification JFrog Trusted Release evidence is a special type of evidence generated exclusively for the release stage. This evidence creates a distinct certification artifact that verifies an application version has been officially released with policy evaluation. It is possible to validate this type of evidence with Kargo and use it as a promotion gating mechanism for promotions in production environments. ##### Configuration This example demonstrates verifying trusted release evidence before production deployment. The `verifyExpression` can be modified according to your requirements. ```yaml steps: - uses: jfrog-evidence as: verify-trusted-release config: credentials: secretName: jfrog-credentials process: query: packageRepo: ${{ vars.packageRepo }} packageName: ${{ vars.packageName }} packageVersion: ${{ imageFrom(vars.packageRegistry+"/"+vars.packageRepo+"/"+vars.packageName).Tag }} predicateType: "https://jfrog.com/evidence/release_certify/v1" verify: localKeys: - ${{ secret("jfrog-credentials").privateKey }} useArtifactoryKeys: true verifyExpression: | evidence.predicate.release_type == "Trusted Release" && len(evidence.predicate.policy_results) > 0 && evidence.predicate.outcome == "passed" outputs: - name: releaseType expression: evidence.predicate.release_type - name: releaseTimestamp expression: evidence.predicate.evaluated_at # Only proceed with production deployment if trusted release is verified in previous step # Update the Argo CD Application directly. Not ideal for practical purposes. - uses: argocd-update config: apps: - name: production-app namespace: argocd sources: - repoURL: https://github.com/company/app-config.git kustomize: images: - repoURL: ${{ vars.packageRegistry }}/{{ vars.packageRepo }}/{{ vars.packageName }} tag: ${{ imageFrom(vars.packageRegistry+"/"+vars.packageRepo+"/"+vars.packageName).Tag }} ``` This verification ensures that: - The evidence is for a "Trusted Release" - Policy evaluations were performed (policy_results is not empty). You can perform more complex checks on the policy_results according to your requirements. - All compliance checks passed - The evidence signature is valid --- ## `jira` :::info This promotion step is only available in Kargo on the [Akuity Platform](https://akuity.io/akuity-platform), versions v1.6 and above. ::: The `jira` promotion step provides comprehensive integration with Jira, allowing you to create, update, delete, and search for issues, manage comments, and track promotion workflows. This is particularly useful for maintaining traceability between your promotion processes and project management activities. This promotion step supports various operations including issue management, comment handling, and status tracking, making it a powerful tool for promotion workflows that require coordination with project management systems. ## Credentials Configuration All Jira operations require proper authentication credentials stored in a Kubernetes `Secret`. | Name | Type | Required | Description | | ------------------------ | -------- | -------- | -------------------------------------------------------------------------------- | | `credentials.secretName` | `string` | N | Name of the `Secret` containing the Jira credentials in the project namespace. | | `credentials.sharedSecretName` | `string` | N | Name of the `Secret` containing the Jira credentials in the `shared-resources-namespace`. | :::info Either `credentials.secretName` or `credentials.sharedSecretName` must be set, but not both. ::: The referenced `Secret` should contain the following keys: - `domain`: The domain of your Jira instance or Jira api (e.g., `https://yourcompany.atlassian.net`) - `username`: Your Jira username or email - `password`: Your Jira API token or password ## Issue Management ### Create Issue Creates a new Jira issue with specified details. #### Configuration | Name | Type | Required | Description | | ---------------------------- | -------- | -------- | ----------------------------------------------------------------------------------------------------------- | | `createIssue.projectKey` | `string` | Y | The key of the Jira project where the issue will be created. | | `createIssue.summary` | `string` | Y | The summary or title of the issue. | | `createIssue.description` | `string` | N | The description of the issue. Supports markdown formatting. | | `createIssue.adfDescription` | `object` | N | ADF (Atlassian Document Format) content for complex formatting. Alternative to `description`. | | `createIssue.issueType` | `string` | N | The type of issue to create (e.g., 'Bug', 'Task', 'Story'). | | `createIssue.assigneeEmail` | `string` | N | Email of the user to assign the issue to. | | `createIssue.labels` | `array` | N | Labels to add to the issue for categorization. | | `createIssue.customFields` | `object` | N | Custom fields to set. Keys should match Jira custom field IDs. | | `createIssue.issueAlias` | `string` | N | Override for the freight metadata key used to reference the created issue id. Defaults to `jira-issue-key`. | #### Output | Name | Type | Description | | ----- | -------- | ----------------------------------------------------- | | `key` | `string` | The key/id of the created Jira issue (e.g., EXT-123). | #### Example This example creates a new Jira issue to track a promotion, assigns it to a team member, and adds relevant labels. ```yaml steps: - uses: jira as: create-promotion-issue config: credentials: secretName: jira-credentials createIssue: projectKey: PROMOTE summary: "Promote ${{ imageFrom(vars.imageRepo).Tag }} to ${{ ctx.stage }}" description: "Promoteing ${{ imageFrom(vars.imageRepo).RepoURL }}:${{ imageFrom(vars.imageRepo).Tag }} to ${{ ctx.stage }} environment. Promotion ID: ${{ ctx.promotion }}. Freight: ${{ ctx.targetFreight.name }}." issueType: Task assigneeEmail: devops@company.com labels: - promotion - "${{ ctx.stage }}" - "release-${{ imageFrom(vars.imageRepo).Tag }}" # Use the created issue key in subsequent steps - uses: jira config: credentials: secretName: jira-credentials updateIssue: issueKey: "${{ outputs['create-promotion-issue'].key }}" status: "IN PROGRESS" ``` ### Update Issue Updates an existing Jira issue with new information. #### Configuration | Name | Type | Required | Description | | ---------------------------- | -------- | -------- | ----------------------------------------------------------------- | | `updateIssue.issueKey` | `string` | Y | The Jira Issue Key (e.g., EXT-123). | | `updateIssue.summary` | `string` | N | Updated summary or title of the issue. | | `updateIssue.description` | `string` | N | Updated description. Supports markdown formatting. | | `updateIssue.adfDescription` | `object` | N | ADF content for complex formatting. Alternative to `description`. | | `updateIssue.issueType` | `string` | N | Updated issue type. | | `updateIssue.assigneeEmail` | `string` | N | Email of the user to assign the issue to. | | `updateIssue.status` | `string` | N | Status to set for the issue (e.g., 'IN PROGRESS', 'DONE'). | | `updateIssue.addLabels` | `array` | N | Labels to add to the issue. | | `updateIssue.removeLabels` | `array` | N | Labels to remove from the issue. | | `updateIssue.customFields` | `object` | N | Custom fields to update. | #### Output This step does not produce any output. #### Example This example updates an existing issue's status and adds a comment with promotion details. ```yaml steps: - uses: jira config: credentials: secretName: jira-credentials updateIssue: issueKey: PROMOTE-123 status: "IN PROGRESS" summary: "Promote ${{ imageFrom(vars.imageRepo).Tag }} to ${{ ctx.stage }} - IN PROGRESS" addLabels: - promoting - "${{ ctx.stage }}-promotion" customFields: customfield_10000: "${{ ctx.stage }} Environment" customfield_10001: "${{ ctx.promotion }}" ``` ### Delete Issue Deletes a Jira issue and, optionally, its subtasks. #### Configuration | Name | Type | Required | Description | | ---------------------------- | --------- | -------- | ---------------------------------------------- | | `deleteIssue.issueKey` | `string` | Y | The Jira Issue Key (e.g., EXT-123). | | `deleteIssue.deleteSubtasks` | `boolean` | N | If true, all subtasks will be deleted as well. | #### Output This step does not produce any output. #### Example This example deletes a Jira issue and all its subtasks when a promotion fails. ```yaml steps: # existing steps create issue and other promotion steps # .... # .... # on failure cleanup logic - as: on-failure-cleanup-issue uses: jira if: ${{ failure() }} config: credentials: secretName: jira-credentials deleteIssue: issueKey: "${{ freightMetadata(ctx.targetFreight.name)['jira-issue-key'] }}" deleteSubtasks: true ``` ### Search Issues Searches for Jira issues using JQL ([Jira Query Language](https://support.atlassian.com/jira-service-management-cloud/docs/use-advanced-search-with-jira-query-language-jql/)). #### Configuration | Name | Type | Required | Description | | ---------------------------- | --------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------- | | `searchIssue.jql` | `string` | Y | The JQL query to search for issues. | | `searchIssue.expectMultiple` | `boolean` | N | If true, expects multiple results and returns the first matching result. If false, expects single result and fails with >1 results. | | `searchIssue.fields` | `array` | N | List of fields to include in search results. | | `searchIssue.expands` | `array` | N | List of fields to expand in search results. | #### Output | Name | Type | Description | | ------- | -------- | --------------------------------------------------------------------------- | | `issue` | `object` | The found Jira issue object containing all requested fields and expansions. | #### Example This example searches for open promotion issues in a specific project and expects multiple results. ```yaml steps: - uses: jira as: find-open-promotions config: credentials: secretName: jira-credentials searchIssue: jql: 'project = PROMOTE AND status != "Done" AND labels = "${{ ctx.stage }}-promotion" AND created >= -7d' expectMultiple: true fields: - summary - status - assignee - created expands: - changelog # Use search results in subsequent steps to notify team # Note: This is just an example of using search outputs and may not be syntactically valid - uses: http config: method: POST url: https://slack.com/api/chat.postMessage headers: - name: Authorization value: "Bearer ${{ secret('slack-credentials').token }}" - name: Content-Type value: application/json body: | ${{ quote({ "channel": "#promotions", "text": "Found issue" + outputs['find-open-promotions'].issue.key + " for " + ctx.stage + " environment" }) }} ``` ## Comment Management ### Add Comment Adds a comment to an existing Jira issue. #### Configuration | Name | Type | Required | Description | | ------------------------- | -------- | -------- | ---------------------------------------------------------- | | `commentOnIssue.issueKey` | `string` | Y | The Jira Issue Key (e.g., EXT-123). | | `commentOnIssue.body` | `string` | N | Text content of the comment. | | `commentOnIssue.adfBody` | `object` | N | ADF content for complex formatting. Alternative to `body`. | #### Output | Name | Type | Description | | ----------- | -------- | -------------------------------------------------------------------- | | `commentID` | `string` | The ID of the created comment that can be used for later operations. | #### Example This example adds a comment to a Jira issue with promotion progress information. ```yaml steps: - uses: jira as: add-progress-comment config: credentials: secretName: jira-credentials commentOnIssue: issueKey: "${{ freightMetadata(ctx.targetFreight.name)['jira-issue-key'] }}" body: "Promotion started. Environment: ${{ ctx.stage }}. Image: ${{ imageFrom(vars.imageRepo).RepoURL }}:${{ imageFrom(vars.imageRepo).Tag }}. Promotion: ${{ ctx.promotion }}. Status: Promoting to ${{ ctx.stage }} environment..." # Later use the comment ID if needed - uses: jira config: credentials: secretName: jira-credentials deleteComment: issueKey: "${{ freightMetadata(ctx.targetFreight.name)['jira-issue-key'] }}" commentID: "${{ quote(outputs['add-progress-comment'].commentID) }}" ``` ### Delete Comment Removes a specific comment from a Jira issue. #### Configuration | Name | Type | Required | Description | | ------------------------- | -------- | -------- | ----------------------------------- | | `deleteComment.issueKey` | `string` | Y | The Jira Issue Key (e.g., EXT-123). | | `deleteComment.commentID` | `string` | Y | The ID of the comment to delete. | #### Output This step does not produce any output. #### Example This example deletes a specific comment from a Jira issue. ```yaml steps: - uses: jira config: credentials: secretName: jira-credentials deleteComment: issueKey: "${{ freightMetadata(ctx.targetFreight.name)['jira-issue-key'] }}" commentID: "${{ outputs['previous-comment-step'].commentID }}" ``` ## Status Tracking ### Wait for Status Waits for a Jira issue to reach a specific status before proceeding. #### Configuration | Name | Type | Required | Description | | ------------------------------ | --------- | -------- | -------------------------------------------------------------- | | `waitForStatus.issueKey` | `string` | Y | The Jira Issue Key (e.g., EXT-123). | | `waitForStatus.expectedStatus` | `string` | Y | The expected status to wait for (e.g., 'IN PROGRESS', 'DONE'). | | `waitForStatus.pollInterval` | `string` | N | Poll interval for checking issue status, specified as a [Go duration string](https://pkg.go.dev/time#ParseDuration) (e.g., `30s`, `5m`). Overrides the default controller reconciliation interval of 5 minutes when set. Available in Kargo >= v1.10. | #### Output This step does not produce any output. #### Example This example waits for a change request issue to be approved before proceeding with promotion. ```yaml steps: - uses: jira config: credentials: secretName: jira-credentials waitForStatus: issueKey: "${{ freightMetadata(ctx.targetFreight.name)['change-request-key'] }}" expectedStatus: "Approved" # promotion steps continue after approval... - uses: helm-template config: path: ./charts vars: imageTag: "${{ imageFrom(vars.imageRepo).Tag }}" environment: "${{ ctx.stage }}" ``` :::info[Content Formatting] The Jira configuration supports setting issue and comment content using `description` or `body` fields. These are plain text fields that do not support special formatting such as Markdown. For rich formatting capabilities, use the ADF (Atlassian Document Format) alternatives `adfDescription` or `adfBody`. For more information about ADF structure and formatting, see the [Atlassian Document Format documentation](https://developer.atlassian.com/cloud/jira/platform/apis/document/structure/). ::: ## Multi-Stage Workflow The Jira promotion step automatically stores created issue keys in the Freight metadata, allowing subsequent stages to reference the same issue. This enables tracking a single Jira issue across multiple promotion stages. ### Accessing Issue Keys from Freight Metadata Use the [`freightMetadata()`](../40-expressions.md#freightmetadatafreightname) template function to retrieve issue keys stored by previous stages: ```yaml # Access the default issue key issueKey: ${{ freightMetadata(ctx.targetFreight.name)['jira-issue-key'] }} # Access a custom issue key (when issueAlias was used during creation) issueKey: ${{ freightMetadata(ctx.targetFreight.name)['my-custom-alias'] }} ``` ### Example This comprehensive example demonstrates using the Jira promotion step across multiple stages in a promotion pipeline, tracking a single issue from its creation during testing through its final promotion to production: ```yaml --- apiVersion: kargo.akuity.io/v1alpha1 kind: Stage metadata: name: test namespace: kargo-proj spec: requestedFreight: - origin: kind: Warehouse name: nginx sources: direct: true promotionTemplate: spec: vars: - name: imageRepo value: public.ecr.aws/nginx/nginx steps: # Create initial promotion issue - as: create-promotion-issue uses: jira config: credentials: secretName: jira createIssue: projectKey: PROMOTE issueType: Task summary: "Promote Release ${{ imageFrom(vars.imageRepo).Tag }}" assigneeEmail: "devops@company.com" adfDescription: type: doc version: 1 content: - type: paragraph content: - type: text text: " " - type: heading attrs: level: 3 content: - type: text text: "Automated promotion issue for release " - type: text text: "${{ imageFrom(vars.imageRepo).Tag }}" marks: - type: code - type: paragraph content: - type: text text: "Image:" marks: - type: strong - type: text text: " " - type: text text: "${{ imageFrom(vars.imageRepo).RepoURL }}:${{ imageFrom(vars.imageRepo).Tag }}" marks: - type: code - type: paragraph content: - type: text text: "Project:" marks: - type: strong - type: text text: " " - type: text text: "${{ ctx.project }}" marks: - type: code labels: - "automated-promotion" - "env-${{ ctx.stage }}" - "release-${{ imageFrom(vars.imageRepo).Tag }}" - "project-${{ ctx.project }}" # Update the Argo CD Application directly. Not ideal for practical purposes. - as: update-app uses: argocd-update config: apps: - name: test-app namespace: argocd sources: - repoURL: https://github.com/company/app-config.git kustomize: images: - repoURL: public.ecr.aws/nginx/nginx tag: ${{ imageFrom("public.ecr.aws/nginx/nginx").Tag }} # Add progress comment - as: comment-on-issue uses: jira config: credentials: secretName: jira commentOnIssue: issueKey: ${{ outputs['create-promotion-issue'].key }} body: "Release ${{ imageFrom(vars.imageRepo).Tag }} has been promoted to ${{ ctx.stage }} environment. Freight: ${{ ctx.targetFreight.name }}. Ready for testing." # Cleanup on failure - as: on-failure-cleanup-issue uses: jira if: ${{ failure() }} config: credentials: secretName: jira deleteIssue: issueKey: ${{ outputs['create-promotion-issue'].key }} deleteSubtasks: true --- apiVersion: kargo.akuity.io/v1alpha1 kind: Stage metadata: name: uat namespace: kargo-proj spec: requestedFreight: - origin: kind: Warehouse name: nginx sources: stages: - test promotionTemplate: spec: vars: - name: imageRepo value: public.ecr.aws/nginx/nginx steps: # Wait for manual approval to proceed to UAT - as: wait-approval uses: jira config: credentials: secretName: jira waitForStatus: issueKey: ${{ freightMetadata(ctx.targetFreight.name)['jira-issue-key'] }} expectedStatus: UAT # Update the Argo CD Application directly. Not ideal for practical purposes. - as: update-app uses: argocd-update config: apps: - name: uat-app namespace: argocd sources: - repoURL: https://github.com/company/app-config.git kustomize: images: - repoURL: ${{ vars.imageRepo }} tag: ${{ imageFrom(vars.imageRepo).Tag }} # Update issue with UAT progress - as: comment-on-issue uses: jira config: credentials: secretName: jira commentOnIssue: issueKey: ${{ freightMetadata(ctx.targetFreight.name)['jira-issue-key'] }} body: "Release ${{ imageFrom(vars.imageRepo).Tag }} has been promoted to ${{ ctx.stage }} environment. Promotion: ${{ ctx.promotion }}. Status: Promoteed and ready for uat validation." # Update environment labels - as: update-issue-labels uses: jira config: credentials: secretName: jira updateIssue: issueKey: ${{ freightMetadata(ctx.targetFreight.name)['jira-issue-key'] }} removeLabels: - "env-test" addLabels: - "env-${{ ctx.stage }}" - "promotion-${{ ctx.promotion }}" # Cleanup comments on failure - as: on-failure-cleanup-comment uses: jira if: ${{ failure() && status('comment-on-issue') == 'Succeeded' }} config: credentials: secretName: jira deleteComment: issueKey: ${{ freightMetadata(ctx.targetFreight.name)['jira-issue-key'] }} commentID: ${{ quote(outputs['comment-on-issue'].commentID) }} --- apiVersion: kargo.akuity.io/v1alpha1 kind: Stage metadata: name: prod namespace: kargo-proj spec: requestedFreight: - origin: kind: Warehouse name: nginx sources: stages: - uat promotionTemplate: spec: vars: - name: imageRepo value: public.ecr.aws/nginx/nginx steps: # Find the issue by searching for release label - as: search-issue uses: jira config: credentials: secretName: jira searchIssue: jql: "created <= 1d and labels IN (release-${{ imageFrom(vars.imageRepo).Tag }}) ORDER BY created DESC" expectMultiple: true fields: - key # Wait for final production approval - as: wait-approval uses: jira config: credentials: secretName: jira waitForStatus: issueKey: ${{ outputs['search-issue'].key }} expectedStatus: RELEASED # Update the Argo CD Application directly. Not ideal for practical purposes. - as: update-app uses: argocd-update config: apps: - name: prod-app namespace: argocd sources: - repoURL: https://github.com/company/app-config.git kustomize: images: - repoURL: public.ecr.aws/nginx/nginx tag: ${{ imageFrom("public.ecr.aws/nginx/nginx").Tag }} # Add final completion comment - as: comment-on-issue uses: jira config: credentials: secretName: jira commentOnIssue: issueKey: ${{ outputs['search-issue'].key }} body: "Release ${{ imageFrom(vars.imageRepo).Tag }} has been successfully promoted to ${{ ctx.stage }} environment. promotion completed for promotion ${{ ctx.promotion }}. All systems operational and release is live!" # Update to production labels - as: update-issue-labels uses: jira config: credentials: secretName: jira updateIssue: issueKey: ${{ outputs['search-issue'].key }} removeLabels: - "env-stage" addLabels: - "env-${{ ctx.stage }}" - "released-${{ imageFrom(vars.imageRepo).Tag }}" - "promotion-${{ ctx.promotion }}" # Cleanup on failure - as: on-failure-cleanup-comment uses: jira if: ${{ failure() && status('comment-on-issue') == 'Succeeded' }} config: credentials: secretName: jira deleteComment: issueKey: ${{ outputs['search-issue'].key }} commentID: ${{ quote(outputs['comment-on-issue'].commentID) }} ``` This multi-stage workflow demonstrates: - **Issue Creation**: The `test` stage creates a comprehensive Jira issue with ADF formatting - **Freight Metadata**: The issue key is automatically stored in freight metadata for later stages - **Status Tracking**: Each stage waits for specific approval statuses before proceeding - **Progressive Updates**: Labels and comments are updated as the release moves through environments - **Error Handling**: Cleanup steps run on failures to maintain clean state - **Search Functionality**: The `prod` stage demonstrates finding issues by label when freight metadata isn't available --- ## `json-parse` `json-parse` is a utility step that parses a JSON file and extracts values using [expr-lang] expressions. ## Configuration | Name | Type | Required | Description | |------|------|----------|-------------| | `path` | `string` | Y | Path to a JSON file. This path is relative to the temporary workspace that Kargo provisions for use by the promotion process. | | `outputs` | `[]object` | Y | A list of rules for extracting values from the parsed JSON. | | `outputs[].name` | `string` | Y | The name of the output variable. | | `outputs[].fromExpression` | `string` | Y | An [expr-lang] expression that can extract the value from the JSON file. Note that this expression should not be offset by `${{` and `}}`. See [examples](#examples) for more details. | ## Expressions The `fromExpression` field supports [expr-lang] expressions. :::note Expressions should _not_ be offset by `${{` and `}}` to prevent pre-processing evaluation by Kargo. The `json-parse` step itself will evaluate these expressions. ::: An `outputs` object (a `map[string]any`) is available to these expressions. It is structured as follows: | Field | Type | Description | |-------|------|-------------| | `outputs` | `map[string]any` | The parsed JSON object. | ## Outputs The `json-parse` step produces the outputs described by the `outputs` field in its configuration. ## Examples ### Common Usage In this example, a values file is parsed to find the container image tag. After cloning the repository and clearing the output directory, the `json-parse` step parses `config.json` to extract the image tag from the `Freight` being promoted. Using dot notation (`image.tag`), it extracts the nested value from the JSON file. ```yaml vars: - name: gitRepo value: https://github.com/example/repo.git steps: - uses: git-clone config: repoURL: ${{ vars.gitRepo }} checkout: - commit: ${{ commitFrom(vars.gitRepo).ID }} path: ./src - branch: stage/${{ ctx.stage }} create: true path: ./out - uses: git-clear config: path: ./out - uses: json-parse as: values config: path: './src/config.json' outputs: - name: imageTag fromExpression: image.tag # Render manifests to ./out, commit, push, etc... ``` Given the sample input JSON: ```json { "image": { "tag": "v1.2.3" } } ``` The step would produce the following [outputs](../15-promotion-templates.md#step-outputs): | Name | Type | Value | |------|------|-------| | `imageTag` | `string` | `v1.2.3` | [expr-lang]: https://expr-lang.org --- ## `json-update` `json-update` updates the values of specified keys in any JSON file, in-place, without disruption to existing formatting choices. :::note[Limitations] `json-update` updates scalar values only. ::: ## Configuration | Name | Type | Required | Description | |------|------|----------|-------------| | `path` | `string` | Y | Path to a JSON file. This path is relative to the temporary workspace that Kargo provisions for use by the promotion process. | | `updates` | `[]object` | Y | The details of changes to be applied to the file. At least one must be specified. | | `updates[].key` | `string` | Y | The key to update within the file. For nested values, use a JSON dot notation path. See [sjson documentation](https://github.com/tidwall/sjson?tab=readme-ov-file#path-syntax) for supported syntax. | | `updates[].value`| `any` | Y | The new scalar value for the key. Typically specified using an expression. Supports strings, numbers, and booleans. | ## Output | Name | Type | Description | |------|------|-------------| | `commitMessage` | `string` | A description of the change(s) applied by this step. Typically, a subsequent [`git-commit` step](git-commit.md) will reference this output and aggregate this commit message fragment with others like it to build a comprehensive commit message that describes all changes. | ## Writing Keys **Nested keys:** ```json { "image": { "tag": "v1.0.0" } } ``` Update key: `image.tag` **Keys with literal dots:** ```json { "example.com/version": "v1.0.0" } ``` Update key: `example\.com/version` **Sequences:** ```json { "containers": [ { "name": "my-app", "image": "my-app:v1.0" } ] } ``` Update key: `containers.0.image` :::note See the [sjson path syntax documentation](https://github.com/tidwall/sjson?tab=readme-ov-file#path-syntax) for the full description of the syntax. ::: ## Examples ### Common Usage In this example, a JSON file's values are updated according to changes in a container image tag. After cloning the repository and clearing the output directory, the `json-update` step updates the `image.tag` field in `configs/settings.json` to match the tag of the image being promoted. This pattern is commonly seen when managing configuration files that need to stay synchronized with deployed container versions. ```yaml vars: - name: gitRepo value: https://github.com/example/repo.git steps: - uses: git-clone config: repoURL: ${{ vars.gitRepo }} checkout: - commit: ${{ commitFrom(vars.gitRepo).ID }} path: ./src - branch: stage/${{ ctx.stage }} create: true path: ./out - uses: git-clear config: path: ./out - uses: json-update config: path: ./src/configs/settings.json updates: - key: image.tag value: ${{ imageFrom("my/image").Tag }} # Render manifests to ./out, commit, push, etc... ``` :::info For more information on `imageFrom()` and expressions used in the example above, see the [Expressions](../40-expressions.md#functions) documentation. ::: --- ## `kustomize-build` `kustomize-build` renders manifests from a specified directory containing a `kustomization.yaml` file to a specified file or to many files in a specified directory. This step is useful for the common scenario of rendering Stage-specific manifests to a Stage-specific branch. This step is commonly preceded by a [`git-clear`](git-clear.md) step and followed by [`git-commit`](git-commit.md) and [`git-push`](git-push.md) steps. ## Configuration | Name | Type | Required | Description | |------|------|----------|-------------| | `path` | `string` | Y | Path to a directory containing a `kustomization.yaml` file. This path is relative to the temporary workspace that Kargo provisions for use by the promotion process. | | `outPath` | `string` | Y | Path to the file or directory where rendered manifests are to be written. If the path ends with `.yaml` or `.yml` it is presumed to indicate a file and is otherwise presumed to indicate a directory. This path is relative to the temporary workspace that Kargo provisions for use by the promotion process. | | `outputFormat` | `string` | N | Specifies the naming convention for output files when writing to a directory. `kargo` (default) uses `[namespace-]kind-name.yaml` format (e.g., `deployment-myapp.yaml` or `default-deployment-myapp.yaml`). `kustomize` matches the naming convention of `kustomize build -o dir/`, using `[namespace_]group_version_kind_name.yaml` format (e.g., `apps_v1_deployment_myapp.yaml` or `default_apps_v1_deployment_myapp.yaml`). | | `plugin.helm.apiVersions` | `[]string` | N | Optionally specifies a list of supported API versions to be used when rendering manifests using Kustomize's Helm chart plugin. This is useful for charts that may contain logic specific to different Kubernetes API versions. | | `plugin.helm.kubeVersion` | `string` | N | Optionally specifies a Kubernetes version to be assumed when rendering manifests using Kustomize's Helm chart plugin. This is useful for charts that may contain logic specific to different Kubernetes versions. | ## Examples ### Rendering to a File In this example, Kustomize manifests are rendered to a single output file. After cloning the repository and clearing the output directory, the `kustomize-build` step processes the Kustomize configuration from the stage-specific directory (`./src/stages/${{ ctx.stage }}`) and writes all rendered manifests to a single file at `./out/manifests.yaml`. This approach is useful when you want to maintain all manifests in a single file for easier tracking or when working with tools that expect a consolidated input file. ```yaml vars: - name: gitRepo value: https://github.com/example/repo.git steps: - uses: git-clone config: repoURL: ${{ vars.gitRepo }} checkout: - commit: ${{ commitFrom(vars.gitRepo).ID }} path: ./src - branch: stage/${{ ctx.stage }} create: true path: ./out - uses: git-clear config: path: ./out - uses: kustomize-build config: path: ./src/stages/${{ ctx.stage }} outPath: ./out/manifests.yaml # Commit, push, etc... ``` ### Rendering to a Directory In this example, Kustomize manifests are rendered with output split across multiple files in a directory. Similar to the [previous example](#rendering-to-a-file), it clones the repository and clears the output directory, but instead of specifying a single output file, it directs the `kustomize-build` step to write to the `./out` directory. This means each Kubernetes resource will be written to its own file, maintaining a clear separation between resources. This pattern is useful when you want to maintain distinct files for different resources or when working with tools that expect resources to be in separate files. ```yaml vars: - name: gitRepo value: https://github.com/example/repo.git steps: - uses: git-clone config: repoURL: ${{ vars.gitRepo }} checkout: - commit: ${{ commitFrom(vars.gitRepo).ID }} path: ./src - branch: stage/${{ ctx.stage }} create: true path: ./out - uses: git-clear config: path: ./out - uses: kustomize-build config: path: ./src/stages/${{ ctx.stage }} outPath: ./out # Commit, push, etc... ``` ### Using Kustomize Output Format By default, `kustomize-build` uses the Kargo naming convention for output files (`[namespace-]kind-name.yaml`). If you prefer filenames that match the output of `kustomize build -o dir/`, you can set `outputFormat: kustomize` to use the Kustomize naming convention (`[namespace_]group_version_kind_name.yaml`). ```yaml vars: - name: gitRepo value: https://github.com/example/repo.git steps: - uses: git-clone config: repoURL: ${{ vars.gitRepo }} checkout: - commit: ${{ commitFrom(vars.gitRepo).ID }} path: ./src - branch: stage/${{ ctx.stage }} create: true path: ./out - uses: git-clear config: path: ./out - uses: kustomize-build config: path: ./src/stages/${{ ctx.stage }} outPath: ./out outputFormat: kustomize # Commit, push, etc... ``` --- ## `kustomize-set-image` `kustomize-set-image` updates the `kustomization.yaml` file in a specified directory to reflect a different revision of a container image. It is equivalent to executing `kustomize edit set image`. This step is commonly followed by a [`kustomize-build` step](kustomize-build). ## Configuration | Name | Type | Required | Description | |------|------|----------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `path` | `string` | Y | Path to a directory containing a `kustomization.yaml` file. This path is relative to the temporary workspace that Kargo provisions for use by the promotion process. | | `images` | `[]object` | N | The details of changes to be applied to the `kustomization.yaml` file. When left unspecified, all images from the Freight collection will be set in the Kustomization file. Unless there is an ambiguous image name (for example, due to two Warehouses subscribing to the same repository), which requires manual configuration. | | `images[].image` | `string` | Y | Name/URL of the image being updated. | | `images[].tag` | `string` | N | A tag naming a specific revision of `image`. Mutually exclusive with `digest`. This field can be specified directly or through an [expression](../../60-reference-docs/40-expressions.md#imagefrom). Either `tag` or `digest` must be provided, but not both. | | `images[].digest` | `string` | N | A digest naming a specific revision of `image`. Mutually exclusive with `tag`. This field can be specified directly or through an [expression](../../60-reference-docs/40-expressions.md#imagefrom). Either `digest` or `tag` must be provided, but not both. | | `images[].newName` | `string` | N | A substitution for the name/URL of the image being updated. This is useful when different Stages have access to different container image repositories (assuming those different repositories contain equivalent images that are tagged identically). This may be a frequent consideration for users of Amazon's Elastic Container Registry. | ## Output | Name | Type | Description | |------|------|-------------| | `commitMessage` | `string` | A description of the change(s) applied by this step. Typically, a subsequent [`git-commit` step](git-commit.md) will reference this output and aggregate this commit message fragment with others like it to build a comprehensive commit message that describes all changes. | ## Examples ### Common Usage In this example, a Kustomize overlay is updated to use a new container image tag. After cloning the repository and clearing the output directory, the `kustomize-set-image` step modifies the `kustomization.yaml` file to use the image tag from the Freight being promoted. The image reference is parameterized using variables to make the configuration more maintainable. This pattern is commonly used when you want to update the version of a container image that will be deployed to a specific stage. :::info For more information on `imageFrom` and expressions, see the [Expressions](../40-expressions.md#functions) documentation. ::: ```yaml vars: - name: gitRepo value: https://github.com/example/repo.git - name: imageRepo value: my/image steps: - uses: git-clone config: repoURL: ${{ vars.gitRepo }} checkout: - commit: ${{ commitFrom(vars.gitRepo).ID }} path: ./src - branch: stage/${{ ctx.stage }} create: true path: ./out - uses: git-clear config: path: ./out - uses: kustomize-set-image config: path: ./src/base images: - image: ${{ vars.imageRepo }} tag: ${{ imageFrom(vars.imageRepo).Tag }} # Render manifests to ./out, commit, push, etc... ``` ### Changing an Image Name For this example, consider the promotion of Freight containing a reference to some revision of the container image `123456789012.dkr.ecr.us-east-1.amazonaws.com/my-image`. This image exists in the `us-east-1` region of Amazon's Elastic Container Registry. However, assuming the Stage targeted by the promotion is backed by environments in the `us-west-2` region, it will be necessary to make a substitution when updating the `kustomization.yaml` file. This can be accomplished like so: ```yaml vars: - name: gitRepo value: https://github.com/example/repo.git steps: - uses: git-clone config: repoURL: ${{ vars.gitRepo }} checkout: - commit: ${{ commitFrom(vars.gitRepo).ID }} path: ./src - branch: stage/${{ ctx.stage }} create: true path: ./out - uses: git-clear config: path: ./out - uses: kustomize-set-image config: path: ./src/base images: - image: 123456789012.dkr.ecr.us-east-1.amazonaws.com/my-image newName: 123456789012.dkr.ecr.us-west-2.amazonaws.com/my-image # Render manifests to ./out, commit, push, etc... ``` --- ## `oci-download` `oci-download` downloads OCI artifacts from a registry and extracts layer content to a specified file. This step is useful for downloading artifacts like Helm charts, configuration files, or other resources packaged as OCI artifacts. The step supports authentication and can target specific layers by media type. :::note Downloads are limited to 100MB to prevent resource exhaustion. ::: ## Configuration | Name | Type | Required | Description | |------|------|----------|-------------| | `imageRef` | `string` | Y | Reference to the OCI artifact to download. Supports both tag format `registry/repository:tag` and digest format `registry/repository@sha256:digest`. For Helm OCI artifacts, the `oci://` prefix is supported (e.g., `oci://registry/repository:tag`) and will use Helm-specific credential lookup. | | `outPath` | `string` | Y | Path to the destination file where the extracted artifact will be saved. This path is relative to the temporary workspace that Kargo provisions for use by the promotion process. | | `allowOverwrite` | `boolean` | N | Whether to allow overwriting an existing file at the specified path. If `false` and the file exists, the download will fail. Defaults to `false`. | | `mediaType` | `string` | N | Media type of the layer to download. Selects the first layer matching this type. If not specified, selects the first layer available. | | `insecureSkipTLSVerify` | `boolean` | N | Whether to skip TLS verification when downloading the artifact. Defaults to `false`. | ## Examples ### Downloading a Helm Chart In this example, an OCI-packaged Helm chart is downloaded from a registry and saved to a local file. This is useful when you need to download charts that are distributed as OCI artifacts. ```yaml steps: - uses: oci-download config: imageRef: registry.example.com/charts/my-app:1.0.0 outPath: ./charts/my-app-1.0.0.tgz ``` ### Downloading a Helm Chart with OCI Protocol This example shows downloading a Helm chart using the `oci://` prefix, which ensures that [Helm-specific credentials](../../50-security/30-managing-secrets.md) are used for authentication. ```yaml steps: - uses: oci-download config: imageRef: oci://registry.example.com/charts/my-app:1.0.0 outPath: ./charts/my-app-1.0.0.tgz ``` ### Downloading Configuration Files In this example, configuration files packaged as an OCI artifact are downloaded and extracted. The step downloads the first available layer since no specific media type is specified. ```yaml steps: - uses: oci-download config: imageRef: registry.example.com/configs/app-config@sha256:abc123def456789 outPath: ./config/app-config.yaml ``` ### Downloading with Digest Reference In this example, an artifact is downloaded using a digest reference for immutable content addressing. This ensures you get exactly the same content every time, regardless of tag mutations. ```yaml steps: - uses: oci-download config: imageRef: ghcr.io/example/artifacts@sha256:1234567890abcdef outPath: ./artifacts/data.tar.gz ``` ### Downloading with a Specific Media Type In this example, an artifact is downloaded by specifying a media type. This is useful when the OCI artifact contains multiple layers, and you want to target a specific one, such as a configuration file attached to a container image. ```yaml steps: - uses: oci-download config: imageRef: registry.example.com/artifacts/my-app:v1.2.3 outPath: ./artifacts/config.json mediaType: application/vnd.example.config.v1+json ``` ### Downloading with TLS Verification Disabled In this example, an artifact is downloaded from a registry with self-signed certificates by disabling TLS verification. This should only be used in development or testing environments where the registry is trusted. ```yaml steps: - uses: oci-download config: imageRef: internal-registry.local/artifacts/data:latest outPath: ./data/artifact.tar.gz insecureSkipTLSVerify: true ``` ### Downloading and Rendering Helm Charts This example shows how `oci-download` can be combined with [`helm-template`](helm-template.md) to download Helm charts from OCI registries and render them to manifests. After downloading the chart archive, it's rendered directly with Stage-specific values before being committed to a Git repository. ```yaml vars: - name: gitRepo value: https://github.com/example/manifests.git - name: chart value: oci://registry.example.com/charts/my-app steps: - uses: git-clone config: repoURL: ${{ vars.gitRepo }} checkout: - branch: stage/${{ ctx.stage }} create: true path: ./out - uses: git-clear config: path: ./out - uses: oci-download config: imageRef: ${{ vars.chart }}:1.0.0 outPath: ./chart.tgz - uses: helm-template config: path: ./chart.tgz releaseName: my-app namespace: ${{ ctx.stage }} outPath: ./out - uses: git-commit config: path: ./out message: "Update manifests for ${{ ctx.stage }} stage" - uses: git-push config: path: ./out ``` --- ## `oci-push` `oci-push` copies or retags OCI artifacts between registries or within the same registry. This step supports container images and Helm charts stored in OCI registries, making it useful for promoting artifacts through a pipeline — for example, retagging an image with a release version or copying it to a production registry. Multi-arch image indexes are copied in full. Registry authentication is supported for both source and destination. ## Configuration | Name | Type | Required | Description | |------|------|----------|-------------| | `srcRef` | `string` | Y | Reference to the source OCI artifact. Supports both tag format `registry/repository:tag` and digest format `registry/repository@sha256:digest`. For Helm OCI artifacts, the `oci://` prefix is supported (e.g., `oci://registry/repository:tag`) and will use Helm-specific credential lookup. | | `destRef` | `string` | Y | Destination reference including tag (e.g., `registry/repository:tag`). For Helm OCI artifacts, the `oci://` prefix is supported. For retag-in-place, use the same repository as `srcRef` with the new tag. | | `annotations` | `object` | N | Annotations to set on the destination artifact. Keys may be prefixed with `index:` or `manifest:` to scope them to the index or image manifest respectively. Unprefixed keys default to the image manifest. For single images, `index:`-prefixed keys are ignored. Values support expressions. Existing annotations on the source artifact are preserved; specified annotations are added or overwritten. | | `insecureSkipTLSVerify` | `boolean` | N | Whether to skip TLS verification for both source and destination registries. Defaults to `false`. | ## Outputs | Name | Type | Description | |------|------|-------------| | `image` | `string` | Full destination reference with tag (e.g., `prod.example.com/myapp:v1.2.3`). | | `digest` | `string` | Digest of the pushed artifact (e.g., `sha256:abc123...`). | | `tag` | `string` | Tag that was applied, parsed from `destRef`. | ## Limits The total compressed size of the artifact (config blob and all layers) must not exceed 1 GiB (or as configured by your administrator). For multi-arch image indexes, this includes the sum across all child images. Exceeding this limit causes a terminal (non-retryable) error. This limit is not enforced when `srcRef` and `destRef` refer to the same repository (i.e. retagging within the same registry and path), since no blob transfer occurs in that case. ## Examples ### Retagging an Image with a Release Version In this example, a dedicated "release" Stage sits downstream from a testing Stage. When verified Freight is promoted into this Stage, its single step retags the image with a semver release version in the same repository. Because the source and destination are in the same repository, this is a lightweight metadata operation with no blob transfer. ```yaml steps: - uses: oci-push config: srcRef: registry.example.com/myapp@${{ imageFrom("registry.example.com/myapp").digest }} destRef: registry.example.com/myapp:v1.2.3 ``` ### Copying to a Production Registry In this example, a verified image is copied from a sandbox registry to a production registry, preserving its original tag. Credentials for each registry are resolved independently. ```yaml steps: - uses: oci-push config: srcRef: sandbox.example.com/myapp@${{ imageFrom("sandbox.example.com/myapp").digest }} destRef: prod.example.com/myapp:${{ imageFrom("sandbox.example.com/myapp").tag }} ``` ### Copying to a Per-Stage Repository In this example, images are copied to stage-specific repositories (e.g., for garbage collection policies that limit images per repository). The step output is then used by `kustomize-set-image` to update the deployment manifest. ```yaml steps: - uses: oci-push as: push-to-stage config: srcRef: registry.example.com/widget-service@${{ imageFrom("registry.example.com/widget-service").digest }} destRef: registry.example.com/widget-service/${{ ctx.stage }}:${{ imageFrom("registry.example.com/widget-service").tag }} - uses: kustomize-set-image config: path: ./out images: - image: registry.example.com/widget-service newName: registry.example.com/widget-service/${{ ctx.stage }} digest: ${{ outputs["push-to-stage"].digest }} ``` ### Promoting an OCI Helm Chart In this example, a Helm chart stored in an OCI registry is copied to a production registry. The `oci://` prefix ensures Helm-specific [credentials](../../50-security/30-managing-secrets.md) are used for authentication. ```yaml steps: - uses: oci-push config: srcRef: oci://registry.example.com/charts/my-app:${{ chartFrom("oci://registry.example.com/charts/my-app").version }} destRef: oci://prod-registry.example.com/charts/my-app:${{ chartFrom("oci://registry.example.com/charts/my-app").version }} ``` ### Adding Annotations In this example, OCI annotations are stamped onto the destination manifest during the push. This can be used to record provenance metadata such as the source repository or Kargo promotion name. ```yaml steps: - uses: oci-push config: srcRef: registry.example.com/myapp@${{ imageFrom("registry.example.com/myapp").digest }} destRef: registry.example.com/myapp:v1.2.3 annotations: org.opencontainers.image.source: "https://github.com/example/myapp" io.kargo.promotion: ${{ ctx.promotion }} ``` ### Scoped Annotations for Multi-Arch Images When pushing image indexes (multi-arch), annotation keys can be prefixed with `index:` or `manifest:` to control where they are applied. Unprefixed keys default to the image manifest. ```yaml steps: - uses: oci-push config: srcRef: registry.example.com/myapp@${{ imageFrom("registry.example.com/myapp").digest }} destRef: registry.example.com/myapp:v1.2.3 annotations: org.opencontainers.image.source: "https://github.com/example/myapp" index:org.opencontainers.image.revision: ${{ commitFrom("https://github.com/example/myapp").id }} manifest:org.opencontainers.image.description: "my app image" ``` ### Copying with TLS Verification Disabled In this example, an artifact is copied between registries with self-signed certificates by disabling TLS verification. This should only be used in development or testing environments where the registries are trusted. ```yaml steps: - uses: oci-push config: srcRef: internal-registry.local/myapp:latest destRef: staging-registry.local/myapp:latest insecureSkipTLSVerify: true ``` --- ## `send-message` :::info This promotion step is only available in Kargo on the [Akuity Platform](https://akuity.io/akuity-platform), versions v1.8 and above. ::: The `send-message` step allows you to send messages to various destinations, such as Slack channels or email addresses, during the promotion process. This can be useful for notifying team members about promotion events, approvals, or other important information. This step is useful when you want to imperatively send a notification as part of your promotion workflow. For an event-driven option that can send notifications based on specific events, consider using the [Notifications feature](../90-events/100-notifications/index.md). :::note This step supports `MessageChannel` and `ClusterMessageChannel` resources configured with Slack or SMTP. HTTP channels are only supported by [`EventRouter`s](../90-events/100-notifications/10-configuring-routers.md). To make HTTP requests as part of a promotion workflow, use the [`http`](./http.md) promotion step instead. ::: This feature is evolving quickly and more configuration options and destinations will continue to be added in future releases. ## Configuration | Name | Type | Required | Description | | ----------------- | --------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `channel` | `object` | Y | Configuration for the destination channel where the message will be sent. | | `channel.kind` | `string` | Y | The type of [channel](../../20-how-to-guides/20-working-with-projects.md#message-channels) to send the message to. Supported values are `MessageChannel` and `ClusterMessageChannel` | | `channel.name` | `string` | Y | The name of the [channel](../../20-how-to-guides/20-working-with-projects.md#message-channels) to send the message to. This should correspond to a configured channel in your Kargo instance. | | `message` | `string` | Y | The content of the message to be sent. This can include plain text or [richly formatted content](#sending-richly-formatted-messages). | | `encodingType` | `string` | N | The encoding format of the message. Used when sending [richly formatted messages](#sending-richly-formatted-messages). If omitted or an empty string, it means the message is plaintext and should be used verbatim. Supported values are `json`, `yaml`, and `xml`. | | `slack` | `object` | N | Optional configuration specific to Slack messages. | | `slack.channelID` | `string` | N | The ID of the Slack channel to send the message to. If omitted, the default channel configured in the Slack channel configuration will be used. | | `slack.threadTS` | `string` | N | The timestamp of the Slack thread to send the message in. If omitted, the message will be sent as a new message in the channel. This can be set by using `set-metadata` and `freightMetadata()` to ensure you set and fetch the timestamp according to some criteria (such as freight ID) | | `smtp` | `object` | N | Optional configuration specific to email messages. | | `smtp.to` | `string` | N | The email address to send the message to. If omitted, the default recipient configured in the SMTP channel configuration will be used. | | `smtp.subject` | `string` | N | The subject line for the email. If omitted, a default subject will be used. | | `smtp.html` | `boolean` | N | Whether the email body should be sent as HTML. If `true`, the message will be treated as HTML content. If `false` or omitted, the message will be treated as plain text. | ### Sending richly formatted messages By default, the `send-message` step will use the given message and send a plain formatting message to the configured channel. However, in some cases you may want to send a richly formatted message, such as a Slack message with [blocks](https://docs.slack.dev/block-kit/). To do this, you can use the `encodingType` field to specify the format of the message. The supported structures for each message type can be found in the [Notification documentation](../90-events/100-notifications/20-message-formatting.md). See the example section below for an example of sending a richly formatted Slack message. :::warning If `encodingType` is set for a message, it assumes that all configuration options will be passed in the encoded body passed as the `message` field (see [the Slack example below](#sending-a-richly-formatted-slack-message)) rather than using the config specific fields such as `slack.channelID` or `smtp.html`. Any options that are set in the `config` block other than `message` and `encodingType` will be ignored. ::: Please note that if you use `json` encoding as a message format, you must ensure that the JSON is properly quoted. The step runner assumes that any JSON content should be decoded, so unquoted JSON will result in an error. This can be done like so: ```yaml steps: - uses: send-message config: channel: kind: MessageChannel name: smtp encodingType: json message: | ${{ quote({ "subject": "🚀 Deployment Promotion Started", "body": "Kargo has kicked off promotion to stage: " + ctx.stage, "to": [ "email@example.com" ], "html": false }) }} ``` ## Outputs The `send-message` step returns different outputs depending on the type of channel used. Each output is nested under a key corresponding to the channel kind (e.g., `slack`, `smtp`). ### Slack Output When sending a message to a Slack channel, the output will include the following fields: | Name | Type | Description | | ---------- | ------ | ----------------------------------------------------------------------------------------------------------------------------------- | | `threadTS` | string | When starting a new thread, the timestamp of the newly created message. When replying to an existing thread, the provided threadTS. | ### SMTP Output Currently SMTP messages do not return any specific outputs. This may change in future releases. ## Examples ### Sending a plain text message to Slack ```yaml steps: - if: ${{ always() }} uses: send-message config: channel: kind: MessageChannel name: slack message: "Kargo promotion to ${{ ctx.stage }} ${{ success() ? 'succeeded' : 'failed' }}" ``` ### Overriding the Slack channel ID ```yaml steps: - uses: send-message config: channel: kind: MessageChannel name: slack message: "Kargo has kicked off promotion to stage: ${{ ctx.stage }}." slack: channelID: C1234567890 ``` ### Sending an email with all custom options ```yaml steps: - uses: send-message config: channel: kind: MessageChannel name: smtp message: "Kargo has kicked off promotion to stage: ${{ ctx.stage }}." smtp: to: [foo@bar.com] subject: "🚀 Deployment Promotion Started" html: true ``` ### Sending a richly formatted Slack message This shows an example of using inline YAML to send a richly formatted Slack message with blocks. The `encodingType` is set to `yaml` to indicate that the message content is in YAML format. Note that the `channel` is also set in the body as all to override the default channel configured in the Slack channel configuration. ```yaml steps: - uses: send-message config: channel: kind: MessageChannel name: slack message: | blocks: - type: section text: type: mrkdwn text: "*Kargo Promotion Started* :rocket:" - type: section fields: - type: mrkdwn text: "*Stage:*\n${{ ctx.stage }}" - type: mrkdwn text: "*Time:*\n${{ ctx.timestamp }}" icon_emoji: ":fire:" channel: C1234567890 encodingType: yaml ``` ### Sending a message to a thread in Slack This shows an example of sending a message to a specific thread in Slack by using the `threadTS` field. It leverages the `set-metadata` step to store and retrieve the thread timestamp associated with a specific freight. This allows messages related to the same freight to be grouped together in a single thread. ```yaml steps: - uses: send-message as: send-slack config: channel: kind: MessageChannel name: slack message: "Kargo has kicked off promotion to stage: ${{ ctx.stage }}." slack: # NOTE: Make sure to wrap this in quotes with `quote` otherwise it will get rendered as # a number threadTS: "${{ quote(freightMetadata(ctx.targetFreight.name)?.slackThreadTS ?? '') }}" - uses: set-metadata config: updates: - kind: Freight name: ${{ ctx.targetFreight.name }} values: # NOTE: Make sure to wrap this in quotes with `quote` otherwise it will get rendered # as a number slackThreadTS: "${{ quote(outputs['send-slack']?.slack?.threadTS ?? '') }}" ``` --- ## `set-freight-alias` `set-freight-alias` updates the alias of a `Freight` resource. When a `Warehouse` produces a new `Freight` resource, it computes a SHA-1 hash from a canonical representation of the artifacts referenced by that resource. This unique "fingerprint" for that collection of artifacts becomes the value of the `metadata.name`. Because this is not a very human-friendly name, every `Freight` resource is also automatically assigned a human-friendly alias. Users may sometimes wish to update aliases at various points in their pipelines. This can make it easier to identify a particularly important (or problematic) `Freight` resource as it progresses through the `Stage`s of a pipeline. ## Configuration | Name | Type | Required | Description | |---------|----------|----------|-------------------------------------------------------------| | `name` | `string` | Y | The name of a `Freight` resource to update. | | `alias` | `string` | Y | The desired new alias to set on the `Freight`. | :::note Alias uniqueness is enforced by the Kargo API. If the requested alias is already in use by another `Freight` resource in the project, this step will fail. ::: ## Examples In this example, a `Freight` resource that references only a single container image and no other artifacts is updated to reflect the image's tag. This makes it easier to identify such resources and make inferences about their payload. :::caution This example is safe only because the `Freight` resource contains only a single artifact. If that were not the case, contention over a single alias would arise when two or more Freight resources referenced the same version of the container image. ::: ```yaml steps: - uses: set-freight-alias config: name: ${{ ctx.targetFreight.name }} alias: ${{ imageFrom('some/repo').Tag }} ``` --- ## `set-metadata` `set-metadata` updates metadata on `Stage` or `Freight` resources during the promotion process. This step allows you to attach arbitrary key/value pairs to the status of these resources, which can be useful for tracking promotion state, timing, version details, or any other relevant metadata. ## Configuration | Name | Type | Required | Description | |------|------|----------|-------------| | `updates` | `[]object` | Y | List of metadata updates to apply. | | `updates[].kind` | `string` | Y | Kind of resource to update metadata for. Must be either `Stage` or `Freight`. | | `updates[].name` | `string` | Y | Name of the resource to update metadata for. | | `updates[].values` | `object` | Y | Key/value pairs to set as metadata on the resource. Must contain at least one key/value pair. This field supports various types of values including strings, numbers, booleans, arrays, and nested objects. | ## Examples ### Common Usage This example shows how to add simple metadata to both a `Stage` and a `Freight` resource: ```yaml steps: - uses: set-metadata config: updates: - kind: Stage name: production values: foo: "hello" bar: 42 - kind: Freight name: my-app-freight values: baz: true qux: ["a", "b", "c"] ``` ### Complex Metadata This example demonstrates more complex metadata structures, including nested objects and arrays: ```yaml steps: - uses: set-metadata config: updates: - kind: Stage name: staging values: foo: nested: "value" numbers: [1, 2, 3] items: - name: "item1" value: "abc" - name: "item2" value: "xyz" bar: alpha: "one" beta: "two" ``` ### Using Metadata Once metadata is set (as shown in the examples above), it can be retrieved using [`freightMetadata()`](../40-expressions.md#freightmetadatafreightname) and [`stageMetadata()`](../40-expressions.md#stagemetadatastagename) functions available within expressions used in other promotion steps or in verification processes. --- ## `snow-create` :::info This promotion step is only available in Kargo on the [Akuity Platform](https://akuity.io/akuity-platform), versions v1.9.0 and above. ::: This page is for `snow-create` step. You can create a ServiceNow record using this step. Check below for other ServiceNow promotion steps. ServiceNow integration for kargo is a group of promotion steps: 1. [snow-create](./snow-create.md) 2. [snow-update](./snow-update.md) 3. [snow-delete](./snow-delete.md) 4. [snow-query-for-records](./snow-query-for-records.md) 5. [snow-wait-for-condition](./snow-wait-for-condition.md) These steps provide a comprehensive integration with ServiceNow, allowing you to create, update, delete, query for records, and track promotion workflows. This is particularly useful for maintaining traceability between your promotion processes and project management activities. All the steps above support different record types in ServiceNow including managing Change Requests, Incidents, Problems etc.,. This integration also provides condition tracking through `snow-wait-for-condition`, making it a powerful tool for promotion workflows that require coordination with project management systems. :::info This step supports [Change Management API](https://www.servicenow.com/docs/bundle/zurich-api-reference/page/integrate/inbound-rest/concept/change-management-api.html) for managing Change Requests and [Table API](https://www.servicenow.com/docs/bundle/zurich-api-reference/page/integrate/inbound-rest/concept/c_TableAPI.html) for all record types. ::: ## Finding field values for step configuration Most of the time you cannot use the field labels you see in the ServiceNow UI as keys in the step configuration. For example, if you want to set the value for the “Short description” field: ![](./images/snow-short-des.png) You can't use `Short description` in the step configuration parameters. You need to use `short_description` as the key. To find the correct key for a field, right-click on the field and click `Configure Dictionary`: ![](./images/snow-configure-dict.png) `column_name` is the key: ![](./images/snow-col-name.png) To see choices available for a particular field e.g., `State`, right click on the field and click on `Show Choice List` to see the available choices for use in the steps: ![](./images/show-choices.png) For example `New` is `-1` and `Scheduled` is `-2` ![](./images/choice-list.png) ## Credentials All ServiceNow operations require proper authentication credentials stored in a Kubernetes `Secret`. | Name | Type | Required | Description | | ------------------------ | -------- | -------- | ----------------------------------------------------------------------------------------------- | | `credentials.secretName` | `string` | N | Name of the `Secret` containing the ServiceNow credentials in the project namespace. | | `credentials.sharedSecretName` | `string` | N | Name of the `Secret` containing the ServiceNow credentials in the `shared-resources-namespace`. | | `credentials.type` | `string` | Y | Type of ServiceNow credentials to use for authentication (either `api-token` or `basic`). | :::info Either `credentials.secretName` or `credentials.sharedSecretName` must be set, but not both. ::: For `credentials.type: api-token` the referenced `Secret` should contain the following keys: - `apiToken`: ServiceNow API Token (see [this blog post](https://www.servicenow.com/community/developer-advocate-blog/inbound-rest-api-keys/ba-p/2854924) for how to create an API token in ServiceNow). - `instanceURL`: Your ServiceNow instance URL. ![](./images/snow-instance-url.png) For `credentials.type: basic` the referenced `Secret` should contain the following keys: - `username`: Username of the ServiceNow user (you may want to [create a user](https://www.servicenow.com/docs/bundle/zurich-platform-administration/page/administer/users-and-groups/task/t_CreateAUser.html) specifically for this integration). - `password`: Password of the ServiceNow user (for how to set the password for a user, see [this](https://www.servicenow.com/docs/bundle/zurich-platform-security/page/integrate/authentication/task/reset-your-password.html)). - `instanceURL`: Your ServiceNow instance URL. ## Configuration | Name | Type | Required | Description | | -------------------- | ------------ | -------- | ------------------------------------------------------------------------------------------------------------------------------- | | `parameters` | `string map` | Y | Parameters/fields of the record. | | `tableName` | `string` | Y | Table name of the record type you want to create (e.g., `incident`, `problem`, `change_request`). | | `template` | `object` | N | Specify this and the subfields below to create a Change Request using the Change Management API (if omitted, the Table API is used). | | `template.type` | `string` | Y/N | Template type (`standard`, `emergency`, or `normal`). Required if `template` is specified; otherwise optional. | | `template.templateId`| `string` | Y/N | Template ID of the standard template (required if `template.type` is `standard`; otherwise optional). | ## Output | Name | Type | Description | | -------- | -------- | ----------------------------------------------------------------------------- | | `sys_id` | `string` | The `sys_id` of the created ServiceNow record (e.g., `ed89b72c83c172104517e470ceaad30a`). | | `number` | `string` | The `number` of the created ServiceNow record (e.g., `CH-123`). | ## Example This example creates a new Change Request using the Change Management API: ```yaml steps: - as: snowcreate config: credentials: secretName: snow-creds type: api-token parameters: impact: "3" short_description: Deploy to ${{ ctx.stage }} complete for ${{ vars.imageRepo }}:${{ imageFrom(vars.imageRepo).Tag }} urgency: "3" tableName: change_request template: templateId: ed89b72c83c172104517e470ceaad30a # example type: standard uses: snow-create # Use the created snow record number in subsequent steps - as: snowquery config: credentials: namespace: kargo-demo secretName: snow-creds type: api-token query: number: ${{ task.outputs.snowcreate.number }} tableName: change_request uses: snow-query-records ``` This example creates a new Incident using the Table API: ```yaml steps: - as: snowcreate config: credentials: secretName: snow-creds type: api-token parameters: impact: "3" short_description: Deploy to ${{ ctx.stage }} complete for ${{ vars.imageRepo }}:${{ imageFrom(vars.imageRepo).Tag }} urgency: "3" tableName: incident uses: snow-create # Use the created snow record number in subsequent steps - as: snowquery config: credentials: namespace: kargo-demo secretName: snow-creds type: api-token query: number: ${{ task.outputs.snowcreate.number }} tableName: incident uses: snow-query-records ``` This example creates a new Change Request using the Table API (notice how the `template` field is omitted): ```yaml steps: - as: snowcreate config: credentials: secretName: snow-creds type: api-token parameters: impact: "3" short_description: Deploy to ${{ ctx.stage }} complete for ${{ vars.imageRepo }}:${{ imageFrom(vars.imageRepo).Tag }} urgency: "3" tableName: change_request uses: snow-create # Use the created snow record number in subsequent steps - as: snowquery config: credentials: namespace: kargo-demo secretName: snow-creds type: api-token query: number: ${{ task.outputs.snowcreate.number }} tableName: change_request uses: snow-query-records ``` ## Different E2E Workflows These examples demonstrate the different steps supported for ServiceNow integration. ### Change API Workflow ```yaml - as: snowcreate config: credentials: secretName: snow-creds type: api-token parameters: impact: "3" short_description: Deploy to ${{ ctx.stage }} complete for ${{ vars.imageRepo }}:${{ imageFrom(vars.imageRepo).Tag }} urgency: "3" tableName: change_request template: templateId: ed89b72c83c172104517e470ceaad30a type: standard uses: snow-create - as: snowquery config: credentials: namespace: kargo-demo secretName: snow-creds type: api-token query: number: ${{ task.outputs.snowcreate.number }} tableName: change_request uses: snow-query-records - as: snowupdate config: credentials: secretName: snow-creds type: api-token parameters: short_description: Update deployment for ${{ vars.imageRepo }}:${{ imageFrom(vars.imageRepo).Tag }} to resolved tableName: change_request template: type: standard ticketId: ${{ task.outputs.snowquery.sys_id }} uses: snow-update - as: snow-wait-for-condition config: condition: state=-2 credentials: namespace: kargo-demo secretName: snow-creds type: api-token tableName: change_request ticketId: ${{ task.outputs.snowquery.sys_id }} uses: snow-wait-for-condition - as: snowdelete config: credentials: secretName: snow-creds type: api-token tableName: change_request template: type: standard ticketId: ${{ task.outputs.snowquery.sys_id }} uses: snow-delete ``` ### Table API Workflow Here's a Table API workflow with Change Request: ```yaml - as: snowcreate config: credentials: secretName: snow-creds type: api-token parameters: impact: "3" short_description: Deploy to ${{ ctx.stage }} complete for ${{ vars.imageRepo }}:${{ imageFrom(vars.imageRepo).Tag }} urgency: "3" tableName: change_request uses: snow-create - as: snowquery config: credentials: namespace: kargo-demo secretName: snow-creds type: api-token query: number: ${{ task.outputs.snowcreate.number }} tableName: change_request uses: snow-query-records - as: snowupdate config: credentials: secretName: snow-creds type: api-token parameters: short_description: Update deployment incident for ${{ vars.imageRepo }}:${{ imageFrom(vars.imageRepo).Tag }} to resolved tableName: change_request ticketId: ${{ task.outputs.snowquery.sys_id }} uses: snow-update - as: snow-wait-for-condition config: condition: state=-2 credentials: namespace: kargo-demo secretName: snow-creds type: api-token tableName: change_request ticketId: ${{ task.outputs.snowquery.sys_id }} uses: snow-wait-for-condition - as: snowdelete config: credentials: secretName: snow-creds type: api-token tableName: change_request ticketId: ${{ task.outputs.snowquery.sys_id }} uses: snow-delete ``` --- ## `snow-delete` :::info This promotion step is only available in Kargo on the [Akuity Platform](https://akuity.io/akuity-platform), versions v1.9.0 and above. ::: This page is for `snow-delete` step. You can delete a ServiceNow record using this step. Check below for other ServiceNow promotion steps. ServiceNow integration for kargo is a group of promotion steps: 1. [snow-create](./snow-create.md) 2. [snow-update](./snow-update.md) 3. [snow-delete](./snow-delete.md) 4. [snow-query-for-records](./snow-query-for-records.md) 5. [snow-wait-for-condition](./snow-wait-for-condition.md) These steps provide a comprehensive integration with ServiceNow, allowing you to create, update, delete, query for records, and track promotion workflows. This is particularly useful for maintaining traceability between your promotion processes and project management activities. All the steps above support different record types in ServiceNow including managing Change Requests, Incidents, Problems etc.,. This integration also provides condition tracking through `snow-wait-for-condition`, making it a powerful tool for promotion workflows that require coordination with project management systems. ## Finding field values for step configuration Most of the time you cannot use the field labels you see in the ServiceNow UI as keys in the step configuration. For example, if you want to set the value for the “Short description” field: ![](./images/snow-short-des.png) You can't use `Short description` in the step configuration parameters. You need to use `short_description` as the key. To find the correct key for a field, right-click on the field and click `Configure Dictionary`: ![](./images/snow-configure-dict.png) `column_name` is the key: ![](./images/snow-col-name.png) To see choices available for a particular field e.g., `State`, right click on the field and click on `Show Choice List` to see the available choices for use in the steps: ![](./images/show-choices.png) For example `New` is `-1` and `Scheduled` is `-2` ![](./images/choice-list.png) ## Credentials All ServiceNow operations require proper authentication credentials stored in a Kubernetes `Secret`. | Name | Type | Required | Description | | ------------------------ | -------- | -------- | ----------------------------------------------------------------------------------------------- | | `credentials.secretName` | `string` | N | Name of the `Secret` containing the ServiceNow credentials in the project namespace. | | `credentials.sharedSecretName` | `string` | N | Name of the `Secret` containing the ServiceNow credentials in the `shared-resources-namespace`. | | `credentials.type` | `string` | Y | Type of ServiceNow credentials to use for authentication (either `api-token` or `basic`). | :::info Either `credentials.secretName` or `credentials.sharedSecretName` must be set, but not both. ::: For `credentials.type: api-token` the referenced `Secret` should contain the following keys: - `apiToken`: ServiceNow API Token (see [this blog post](https://www.servicenow.com/community/developer-advocate-blog/inbound-rest-api-keys/ba-p/2854924) for how to create an API token in ServiceNow). - `instanceURL`: Your ServiceNow instance URL. ![](./images/snow-instance-url.png) For `credentials.type: basic` the referenced `Secret` should contain the following keys: - `username`: Username of the ServiceNow user (you may want to [create a user](https://www.servicenow.com/docs/bundle/zurich-platform-administration/page/administer/users-and-groups/task/t_CreateAUser.html) specifically for this integration). - `password`: Password of the ServiceNow user (for how to set the password for a user, see [this](https://www.servicenow.com/docs/bundle/zurich-platform-security/page/integrate/authentication/task/reset-your-password.html)). - `instanceURL`: Your ServiceNow instance URL. ## Configuration | Name | Type | Required | Description | | --------------- | -------- | -------- | ------------------------------------------------------------------------------------------------------------------------------- | | `tableName` | `string` | Y | Table name (e.g., `incident`, `problem`, `change_request`). | | `ticketId` | `string` | Y | Record `sys_id`. | | `template` | `object` | N | Use Change Management API for Change Requests; otherwise omit. | | `template.type` | `string` | Y/N | Required if `template` is specified (`standard`, `emergency`, `normal`). | ## Output This step does not produce output. ## Examples This example deletes a Change Request using the Change Management API (notice how `template` is specified): ```yaml steps: - as: snowdelete config: credentials: secretName: snow-creds type: api-token tableName: change_request template: type: standard ticketId: d457fbac6112287007379b57c6b2e60 # example uses: snow-delete ``` This example deletes an Incident using the Table API (notice how the `template` field is omitted): ```yaml steps: - as: snowdelete config: credentials: secretName: snow-creds type: api-token tableName: incident ticketId: d457fbac6112287007379b57c6b2e60 # example uses: snow-delete ``` This example deletes a Change Request using the Table API (notice how the `template` field is omitted): ```yaml steps: - as: snowdelete config: credentials: secretName: snow-creds type: api-token tableName: change_request ticketId: d457fbac6112287007379b57c6b2e60 # example uses: snow-delete ``` ## Different E2E Workflows These examples demonstrate the different steps supported for ServiceNow integration. ### Change API Workflow ```yaml - as: snowcreate config: credentials: secretName: snow-creds type: api-token parameters: impact: "3" short_description: Deploy to ${{ ctx.stage }} complete for ${{ vars.imageRepo }}:${{ imageFrom(vars.imageRepo).Tag }} urgency: "3" tableName: change_request template: templateId: ed89b72c83c172104517e470ceaad30a type: standard uses: snow-create - as: snowquery config: credentials: namespace: kargo-demo secretName: snow-creds type: api-token query: number: ${{ task.outputs.snowcreate.number }} tableName: change_request uses: snow-query-records - as: snowupdate config: credentials: secretName: snow-creds type: api-token parameters: short_description: Update deployment for ${{ vars.imageRepo }}:${{ imageFrom(vars.imageRepo).Tag }} to resolved tableName: change_request template: type: standard ticketId: ${{ task.outputs.snowquery.sys_id }} uses: snow-update - as: snow-wait-for-condition config: condition: state=-2 credentials: namespace: kargo-demo secretName: snow-creds type: api-token tableName: change_request ticketId: ${{ task.outputs.snowquery.sys_id }} uses: snow-wait-for-condition - as: snowdelete config: credentials: secretName: snow-creds type: api-token tableName: change_request template: type: standard ticketId: ${{ task.outputs.snowquery.sys_id }} uses: snow-delete ``` ### Table API Workflow Here's a Table API workflow with Change Request: ```yaml - as: snowcreate config: credentials: secretName: snow-creds type: api-token parameters: impact: "3" short_description: Deploy to ${{ ctx.stage }} complete for ${{ vars.imageRepo }}:${{ imageFrom(vars.imageRepo).Tag }} urgency: "3" tableName: change_request uses: snow-create - as: snowquery config: credentials: namespace: kargo-demo secretName: snow-creds type: api-token query: number: ${{ task.outputs.snowcreate.number }} tableName: change_request uses: snow-query-records - as: snowupdate config: credentials: secretName: snow-creds type: api-token parameters: short_description: Update deployment incident for ${{ vars.imageRepo }}:${{ imageFrom(vars.imageRepo).Tag }} to resolved tableName: change_request ticketId: ${{ task.outputs.snowquery.sys_id }} uses: snow-update - as: snow-wait-for-condition config: condition: state=-2 credentials: namespace: kargo-demo secretName: snow-creds type: api-token tableName: change_request ticketId: ${{ task.outputs.snowquery.sys_id }} uses: snow-wait-for-condition - as: snowdelete config: credentials: secretName: snow-creds type: api-token tableName: change_request ticketId: ${{ task.outputs.snowquery.sys_id }} uses: snow-delete ``` --- ## `snow-query-for-records` :::info This promotion step is only available in Kargo on the [Akuity Platform](https://akuity.io/akuity-platform), versions v1.9.0 and above. ::: This page is for `snow-query-for-records` step. You can query for a ServiceNow record (returns the first matching record) using this step. Check below for other ServiceNow promotion steps. ServiceNow integration for kargo is a group of promotion steps: 1. [snow-create](./snow-create.md) 2. [snow-update](./snow-update.md) 3. [snow-delete](./snow-delete.md) 4. [snow-query-for-records](./snow-query-for-records.md) 5. [snow-wait-for-condition](./snow-wait-for-condition.md) These steps provide a comprehensive integration with ServiceNow, allowing you to create, update, delete, query for records, and track promotion workflows. This is particularly useful for maintaining traceability between your promotion processes and project management activities. All the steps above support different record types in ServiceNow including managing Change Requests, Incidents, Problems etc.,. This integration also provides condition tracking through `snow-wait-for-condition`, making it a powerful tool for promotion workflows that require coordination with project management systems. ## Finding field values for step configuration Most of the time you cannot use the field labels you see in the ServiceNow UI as keys in the step configuration. For example, if you want to set the value for the “Short description” field: ![](./images/snow-short-des.png) You can't use `Short description` in the step configuration parameters. You need to use `short_description` as the key. To find the correct key for a field, right-click on the field and click `Configure Dictionary`: ![](./images/snow-configure-dict.png) `column_name` is the key: ![](./images/snow-col-name.png) To see choices available for a particular field e.g., `State`, right click on the field and click on `Show Choice List` to see the available choices for use in the steps: ![](./images/show-choices.png) For example `New` is `-1` and `Scheduled` is `-2` ![](./images/choice-list.png) ## Credentials All ServiceNow operations require proper authentication credentials stored in a Kubernetes `Secret`. | Name | Type | Required | Description | | ------------------------ | -------- | -------- | ----------------------------------------------------------------------------------------------- | | `credentials.secretName` | `string` | N | Name of the `Secret` containing the ServiceNow credentials in the project namespace. | | `credentials.sharedSecretName` | `string` | N | Name of the `Secret` containing the ServiceNow credentials in the `shared-resources-namespace`. | | `credentials.type` | `string` | Y | Type of ServiceNow credentials to use for authentication (either `api-token` or `basic`). | :::info Either `credentials.secretName` or `credentials.sharedSecretName` must be set, but not both. ::: For `credentials.type: api-token` the referenced `Secret` should contain the following keys: - `apiToken`: ServiceNow API Token (see [this blog post](https://www.servicenow.com/community/developer-advocate-blog/inbound-rest-api-keys/ba-p/2854924) for how to create an API token in ServiceNow). - `instanceURL`: Your ServiceNow instance URL. ![](./images/snow-instance-url.png) For `credentials.type: basic` the referenced `Secret` should contain the following keys: - `username`: Username of the ServiceNow user (you may want to [create a user](https://www.servicenow.com/docs/bundle/zurich-platform-administration/page/administer/users-and-groups/task/t_CreateAUser.html) specifically for this integration). - `password`: Password of the ServiceNow user (for how to set the password for a user, see [this](https://www.servicenow.com/docs/bundle/zurich-platform-security/page/integrate/authentication/task/reset-your-password.html)). - `instanceURL`: Your ServiceNow instance URL. ## Configuration | Name | Type | Required | Description | | ----------- | -------- | -------- | ------------------------------------------------------------------------------------------------------------------------------- | | `tableName` | `string` | Y | Table name of the record type you want to query (e.g., `incident`, `problem`, `change_request`). | | `query` | `object` | Y | Specify your query parameters on which you want to filter the records. For supported parameters, please see the official documentation [here](https://www.servicenow.com/docs/bundle/zurich-api-reference/page/integrate/inbound-rest/concept/c_TableAPI.html#title_table-GET). | ## Output The output is a ServiceNow record. All fields are available for use in subsequent steps. | Name | Type | Description | | -------- | -------- | ----------------------------------------------------- | | `record` | `object` | The found ServiceNow record object containing all fields. | ## Example This example searches for Change Requests with a specific record `number`: ```yaml steps: - as: snowquery config: credentials: namespace: kargo-demo secretName: snow-creds type: api-token query: number: CHG0000007 tableName: change_request uses: snow-query-records # Use the queried record in subsequent steps - as: snowupdate config: credentials: secretName: snow-creds type: api-token parameters: short_description: Update incident for ${{ vars.imageRepo }}:${{ imageFrom(vars.imageRepo).Tag }} to resolved tableName: change_request template: type: standard ticketId: ${{ task.outputs.snowquery.sys_id }} uses: snow-update ``` ## Different E2E Workflows These examples demonstrate the different steps supported for ServiceNow integration. ### Change API Workflow ```yaml - as: snowcreate config: credentials: secretName: snow-creds type: api-token parameters: impact: "3" short_description: Deploy to ${{ ctx.stage }} complete for ${{ vars.imageRepo }}:${{ imageFrom(vars.imageRepo).Tag }} urgency: "3" tableName: change_request template: templateId: ed89b72c83c172104517e470ceaad30a type: standard uses: snow-create - as: snowquery config: credentials: namespace: kargo-demo secretName: snow-creds type: api-token query: number: ${{ task.outputs.snowcreate.number }} tableName: change_request uses: snow-query-records - as: snowupdate config: credentials: secretName: snow-creds type: api-token parameters: short_description: Update deployment for ${{ vars.imageRepo }}:${{ imageFrom(vars.imageRepo).Tag }} to resolved tableName: change_request template: type: standard ticketId: ${{ task.outputs.snowquery.sys_id }} uses: snow-update - as: snow-wait-for-condition config: condition: state=-2 credentials: namespace: kargo-demo secretName: snow-creds type: api-token tableName: change_request ticketId: ${{ task.outputs.snowquery.sys_id }} uses: snow-wait-for-condition - as: snowdelete config: credentials: secretName: snow-creds type: api-token tableName: change_request template: type: standard ticketId: ${{ task.outputs.snowquery.sys_id }} uses: snow-delete ``` ### Table API Workflow Here's a Table API workflow with Change Request: ```yaml - as: snowcreate config: credentials: secretName: snow-creds type: api-token parameters: impact: "3" short_description: Deploy to ${{ ctx.stage }} complete for ${{ vars.imageRepo }}:${{ imageFrom(vars.imageRepo).Tag }} urgency: "3" tableName: change_request uses: snow-create - as: snowquery config: credentials: namespace: kargo-demo secretName: snow-creds type: api-token query: number: ${{ task.outputs.snowcreate.number }} tableName: change_request uses: snow-query-records - as: snowupdate config: credentials: secretName: snow-creds type: api-token parameters: short_description: Update deployment incident for ${{ vars.imageRepo }}:${{ imageFrom(vars.imageRepo).Tag }} to resolved tableName: change_request ticketId: ${{ task.outputs.snowquery.sys_id }} uses: snow-update - as: snow-wait-for-condition config: condition: state=-2 credentials: namespace: kargo-demo secretName: snow-creds type: api-token tableName: change_request ticketId: ${{ task.outputs.snowquery.sys_id }} uses: snow-wait-for-condition - as: snowdelete config: credentials: secretName: snow-creds type: api-token tableName: change_request ticketId: ${{ task.outputs.snowquery.sys_id }} uses: snow-delete ``` --- ## `snow-update` :::info This promotion step is only available in Kargo on the [Akuity Platform](https://akuity.io/akuity-platform), versions v1.9.0 and above. ::: This page is for `snow-update` step. You can update a ServiceNow record using this step. Check below for other ServiceNow promotion steps. ServiceNow integration for kargo is a group of promotion steps: 1. [snow-create](./snow-create.md) 2. [snow-update](./snow-update.md) 3. [snow-delete](./snow-delete.md) 4. [snow-query-for-records](./snow-query-for-records.md) 5. [snow-wait-for-condition](./snow-wait-for-condition.md) These steps provide a comprehensive integration with ServiceNow, allowing you to create, update, delete, query for records, and track promotion workflows. This is particularly useful for maintaining traceability between your promotion processes and project management activities. All the steps above support different record types in ServiceNow including managing Change Requests, Incidents, Problems etc.,. This integration also provides condition tracking through `snow-wait-for-condition`, making it a powerful tool for promotion workflows that require coordination with project management systems. :::info This step supports [Change Management API](https://www.servicenow.com/docs/bundle/zurich-api-reference/page/integrate/inbound-rest/concept/change-management-api.html) for managing Change Requests and [Table API](https://www.servicenow.com/docs/bundle/zurich-api-reference/page/integrate/inbound-rest/concept/c_TableAPI.html) for all record types. ::: ## Finding field values for step configuration Most of the time you cannot use the field labels you see in the ServiceNow UI as keys in the step configuration. For example, if you want to set the value for the “Short description” field: ![](./images/snow-short-des.png) You can't use `Short description` in the step configuration parameters. You need to use `short_description` as the key. To find the correct key for a field, right-click on the field and click `Configure Dictionary`: ![](./images/snow-configure-dict.png) `column_name` is the key: ![](./images/snow-col-name.png) To see choices available for a particular field e.g., `State`, right click on the field and click on `Show Choice List` to see the available choices for use in the steps: ![](./images/show-choices.png) For example `New` is `-1` and `Scheduled` is `-2` ![](./images/choice-list.png) ## Credentials All ServiceNow operations require proper authentication credentials stored in a Kubernetes `Secret`. | Name | Type | Required | Description | | ------------------------ | -------- | -------- | ----------------------------------------------------------------------------------------------- | | `credentials.secretName` | `string` | N | Name of the `Secret` containing the ServiceNow credentials in the project namespace. | | `credentials.sharedSecretName` | `string` | N | Name of the `Secret` containing the ServiceNow credentials in the `shared-resources-namespace`. | | `credentials.type` | `string` | Y | Type of ServiceNow credentials to use for authentication (either `api-token` or `basic`). | :::info Either `credentials.secretName` or `credentials.sharedSecretName` must be set, but not both. ::: For `credentials.type: api-token` the referenced `Secret` should contain the following keys: - `apiToken`: ServiceNow API Token (see [this blog post](https://www.servicenow.com/community/developer-advocate-blog/inbound-rest-api-keys/ba-p/2854924) for how to create an API token in ServiceNow). - `instanceURL`: Your ServiceNow instance URL. ![](./images/snow-instance-url.png) For `credentials.type: basic` the referenced `Secret` should contain the following keys: - `username`: Username of the ServiceNow user (you may want to [create a user](https://www.servicenow.com/docs/bundle/zurich-platform-administration/page/administer/users-and-groups/task/t_CreateAUser.html) specifically for this integration). - `password`: Password of the ServiceNow user (for how to set the password for a user, see [this](https://www.servicenow.com/docs/bundle/zurich-platform-security/page/integrate/authentication/task/reset-your-password.html)). - `instanceURL`: Your ServiceNow instance URL. ## Configuration | Name | Type | Required | Description | | --------------- | ------------ | -------- | ------------------------------------------------------------------------------------------------------------------------------- | | `parameters` | `string map` | Y | Parameters/fields of the record. | | `tableName` | `string` | Y | Table name of the record type you want to update (e.g., `incident`, `problem`, `change_request`). | | `ticketId` | `string` | Y | Ticket ID (`sys_id`) of the record you want to update. | | `template` | `object` | N | Specify this to update a Change Request using the Change Management API (if omitted, the Table API is used). | | `template.type` | `string` | Y/N | Template type (`standard`, `emergency`, or `normal`). Required if `template` is specified; otherwise optional. | ## Output This step does not produce any output. ## Example This example updates a Change Request using the Change Management API: ```yaml steps: - as: snowupdate config: credentials: secretName: snow-creds type: api-token parameters: short_description: Update deployment for ${{ vars.imageRepo }}:${{ imageFrom(vars.imageRepo).Tag }} to done tableName: change_request template: type: standard ticketId: 9d41c061c611228700edc88b231ec47c uses: snow-update ``` This example updates an Incident using the Table API: ```yaml steps: - config: ticketId: ${{ task.outputs.snowquery.sys_id }} credentials: secretName: snow-creds type: api-token parameters: short_description: Update deployment incident for ${{ vars.imageRepo }}:${{ imageFrom(vars.imageRepo).Tag }} to resolved tableName: incident uses: snow-update ``` This example updates a Change Request using the Table API (notice how the `template` field is omitted): ```yaml steps: - as: snowupdate config: credentials: secretName: snow-creds type: api-token parameters: short_description: Update deployment incident for ${{ vars.imageRepo }}:${{ imageFrom(vars.imageRepo).Tag }} to resolved tableName: change_request ticketId: ${{ task.outputs.snowquery.sys_id }} uses: snow-update ``` ## Different E2E Workflows These examples demonstrate the different steps supported for ServiceNow integration. ### Change API Workflow ```yaml - as: snowcreate config: credentials: secretName: snow-creds type: api-token parameters: impact: "3" short_description: Deploy to ${{ ctx.stage }} complete for ${{ vars.imageRepo }}:${{ imageFrom(vars.imageRepo).Tag }} urgency: "3" tableName: change_request template: templateId: ed89b72c83c172104517e470ceaad30a type: standard uses: snow-create - as: snowquery config: credentials: namespace: kargo-demo secretName: snow-creds type: api-token query: number: ${{ task.outputs.snowcreate.number }} tableName: change_request uses: snow-query-records - as: snowupdate config: credentials: secretName: snow-creds type: api-token parameters: short_description: Update deployment for ${{ vars.imageRepo }}:${{ imageFrom(vars.imageRepo).Tag }} to resolved tableName: change_request template: type: standard ticketId: ${{ task.outputs.snowquery.sys_id }} uses: snow-update - as: snow-wait-for-condition config: condition: state=-2 credentials: namespace: kargo-demo secretName: snow-creds type: api-token tableName: change_request ticketId: ${{ task.outputs.snowquery.sys_id }} uses: snow-wait-for-condition - as: snowdelete config: credentials: secretName: snow-creds type: api-token tableName: change_request template: type: standard ticketId: ${{ task.outputs.snowquery.sys_id }} uses: snow-delete ``` ### Table API Workflow Here's a Table API workflow with Change Request: ```yaml - as: snowcreate config: credentials: secretName: snow-creds type: api-token parameters: impact: "3" short_description: Deploy to ${{ ctx.stage }} complete for ${{ vars.imageRepo }}:${{ imageFrom(vars.imageRepo).Tag }} urgency: "3" tableName: change_request uses: snow-create - as: snowquery config: credentials: namespace: kargo-demo secretName: snow-creds type: api-token query: number: ${{ task.outputs.snowcreate.number }} tableName: change_request uses: snow-query-records - as: snowupdate config: credentials: secretName: snow-creds type: api-token parameters: short_description: Update deployment incident for ${{ vars.imageRepo }}:${{ imageFrom(vars.imageRepo).Tag }} to resolved tableName: change_request ticketId: ${{ task.outputs.snowquery.sys_id }} uses: snow-update - as: snow-wait-for-condition config: condition: state=-2 credentials: namespace: kargo-demo secretName: snow-creds type: api-token tableName: change_request ticketId: ${{ task.outputs.snowquery.sys_id }} uses: snow-wait-for-condition - as: snowdelete config: credentials: secretName: snow-creds type: api-token tableName: change_request ticketId: ${{ task.outputs.snowquery.sys_id }} uses: snow-delete ``` --- ## `snow-wait-for-condition` :::info This promotion step is only available in Kargo on the [Akuity Platform](https://akuity.io/akuity-platform), versions v1.9.0 and above. ::: This page is for `snow-wait-for-condition` step. You can wait for a record field on a ServiceNow record to be set to a particular value before proceeding using this step. Check below for other ServiceNow promotion steps. ServiceNow integration for kargo is a group of promotion steps: 1. [snow-create](./snow-create.md) 2. [snow-update](./snow-update.md) 3. [snow-delete](./snow-delete.md) 4. [snow-query-for-records](./snow-query-for-records.md) 5. [snow-wait-for-condition](./snow-wait-for-condition.md) These steps provide a comprehensive integration with ServiceNow, allowing you to create, update, delete, query for records, and track promotion workflows. This is particularly useful for maintaining traceability between your promotion processes and project management activities. All the steps above support different record types in ServiceNow including managing Change Requests, Incidents, Problems etc.,. This integration also provides condition tracking through `snow-wait-for-condition`, making it a powerful tool for promotion workflows that require coordination with project management systems. ## Finding field values for step configuration Most of the time you cannot use the field labels you see in the ServiceNow UI as keys in the step configuration. For example, if you want to set the value for the “Short description” field: ![](./images/snow-short-des.png) You can't use `Short description` in the step configuration parameters. You need to use `short_description` as the key. To find the correct key for a field, right-click on the field and click `Configure Dictionary`: ![](./images/snow-configure-dict.png) `column_name` is the key: ![](./images/snow-col-name.png) To see choices available for a particular field e.g., `State`, right click on the field and click on `Show Choice List` to see the available choices for use in the steps: ![](./images/show-choices.png) For example `New` is `-1` and `Scheduled` is `-2` ![](./images/choice-list.png) ## Credentials All ServiceNow operations require proper authentication credentials stored in a Kubernetes `Secret`. | Name | Type | Required | Description | | ------------------------ | -------- | -------- | ----------------------------------------------------------------------------------------------- | | `credentials.secretName` | `string` | N | Name of the `Secret` containing the ServiceNow credentials in the project namespace. | | `credentials.sharedSecretName` | `string` | N | Name of the `Secret` containing the ServiceNow credentials in the `shared-resources-namespace`. | | `credentials.type` | `string` | Y | Type of ServiceNow credentials to use for authentication (either `api-token` or `basic`). | :::info Either `credentials.secretName` or `credentials.sharedSecretName` must be set, but not both. ::: For `credentials.type: api-token` the referenced `Secret` should contain the following keys: - `apiToken`: ServiceNow API Token (see [this blog post](https://www.servicenow.com/community/developer-advocate-blog/inbound-rest-api-keys/ba-p/2854924) for how to create an API token in ServiceNow). - `instanceURL`: Your ServiceNow instance URL. ![](./images/snow-instance-url.png) For `credentials.type: basic` the referenced `Secret` should contain the following keys: - `username`: Username of the ServiceNow user (you may want to [create a user](https://www.servicenow.com/docs/bundle/zurich-platform-administration/page/administer/users-and-groups/task/t_CreateAUser.html) specifically for this integration). - `password`: Password of the ServiceNow user (for how to set the password for a user, see [this](https://www.servicenow.com/docs/bundle/zurich-platform-security/page/integrate/authentication/task/reset-your-password.html)). - `instanceURL`: Your ServiceNow instance URL. ## Configuration | Name | Type | Required | Description | | -------------- | --------- | -------- | ------------------------------------------------------------------------------------------------------------------------------- | | `tableName` | `string` | Y | Table name of the record type you want to wait on (e.g., `incident`, `problem`, `change_request`). | | `ticketId` | `string` | Y | Ticket ID (`sys_id`) of the record you want to monitor. | | `condition` | `string` | Y | Condition to wait on before proceeding. | | `pollInterval` | `string` | N | Poll interval for checking record condition, specified as a [Go duration string](https://pkg.go.dev/time#ParseDuration) (e.g., `30s`, `5m`). Overrides the default controller reconciliation interval of 5 minutes when set. Available in Kargo >= v1.10. | ## Output This step does not produce any output. ## Example This example waits for a Change Request to be Scheduled (`state=-2`) before proceeding with promotion: ```yaml steps: - as: snow-wait-for-condition config: condition: state=-2 credentials: namespace: kargo-demo secretName: snow-creds type: api-token tableName: change_request ticketId: d457fbac6112287007379b57c6b2e60 uses: snow-wait-for-condition # promotion steps continue after approval... - uses: helm-template config: path: ./charts vars: imageTag: "${{ imageFrom(vars.imageRepo).Tag }}" environment: "${{ ctx.stage }}" ``` You can write more complex conditions as well. See the list of supported operators [here](https://www.servicenow.com/docs/bundle/zurich-platform-user-interface/page/use/common-ui-elements/reference/r_OpAvailableFiltersQueries.html). ## Different E2E Workflows These examples demonstrate the different steps supported for ServiceNow integration. ### Change API Workflow ```yaml - as: snowcreate config: credentials: secretName: snow-creds type: api-token parameters: impact: "3" short_description: Deploy to ${{ ctx.stage }} complete for ${{ vars.imageRepo }}:${{ imageFrom(vars.imageRepo).Tag }} urgency: "3" tableName: change_request template: templateId: ed89b72c83c172104517e470ceaad30a type: standard uses: snow-create - as: snowquery config: credentials: namespace: kargo-demo secretName: snow-creds type: api-token query: number: ${{ task.outputs.snowcreate.number }} tableName: change_request uses: snow-query-records - as: snowupdate config: credentials: secretName: snow-creds type: api-token parameters: short_description: Update deployment for ${{ vars.imageRepo }}:${{ imageFrom(vars.imageRepo).Tag }} to resolved tableName: change_request template: type: standard ticketId: ${{ task.outputs.snowquery.sys_id }} uses: snow-update - as: snow-wait-for-condition config: condition: state=-2 credentials: namespace: kargo-demo secretName: snow-creds type: api-token tableName: change_request ticketId: ${{ task.outputs.snowquery.sys_id }} uses: snow-wait-for-condition - as: snowdelete config: credentials: secretName: snow-creds type: api-token tableName: change_request template: type: standard ticketId: ${{ task.outputs.snowquery.sys_id }} uses: snow-delete ``` ### Table API Workflow Here's a Table API workflow with Change Request: ```yaml - as: snowcreate config: credentials: secretName: snow-creds type: api-token parameters: impact: "3" short_description: Deploy to ${{ ctx.stage }} complete for ${{ vars.imageRepo }}:${{ imageFrom(vars.imageRepo).Tag }} urgency: "3" tableName: change_request uses: snow-create - as: snowquery config: credentials: namespace: kargo-demo secretName: snow-creds type: api-token query: number: ${{ task.outputs.snowcreate.number }} tableName: change_request uses: snow-query-records - as: snowupdate config: credentials: secretName: snow-creds type: api-token parameters: short_description: Update deployment for ${{ vars.imageRepo }}:${{ imageFrom(vars.imageRepo).Tag }} to resolved tableName: change_request ticketId: ${{ task.outputs.snowquery.sys_id }} uses: snow-update - as: snow-wait-for-condition config: condition: state=-2 credentials: namespace: kargo-demo secretName: snow-creds type: api-token tableName: change_request ticketId: ${{ task.outputs.snowquery.sys_id }} uses: snow-wait-for-condition - as: snowdelete config: credentials: secretName: snow-creds type: api-token tableName: change_request ticketId: ${{ task.outputs.snowquery.sys_id }} uses: snow-delete ``` --- ## `tf-apply` :::info This promotion step is only available in Kargo on the [Akuity Platform](https://akuity.io/akuity-platform), versions v1.9 and above. Additionally, it requires enabling of the Promotion Controller to allow for Pod-based promotions. ::: `tf-apply` applies an OpenTofu configuration to create, update, or delete infrastructure. This step can apply changes directly from configuration files or from a saved plan file generated by the [`tf-plan`](tf-plan.md) step. :::info For a safer workflow, use [`tf-plan`](tf-plan.md) with the `out` option to generate a plan file, then apply that specific plan with `tf-apply`. This ensures that exactly the changes previewed in the plan are applied. ::: ## Configuration | Name | Type | Required | Description | |------|------|----------|-------------| | `dir` | `string` | Y | Directory containing OpenTofu configuration files. This path is relative to the temporary workspace that Kargo provisions for use by the promotion process. | | `plan` | `string` | N | Path to a plan file to apply, relative to the `dir` directory. Typically generated by the [`tf-plan`](tf-plan.md) step. When not specified, the apply runs directly against the configuration. | | `vars` | `[]object` | N | Variables to pass to OpenTofu. Ignored when applying a saved plan file. | | `vars[].file` | `string` | N | Path to a variables file (`.tfvars`), relative to the `dir` directory. Mutually exclusive with `name`/`value`. | | `vars[].name` | `string` | N | Variable name. Required when not using `file`. Mutually exclusive with `file`. | | `vars[].value` | `string` | N | Variable value. Required when not using `file`. Mutually exclusive with `file`. | | `env` | `[]object` | N | Environment variables to set during OpenTofu execution. | | `env[].name` | `string` | Y | Environment variable name. Must match the pattern `^[a-zA-Z_][a-zA-Z0-9_]*$`. | | `env[].value` | `string` | Y | Environment variable value. | ## Output | Name | Type | Description | |------|------|-------------| | `result` | `string` | The apply output as text, showing what changes were made. | ## Examples ### Common Usage The most common usage of this step is to apply Stage-specific OpenTofu configuration directly, such as deploying an AWS Lambda function. This example shows a complete direct-apply workflow where configuration is updated, applied, and state changes are committed back to Git. ```yaml vars: - name: repoURL value: https://github.com/example/infra.git - name: image value: 123456789.dkr.ecr.us-west-2.amazonaws.com/my-app steps: - uses: git-clone config: repoURL: ${{ vars.repoURL }} checkout: - branch: main path: ./src - uses: hcl-update config: path: ./src/opentofu/${{ ctx.stage }}/env.auto.tfvars updates: - key: image_uri value: ${{ vars.image }}:${{ imageFrom(vars.image).Tag }} - uses: tf-apply config: dir: ./src/opentofu/${{ ctx.stage }} env: - name: AWS_REGION value: us-west-2 - name: AWS_ACCESS_KEY_ID value: ${{ secret('aws-creds').awsAccessKeyID }} - name: AWS_SECRET_ACCESS_KEY value: ${{ secret('aws-creds').awsSecretAccessKey }} - uses: git-commit config: path: ./src message: Update ${{ ctx.stage }} to ${{ imageFrom(vars.image).Tag }} - uses: git-push config: path: ./src ``` ### Post-PR Merge Apply This example shows the second phase of a PR-based workflow. After a pull request containing updated configuration has been reviewed and merged, a fresh clone of the merged branch is made and then applied. This pattern ensures that production changes are reviewed before being applied. ```yaml vars: - name: repoURL value: https://github.com/example/infra.git - name: image value: 123456789.dkr.ecr.us-west-2.amazonaws.com/my-app steps: # First phase: Create and wait for PR (see tf-plan examples) # ... - uses: git-wait-for-pr config: prNumber: ${{ task.outputs['open-pr'].prNumber }} repoURL: ${{ vars.repoURL }} # Second phase: Clone merged main and apply - uses: git-clone config: repoURL: ${{ vars.repoURL }} checkout: - branch: main path: ./merged - uses: tf-apply config: dir: ./merged/opentofu/${{ ctx.stage }} env: - name: AWS_REGION value: us-west-2 - name: AWS_ACCESS_KEY_ID value: ${{ secret('aws-creds').awsAccessKeyID }} - name: AWS_SECRET_ACCESS_KEY value: ${{ secret('aws-creds').awsSecretAccessKey }} - uses: git-commit config: path: ./merged message: "State files for: Update ${{ ctx.stage }} to ${{ imageFrom(vars.image).Tag }}" - uses: git-push config: path: ./merged ``` ### Applying a Saved Plan This example demonstrates applying a saved plan file. By saving the plan to a file, you ensure that exactly the changes shown in the plan are applied, with no possibility of drift between planning and applying. ```yaml steps: # Clone, update configuration, etc... - uses: tf-plan as: plan config: dir: ./src/opentofu/${{ ctx.stage }} out: tfplan env: - name: AWS_REGION value: us-west-2 - name: AWS_ACCESS_KEY_ID value: ${{ secret('aws-creds').awsAccessKeyID }} - name: AWS_SECRET_ACCESS_KEY value: ${{ secret('aws-creds').awsSecretAccessKey }} - uses: tf-apply config: dir: ./src/opentofu/${{ ctx.stage }} plan: tfplan env: - name: AWS_REGION value: us-west-2 - name: AWS_ACCESS_KEY_ID value: ${{ secret('aws-creds').awsAccessKeyID }} - name: AWS_SECRET_ACCESS_KEY value: ${{ secret('aws-creds').awsSecretAccessKey }} ``` --- ## `tf-output` :::info This promotion step is only available in Kargo on the [Akuity Platform](https://akuity.io/akuity-platform), versions v1.9 and above. Additionally, it requires enabling of the Promotion Controller to allow for Pod-based promotions. ::: `tf-output` retrieves output values from OpenTofu state. These outputs can be used in subsequent promotion steps or written to a JSON file. This step is typically used after [`tf-apply`](tf-apply.md) to access values such as resource IDs, endpoints, or other computed attributes. :::note By default, sensitive outputs are filtered from the results when retrieving all outputs. Set `sensitive: true` to include them, or retrieve a specific output by name to bypass filtering. ::: ## Configuration | Name | Type | Required | Description | |------|------|----------|-------------| | `dir` | `string` | Y | Directory containing OpenTofu configuration files. This path is relative to the temporary workspace that Kargo provisions for use by the promotion process. | | `name` | `string` | N | Name of a specific output to retrieve. When specified, only this output is returned. When omitted, all outputs are retrieved. | | `out` | `string` | N | Path where outputs will be written as JSON, relative to the temporary workspace. When specified, outputs are written to this file instead of being returned in the step output. | | `state` | `string` | N | Path to a custom state file, relative to the `dir` directory. When omitted, the default state file is used. | | `sensitive` | `boolean` | N | Whether to include sensitive outputs in the results. Defaults to `false`. Only applies when retrieving all outputs (when `name` is not specified). | | `vars` | `[]object` | N | Variables to pass to OpenTofu. | | `vars[].file` | `string` | N | Path to a variables file (`.tfvars`), relative to the `dir` directory. Mutually exclusive with `name`/`value`. | | `vars[].name` | `string` | N | Variable name. Required when not using `file`. Mutually exclusive with `file`. | | `vars[].value` | `string` | N | Variable value. Required when not using `file`. Mutually exclusive with `file`. | | `env` | `[]object` | N | Environment variables to set during OpenTofu execution. | | `env[].name` | `string` | Y | Environment variable name. Must match the pattern `^[a-zA-Z_][a-zA-Z0-9_]*$`. | | `env[].value` | `string` | Y | Environment variable value. | ## Output The output format depends on the configuration: **When `out` is specified:** The step writes outputs to the specified file as JSON and returns an empty map. **When `out` is not specified and `name` is specified:** | Name | Type | Description | |------|------|-------------| | `{name}` | `any` | The value of the specified output. The key matches the `name` parameter. | **When `out` is not specified and `name` is not specified:** | Name | Type | Description | |------|------|-------------| | `{output_name}` | `object` | Each output is returned as an object containing `value`, `type`, and `sensitive` fields. Sensitive outputs are excluded unless `sensitive: true` is set. | ## Examples ### Common Usage The most common usage of this step is to retrieve outputs from the OpenTofu state after applying configuration. This example retrieves the function URL from an AWS Lambda deployment for use in subsequent steps. ```yaml vars: - name: repoURL value: https://github.com/example/infra.git - name: image value: 123456789.dkr.ecr.us-west-2.amazonaws.com/my-app steps: - uses: git-clone config: repoURL: ${{ vars.repoURL }} checkout: - branch: main path: ./src - uses: hcl-update config: path: ./src/opentofu/${{ ctx.stage }}/env.auto.tfvars updates: - key: image_uri value: ${{ vars.image }}:${{ imageFrom(vars.image).Tag }} - uses: tf-apply config: dir: ./src/opentofu/${{ ctx.stage }} env: - name: AWS_REGION value: us-west-2 - name: AWS_ACCESS_KEY_ID value: ${{ secret('aws-creds').awsAccessKeyID }} - name: AWS_SECRET_ACCESS_KEY value: ${{ secret('aws-creds').awsSecretAccessKey }} - uses: tf-output as: infra config: dir: ./src/opentofu/${{ ctx.stage }} env: - name: AWS_REGION value: us-west-2 - name: AWS_ACCESS_KEY_ID value: ${{ secret('aws-creds').awsAccessKeyID }} - name: AWS_SECRET_ACCESS_KEY value: ${{ secret('aws-creds').awsSecretAccessKey }} # Commit and push state changes... ``` The outputs can then be referenced in subsequent steps: ```yaml - uses: http config: url: ${{ outputs.infra.function_url.value }} ``` ### Retrieving a Specific Output This example retrieves a single output by name. When retrieving by name, the step returns only the value without the metadata wrapper, making it easier to use in subsequent steps. ```yaml steps: # Clone, plan, apply, etc... - uses: tf-output as: endpoint config: dir: ./src/opentofu/${{ ctx.stage }} name: function_url env: - name: AWS_REGION value: us-west-2 - name: AWS_ACCESS_KEY_ID value: ${{ secret('aws-creds').awsAccessKeyID }} - name: AWS_SECRET_ACCESS_KEY value: ${{ secret('aws-creds').awsSecretAccessKey }} ``` The output value can be referenced directly: ```yaml - uses: http config: url: ${{ outputs.endpoint.function_url }} ``` ### Writing Outputs to a File This example writes all outputs to a JSON file. This is useful when outputs need to be consumed by external tools or processes outside of Kargo's promotion workflow. ```yaml steps: # Clone, plan, apply, etc... - uses: tf-output config: dir: ./src/opentofu/${{ ctx.stage }} out: ./outputs.json env: - name: AWS_REGION value: us-west-2 - name: AWS_ACCESS_KEY_ID value: ${{ secret('aws-creds').awsAccessKeyID }} - name: AWS_SECRET_ACCESS_KEY value: ${{ secret('aws-creds').awsSecretAccessKey }} ``` ### Including Sensitive Outputs This example demonstrates how to include sensitive outputs in the results. This is useful when you need access to values that OpenTofu marks as sensitive, such as generated passwords or API keys. :::warning Exercise caution when including sensitive outputs, as they may contain secrets or other confidential information. ::: ```yaml steps: # Clone, plan, apply, etc... - uses: tf-output as: outputs config: dir: ./src/opentofu/${{ ctx.stage }} sensitive: true env: - name: AWS_REGION value: us-west-2 - name: AWS_ACCESS_KEY_ID value: ${{ secret('aws-creds').awsAccessKeyID }} - name: AWS_SECRET_ACCESS_KEY value: ${{ secret('aws-creds').awsSecretAccessKey }} ``` --- ## `tf-plan` :::info This promotion step is only available in Kargo on the [Akuity Platform](https://akuity.io/akuity-platform), versions v1.9 and above. Additionally, it requires enabling of the Promotion Controller to allow for Pod-based promotions. ::: `tf-plan` executes an OpenTofu plan operation to preview what infrastructure changes would be made. This step initializes the OpenTofu working directory and generates an execution plan, which can optionally be saved to a file for use with the [`tf-apply`](tf-apply.md) step. :::note The step returns `Succeeded` when changes are detected and `Skipped` when no changes are detected. This allows you to conditionally execute subsequent steps based on whether infrastructure changes are needed. ::: ## Configuration | Name | Type | Required | Description | |------|------|----------|-------------| | `dir` | `string` | Y | Directory containing OpenTofu configuration files. This path is relative to the temporary workspace that Kargo provisions for use by the promotion process. | | `out` | `string` | N | Path where the generated plan file will be saved, relative to the `dir` directory. When specified, the plan can be applied using the [`tf-apply`](tf-apply.md) step. | | `vars` | `[]object` | N | Variables to pass to OpenTofu. | | `vars[].file` | `string` | N | Path to a variables file (`.tfvars`), relative to the `dir` directory. Mutually exclusive with `name`/`value`. | | `vars[].name` | `string` | N | Variable name. Required when not using `file`. Mutually exclusive with `file`. | | `vars[].value` | `string` | N | Variable value. Required when not using `file`. Mutually exclusive with `file`. | | `env` | `[]object` | N | Environment variables to set during OpenTofu execution. | | `env[].name` | `string` | Y | Environment variable name. Must match the pattern `^[a-zA-Z_][a-zA-Z0-9_]*$`. | | `env[].value` | `string` | Y | Environment variable value. | ## Output | Name | Type | Description | |------|------|-------------| | `plan` | `string` | The plan output as text, showing what changes would be made. | ## Examples ### Common Usage The most common usage of this step is to run a plan operation on Stage-specific OpenTofu configuration. This example shows planning infrastructure changes for an AWS Lambda deployment, with credentials provided via Kargo secrets. ```yaml vars: - name: repoURL value: https://github.com/example/infra.git - name: image value: 123456789.dkr.ecr.us-west-2.amazonaws.com/my-app steps: - uses: git-clone config: repoURL: ${{ vars.repoURL }} checkout: - branch: main path: ./src - uses: hcl-update config: path: ./src/opentofu/${{ ctx.stage }}/env.auto.tfvars updates: - key: image_uri value: ${{ vars.image }}:${{ imageFrom(vars.image).Tag }} - uses: tf-plan as: plan config: dir: ./src/opentofu/${{ ctx.stage }} env: - name: AWS_REGION value: us-west-2 - name: AWS_ACCESS_KEY_ID value: ${{ secret('aws-creds').awsAccessKeyID }} - name: AWS_SECRET_ACCESS_KEY value: ${{ secret('aws-creds').awsSecretAccessKey }} ``` ### Plan Output in Pull Request This example demonstrates a review-based workflow where the plan output is included in a pull request description. This allows reviewers to see exactly what infrastructure changes will be made before approving the PR. The step alias (`as: tf-plan`) allows referencing the plan output in subsequent steps. ```yaml vars: - name: repoURL value: https://github.com/example/infra.git - name: image value: 123456789.dkr.ecr.us-west-2.amazonaws.com/my-app steps: - uses: git-clone config: repoURL: ${{ vars.repoURL }} checkout: - branch: main path: ./src - uses: hcl-update config: path: ./src/opentofu/${{ ctx.stage }}/env.auto.tfvars updates: - key: image_uri value: ${{ vars.image }}:${{ imageFrom(vars.image).Tag }} - uses: tf-plan as: tf-plan config: dir: ./src/opentofu/${{ ctx.stage }} env: - name: AWS_REGION value: us-west-2 - name: AWS_ACCESS_KEY_ID value: ${{ secret('aws-creds').awsAccessKeyID }} - name: AWS_SECRET_ACCESS_KEY value: ${{ secret('aws-creds').awsSecretAccessKey }} - uses: git-commit config: path: ./src message: Update ${{ ctx.stage }} to ${{ imageFrom(vars.image).Tag }} - uses: git-push as: push config: generateTargetBranch: true path: ./src - uses: git-open-pr as: open-pr config: repoURL: ${{ vars.repoURL }} sourceBranch: ${{ task.outputs.push.branch }} targetBranch: main description: |- ## OpenTofu Plan Output ${{ task.outputs['tf-plan'].plan }} - uses: git-wait-for-pr config: prNumber: ${{ task.outputs['open-pr'].prNumber }} repoURL: ${{ vars.repoURL }} ``` ### Saving Plan for Apply This example saves the plan to a file that can be used with the [`tf-apply`](tf-apply.md) step. This ensures that exactly the changes shown in the plan are applied, with no possibility of drift between planning and applying. ```yaml steps: # Clone, update configuration, etc... - uses: tf-plan as: plan config: dir: ./src/opentofu/${{ ctx.stage }} out: tfplan env: - name: AWS_REGION value: us-west-2 - name: AWS_ACCESS_KEY_ID value: ${{ secret('aws-creds').awsAccessKeyID }} - name: AWS_SECRET_ACCESS_KEY value: ${{ secret('aws-creds').awsSecretAccessKey }} - uses: tf-apply config: dir: ./src/opentofu/${{ ctx.stage }} plan: tfplan env: - name: AWS_REGION value: us-west-2 - name: AWS_ACCESS_KEY_ID value: ${{ secret('aws-creds').awsAccessKeyID }} - name: AWS_SECRET_ACCESS_KEY value: ${{ secret('aws-creds').awsSecretAccessKey }} ``` --- ## `toml-parse` `toml-parse` is a utility step that parses a TOML file and extracts values using [expr-lang] expressions. ## Configuration | Name | Type | Required | Description | |------|------|----------|-------------| | `path` | `string` | Y | Path to a TOML file. This path is relative to the temporary workspace that Kargo provisions for use by the promotion process. | | `outputs` | `[]object` | Y | A list of rules for extracting values from the parsed TOML. | | `outputs[].name` | `string` | Y | The name of the output variable. | | `outputs[].fromExpression` | `string` | Y | An [expr-lang] expression that can extract the value from the TOML file. Note that this expression should not be offset by `${{` and `}}`. See [examples](#examples) for more details. | ## Expressions The `fromExpression` field supports [expr-lang] expressions. :::note Expressions should _not_ be offset by `${{` and `}}` to prevent pre-processing evaluation by Kargo. The `toml-parse` step itself will evaluate these expressions. ::: An `outputs` object (a `map[string]any`) is available to these expressions. It is structured as follows: | Field | Type | Description | |-------|------|-------------| | `outputs` | `map[string]any` | The parsed TOML object. | ## Outputs The `toml-parse` step produces the outputs described by the `outputs` field in its configuration. ## Examples ### Common Usage In this example, a TOML file is parsed to find the container image tag. After cloning the repository and clearing the output directory, the `toml-parse` step parses `config.toml` to extract the image tag from the `Freight` being promoted. Using dot notation (`image.tag`), it extracts the nested value from the TOML file. ```yaml vars: - name: gitRepo value: https://github.com/example/repo.git steps: - uses: git-clone config: repoURL: ${{ vars.gitRepo }} checkout: - commit: ${{ commitFrom(vars.gitRepo).ID }} path: ./src - branch: stage/${{ ctx.stage }} create: true path: ./out - uses: git-clear config: path: ./out - uses: toml-parse as: values config: path: './src/config.toml' outputs: - name: imageTag fromExpression: image.tag # Render manifests to ./out, commit, push, etc... ``` Given the sample input TOML: ```toml [image] tag = "v1.2.3" [rbac] installClusterRoles = true ``` The step would produce the following [outputs](../15-promotion-templates.md#step-outputs): | Name | Type | Value | |------|------|-------| | `imageTag` | `string` | `v1.2.3` | [expr-lang]: https://expr-lang.org --- ## `toml-update` `toml-update` updates the values of specified keys in any TOML file, in-place, without disruption to existing formatting choices. :::note[Limitations] `toml-update` updates scalar values only and can only update the values of existing keys. ::: ## Configuration | Name | Type | Required | Description | |------|------|----------|-------------| | `path` | `string` | Y | Path to a TOML file. This path is relative to the temporary workspace that Kargo provisions for use by the promotion process. | | `updates` | `[]object` | Y | The details of changes to be applied to the file. At least one must be specified. | | `updates[].key` | `string` | Y | The key to update within the file. For nested values, use dots to delimit key parts. e.g. `image.tag`. The syntax is identical to that supported by the `json-update` step and is documented in more detail [here](https://github.com/tidwall/sjson?tab=readme-ov-file#path-syntax). | | `updates[].value` | `any` | Y | The new scalar value for the key. Typically specified using an expression. Supports strings, numbers, and booleans. | ## Output | Name | Type | Description | |------|------|-------------| | `commitMessage` | `string` | A description of the change(s) applied by this step. Typically, a subsequent [`git-commit` step](git-commit.md) will reference this output and aggregate this commit message fragment with others like it to build a comprehensive commit message that describes all changes. | ## Writing Keys **Nested keys:** ```toml [package] version = "1.0.0" ``` Update key: `package.version` **Keys with literal dots:** ```toml [labels] "example.com/version" = "1.0.0" ``` Update key: `labels.example\.com/version` **Sequences:** ```toml values = [1, 2, 3] ``` Update key: `values.1` :::note See the [sjson path syntax documentation](https://github.com/tidwall/sjson?tab=readme-ov-file#path-syntax) for the full description of the syntax. ::: ## Examples ### Common Usage In this example, a TOML file's values are updated according to changes in a container image tag. After cloning the repository and clearing the output directory, the `toml-update` step updates the `image.tag` field in `configs/settings.toml` to match the tag of the image being promoted. This pattern is commonly seen when managing configuration files that need to stay synchronized with deployed container versions. ```yaml vars: - name: gitRepo value: https://github.com/example/repo.git steps: - uses: git-clone config: repoURL: ${{ vars.gitRepo }} checkout: - commit: ${{ commitFrom(vars.gitRepo).ID }} path: ./src - branch: stage/${{ ctx.stage }} create: true path: ./out - uses: git-clear config: path: ./out - uses: toml-update config: path: ./src/configs/settings.toml updates: - key: image.tag value: ${{ imageFrom("my/image").Tag }} # Render manifests to ./out, commit, push, etc... ``` :::info For more information on `imageFrom()` and expressions used in the example above, see the [Expressions](../40-expressions.md#functions) documentation. ::: --- ## `untar` `untar` extracts the contents of a tar file (including gzipped tar files) to a specified location. It automatically detects if the file is gzipped and handles it accordingly. ## Configuration | Name | Type | Required | Description | |------|------|----------|-------------| | `inPath` | `string` | Y | Path to the tar file to extract. This path is relative to the temporary workspace that Kargo provisions for use by the promotion process. | | `outPath` | `string` | Y | Path to the destination directory where contents will be extracted. This path is relative to the temporary workspace that Kargo provisions for use by the promotion process. The outPath content will be overriden if it already exists. | | `stripComponents` | `integer` | N | Number of leading path components to strip from file names in the archive. Similar to the `--strip-components` option in the tar command. | | `ignore` | `string` | N | A multiline string of glob patterns to ignore when extracting files. It accepts the same syntax as `.gitignore` files. | ## Examples ### Basic Usage Extract a tarball to a specific directory: ```yaml steps: - uses: untar config: inPath: ./artifacts/bundle.tar.gz outPath: ./extracted ``` ### Strip Path Components Extract a tarball while removing leading directory components: ```yaml steps: - uses: untar config: inPath: ./artifacts/bundle.tar.gz outPath: ./extracted stripComponents: 1 ``` ### Ignore Patterns Extract a tarball while ignoring specific files: ```yaml steps: - uses: untar config: inPath: ./artifacts/bundle.tar.gz outPath: ./extracted ignore: | *.log temp/ .DS_Store ``` ### Extract helm chart dependency This example shows how to update dependencies for a given helm chart and then extract the downloaded archive to have the raw manifests as dependencies instead of the tgz. ```yaml vars: - name: gitRepo value: https://github.com/example/repo.git - name: chartRepo value: https://example.github.io/helm-charts - name: chartName value: myChart steps: - uses: git-clone config: checkout: - branch: main path: ./target repoURL: ${{ vars.gitRepo }} - uses: helm-update-chart config: path: ./target/pathToMyUmbrellaChart charts: - repository: ${{ vars.chartRepo }} name: ${{ vars.chartName }} version: ${{ chartFrom(vars.chartRepo, vars.chartName).Version }} - as: deleteFormerChart uses: delete config: path: ./target/pathToMyUmbrellaChart/charts/${{ vars.chartName }} - as: untar uses: untar config: inPath: ./target/pathToMyUmbrellaChart/charts/${{ vars.chartName }}-${{ chartFrom(vars.chartRepo, "kube-prometheus-stack").Version }}.tgz outPath: ./target/pathToMyUmbrellaChart/charts/${{ vars.chartName }} - as: deleteTgz uses: delete config: path: ./target/pathToMyUmbrellaChart/charts/${{ vars.chartName }}-${{ chartFrom(vars.chartRepo, "kube-prometheus-stack").Version }}.tgz # Then commit, push ... ``` --- ## `yaml-merge` `yaml-merge` merges multiple YAML files into a single file. YAML files are merged in order. The first file in the list is the source, and all subsequent files are applied over it. When `ignoreMissingFiles` is false (default), the step will fail if any file from `inFiles` does not exist. :::note Merging is performed as follows: - **Scalar values:** If both documents define a scalar (string, number, boolean) at the same key, the value from the second document overrides the first. - **Mapping (object) values:** If both documents define a mapping at the same key, the mappings are merged recursively. - **Sequence (array) values:** If both documents define a sequence at the same key, the sequence from the second document replaces the first (no merging). - **Keys present only in one document:** Keys that exist in only one document are included as-is in the result. - **Null values:** If a key is set to `null` in the second document, it removes or overrides the value from the first document. ::: ## Configuration | Name | Type | Required | Description | |------|------|----------|-------------| | `inFiles` | `[]string` | Y | Paths to a YAML files. This path is relative to the temporary workspace that Kargo provisions for use by the promotion process. | | `outFile` | `string` | Y | The path to the output file. This path is relative to the temporary workspace that Kargo provisions for use by the promotion process. | | `ignoreMissingFiles` | `bool` | N | When set to `true`, the directive will skip input files that do not exist. If all input files are missing, an empty output file will be created. Defaults to `false`. | ## Output | Name | Type | Description | |------|------|-------------| | `commitMessage` | `string` | A description of the change(s) applied by this step. Typically, a subsequent [`git-commit` step](git-commit.md) will reference this output and aggregate this commit message fragment with others like it to build a comprehensive commit message that describes all changes. | ## Examples ### Common Usage In the following example, multiple Helm values files (one "base", a second with environment-specific overrides, and a third with cluster-specific overrides) are merged into a new, single file, which is then committed to a `Stage`-specific branch: ```yaml vars: - name: gitRepo value: https://github.com/example/repo.git steps: - uses: git-clone config: repoURL: ${{ vars.gitRepo }} checkout: - commit: ${{ commitFrom(vars.gitRepo).ID }} path: ./src - branch: stage/${{ ctx.stage }} create: true path: ./out - uses: git-clear config: path: ./out - uses: yaml-merge config: inFiles: - ./src/charts/my-chart/values.yaml - ./src/charts/qa/values.yaml - ./src/charts/qa/cluster-a/values.yaml outFile: ./out/charts/my-chart/values.yaml # Render manifests to ./out, commit, push, etc... ``` Given the following sample input files: **charts/my-chart/values.yaml** ```yaml service: enabled: false ``` **charts/qa/values.yaml** ```yaml service: enabled: true ``` **charts/qa/cluster-a/values.yaml** ```yaml service: type: LoadBalancer ``` The step would produce the following output: **charts/my-chart/values.yaml** ```yaml service: enabled: true type: LoadBalancer ``` --- ## `yaml-parse` `yaml-parse` is a utility step that parses a YAML file and extracts values using [expr-lang] expressions. ## Configuration | Name | Type | Required | Description | |------|------|----------|-------------| | `path` | `string` | Y | Path to a YAML file. This path is relative to the temporary workspace that Kargo provisions for use by the promotion process. | | `outputs` | `[]object` | Y | A list of rules for extracting values from the parsed YAML. | | `outputs[].name` | `string` | Y | The name of the output variable. | | `outputs[].fromExpression` | `string` | Y | An [expr-lang] expression that can extract the value from the YAML file. Note that this expression should not be offset by `${{` and `}}`. See [examples](#examples) for more details. | ## Expressions The `fromExpression` field supports [expr-lang] expressions. :::note Expressions should _not_ be offset by `${{` and `}}` to prevent pre-processing evaluation by Kargo. The `yaml-parse` step itself will evaluate these expressions. ::: An `outputs` object (a `map[string]any`) is available to these expressions. It is structured as follows: | Field | Type | Description | |-------|------|-------------| | `outputs` | `map[string]any` | The parsed YAML object. | ## Outputs The `yaml-parse` step produces the outputs described by the `outputs` field in its configuration. ## Examples ### Common Usage In this example, a Helm values file is parsed to find the container image tag. After cloning the repository and clearing the output directory, the `yaml-parse` step parses `values.yaml` to extract the image tag from the `Freight` being promoted. Using dot notation (`image.tag`), it extracts the nested value from the YAML file. ```yaml vars: - name: gitRepo value: https://github.com/example/repo.git steps: - uses: git-clone config: repoURL: ${{ vars.gitRepo }} checkout: - commit: ${{ commitFrom(vars.gitRepo).ID }} path: ./src - branch: stage/${{ ctx.stage }} create: true path: ./out - uses: git-clear config: path: ./out - uses: yaml-parse as: values config: path: './src/charts/my-chart/values.yaml' outputs: - name: imageTag fromExpression: image.tag # Render manifests to ./out, commit, push, etc... ``` Given the sample input YAML: ```yaml image: tag: "v1.2.3" ``` The step would produce the following [outputs](../15-promotion-templates.md#step-outputs): | Name | Type | Value | |------|------|-------| | `imageTag` | `string` | `v1.2.3` | [expr-lang]: https://expr-lang.org --- ## `yaml-update` `yaml-update` updates the values of specified keys in any YAML file, in-place, without disruption to existing formatting choices. :::note[Limitations] `yaml-update` updates scalar values only and can only update the values of existing keys. ::: ## Configuration | Name | Type | Required | Description | |------|------|----------|-------------| | `path` | `string` | Y | Path to a YAML file. This path is relative to the temporary workspace that Kargo provisions for use by the promotion process. | | `updates` | `[]object` | Y | The details of changes to be applied to the file. At least one must be specified. | | `updates[].key` | `string` | Y | The key to update within the file. For nested values, use dots to delimit key parts. e.g. `image.tag`. The syntax is identical to that supported by the `json-update` step and is documented in more detail [here](https://github.com/tidwall/sjson?tab=readme-ov-file#path-syntax). | | `updates[].value` | `any` | Y | The new scalar value for the key. Typically specified using an expression. Supports strings, numbers, and booleans. | ## Output | Name | Type | Description | |------|------|-------------| | `commitMessage` | `string` | A description of the change(s) applied by this step. Typically, a subsequent [`git-commit` step](git-commit.md) will reference this output and aggregate this commit message fragment with others like it to build a comprehensive commit message that describes all changes. | ## Writing Keys **Nested keys:** ```yaml image: tag: v1.0.0 ``` Update key: `image.tag` **Keys with literal dots:** ```yaml example.com/version: v1.0.0 ``` Update key: `example\.com/version` **Sequences:** ```yaml containers: - name: my-app image: my-app:v1.0 ``` Update key: `containers.0.image` :::note See the [sjson path syntax documentation](https://github.com/tidwall/sjson?tab=readme-ov-file#path-syntax) for the full description of the syntax. ::: ## Examples ### Common Usage In this example, a Helm values file is updated to use a new container image tag. After cloning the repository and clearing the output directory, the `yaml-update` step modifies `values.yaml` to use the image tag from the `Freight` being promoted. This pattern is commonly seen when managing configuration files that need to stay synchronized with deployed container versions. ```yaml vars: - name: gitRepo value: https://github.com/example/repo.git steps: - uses: git-clone config: repoURL: ${{ vars.gitRepo }} checkout: - commit: ${{ commitFrom(vars.gitRepo).ID }} path: ./src - branch: stage/${{ ctx.stage }} create: true path: ./out - uses: git-clear config: path: ./out - uses: yaml-update config: path: ./src/charts/my-chart/values.yaml updates: - key: image.tag value: ${{ imageFrom("my/image").Tag }} # Render manifests to ./out, commit, push, etc... ``` :::info For more information on `imageFrom()` and expressions used in the example above, see the [Expressions](../40-expressions.md#functions) documentation. ::: --- ## Expression Language Reference The [steps](15-promotion-templates.md#steps) of a user-defined promotion process as well as a `Stage`'s [verification arguments](../20-how-to-guides/60-verification.md#arguments-and-metadata) may take advantage of expressions in their configuration. :::info The documentation on this page assumes a general familiarity with the concepts of Promotion Templates and Analysis Templates, as well as some knowledge of how a promotion process is defined as a sequence of discrete steps and how verification is defined in a `Stage`. For more information on Promotion Templates, refer to the [Promotion Templates Reference](15-promotion-templates.md). For detailed coverage of individual promotion steps, refer to the [Promotion Steps Reference](30-promotion-steps/index.md). For information on Analysis Templates, refer to the [Verification Guide](../20-how-to-guides/60-verification.md) and [Analysis Templates Reference](50-analysis-templates.md). ::: ## Syntax All steps in a user-defined promotion processes (i.e. those described by a [Promotion Template](15-promotion-templates.md) and [PromotionTasks](20-promotion-tasks.md)) support the use of an [Expression Language](https://expr-lang.org) as a means of dynamically resolving values in their configuration at promotion time. In addition, [`Stage` verification arguments](../20-how-to-guides/60-verification.md#arguments-and-metadata) may also use expressions to inject dynamic values into the `AnalysisRun` that is created from an `AnalysisTemplate`. All expressions must be enclosed within the `${{` and `}}` delimiters. This is not universally true for all applications of expr-lang. Kargo selected these specific delimiters to mimic GitHub Actions expression syntax, which many users will already be familiar with. Basic example: ```yaml config: message: ${{ "Hello, world!" }} ``` The above example will be evaluated as the following: ```yaml config: message: Hello, world! ``` :::info What if? If an expression must contain the literal character sequence `}}`, that sequence can be mistaken as marking the end of the expression, cutting the expression short, and resulting in an error or, at least, unexpected results. Inconveniently, the `}}` character sequence is a common one occurring in the definition of JSON objects or string representations of such an objects. The following would fail, because the expression would be misidentified as `{"labels": {"app": "nginx"`, which is not valid JSON: ```yaml config: jsonObject: ${{ {"labels": {"app": "nginx"}} }} ``` As a workaround for the above, a secondary set of delimiters, `${%` and `%}`, are supported as well. Opt for these delimiters when an expression must contain the literal character sequence `}}`. Similarly, always opt for `${{` and `}}` as delimiters when an expression must contain the literal character sequence `%}`. ```yaml config: jsonData: ${% {"labels": {"app": "nginx"}} %} ``` ::: The [expr-lang language definition docs](https://expr-lang.org/docs/language-definition) provide a comprehensive overview of the language's syntax and capabilities, so this reference will continue to focus only on Kargo-specific extensions and usage. :::info You can test your expressions using the [expr-lang playground](https://expr-lang.org/playground). The playground allows you to evaluate expressions against sample data and see the results in real-time. This is especially useful for debugging and validating your expressions before using them in your Kargo configuration. ::: ## Structure and Behavior ### Config Blocks In promotion steps, expressions appear within configuration blocks that can have nested values. Kargo will evaluate expressions just-in-time as each step of a promotion process is executed. It will _only_ evaluate expressions within _values_ of a configuration block and will _not_ evaluate expressions within keys. Expressions in values are evaluated recursively, so expressions may be nested any number of levels deep within a configuration block. ```yaml config: nested: value: ${{ foo.bar }} other: ${{ baz.qux }} ``` ### Variables In promotion step and verification arguments, expressions appear in a flat list of argument name-value pairs. Each argument has a name and a single value that can contain an expression. Unlike configuration blocks, these arguments do not support nested values. ### Validation Kargo parses configuration blocks _before_ evaluating expressions, so any configuration containing expressions _must_ be well-formed YAML even prior to evaluation. Further validation (e.g. for adherence to a step-specific schema) is performed only _after_ expressions are evaluated. ### Types Due to the requirement that configuration blocks be well-formed YAML, all fields containing expressions must be `string`s. Internally, all expressions will also evaluate to `string`s, however, Kargo will attempt to coerce the results to other valid JSON types (YAML is a superset of JSON) including `object`, `array`, `number`, `boolean`, and `null` before concluding that the evaluated expression should continue to be treated as a `string`. This behavior should be unsurprising and perhaps even familiar to experienced YAML users, as YAML parsers behave in the same way. `42`, for example, is interpreted as a JSON `number` unless it is explicitly quoted (i.e. `"42"`) to specify that it should be interpreted as a `string`. In practice, this means care should be taken to use Kargo's built-in `quote()` function in cases where an evaluated expression may appear to be a `number` or `boolean`, for instance, but should be treated as a `string`. For example: ```yaml config: numField: ${{ 40 + 2 }} # Will be treated as a number strField: ${{ quote(40 + 2) }} # Will be treated as a string ``` The above example will be evaluated to the following: ```yaml config: numField: 42 strField: "42" ``` ## Pre-Defined Variables Kargo provides a number of pre-defined variables that are accessible within expressions. This section enumerates these variables, their structure, and use. :::info Expect other useful variables to be added in the future. ::: ### Promotion Variables | Name | Type | Description | |------|------|-------------| | `ctx` | `object` | Contains contextual information about the promotion. See detailed structure below. | | `outputs` | `object` | A map of output from previous promotion steps indexed by step aliases. | | `vars` | `object` | A user-defined map of variable names to static values of any type. The map is derived from both the `Stage`'s `spec.vars` field and the `Promotion`'s `spec.promotionTemplate.spec.vars` field, with promotion template variables taking precedence over Stage-level variables for any conflicting names. Variable names must observe standard Go variable-naming rules. Variables values may, themselves, be defined using an expression. `vars` (contains previously defined variables) and `ctx` are available to expressions defining the values of variables, however, `outputs` and `secrets` are not. | | `task` | `object` | A map containing output from previous steps within the same PromotionTask under the `outputs` field, indexed by step aliases. Only available within `(Cluster)PromotionTask` steps. | #### Context (`ctx`) Object Structure The `ctx` object has the following structure: ```console ctx ├── project: string # The name of the Project ├── stage: string # The name of the Stage ├── promotion: string # The name of the Promotion ├── targetFreight │ ├── name: string # The name of the Freight that is initiated this Promotion │ └── origin │ └── name: string # The name of the Warehouse that contains the Freight └── meta └── promotion └── actor: string # The creator of the Promotion ``` The following example promotion process clones a repository and checks out two branches to different directories, uses Kustomize with source from one branch to render some Kubernetes manifests that it commits to the other branch, and pushes back to the repository. These steps make extensive use of the pre-defined variables `ctx`, `outputs`, and `vars`. ```yaml promotionTemplate: spec: vars: - name: gitRepo value: https://github.com/example/repo.git - name: srcPath value: ./src - name: outPath value: ./out - name: targetBranch value: stage/${{ ctx.stage }} steps: - uses: git-clone config: repoURL: ${{ vars.gitRepo }} checkout: - fromFreight: true path: ${{ vars.srcPath }} - branch: ${{ vars.targetBranch }} create: true path: ${{ vars.outPath }} - uses: git-clear config: path: ${{ vars.outPath }} - uses: kustomize-set-image as: update-image config: path: ${{ vars.srcPath }}/base images: - image: public.ecr.aws/nginx/nginx - uses: kustomize-build config: path: ${{ vars.srcPath }}/stages/${{ ctx.stage }} outPath: ${{ vars.outPath }} - uses: git-commit as: commit config: path: ${{ vars.outPath }} message: ${{ outputs['update-image'].commitMessage }} - uses: git-push config: path: ${{ vars.outPath }} targetBranch: ${{ vars.targetBranch }} - uses: argocd-update config: apps: - name: example-${{ ctx.stage }} sources: - repoURL: ${{ vars.gitRepo }} desiredRevision: ${{ outputs.commit.commit }} ``` :::info Since the usage of expressions and pre-defined variables effectively parameterizes the promotion process, the same promotion process can be reused in other `Projects` or `Stages` with few, if any, modifications (other than the definition of the static variables). ::: ### Verification Variables | Name | Type | Description | |------|------|-------------| | `ctx` | `object` | Contains contextual information about the stage. See structure below. | | `vars` | `object` | A user-defined map of variable names to static values of any type. The map is derived from the `Stage`'s `spec.vars` field. Variable names must observe standard Go variable-naming rules. Stage-level variables can be referenced in verification arguments to pass dynamic values to AnalysisRuns. | :::note Verification processes evaluate a `Stage`'s current state and, while frequently executed immediately following a promotion, are not intrinsically part of the promotion process itself. Therefore, promotion-level variables (such as those defined in `spec.promotionTemplate.spec.vars`) and promotion context (like `outputs` from promotion steps) are not accessible during verification. Only Stage-level variables and context are available. ::: #### Context (`ctx`) Object Structure for Verification The `ctx` object for verification has the following structure: ```console ctx ├── project: string # The name of the Project └── stage: string # The name of the Stage ``` ## Functions Besides the [built-in functions](https://expr-lang.org/docs/language-definition) provided by expr-lang itself, Kargo provides a number of additional functions that can be used within expressions in promotion steps and verification arguments. These functions allow you to access Kubernetes resources, manipulate strings, and retrieve information about the current promotion context, among other things. They are designed to be used in conjunction with the [pre-defined variables](#pre-defined-variables) to create dynamic and flexible promotion processes and verification steps. ### `quote(value)` The `quote()` function converts any value to its string representation. It has one required argument: - `value` (Required): A value of any type to be converted to a string. This is useful for scenarios where an expression evaluates to a non-`string` JSON type, but you wish to treat it as a `string` regardless. For string inputs, it produces clean output without visible quotation marks. Example: ```yaml config: numField: ${{ 40 + 2 }} # Will be treated as a number (42) strField: ${{ quote(40 + 2) }} # Will be treated as a string ("42") rawField: ${{ quote("string") }} # Will result in "string" ``` ### `unsafeQuote(value)` The `unsafeQuote()` function converts any value to its string representation. It has one required argument: - `value` (Required): A value of any type to be converted to a string. Compared to [`quote()`](#quotevalue) this function is considered "unsafe" because it adds escaped quotes around input values that are already considered strings. For non-string values, it behaves similarly to `quote()`. Example: ```yaml config: numField: ${{ 40 + 2 }} # Will be treated as a number (42) strField: ${{ unsafeQuote(40 + 2) }} # Will result in "42" rawField: ${{ unsafeQuote("string") }} # Will result in "\"string\"" ``` ### `configMap(name)` The `configMap()` function returns the `Data` field (a `map[string]string`) of a Kubernetes `ConfigMap` with the specified name from the `Project`'s namespace. If no such `ConfigMap` exists, an empty map is returned. Example: ```yaml config: repoURL: ${{ configMap('my-config').repoURL }} ``` ### `sharedConfigMap(name)` The `sharedConfigMap()` function returns the `Data` field (a `map[string]string`) of a Kubernetes `ConfigMap` with the specified name from the shared resources namespace. If no such `ConfigMap` exists, an empty map is returned. Example: ```yaml config: repoURL: ${{ sharedConfigMap('my-config').repoURL }} ``` ### `secret(name)` The `secret()` function returns the `Data` field of a Kubernetes `Secret` with the specified name from the `Project`'s namespace decoded into a `map[string]string`. If no such `Secret` exists, an empty map is returned. Examples: ```yaml config: headers: - name: Authorization value: Bearer ${{ secret('slack').token }} ``` ### `sharedSecret(name)` The `sharedSecret()` function returns the `Data` field of a Kubernetes `Secret` with the specified name from the shared resources namespace decoded into a `map[string]string`. If no such `Secret` exists, an empty map is returned. **By design, this function only works for "generic credentials" (i.e. `Secret`s labeled with `kargo.akuity.io/cred-type: generic`).** Examples: ```yaml config: headers: - name: Authorization value: Bearer ${{ sharedSecret('slack').token }} ``` :::note `sharedSecret()` can only be used to retrieve shared `Secret`s whose `kargo.akuity.io/cred-type` label is `generic`. ::: ### `warehouse(name)` The `warehouse()` function returns a `FreightOrigin` object representing a `Warehouse`. It has one required argument: - `name` (Required): A string representing the name of a `Warehouse` resource in the same `Project` as the `Promotion` being executed. The returned `FreightOrigin` object has the following fields: | Field | Description | |-------|-------------| | `Kind` | The kind of the `FreightOrigin`. Always equals `Warehouse` for this function. | | `Name` | The name of the `Warehouse` resource. | The `FreightOrigin` object can be used as an optional argument to the `commitFrom()`, `imageFrom()`, or `chartFrom()` functions to disambiguate the desired source of an artifact when necessary. These functions return `nil` when relevant `Freight` is not found from the `FreightCollection`. :::info You can handle `nil` values gracefully in Expr using its [nil coalescing](https://expr-lang.org/docs/language-definition#nil-coalescing) and [optional chaining](https://expr-lang.org/docs/language-definition#optional-chaining) features. ::: ### `freightMetadata(freightName)` The `freightMetadata()` function retrieves the map of all metadata stored in a `Freight` resource. It has one required argument: - `freightName` (Required): The name of the `Freight` resource Example: ```yaml config: # Access metadata values using dot notation category: ${{ freightMetadata(ctx.targetFreight.name).category }} # Using nil coalescing (??) to provide default values if metadata is missing settings: ${{ freightMetadata(ctx.targetFreight.name)['settings'] ?? "default-settings" }} # Using optional chaining (?.) with nil coalescing for nested values nested: ${{ freightMetadata(ctx.targetFreight.name)?.config?.settings?.timeoutSeconds ?? 300 }} ``` :::info You can handle `nil` values gracefully in Expr using its [nil coalescing](https://expr-lang.org/docs/language-definition#nil-coalescing) and [optional chaining](https://expr-lang.org/docs/language-definition#optional-chaining) features. ::: ### `stageMetadata(stageName)` The `stageMetadata()` function retrieves metadata stored in a `Stage` resource. It has one required argument: - `stageName` (Required): The name of the `Stage` resource This returns a map containing all metadata key/value pairs stored in the `Stage` resource. Example: ```yaml config: # Access metadata values using ['key-name'] syntax region: ${{ stageMetadata(ctx.stage)['aws-region'] }} # Using nil coalescing (??) to provide default values if metadata is missing tier: ${{ stageMetadata(ctx.stage)['tier'] ?? "default-tier" }} # Using optional chaining (?.) with nil coalescing for nested values nested: ${{ stageMetadata(ctx.stage)?.config?.settings?.timeoutSeconds ?? 300 }} ``` :::info You can handle `nil` values gracefully in Expr using its [nil coalescing](https://expr-lang.org/docs/language-definition#nil-coalescing) and [optional chaining](https://expr-lang.org/docs/language-definition#optional-chaining) features. ::: ### `commitFrom()` The signature and return value of the `commitFrom()` function vary slightly with the context in which it's used. In the context of an optional expression evaluated to determine whether criteria have been met for automatic creation of a `Freight` resource following a `Warehouse`'s artifact discovery process, the function signature is: `commitFrom(repoURL)` It has one required argument: - `repoURL` (Required): The URL of a Git repository. The returned `DiscoveredCommit` object has the following fields: | Field | Description | |-------|-------------| | `ID` | The ID of the Git commit. | | `Branch` | Branch is the branch in which the commit was found. This field is optional, and populated based on the CommitSelectionStrategy of the GitSubscription. | | `Tag` | Tag is the tag that resolved to this commit. This field is optional, and populated based on the CommitSelectionStrategy of the GitSubscription. | | `Subject` | The first line of the commit message. | | `Author` | Author is the author of the commit. | | `Committer` | Committer is the person who committed the commit. | | `CreatorDate` | The creation date of the commit as specified by the commit. | Example: ```yaml spec: freightCreationPolicy: Automatic subscriptions: - git: repoURL: https://github.com/example/frontend.git - git: repoURL: https://github.com/example/backend.git freightCreationCriteria: expression: | commitFrom('https://github.com/example/frontend.git').Tag == commitFrom('https://github.com/example/backend.git').Tag ``` In all other contexts, such as promotion and verification processes, the function signature is: `commitFrom(repoURL, [freightOrigin])` It has one required and one optional argument: - `repoURL` (Required): The URL of a Git repository. - `freightOrigin` (Optional): A `FreightOrigin` object (obtained from [`warehouse()`](#warehousename)) to specify which `Warehouse` should provide the commit information. The returned `GitCommit` object has the following fields: | Field | Description | |-------|-------------| | `RepoURL` | The URL of the Git repository the commit originates from. | | `ID` | The ID of the Git commit. | | `Branch` | The branch of the repository where this commit was found. Only present if the `Warehouse`'s Git subscription is configured to track branches. | | `Tag` | The tag of the repository where this commit was found. Only present if the `Warehouse`'s Git subscription is configured to track tags. | | `Message` | The first line of the commit message (up to 80 characters). | | `Author` | The name and email address of the commit author. | | `Committer` | The name and email address of the committer. | The optional `freightOrigin` argument should be used when a `Stage` requests `Freight` from multiple origins (`Warehouse`s) and more than one can provide a `GitCommit` object from the specified repository. If a commit is not found from the `FreightCollection`, returns `nil`. Examples: ```yaml config: commitID: ${{ commitFrom("https://github.com/example/repo.git").ID }} ``` ```yaml config: commitID: ${{ commitFrom("https://github.com/example/repo.git", warehouse("my-warehouse")).ID }} ``` ### `imageFrom()` The signature of the `imageFrom()` function varies slightly with the context in which it's used. In the context of an optional expression evaluated to determine whether criteria have been met for automatic creation of a `Freight` resource following a `Warehouse`'s artifact discovery process, the function signature is: `imageFrom(repoURL)` It has one required argument: - `repoURL` (Required): The URL of an image repository. The returned `Image` object has the following fields: | Field | Description | |-------|-------------| | `RepoURL` | The URL of the container image repository the image originates from. | | `Tag` | The tag of the image. | | `Digest` | The digest of the image. | | `Annotations` | A map of [annotations](https://specs.opencontainers.org/image-spec/annotations/) discovered for the image. | Example: ```yaml spec: freightCreationPolicy: Automatic subscriptions: - image: repoURL: ghcr.io/example/frontend - image: repoURL: ghcr.io/example/backend freightCreationCriteria: expression: | imageFrom('ghcr.io/example/frontend.git').Tag == imageFrom('ghcr.io/example/backend.git').Tag ``` :::info The returned `Image` object is the same in all contexts. ::: In all other contexts, such as promotion and verification processes, the function signature is: `imageFrom(repoURL, [freightOrigin])` It has one required and one optional argument: - `repoURL` (Required): The URL of a container image repository. - `freightOrigin` (Optional): A `FreightOrigin` object (obtained from [`warehouse()`](#warehousename)) to specify which `Warehouse` should provide the image information. If an image is not found from the `FreightCollection`, returns `nil`. Examples: ```yaml config: imageTag: ${{ imageFrom("public.ecr.aws/nginx/nginx").Tag }} ``` ```yaml config: imageTag: ${{ imageFrom("public.ecr.aws/nginx/nginx", warehouse("my-warehouse")).Tag }} ``` ### `chartFrom()` The signature of the `chartFrom()` function varies slightly with the context in which it's used. In the context of an optional expression evaluated to determine whether criteria have been met for automatic creation of a `Freight` resource following a `Warehouse`'s artifact discovery process, the function signature is: `chartFrom(repoURL, [chartName])` It has one required and one optional argument: - `repoURL` (Required): The URL of a Helm chart repository. - `chartName` (Optional): The name of the chart (required for HTTP/S repositories, not needed for OCI registries). The `chartFrom()` function returns a corresponding `Chart` object. | Field | Description | |-------|-------------| | `RepoURL` | The URL of the Helm chart repository the chart originates from. For HTTP/S repositories, this is the URL of the repository. For OCI repositories, this is the URL of the container image repository including the chart's name. | | `Name` | The name of the Helm chart. Only present for HTTP/S repositories. | | `Version` | The version of the Helm chart. | Example: ```yaml spec: freightCreationPolicy: Automatic subscriptions: - chart: repoURL: oci://example.com/my-chart - chart: repoURL: oci://example.com/my-other-chart freightCreationCriteria: expression: | chartFrom('oci://example.com/my-chart').Version == chartFrom('oci://example.com/my-other-chart').Version ``` :::info The returned `Chart` object is the same in all contexts. ::: In all other contexts, such as promotion and verification processes, the function signature is: `chartFrom(repoURL, [chartName], [freightOrigin])` It has one required and two optional arguments: - `repoURL` (Required): The URL of a Helm chart repository. - `chartName` (Optional): The name of the chart (required for HTTP/S repositories, not needed for OCI registries). - `freightOrigin` (Optional): A `FreightOrigin` object (obtained from [`warehouse()`](#warehousename)) to specify which `Warehouse` should provide the chart information. For Helm charts stored in OCI registries, the URL should be the full path to the repository within that registry. For Helm charts stored in classic (HTTP/S) repositories, which can store multiple different charts within a single repository, the `chartName` argument must be provided to specify the name of the chart within the repository. The optional `freightOrigin` argument should be used when a `Stage` requests `Freight` from multiple origins (`Warehouse`s) and more than one can provide a `Chart` object from the specified repository. If a chart is not found from the `FreightCollection`, returns `nil`. Examples: ```yaml # OCI registry config: chartVersion: ${{ chartFrom("oci://example.com/my-chart").Version }} ``` ```yaml # OCI registry with specific warehouse config: chartVersion: ${{ chartFrom("oci://example.com/my-chart", warehouse("my-warehouse")).Version }} ``` ```yaml # HTTP/S repository config: chartVersion: ${{ chartFrom("https://example.com/charts", "my-chart").Version }} ``` ```yaml # HTTP/S repository with specific warehouse config: chartVersion: ${{ chartFrom("https://example.com/charts", "my-chart", warehouse("my-warehouse")).Version }} ``` ### `success()` The `success()` function checks the status of all preceding steps and returns true if none of them have failed or errored and false otherwise. Examples: ```yaml config: wasSuccessful: ${{ success() }} ``` ### `failure()` The `failure()` function checks the status of all preceding steps and returns true if any of them have failed or errored and false otherwise. Examples: ```yaml config: wasFailure: ${{ failure() }} ``` ### `always()` The `always()` function unconditionally returns true. Examples: ```yaml config: alwaysTrue: ${{ always() }} ``` ### `status(stepAlias)` The `status(stepAlias)` function retrieves the status of a specific step by its alias within the current promotion context; returning its value as a string. Examples: ```yaml config: status: ${{ status("my-step-alias") }} ``` ### `semverParse(version)` The `semverParse()` function parses a semantic version string and returns a struct that provides access to its individual components through method calls. This is useful for scenarios where you need to inspect or manipulate specific version components, such as bumping the major, minor, or patch version. The returned struct provides the following methods: | Method | Return Type | Description | |--------|-------------|-------------| | `Major()` | `number` | Returns the major version component. | | `Minor()` | `number` | Returns the minor version component. | | `Patch()` | `number` | Returns the patch version component. | | `Prerelease()` | `string` | Returns the prerelease identifier (empty string if none). | | `Metadata()` | `string` | Returns the build metadata (empty string if none). | | `IncMajor()` | `Version` | Returns a new version with the major version incremented and minor/patch reset to 0. | | `IncMinor()` | `Version` | Returns a new version with the minor version incremented and patch reset to 0. | | `IncPatch()` | `Version` | Returns a new version with the patch version incremented. | | `String()` | `string` | Returns the full version string. | :::info The function uses the [Semantic Versioning](https://semver.org/) specification to parse versions. It supports versions with or without the `v` prefix, as well as prerelease and build metadata components. ::: Example: ```yaml # Read current chart version from Chart.yaml - uses: yaml-parse as: read-version config: path: ./src/Chart.yaml outputs: - name: currentVersion fromExpression: version # Update Chart.yaml with bumped minor version - uses: yaml-update config: path: ./src/Chart.yaml updates: - key: version value: ${{ semverParse(task.outputs['read-version'].currentVersion).IncMinor().String() }} ``` ### `semverDiff(version1, version2)` The `semverDiff()` function compares two semantic version strings and returns a string indicating the magnitude of difference between them. Possible return values include, and are limited to: | Return Value | Description | |--------------|-------------| | `Major` | The major version components differ. | | `Minor` | The minor version components differ (major versions are the same). | | `Patch` | The patch version components differ (major and minor versions are the same). | | `Metadata` | Only the build metadata differs (major, minor, and patch versions are the same). | | `None` | The versions are identical. | | `Incomparable` | One or both arguments are not valid semantic versions. | :::info The function uses the [Semantic Versioning](https://semver.org/) specification to parse and compare versions. It supports versions with or without the `v` prefix, as well as prerelease and build metadata components. ::: Example: ```yaml # Open a pull request for major version changes of an image; push directly # otherwise... # Presume steps not shown have read and updated the version number of the image # referenced by some manifest. - uses: git-push as: direct-push if: ${{ semverDiff(imageFrom(vars.imageRepo).Tag, outputs['read-version'].currentVersion) != 'Major' }} config: repoURL: ${{ vars.gitRepo }} targetBranch: stage/${{ ctx.stage }} message: ${{ semverDiff(imageFrom(vars.imageRepo).Tag, outputs['read-version'].currentVersion) }} update of ${{ vars.imageRepo }} - uses: git-push as: indirect-push if: ${{ semverDiff(imageFrom(vars.imageRepo).Tag, outputs['read-version'].currentVersion) == 'Major' }} config: repoURL: ${{ vars.gitRepo }} generateTargetBranch: true message: Major update of ${{ vars.imageRepo }} - uses: git-open-pr if: ${{ semverDiff(imageFrom(vars.imageRepo).Tag, outputs['read-version'].currentVersion) == 'Major' }} config: repoURL: https://github.com/example/config-repo.git sourceBranch: ${{ outputs['indirect-push'].branch }} targetBranch: stage/${{ ctx.stage }} title: Major update of ${{ vars.imageRepo }} labels: - breaking-change - needs-review ``` --- ## Analysis Templates Reference An `AnalysisTemplate` is a resource that defines how to perform verification testing, including: * Container images and commands to run * Queries to external monitoring tools * How to interpret results from metric providers * Success or failure criteria * Frequency and duration of measurements `AnalysisTemplate` resources (and the `AnalysisRun` resources that are spawned from them) are CRDs re-used from the [Argo Rollouts](https://argoproj.github.io/argo-rollouts) project. They were intentionally built to be useful in contexts other than Argo Rollouts. Re-using this resource type to define verification processes means those processes benefit from this rich and battle-tested feature of Argo Rollouts. :::info This reference guide is intended to give a brief introduction to `AnalysisTemplate`s for some common use cases. Please consult the [relevant sections](https://argoproj.github.io/argo-rollouts/features/analysis/) of the Argo Rollouts documentation for comprehensive coverage of the full range of `AnalysisTemplate` capabilities. ::: `AnalysisTemplate`s integrate natively with many popular open-source and commercial monitoring tools, including: * [Prometheus](https://prometheus.io/) * [DataDog](https://www.datadoghq.com/) * [Amazon CloudWatch](https://aws.amazon.com/cloudwatch/) * [NewRelic](https://newrelic.com/) * [InfluxDB](https://www.influxdata.com/) * [Apache SkyWalking](https://skywalking.apache.org/) * [Graphite](https://graphiteapp.org/) In addition to monitoring tools, analysis can integrate with internal systems by: * Running containerized processes as Kubernetes `Job`s * Making HTTP requests and interpreting JSON responses ## Arguments `AnalysisTemplate`s may declare a set of arguments that can be "passed" in by the `Stage`. The arguments are resolved at the time the `AnalysisRun` is created and can then be referenced in metrics configuration. Arguments are dereferenced using the syntax: `{{ args. }}`. :::caution Unlike Kargo promotion processes, which require expressions to be enclosed within `${{ }}`, Argo Rollouts `AnalysisTemplate`s require expressions to be enclosed within `{{ }}` (i.e. without `$`). ::: The following example shows an `AnalysisTemplate` with three arguments. Values for arguments can have a default value, supplied by the `Stage`, or obtained from a `Secret` if the value is sensitive (e.g. a bearer token for an HTTP request): ```yaml apiVersion: argoproj.io/v1alpha1 kind: AnalysisTemplate metadata: name: args-example spec: args: # An argument can specify a value to be used as its default. # This will be overridden by a value supplied by the Stage. - name: api-url value: http://example/measure # If an argument specifies no value, it is considered a required # argument and must be supplied by the Stage. - name: service-name # Arguments can be obtained from a Secret in the Project Namespace - name: api-token valueFrom: secretKeyRef: name: token-secret key: apiToken metrics: - name: webmetric successCondition: result == 'true' provider: web: # placeholders are resolved when an AnalysisRun is created url: "{{ args.api-url }}?service={{ args.service-name }}" headers: - key: Authorization value: "Bearer {{ args.api-token }}" jsonPath: "{$.results.ok}" ``` ## Interval and Count To collect multiple measurements over a longer duration, use the `count` and `interval` fields. This allows you to define how many measurements to take and how frequently to take them. In the example below, the analysis is configured to take 20 measurements at 3-minute intervals, resulting in a total duration of approximately 1 hour: ```yaml apiVersion: argoproj.io/v1alpha1 kind: AnalysisTemplate metadata: name: count-interval-example spec: metrics: - name: test # Number of measurements to take count: 20 # Time interval between each measurement as a duration string interval: 3m provider: web: url: https://pokeapi.co/api/v2/pokemon/pikachu ``` :::caution When specifying an `interval`, you must also specify a `count`. Without it, the AnalysisRun would collect an indefinite amount of measurements and never complete (until terminated). ::: ## Success Condition When interpreting the result of a query, an [Expression Language](https://expr-lang.org) expression can be used to evaluate the response. The response payload is set in a variable `result`. The following will interpret the response of a Prometheus query, and require that the element of the returned vector is greater than or equal to `0.95`: ```yaml apiVersion: argoproj.io/v1alpha1 kind: AnalysisTemplate metadata: name: success-rate spec: metrics: - name: success-rate # Expr expression that can be evaluated to true or false # NOTE: prometheus queries return results in the form of a vector. # So it is common to access the index 0 of the returned array to obtain the value successCondition: result[0] >= 0.95 provider: prometheus: address: "http://prometheus.example.com:9090" query: | sum(irate( istio_requests_total{reporter="source",response_code!~"5.*"}[5m] )) / sum(irate( istio_requests_total{reporter="source"}[5m] )) ``` ## Failure Conditions and Limits As an alternative to `successCondition`, a `failureCondition` can be used to describe when a measurement is considered failed. Additionally, `failureLimit` can also be used to specify the maximum number of failed measurements that are allowed before the entire `AnalysisRun` is considered `Failed`. The following example continually polls a Prometheus server to get the total number of errors (i.e., HTTP response code >= 500) every five minutes, causing the measurement to fail if ten or more errors are encountered. The entire `AnalysisRun` is considered to have `Failed` after three failed measurements. ```yaml apiVersion: argoproj.io/v1alpha1 kind: AnalysisTemplate metadata: name: failure-condition-example spec: metrics: - name: total-errors interval: 5m count: 3 failureCondition: result[0] >= 10 failureLimit: 3 provider: prometheus: address: http://prometheus.example.com:9090 query: | sum(irate( istio_requests_total{reporter="source",response_code=~"5.*"}[5m] )) ``` ## Delaying Measurements In some scenarios, it may be necessary to delay the start of a metric measurement. For example, some time may need to pass after an update in order for new data to populate in the monitoring services. The `initialDelay` option can be used to delay the start of measurements. Each metric can be configured to have a different delay. ```yaml apiVersion: argoproj.io/v1alpha1 kind: AnalysisTemplate metadata: name: initial-delay-example spec: metrics: - name: success-rate # Duration before measurement collection. Default is no delay initialDelay: 5m successCondition: result[0] >= 0.90 provider: prometheus: address: http://prometheus.example.com:9090 query: ... ``` ## Example Metric Types ### Web An HTTP request can be performed against some external service to obtain the measurements. ```yaml apiVersion: argoproj.io/v1alpha1 kind: AnalysisTemplate metadata: name: web-metric-example spec: args: - name: api-token valueFrom: secretKeyRef: name: token-secret key: apiToken metrics: - name: webmetric successCondition: result == true provider: web: url: "http://example.com/api/v1/measurement" # HTTP Method. valid values are GET|POST|PUT. Defaults to GET method: POST # Timeout for the request. Defaults to 10 seconds timeoutSeconds: 20 headers: - key: Authorization value: "Bearer {{ args.api-token }}" # if body is a json, it is recommended to set the Content-Type - key: Content-Type value: "application/json" # Requst body to send. body: | {"foo": "bar"} # Optional JSON path to set the value of `result` in successCondition/failureCondition jsonPath: "{$.data.ok}" ``` ### Job A Kubernetes `Job` can be used to perform analysis. When a `Job` is used, the metric is considered successful if the `Job` completes with an exit code of zero and is otherwise considered to have failed. ```yaml apiVersion: argoproj.io/v1alpha1 kind: AnalysisTemplate metadata: name: integration-test namespace: guestbook spec: metrics: - name: integration-test provider: job: spec: template: spec: containers: - name: sleep image: alpine:latest command: [sleep, "10"] restartPolicy: Never backoffLimit: 1 ``` --- ## Annotations and Labels used by Kargo This page documents certain annotations and labels that are applicable to Kargo resource types and other Kubernetes resource types (e.g. `Secret`s or `ServiceAccounts`) and affect how Kargo handles those resources. The subset documented here are those that the maintainers have judged most likely to be of use to advanced users who may, for instance, be looking to interact programmatically with Kargo. ## Annotations | Key | Target Resource(s) | Possible Values | Description | |-----|--------------------|-----------------|-------------| | `kargo.akuity.io/abort` | `Stage` | A plain string (verification ID from `.status.verifications[*].id` of the `Stage`). | Aborts an in-progress `Freight` verification. | | `kargo.akuity.io/authorized-stage` | `Argo CD Application` | `:` | Indicates which `Stage` is authorized to manage the `Application` resource. | | `kargo.akuity.io/color` | `Stage` | Hex color code (e.g. `#ff8800`) | Optional cosmetic color used in the UI's pipeline view. | | `kargo.akuity.io/description` | Any | Any string | Optional human-readable description of the resource. May be used by the Kargo UI to display additional context or details. | | `kargo.akuity.io/refresh` | `Warehouse`, `Stage`, `Promotion` | A string that is unique or at least unlikely to repeat, such as a UUID or a timestamp of "now" | Triggers reconciliation of the resource when its value changes. | | `kargo.akuity.io/reverify` | `Stage` | Either a plain string (verification ID from `.status.verifications[*].id` of the `Stage`) or a JSON object with `id` (required), `actor`, and `controlPlane` fields. If a JSON object is provided, it is parsed as a `VerificationRequest`. | Triggers re-verification of a previously completed verification for the current `Freight`. | | `rbac.kargo.akuity.io/claim.` | `ServiceAccount` | Any valid OIDC claim value (e.g., `sub`, `email`, or `groups`) | Maps an OIDC claim to a `ServiceAccount`, enabling user-to-ServiceAccount mappings. For more details, refer to the access control sections of the [Operator Guide](../../40-operator-guide/40-security/30-access-controls.md) and [User Guide](../50-security/20-access-controls/index.md). | | `rbac.kargo.akuity.io/claims` | `ServiceAccount` | Any valid OIDC claim values (e.g., `sub`, `email`, or `groups`) | Maps multiple OIDC claims to a `ServiceAccount`, enabling user-to-ServiceAccount mappings. This method is preferred over `rbac.kargo.akuity.io/claim.`. For more details, refer to the access control sections of the [Operator Guide](../../40-operator-guide/40-security/30-access-controls.md) and [User Guide](../50-security/20-access-controls/index.md). | `rbac.kargo.akuity.io/managed` | `ServiceAccount`, `Role`, `RoleBinding` | `"true"` | Permits the UI or CLI (via the API server) to programmatically manage trios of `ServiceAccount`, `Role`, and `RoleBinding` resources via Kargo's own ["roles" abstraction](../50-security/20-access-controls/index.md#managing-mappings-and-permissions). Omit this annotation if you wish to exclusively manage these resources [declaratively](../50-security/20-access-controls/index.md#managing-kargo-roles-declaratively). | | `kargo.akuity.io/replicate-to` | `Secret`, `ConfigMap` (in the shared resources namespace) | `"*"` | Enables automatic replication of the resource to all Project namespaces. For more details, see the [Replicating Shared Resources](../../40-operator-guide/40-security/40-managing-secrets.md#replicating-shared-resources-to-project-namespaces) section of the Operator Guide. | | `kargo.akuity.io/replicated-at` | `Secret`, `ConfigMap` (replicated copies) | UTC timestamp | Records when the resource was last replicated from the source. Managed by the system. | ## Labels | Key | Target Resource(s) | Possible Values | Description | |-----|--------------------|-----------------|-------------| | `kargo.akuity.io/alias` | `Freight` | Any string that is unique within the project | Mutable, human-readable alias for a piece of `Freight`. This label is automatically synced from the resource's `alias` field. Users are discouraged from modifying the label directly. The label exists primarily to enable querying for `Freight` by alias using `kubectl`. | | `kargo.akuity.io/cred-type` | `Secret` | `git`, `helm`, `image`, `generic` | Indicates a `Secret` represents credentials for a repository of the specified type. For more details, see the [Managing Secrets](../50-security/30-managing-secrets.md#repository-credentials). | | `kargo.akuity.io/project` | `Namespace` | `"true"` | Indicates that the `Namespace` is eligible for adoption by a `Project` with the same name. This label is useful when `Namespace`s are unavoidably pre-created by some other agent. For more details, see the [Working with Projects](../20-how-to-guides/20-working-with-projects.md#namespace-adoption) section. | | `kargo.akuity.io/replicated-from` | `Secret`, `ConfigMap` (replicated copies) | Source resource name | Identifies the resource as a replica and names its source in the shared resources namespace. Managed by the system. | | `kargo.akuity.io/replicated-sha` | `Secret`, `ConfigMap` (replicated copies) | 16-character truncated SHA-256 | Hash of the source resource's data at the time of last replication, used for change detection. Managed by the system. | | `kargo.akuity.io/shard` | `Promotion`, `Stage`, `Warehouse` | Shard ID | Indicates a specific controller instance responsible for reconciling the resource. For `Warehouse` and `Stage` resources, this label is automatically synced from the resource's `spec.shard` field. Users are discouraged from modifying the label directly. The label exists primarily to enable querying for resources by shard using `kubectl`. | --- ## Artifactory Webhook Receiver The Artifactory Webhook Receiver responds to `pushed` events originating from Artifactory repositories by _refreshing_ all `Warehouse` resources subscribed to those repositories. :::warning This webhook receiver does not respond to events where `domain` is `artifact` and `event_type` is `deployed`. ::: :::info "Refreshing" a `Warehouse` resource means enqueuing it for immediate reconciliation by the Kargo controller, which will execute the discovery of new artifacts from all repositories to which that `Warehouse` subscribes. ::: ## Self-Hosted Artifactory :::info If you are not using a self-hosted Artifactory instance, skip to [the configuring the receiver](#configuring-the-receiver) section. ::: In order for a webhook initiated `Warehouse` refresh to successfully occur, it is required that you set a Custom Base URL for your instance. When this setting hasn't been configured, critical information will be missing from the webhook payloads. 1. Navigate to `https:///ui/admin/configuration/general`, where `` has been replaced with the base URL of your self-hosted Artifactory instance. 1. Set the Custom Base URL field to the base URL of your self-hosted Artifactory instance. ![Custom Base URL](./img/custom_base_url.png "Custom Base URL") 1. At the bottom of the form, click Save. :::info For additional information on configuring your Custom Base URL refer directly to the [Artifactory Docs](https://jfrog.com/help/r/jfrog-platform-administration-documentation/general-settings). ::: ## Configuring the Receiver An Artifactory webhook receiver must reference a Kubernetes `Secret` resource with a `secret-token` key in its data map. This [shared secret](https://en.wikipedia.org/wiki/Shared_secret) will be used by Artifactory to sign requests and by the receiver to verify those signatures. :::note The following commands are suggested for generating and base64-encoding a complex secret: ```shell secret_token=$(openssl rand -base64 48 | tr -d '=+/' | head -c 32) echo "Secret token: $secret_token" echo "Encoded secret token: $(echo -n $secret_token | base64)" ``` ::: ```yaml apiVersion: v1 kind: Secret metadata: name: artifactory-wh-secret namespace: kargo-demo labels: kargo.akuity.io/cred-type: generic data: secret-token: --- apiVersion: kargo.akuity.io/v1alpha1 kind: ProjectConfig metadata: name: kargo-demo namespace: kargo-demo spec: webhookReceivers: - name: artifactory-wh-receiver artifactory: secretRef: name: artifactory-wh-secret ``` ### Virtual Repositories When Warehouses intended to be refreshed by an Artifactory webhook receiver subscribe to Artifactory [virtual repositories](https://jfrog.com/help/r/jfrog-artifactory-documentation/virtual-repositories) there will be discrepancies between the URLs the receiver will infer for the [local repositories](https://jfrog.com/help/r/jfrog-artifactory-documentation/local-repositories) from which push events have originated and the URLs actually used by those Warehouses' subscriptions. To compensate for this, a value can be provided for the Artifactory webhook receiver configuration's `virtualRepoName` field. When specified, its value supersedes the local repository name found in the webhook's payload, which allows the receiver to infer the correct virtual repository URL for which all subscribed Warehouses should be refreshed. In practice, when using virtual repositories, a separate Artifactory webhook receiver should be configured _for each_, but one such receiver can handle events originating from _any number_ of local repositories that are aggregated by that virtual repository. For example, if a virtual repository `proj-virtual` aggregates container images from all of the `proj` Artifactory project's local image repositories, with a single webhook configured to post to the following receiver, an image pushed to `example.frog.io/proj-//image`, will correctly cause that receiver to refresh all Warehouses subscribed to `example.frog.io/proj-virtual//image`. ```yaml apiVersion: v1 kind: Secret metadata: name: artifactory-wh-secret namespace: kargo-demo labels: kargo.akuity.io/cred-type: generic data: secret-token: --- apiVersion: kargo.akuity.io/v1alpha1 kind: ProjectConfig metadata: name: kargo-demo namespace: kargo-demo spec: webhookReceivers: - name: proj-virtual-wh-receiver artifactory: secretRef: name: artifactory-wh-secret virtualRepoName: proj-virtual ``` ## Retrieving the Receiver's URL Kargo will generate a hard-to-guess URL from the receiver's configuration. This URL can be obtained using a command such as the following: ```shell kubectl get projectconfigs kargo-demo \ -n kargo-demo \ -o=jsonpath='{.status.webhookReceivers}' ``` ## Registering with Artifactory 1. Navigate to `https://.jfrog.io/ui/admin/configuration/webhooks`, where `` has been replaced with an Artifactory instance for which you are an administrator. 1. Click New Webhook. ![Webhooks Dashboard](./img/webhooks.png "Webhooks Dashboard") 1. Complete the Create new webhook form: ![Add Webhook](./img/add-webhook.png "Add Webhook") 1. Enter a descriptive name in the Name field. 1. Complete the URL field using the URL [for the webhook receiver](#retrieving-the-receivers-url). 1. Under Execution Results check Show status of successful executions in the Troubleshooting tab. :::info Although Artifactory supports sending test events to the URL, such events are _not_ displayed in the troubleshooting tab; only actual events are. ::: 1. In the Events drop-down, select Docker and OCI ⃗ Tag was pushed. ![Select Trigger](./img/select-trigger.png "Select Trigger") :::info Artifactory supports many different types of registries and repositories. This webhook responds only to events originating from repositories in OCI registries. No other type of repository, including legacy (HTTP/S) Helm chart repositories, is supported. ::: 1. Complete the dialog that appears: ![Select Repos](./img/select-repos.png "Select Repos") 1. Select repositories from which you would like to receive events from those listed on the left. 1. Click > to move your selections to the right. Upon doing so, the Save button will be enabled. ![Repos Selected](./img/repos-selected.png "Repos Selected") 1. Click Save. 1. Under Authentication, complete the Secret token field using the (unencoded) value of the `secret-token` key in the `Secret` resource referenced by the [webhook receiver's configuration](#configuring-the-receiver). ![Setup Auth](./img/setup-auth.png "Setup Auth") 1. Select Use secret for payload signing. :::caution The webhook receiver won't accept unsigned requests. ::: 1. Click Save. You will be redirected to the Webhooks page where the newly created webhook will appear. ![Created](./img/created.png "Created") --- ## Azure Webhook Receiver The Azure webhook receiver responds to `push` and `ping` events originating from Azure Container Registry repositories and `git.push` events originating from Azure DevOps repositories. The receiver unconditionally responds to `ping` events with an HTTP `200` status code. The receiver responds to `push` and `git.push` events by _refreshing_ all `Warehouse` resources subscribed to the repositories that correspond to the event. :::info "Refreshing" a `Warehouse` resource means enqueuing it for immediate reconciliation by the Kargo controller, which will execute the discovery of new artifacts from all repositories to which that `Warehouse` subscribes. ::: ## Configuring the Receiver An Azure webhook receiver must reference a Kubernetes `Secret` resource with a `secret` key in its data map. :::info _This secret will not be shared directly with Azure._ Azure does not natively implement any mechanism whereby receivers may authenticate inbound webhook requests. To compensate for this, Kargo incorporates the secret into the generation of a hard-to-guess URL for the receiver. This URL serves as a _de facto_ [shared secret](https://en.wikipedia.org/wiki/Shared_secret) and authentication mechanism. ::: :::note The following commands are suggested for generating and base64-encoding a complex secret: ```shell secret=$(openssl rand -base64 48 | tr -d '=+/' | head -c 32) echo "Secret: $secret" echo "Encoded secret: $(echo -n $secret | base64)" ``` ::: ```yaml apiVersion: v1 kind: Secret metadata: name: azure-wh-secret namespace: kargo-demo labels: kargo.akuity.io/cred-type: generic data: secret: --- apiVersion: kargo.akuity.io/v1alpha1 kind: ProjectConfig metadata: name: kargo-demo namespace: kargo-demo spec: webhookReceivers: - name: azure-wh-receiver azure: secretRef: name: azure-wh-secret ``` ## Retrieving the Receiver's URL Kargo will generate a hard-to-guess URL from the receiver's configuration. This URL can be obtained using a command such as the following: ```shell kubectl get projectconfigs kargo-demo \ -n kargo-demo \ -o=jsonpath='{.status.webhookReceivers}' ``` ## Registering with Azure The Azure webhook receiver supports both Azure Container Registry and Azure DevOps services. ### Azure Container Registry 1. In your Azure Container Registry portal, on the left-hand side, navigate to Services → Webhooks. 1. At the top of the screen, click Add. ![Webhooks](./img/acr/webhooks.png "Webhooks") 1. Complete the Create webhook form: ![Create Webhook](./img/acr/create-webhook.png "Create Webhook") 1. Enter a descriptive name in the Webhook name field. 1. Select the Location closest to where your Kargo instance is running. 1. Complete the Service URI field using the URL [for the webhook receiver](#retrieving-the-receivers-url). 1. From the Actions drop-down menu, select `push`. 1. Ensure Active is checked. 1. Set the Scope using the format `:`. :::note Wildcards are supported in both the `repository` and `tag` part of the Scope string. ::: 1. Click Create. 1. Return to the Webhooks dashboard and select the newly created webhook. ![Created](./img/acr/created.png "Created") 1. Click Ping. 1. Click Refresh. ![Ping](./img/acr/ping.png "Ping") 1. Confirm a 200 Http Status for the Ping action. ![Success](./img/acr/success.png "Success") :::info For additional information on configuring ACR webhooks, refer directly to the [ACR Docs](https://learn.microsoft.com/en-us/azure/container-registry/container-registry-webhook#create-webhook---azure-portal). ::: ### Azure DevOps 1. Navigate to `https://dev.azure.com///_settings/serviceHooks`, where`` has been replaced with an organization name and `` has been replaced with the name of a project belonging to that account and for which you are an administrator. 1. On the left-hand-side menu under General, click Service Hooks. 1. Click Create Subscription. ![Create Sub](./img/devops/create-subscription.png "Create Sub") 1. In the New Service Hooks Subscription modal, scroll down to Webhooks on the left-hand-side. 1. Click Webhooks. 1. Click Next. ![New Service Hook](./img/devops/new-service-hook.png "New Service Hook") 1. Complete the Trigger form: ![Trigger](./img/devops/trigger.png "Trigger") 1. Select Code Pushed from the drop-down menu labeled Trigger on this type of event. 1. Select the Repositories, Branches, and Users for which the event will trigger for. 1. Click Next. 1. Complete the Action form: ![Action](./img/devops/action.png "Action") 1. Complete the Service URI field using the URL [for the webhook receiver](#retrieving-the-receivers-url). 1. Click Test. ![Success](./img/devops/success.png "Success") 1. After receiving a Succeeded notification, click Close. 1. Click Finish. :::info For additional information on configuring Azure DevOps webhooks, refer directly to the [Azure DevOps Docs](https://learn.microsoft.com/en-us/azure/devops/service-hooks/services/webhooks?view=azure-devops). ::: --- ## Bitbucket Webhook Receiver The Bitbucket webhook receiver responds to `repo:push` events originating from Bitbucket repositories by _refreshing_ all `Warehouse` resources subscribed to those repositories. :::info "Refreshing" a `Warehouse` resource means enqueuing it for immediate reconciliation by the Kargo controller, which will execute the discovery of new artifacts from all repositories to which that `Warehouse` subscribes. ::: :::info The Bitbucket webhook receiver also works with Bitbucket Server and Bitbucket Data Center, although some URLs in this document may need to be adjusted accordingly. ::: ## Configuring the Receiver A Bitbucket webhook receiver must reference a Kubernetes `Secret` resource with a `secret` key in its data map. This [shared secret](https://en.wikipedia.org/wiki/Shared_secret) will be used by Bitbucket to sign requests and by the receiver to verify those signatures. :::note The following commands are suggested for generating and base64-encoding a complex secret: ```shell secret=$(openssl rand -base64 48 | tr -d '=+/' | head -c 32) echo "Secret: $secret" echo "Encoded secret: $(echo -n $secret | base64)" ``` ::: ```yaml apiVersion: v1 kind: Secret metadata: name: bb-wh-secret namespace: kargo-demo labels: kargo.akuity.io/cred-type: generic data: secret: --- apiVersion: kargo.akuity.io/v1alpha1 kind: ProjectConfig metadata: name: kargo-demo namespace: kargo-demo spec: webhookReceivers: - name: bb-wh-receiver bitbucket: secretRef: name: bb-wh-secret ``` ## Retrieving the Receiver's URL Kargo will generate a hard-to-guess URL from the receiver's configuration. This URL can be obtained using a command such as the following: ```shell kubectl get projectconfigs kargo-demo \ -n kargo-demo \ -o=jsonpath='{.status.webhookReceivers}' ``` ## Registering with Bitbucket To configure a single Bitbucket repository to notify a receiver of `repo:push` events: 1. Navigate to `https://bitbucket.org///admin/webhooks` where `` has been replaced with a Bitbucket workspace and `` has been replaced with the name of a repository belonging to that workspace and for which you are an administrator. 1. Click Add webhook. 1. Complete the Add new webhook form: ![Add New Webhook Form](./img/add-new-webhook-form.png "Add New Webhook Form") 1. Enter a descriptive name in the Title field. 1. Complete the URL field using the URL [for the webhook receiver](#retrieving-the-receivers-url). 1. Complete the Secret field using the (unencoded) value assigned to the `secret` key of the `Secret` resource referenced by the [webhook receiver's configuration](#configuring-the-receiver). :::danger Do not use the Generate secret button in the Bitbucket UI. Kargo incorporates the secret's value into the URL it generates for the webhook receiver. Using a secret in this field other than the one already referenced by the receiver's configuration will require revisiting that configuration _and doing so will change the receiver's URL._ ::: 1. Under Status, ensure Active is selected. 1. Under Triggers → Repository, ensure Push is selected. 1. Click Save. 1. Verify that the new webhook appears under Repository hooks. 1. If you'd like to record outbound webhook requests for troubleshooting purposes: 1. Click the View requests link next to your webhook. 1. Click on Enable History. ![Enable History](./img/enable-history.png "Enabled History") When these steps are complete, the repository will send events to the webhook receiver. :::info For additional information on configuring webhooks, refer directly to the [Bitbucket Docs](https://support.atlassian.com/bitbucket-cloud/docs/manage-webhooks/). ::: --- ## Docker Hub Webhook Receiver The Docker Hub webhook receiver responds to events originating from Docker Hub repositories by _refreshing_ all `Warehouse` resources subscribed to those repositories. :::info "Refreshing" a `Warehouse` resource means enqueuing it for immediate reconciliation by the Kargo controller, which will execute the discovery of new artifacts from all repositories to which that `Warehouse` subscribes. ::: ## Configuring the Receiver A Docker Hub webhook receiver must reference a Kubernetes `Secret` resource with a `secret` key in its data map. :::info _This secret will not be shared directly with Docker Hub._ Docker Hub does not natively implement any mechanism whereby receivers may authenticate inbound webhook requests. To compensate for this, Kargo incorporates the secret into the generation of a hard-to-guess URL for the receiver. This URL serves as a _de facto_ [shared secret](https://en.wikipedia.org/wiki/Shared_secret) and authentication mechanism. ::: :::note The following commands are suggested for generating and base64-encoding a complex secret: ```shell secret=$(openssl rand -base64 48 | tr -d '=+/' | head -c 32) echo "Secret: $secret" echo "Encoded secret: $(echo -n $secret | base64)" ``` ::: ```yaml apiVersion: v1 kind: Secret metadata: name: dh-wh-secret namespace: kargo-demo labels: kargo.akuity.io/cred-type: generic data: secret: --- apiVersion: kargo.akuity.io/v1alpha1 kind: ProjectConfig metadata: name: kargo-demo namespace: kargo-demo spec: webhookReceivers: - name: dh-wh-receiver dockerhub: secretRef: name: dh-wh-secret ``` ## Retrieving the Receiver's URL Kargo will generate a hard-to-guess URL from the receiver's configuration. This URL can be obtained using a command such as the following: ```shell kubectl get projectconfigs kargo-demo \ -n kargo-demo \ -o=jsonpath='{.status.webhookReceivers}' ``` ## Registering with Docker Hub To configure a single Docker Hub repository to notify a receiver when objects (like container images or Helm charts) have been pushed to it: 1. Navigate to a Docker Hub repository for which you are an administrator. 1. Select the Webhooks tab. ![Webhooks Tab](./img/webhooks-tab.png "Webhooks Tab") 1. Complete the New webhook form: ![New Webhook Form](./img/new-webhook.png "New Webhook Form") 1. Enter a descriptive name in the Webhook name field. 1. Complete the Webhook URL field using the URL [for the webhook receiver](#retrieving-the-receivers-url). 1. Click +. ![Create Webhook Button](./img/create-webhook.png "Create Webhook Button") 1. Verify the new webhook appears under Current webhooks. :::note If you'd like to review outbound webhook requests for troubleshooting purposes, select the three dots to the right of a webhook, then select View history from the context menu. ::: When these steps are complete, the repository will send events to the webhook receiver. :::info For additional information on configuring webhooks, refer directly to the [Docker Hub Docs](https://docs.docker.com/docker-hub/repos/manage/webhooks/). ::: --- ## Generic Webhook Receiver The generic webhook receiver responds to any inbound POST request by determining whether it meets user-defined criteria, then executing user-defined actions on a user-defined set of resources when it does. :::note Currently, these actions are limited to "refreshing" `Warehouse` resources, which triggers their artifact discovery processes, so a typical use of this component is responding to "push" events from artifact repositories that lack dedicated webhook receiver implementations. Since this component effectively enables imperatively refreshing a `Warehouse` from any external process, other uses are possible and practical. ::: ## Configuring the Receiver ### Base Configuration A Generic webhook receiver must reference a Kubernetes `Secret` resource with a `secret` key in its data map. :::info Kargo incorporates the secret into the generation of a hard-to-guess URL for the receiver. This URL serves as a _de facto_ [shared secret](https://en.wikipedia.org/wiki/Shared_secret) and authentication mechanism. ::: :::note The following commands are suggested for generating and base64-encoding a complex secret: ```shell secret=$(openssl rand -base64 48 | tr -d '=+/' | head -c 32) echo "Secret: $secret" echo "Encoded secret: $(echo -n $secret | base64)" ``` ::: ```yaml apiVersion: v1 kind: Secret metadata: name: wh-secret namespace: kargo-demo labels: kargo.akuity.io/cred-type: generic data: secret: --- apiVersion: kargo.akuity.io/v1alpha1 kind: ProjectConfig metadata: name: kargo-demo namespace: kargo-demo spec: webhookReceivers: - name: my-receiver generic: secretRef: name: wh-secret ``` ### Defining Actions Actions are defined by: 1. [`action`](#action) 1. [`whenExpression`](#whenexpression) 1. [`targetSelectionCriteria`](#targetselectioncriteria) #### action The `action` field specifies the action that should be performed. ```yaml apiVersion: kargo.akuity.io/v1alpha1 kind: ProjectConfig metadata: name: kargo-demo namespace: kargo-demo spec: webhookReceivers: - name: my-receiver generic: secretRef: name: wh-secret actions: - action: Refresh ``` :::note The only currently supported `action` is `Refresh`. ::: #### whenExpression Use `whenExpression` to ensure that an action is only executed when specific criteria are met, providing fine-grained control over webhook behavior. The following example depicts an action that is executed only when the request contains an `X-Event-Type` header with the value `push`: ```yaml apiVersion: kargo.akuity.io/v1alpha1 kind: ProjectConfig metadata: name: kargo-demo namespace: kargo-demo spec: webhookReceivers: - name: my-receiver generic: secretRef: name: wh-secret actions: - action: Refresh whenExpression: "request.header("X-Event-Type") == 'push'" ``` :::note This can be left empty if the action should run unconditionally. ::: #### targetSelectionCriteria `targetSelectionCriteria` is used to select resources that an action needs to be performed on. There are three ways to define `targetSelectionCriteria`: 1. [By Name](#by-name) 1. [By Labels](#by-labels) 1. [By Values in an Index](#by-values-in-an-index) All methods support both static and [dynamic](#expression-reference) values. :::note Using more than one of the above selects resources at the logical intersection of the criteria. (i.e. Criteria are "ANDed.") ::: ##### By Name The simplest way to select a resource is by specifying its `name`. The following example depicts `targetSelectionCriteria` that selects `Warehouse` resources by a static `name`. ```yaml apiVersion: kargo.akuity.io/v1alpha1 kind: ProjectConfig metadata: name: kargo-demo namespace: kargo-demo spec: webhookReceivers: - name: my-receiver generic: secretRef: name: wh-secret actions: - action: Refresh whenExpression: "request.header('X-Event-Type') == 'push'" targetSelectionCriteria: - kind: Warehouse name: my-warehouse ``` The following example depicts `targetSelectionCriteria` that selects `Warehouse` resources by a `name` that is dynamically sourced from a repository name in the request body. ```yaml apiVersion: kargo.akuity.io/v1alpha1 kind: ProjectConfig metadata: name: kargo-demo namespace: kargo-demo spec: webhookReceivers: - name: my-receiver generic: secretRef: name: wh-secret actions: - action: Refresh whenExpression: "request.header('X-Event-Type') == 'push'" targetSelectionCriteria: - kind: Warehouse name: "${{ normalizeGit(request.body.repository.name) }}" ``` ##### By Labels The following example depicts `targetSelectionCriteria` that selects `Warehouse` resources with an `environment` label whose value is `prod`: ```yaml apiVersion: kargo.akuity.io/v1alpha1 kind: ProjectConfig metadata: name: kargo-demo namespace: kargo-demo spec: webhookReceivers: - name: my-receiver generic: secretRef: name: wh-secret actions: - action: Refresh whenExpression: "request.header('X-Event-Type') == 'push'" targetSelectionCriteria: - kind: Warehouse labelSelector: matchLabels: environment: prod ``` The following example depicts `targetSelectionCriteria` that selects `Warehouse` resources with a `service` label whose value is either `ui` or `api`: ```yaml apiVersion: kargo.akuity.io/v1alpha1 kind: ProjectConfig metadata: name: kargo-demo namespace: kargo-demo spec: webhookReceivers: - name: my-receiver generic: secretRef: name: wh-secret actions: - action: Refresh whenExpression: "request.header('X-Event-Type') == 'push'" targetSelectionCriteria: - kind: Warehouse labelSelector: matchExpressions: - key: service operator: In values: ["ui", "api"] ``` ##### By Values in an Index Use `indexSelector` to retrieve resources by a cached index. The following example depicts `targetSelectionCriteria` that dynamically selects `Warehouse` resources that contain subscriptions to the normalized git URL from the request body. ```yaml actions: - action: Refresh whenExpression: "request.header('X-Event-Type') == 'push'" targetSelectionCriteria: - kind: Warehouse indexSelector: matchIndices: - key: subscribedURLs operator: Equals value: "${{ normalizeGit(request.body.repository.url) }}" ``` :::note `subscribedURLs` is the only available index. It refers to `Warehouse` resources that contain subscriptions for a provided repository URL. ::: ### Expression Reference The Generic webhook receiver extends [built-in expr-lang support](https://expr-lang.org/docs/language-definition) with utilities that can be used to help resolve `targetSelectionCriteria` information from incoming requests. The following reference contains the variables and functions available for yielding dynamic values. - [request.body](#requestbody) - [request.header](#requestheaderheaderkey) - [request.headers](#requestheadersheaderkey) - [request.params](#requestparamsqueryparamkey) - [normalizeGit](#normalizegiturl) - [normalizeImage](#normalizeimageurl) - [normalizeChart](#normalizecharturl) #### request.body `request.body` is structured data parsed from the original request body which is assumed to contain valid JSON. Fields can be accessed using bracket or dot-notation. For example, `data.address.city` would access the `city` property nested within the `address` object, and `data.users[0]` would access the first item in a `users` array. #### request.header(headerKey) Retrieves first value for `headerKey`. It has one argument: - `headerKey` (Required): Case-insensitive header key. If `headerKey` is not present in the request headers, an empty `string` will be returned. #### request.headers(headerKey) Retrieves all values for `headerKey`. It has one argument: - `headerKey` (Required): Case-insensitive header key. If `headerKey` is not present in the request headers, an empty `string` array will be returned. #### request.params(queryParamKey) Retrieves the query param value for the provided query param key. It has one argument: - `queryParamKey` (Required): URL query parameter key. If `queryParamKey` is not present in the request headers, an empty `string` will be returned. #### normalizeGit(url) Normalizes Git URLs of the following forms: - `http[s]://[proxy-user:proxy-pass@]host.xz[:port][/path/to/repo[.git][/]]` - `ssh://[user@]host.xz[:port][/path/to/repo[.git][/]]` - `[user@]host.xz[:path/to/repo[.git][/]]` This is useful for the purposes of comparison and also in cases where a canonical representation of a Git URL is needed. Any URL that cannot be normalized will be returned as-is. It has one argument: - `url` (Required): The URL of a git repository. The returned value is a `string`. #### normalizeImage(url) Normalizes image repository URLs. Notably, hostnames docker.io and index.docker.io, if present, are dropped. The optional /library prefix for official images from Docker Hub, if included, is also dropped. Valid, non-Docker Hub repository URLs will be returned unchanged. This is useful for the purposes of comparison and also in cases where a canonical representation of a repository URL is needed. Any URL that cannot be normalized will be returned as-is. It has one argument: - `url` (Required): The URL of an image repository. The returned value is a `string`. #### normalizeChart(url) Normalizes a chart repository URL for purposes of comparison. Crucially, this function removes the oci:// prefix from the URL if there is one. It has one argument: - `url` (Required): The URL of a chart repository. The returned value is a `string`. ## Retrieving the Receiver's URL Kargo will generate a hard-to-guess URL from the receiver's configuration. This URL can be obtained using a command such as the following: ```shell kubectl get projectconfigs kargo-demo \ -n kargo-demo \ -o=jsonpath='{.status.webhookReceivers}' ``` --- ## Receiving Webhooks from the GitHub Container Registry Webhooks cannot be registered directly on a GHCR image repository. Instead, `package` events are delivered from an associated source code repository as if the event precipitating the delivery had occurred there. Refer to documentation for the [GitHub Webhooks Receiver](./github/index.md) for further instructions. :::note If your GHCR image repository has not yet been associated with a source code repository, [refer to these instructions](https://docs.github.com/en/packages/learn-github-packages/connecting-a-repository-to-a-package). ::: :::note GitHub can deliver webhooks _only_ for events occurring in a container image repository associated with a source code repository. This is a limitation of GitHub/GHCR and not a limitation of Kargo. ::: --- ## The Gitea Webhook Receiver The Gitea webhook receiver responds to `push` events originating from Gitea repositories by _refreshing_ `Warehouse` resources subscribed to those repositories. When a Warehouse has `includePaths` or `excludePaths` configured, the receiver extracts the list of changed files from the push event and only refreshes the Warehouse if the changed files match those path filters. :::note Gitea limits the `commits` array in push webhook payloads to a configurable number of commits (default: **5**, controlled by `UI.FEED_MAX_COMMIT_NUM` in Gitea's `app.ini`). If a push contains more commits than this limit, Kargo detects the truncation and skips path filtering for that event, refreshing all matching Warehouses unconditionally. ::: :::info "Refreshing" a `Warehouse` resource means enqueuing it for immediate reconciliation by the Kargo controller, which will execute the discovery of new artifacts from all repositories to which that `Warehouse` subscribes. ::: :::info The Gitea webhook receiver also works with Gitea Enterprise and Gitea Cloud. ::: ## Configuring the Receiver A Gitea webhook receiver must reference a Kubernetes `Secret` resource with a `secret` key in its data map. This [shared secret](https://en.wikipedia.org/wiki/Shared_secret) will be used by Gitea to sign requests and by the receiver to verify those signatures. :::note The following commands are suggested for generating and base64-encoding a complex secret: ```shell secret=$(openssl rand -base64 48 | tr -d '=+/' | head -c 32) echo "Secret: $secret" echo "Encoded secret: $(echo -n $secret | base64)" ``` ::: ```yaml apiVersion: v1 kind: Secret metadata: name: gitea-wh-secret namespace: kargo-demo labels: kargo.akuity.io/cred-type: generic data: secret: --- apiVersion: kargo.akuity.io/v1alpha1 kind: ProjectConfig metadata: name: kargo-demo namespace: kargo-demo spec: webhookReceivers: - name: gitea-wh-receiver gitlab: secretRef: name: gitea-wh-secret ``` ## Retrieving the Receiver's URL Kargo will generate a hard-to-guess URL from the receiver's configuration. This URL can be obtained using a command such as the following: ```shell kubectl get projectconfigs kargo-demo \ -n kargo-demo \ -o=jsonpath='{.status.webhookReceivers}' ``` ## Registering with Gitea 1. Navigate to the webhooks dashboard. Where you can find these settings varies based on the scope at which you'd like to enable your webhooks. Webhooks can be enabled for a single repository, for all repositories within an organization, or for all repositories belonging to an individual user. Navigate to `https://gitea.com///settings/hooks`, where `` has been replaced with a Gitea username or group name and `` has been replaced with the name of a project belonging to that namespace and for which you are an administrator. Navigate to `https://gitea.com/org//settings/hooks`, where `` has been replaced by a Gitea organization for which you are an administrator. Navigate to `https://gitea.com/org/user/settings/hooks`. ![Settings](./img/settings.png "Settings") 1. Click Add Webhook. 1. Click Gitea from the dropdown menu. ![Dropdown](./img/dropdown.png "Dropdown") 1. Complete the Webhooks form: ![Webhooks Form](./img/form.png " Webhooks Form") 1. Set the Target URL to the URL [for the webhook receiver](#retrieving-the-receivers-url). 1. Set Secret to the value assigned to the `secret` key of the `Secret` referenced by the [webhook receiver's configuration](#configuring-the-receiver). 1. In the Trigger On section, ensure Push Events is checked. 1. Click Add Webhook. This will return you to the list of all webhooks registered at the selected scope. :::info For additional information on configuring Gitea webhooks, refer directly to the [Gitea Docs](https://docs.gitea.com/usage/webhooks). ::: --- ## GitHub Webhook Receiver The GitHub webhook receiver responds to `ping`, `push`, `package`, and `registry_package` events originating from GitHub. The receiver unconditionally responds to `ping` events with an HTTP `200` status code. The receiver responds to `push` events by _refreshing_ `Warehouse` resources subscribed to the Git repositories from which those events originated. When a Warehouse has `includePaths` or `excludePaths` configured, the receiver extracts the list of changed files from the push event and only refreshes the Warehouse if the changed files match those path filters. The receiver responds to `package` events by _refreshing_ all `Warehouse` resources subscribed to the GHCR repositories from which those events originated. :::info "Refreshing" a `Warehouse` resource means enqueuing it for immediate reconciliation by the Kargo controller, which will execute the discovery of new artifacts from all repositories to which that `Warehouse` subscribes. ::: :::info The GitHub webhook receiver also works with GitHub Enterprise Cloud and GitHub Enterprise Server, although some URLs in this document may need to be adjusted accordingly. ::: ## Configuring the Receiver A GitHub webhook receiver must reference a Kubernetes `Secret` resource with a `secret` key in its data map. This [shared secret](https://en.wikipedia.org/wiki/Shared_secret) will be used by GitHub to sign requests and by the receiver to verify those signatures. :::note The following commands are suggested for generating and base64-encoding a complex secret: ```shell secret=$(openssl rand -base64 48 | tr -d '=+/' | head -c 32) echo "Secret: $secret" echo "Encoded secret: $(echo -n $secret | base64)" ``` ::: ```yaml apiVersion: v1 kind: Secret metadata: name: gh-wh-secret namespace: kargo-demo labels: kargo.akuity.io/cred-type: generic data: secret: --- apiVersion: kargo.akuity.io/v1alpha1 kind: ProjectConfig metadata: name: kargo-demo namespace: kargo-demo spec: webhookReceivers: - name: gh-wh-receiver github: secretRef: name: gh-wh-secret ``` ## Retrieving the Receiver's URL Kargo will generate a hard-to-guess URL from the receiver's configuration. This URL can be obtained using a command such as the following: ```shell kubectl get projectconfigs kargo-demo \ -n kargo-demo \ -o=jsonpath='{.status.webhookReceivers}' ``` ## Registering with GitHub There are three options whereby GitHub repositories can be configured to send events to the webhook receiver: 1. Configure webhooks directly for a single repository. The advantage of this approach is that it is comparatively simple, however, its large disadvantage is that it is tedious and most likely infeasible to repeat this configuration for a large number of repositories. 1. Configure webhooks directly at the organization level. The advantage of this approach is that it is simple and applies to all repositories in the organization, however, its disadvantage is that it requires organization-level permissions and will send events from all repositories in the organization. 1. Create a [GitHub App](https://docs.github.com/en/apps). The disadvantage of this approach is that it is comparatively complex, however, its large advantage is that once created and configured, the App can be easily [installed](https://docs.github.com/en/apps/using-github-apps/installing-your-own-github-app) into any number of GitHub repositories (belonging to the same account that owns the App). In the sections below, you will find instructions for all three options. ### Webhooks from a Single Repository To configure a single GitHub repository to notify a receiver of relevant events: 1. Navigate to `https://github.com///settings/hooks`, where `` has been replaced with a GitHub username or organization name and `` has been replaced with the name of a repository belonging to that account and for which you are an administrator. ![Settings](./img/repos/settings.png "Settings") 1. Click Add webhook. 1. Complete the Add webhook form: ![Add Webhook Form](./img/repos/add-webhook-form.png "Add Webhook Form") 1. Complete the Payload URL field using the URL [for the webhook receiver](#retrieving-the-receivers-url). 1. Set Content type to `application/json`. 1. Complete the Secret field using to the (unencoded) value assigned to the `secret` key of the `Secret` resource referenced by the [webhook receiver's configuration](#configuring-the-receiver). 1. Under Which events would you like to trigger this webhook?: Leave Just the push event. selected, unless you would like to receive events when container images or Helm charts are pushed to associated GHCR repositories. To receive such events, select Let me select individual events., then ensure Pushes and Packages are both selected. :::note You will only receive events from those GHCR repositories explicitly associated with your Git repository. For more information on this topic, refer to [these GitHub docs](https://docs.github.com/en/packages/learn-github-packages/connecting-a-repository-to-a-package). ::: 1. Ensure Active is selected. 1. Click Add webhook. 1. Verify connectivity: 1. From the Webhooks dashboard, select the new webhook. ![Webhooks](./img/repos/webhooks.png "Webhooks") 1. Select the Recent Deliveries tab. ![Recent Deliveries](./img/repos/recent-deliveries.png "Recent Deliveries") 1. Select the ping event and ensure an HTTP response status of `200` was received from the webhook receiver. ![Ping Response](./img/repos/ping-response.png "Ping Response") When these steps are complete, the repository will send events to the webhook receiver. :::info For additional information on configuring webhooks, refer directly to the [GitHub Docs](https://docs.github.com/en/webhooks/using-webhooks/creating-webhooks). ::: ### Webhooks from an Organization To configure an organization to notify the receiver of relevant events from all repositories in the organization: 1. Navigate to `https://github.com/organizations//settings/hooks`, where `` has been replaced with the name of an organization for which you are an administrator. ![Settings](./img/repos/settings.png "Settings") 1. Click Add webhook. 1. Complete the Add webhook form: ![Add Webhook Form](./img/orgs/add-webhook-form.png "Add Webhook Form") 1. Complete the Payload URL field using the URL [for the webhook receiver](#retrieving-the-receivers-url). 1. Set Content type to `application/json`. 1. Complete the Secret field using to the (unencoded) value assigned to the `secret` key of the `Secret` resource referenced by the [webhook receiver's configuration](#configuring-the-receiver). 1. Under Which events would you like to trigger this webhook?: Leave Just the push event. selected, unless you would like to receive events when container images or Helm charts are pushed to associated GHCR repositories. To receive such events, select Let me select individual events., then ensure Pushes and Packages are both selected. :::note You will only receive events from those GHCR repositories explicitly associated with your Git repository. For more information on this topic, refer to [these GitHub docs](https://docs.github.com/en/packages/learn-github-packages/connecting-a-repository-to-a-package). ::: 1. Ensure Active is selected. 1. Click Add webhook. 1. Verify connectivity: 1. From the organization Webhooks dashboard, select the new webhook. ![Webhooks](./img/orgs/webhooks.png "Webhooks") 1. Scroll down to the Recent Deliveries section. ![Recent Deliveries](./img/orgs/recent-deliveries.png "Recent Deliveries") 1. Select the ping event and ensure an HTTP response status of `200` was received from the webhook receiver. ![Ping Response](./img/orgs/ping-response.png "Ping Response") When these steps are complete, all repositories in the organization will send events to the webhook receiver. :::info For additional information on configuring organization webhooks, refer directly to the [GitHub Docs](https://docs.github.com/en/webhooks/using-webhooks/creating-webhooks#creating-organization-webhooks). ::: ### Webhooks from a GitHub App To configure a [GitHub App](https://docs.github.com/en/apps) to notify the receiver of relevant events from any repository into which it's been installed: 1. Navigate to [https://github.com/settings/apps](https://github.com/settings/apps) to create a new GitHub App owned by your own account OR Navigate to `https://github.com/organizations//settings/apps`, where `` has been replaced with an organization for which you are an administrator to create a new GitHub App owned by that organization. 1. Complete the Register new GitHub App form: ![Register New GitHub App Form](./img/apps/register.png "Register New GitHub App Form") 1. Complete the GitHub App name field using a name of your choosing. :::note This name must be _globally unique_ across all of GitHub. Unfortunately, you will not learn whether the name you've selected is available or not until you have submitted the form. ::: 1. Complete the Homepage URL field using a URL of your choosing. :::info This is a required field, but for our purposes, its value is unimportant. ::: 1. Skip the Identifying and authorizing users and Post installation sections of the form. :::info These sections are not relevant to webhook delivery. ::: 1. Complete the Webhook URL field using the URL [for the webhook receiver](#retrieving-the-receivers-url). 1. Complete the Secret field using the (unencoded) value assigned to the `secret` key of the `Secret` resource referenced by the [webhook receiver's configuration](#configuring-the-receiver). 1. In the Permissions section of the form, expand Repository Permissions. 1. Ensure Contents is set to Read-only. 1. If you would like to receive events when container images or Helm charts are pushed to associated GHCR repositories, set Packages to Read-only. :::note You will only receive events from those GHCR repositories explicitly associated with your Git repository. For more information on this topic, refer to [these GitHub docs](https://docs.github.com/en/packages/learn-github-packages/connecting-a-repository-to-a-package). ::: 1. In the Subscribe to events section of the form, ensure Push is selected. If you would like to receive events when container images or Helm charts are pushed to associated GHCR repositories, ensure Registry package is also selected. :::note The events available for selection in this section of the form are dynamic and dependent on your selections in the Permissions section. ::: ![Subscribe to Events](./img/apps/subscribe.png "Subscribe to Events") 1. Under Where can this GitHub App be installed? ensure Only this account is selected. :::danger If you select the other option (Any account), your App will be installable into any repository in GitHub, regardless of what account owns it. _Every_ repository into which your App is installed will send events to the webhook receiver. __You almost certainly do not want this!__ ::: 1. Click Create GitHub App. 1. Verify connectivity: 1. From the new App's settings page, select Advanced from the left sidebar. 1. Under Recent Deliveries, select the ping event and ensure an HTTP response status of `200` was received from the webhook receiver. ![Ping Response](./img/apps/ping-response.png "Ping Response") 1. Install the new GitHub App into one or more repositories: 1. Select Install App from the left sidebar. You will be presented with a list of accounts (users or organizations) into which you may install the App. If you followed the instructions for creating the App as presented above, _only_ the account owning the App should be listed here. 1. Click Install next to the account into which you wish to install the App. An installation dialog will appear. 1. Under for these repositories, select: * All repositories if you wish for all repositories owned by the account to send events to the webhook receiver. :::note Selecting this option will result in the App also being installed in _new_ repositories belonging to the account as they are created. ::: * Only select repositories if you wish for only specific, existing repositories owned by the account to send events to the webhook receiver. :::note If this option is selected, the installation process can be repeated in the future to install the App into additional repositories. ::: 1. Click Install. When these steps are complete, all repositories into which your GitHub App has been installed will send events to the webhook receiver. :::info For additional information on configuring webhooks for GitHub Apps, refer directly to the [GitHub Docs](https://docs.github.com/en/apps/creating-github-apps/registering-a-github-app/using-webhooks-with-github-apps). ::: --- ## GitLab Webhook Receiver The GitLab Webhook Receiver responds to `push` events originating from GitLab repositories by _refreshing_ `Warehouse` resources subscribed to those repositories. When a Warehouse has `includePaths` or `excludePaths` configured, the receiver extracts the list of changed files from the push event and only refreshes the Warehouse if the changed files match those path filters. :::note GitLab limits the `commits` array in push webhook payloads to **20 commits**. If a push contains more than 20 commits, Kargo detects the truncation and skips path filtering for that event, refreshing all matching Warehouses unconditionally. ::: :::info "Refreshing" a `Warehouse` resource means enqueuing it for immediate reconciliation by the Kargo controller, which will execute the discovery of new artifacts from all repositories to which that `Warehouse` subscribes. ::: :::info The GitLab webhook receiver also works with GitLab Dedicated and GitLab Self Managed, although some URLs in this document may need to be adjusted accordingly. ::: ## Configuring the Receiver A GitLab webhook receiver must reference a Kubernetes `Secret` resource with a `secret-token` key in its data map. This [shared secret](https://en.wikipedia.org/wiki/Shared_secret) will be used by GitLab to sign requests and by the receiver to verify those signatures. :::note The following commands are suggested for generating and base64-encoding a complex secret: ```shell secret_token=$(openssl rand -base64 48 | tr -d '=+/' | head -c 32) echo "Secret token: $secret_token" echo "Encoded secret token: $(echo -n $secret_token | base64)" ``` ::: ```yaml apiVersion: kargo.akuity.io/v1alpha1 kind: Project metadata: name: kargo-demo --- apiVersion: v1 kind: Secret metadata: name: gl-wh-secret namespace: kargo-demo labels: kargo.akuity.io/cred-type: generic data: secret-token: --- apiVersion: kargo.akuity.io/v1alpha1 kind: ProjectConfig metadata: name: kargo-demo namespace: kargo-demo spec: webhookReceivers: - name: gl-wh-receiver gitlab: secretRef: name: gl-wh-secret ``` ## Retrieving the Receiver's URL Kargo will generate a hard-to-guess URL from the receiver's configuration. This URL can be obtained using a command such as the following: ```shell kubectl get projectconfigs kargo-demo \ -n kargo-demo \ -o=jsonpath='{.status.webhookReceivers}' ``` ## Registering with GitLab 1. Navigate to the webhooks dashboard. Where you can find these settings varies based on the scope at which you'd like to enable your webhooks. Webhooks can be enabled for a single project (repository) or for all projects (repositories) in a group. Navigate to `https://gitlab.com///-/hooks`, where `` has been replaced with a GitLab username and `` has been replaced with the name of a project belonging to that namespace and for which you are an administrator. Navigate to `https://gitlab.com/groups//-/hooks`, where `` has been replaced with a GitLab group name for which you are an owner of the group. :::caution If you configure identical webhooks affecting a given project at _both_ the project level and group level, both webhooks will be triggered by applicable events in that project. ::: 1. Click Add new webhook. ![Settings](./img/settings.png "Settings") 1. Complete the Webhooks form: ![Webhooks Form](./img/add-webhook-form.png "Webhooks Form") 1. Enter a descriptive name in the Name field. 1. Complete the URL field using the URL [for the webhook receiver](#retrieving-the-receivers-url). 1. Complete the Secret token field using to the (unencoded) value assigned to the `secret-token` key of the `Secret` resource referenced by the [webhook receiver's configuration](#configuring-the-receiver). 1. In the Trigger section, ensure Push events is selected. 1. Click Add webhook. 1. From the Webhooks settings page, verify connectivity by expanding the Test dropdown menu next to your webhook and selecting Push events. ![Webhooks](./img/webhooks.png "Webhooks") ![Test Button](./img/test-button.png "Test Button") :::note This is safe to do because the test payload will lack the necessary details to successfully refresh `Warehouse` resources subscribed to the repository. ::: If the test `push` event was delivered successfully, a success message will appear at the top of the Webhooks settings page. :::info If the test event is not successful, troubleshoot by clicking the Edit button next to your webhook, then scrolling down to the Recent events section to view details of the failed request. ::: :::info For additional information on configuring GitLab webhooks, refer directly to the [GitLab Docs](https://docs.gitlab.com/user/project/integrations/webhooks/). ::: --- ## Harbor Webhook Receiver The Harbor Webhook Receiver responds to `PUSH_ARTIFACT` events originating from Harbor repositories by _refreshing_ all `Warehouse` resources subscribed to those repositories. :::info "Refreshing" a `Warehouse` resource means enqueuing it for immediate reconciliation by the Kargo controller, which will execute the discovery of new artifacts from all repositories to which that `Warehouse` subscribes. ::: ## Configuring the Receiver A Harbor webhook receiver must reference a Kubernetes `Secret` resource with an `auth-header` key in its data map. This [shared secret](https://en.wikipedia.org/wiki/Shared_secret) will be passed in the `Authorization` header of the webhook request sent by Harbor and verified by Kargo. :::info While supplying an "Auth Header" is optional in Harbor, Kargo requires using this. ::: :::note The following commands are suggested for generating and base64-encoding a complex secret: ```shell auth_header=$(openssl rand -base64 48 | tr -d '=+/' | head -c 32) echo "Auth header: $auth_header" echo "Encoded auth header: $(echo -n $auth_header | base64)" ``` ::: ```yaml apiVersion: v1 kind: Secret metadata: name: harbor-wh-secret namespace: kargo-demo labels: kargo.akuity.io/cred-type: generic data: auth-header: --- apiVersion: kargo.akuity.io/v1alpha1 kind: ProjectConfig metadata: name: kargo-demo namespace: kargo-demo spec: webhookReceivers: - name: harbor-wh-receiver harbor: secretRef: name: harbor-wh-secret ``` ## Retrieving the Receiver's URL Kargo will generate a hard-to-guess URL from the receiver's configuration. This URL can be obtained using a command such as the following: ```shell kubectl get projectconfigs kargo-demo \ -n kargo-demo \ -o=jsonpath='{.status.webhookReceivers}' ``` ## Registering with Harbor 1. Navigate to `https:///harbor/projects//webhook`, where `` and `` have been replaced with a Harbor instance and project for which you are an administrator. 1. Click New Webhook. ![Webhooks View](./img/webhooks-view.png "Webhooks View") 1. Complete the Add Webhook form: ![Add Webhook](./img/add-webhook.png "Add Webhook") 1. Enter a descriptive name in the Name field. 1. Check Event type Artifact pushed. Uncheck all other event types. 1. Complete the Endpoint URL field using the URL [for the webhook receiver](#retrieving-the-receivers-url). 1. Complete the Auth Header field using shared secret [generated above](#configuring-the-receiver). :::note Use the unencoded value; not the base64-encoded value. ::: 1. Check Verify Remote Certificate. 1. Click Add 1. The webhook should now be registered with Harbor. You can test it by pushing an artifact to the repository. This should trigger a webhook run, which you can review by clicking the name of your webhook. You can then check the status of the run and verify that the run was successful, as well as review the logs for any errors or issues if it was not. ![Webhooks View After Adding Webhook](./img/webhooks-view-after-adding-webhook.png "Webhooks View After Adding Webhook") --- ## Webhook Receivers Reference Below is an index of documentation for specific webhook receivers that can be used to trigger applicable `Warehouse` discovery processes as new artifacts are published to your repositories. :::info[Not what you were looking for?] For more generalized coverage of Kargo's webhook receivers, developers may refer to the [webhooks section of the Working with Warehouses page](../../20-how-to-guides/30-working-with-warehouses.md#triggering-artifact-discovery-using-webhooks). Operator may refer to the [webhooks section of the Cluster Level Configuration page](../../../40-operator-guide/35-cluster-configuration.md#triggering-artifact-discovery-using-webhooks). ::: --- ## Quay.io Webhook Receiver The Quay.io webhook receiver responds to "Push to Repository" events originating from Quay.io repositories by "refreshing" all `Warehouse` resources subscribed to those repositories. :::info "Refreshing" a `Warehouse` resource means enqueuing it for immediate reconciliation by the Kargo controller, which will execute the discovery of new artifacts from all repositories to which that `Warehouse` subscribes. ::: ## Configuring the Receiver A Quay.io webhook receiver must reference a Kubernetes `Secret` resource with a `secret` key in its data map. :::info _This secret will not be shared directly with Quay.io._ Quay.io does not natively implement any mechanism whereby receivers may authenticate inbound webhook requests. To compensate for this, Kargo incorporates the secret into the generation of a hard-to-guess URL for the receiver. This URL serves as a _de facto_ [shared secret](https://en.wikipedia.org/wiki/Shared_secret) and authentication mechanism. ::: :::note The following commands are suggested for generating and base64-encoding a complex secret: ```shell secret=$(openssl rand -base64 48 | tr -d '=+/' | head -c 32) echo "Secret: $secret" echo "Encoded secret: $(echo -n $secret | base64)" ``` ::: ```yaml apiVersion: v1 kind: Secret metadata: name: q-wh-secret namespace: kargo-demo labels: kargo.akuity.io/cred-type: generic data: secret: --- apiVersion: kargo.akuity.io/v1alpha1 kind: ProjectConfig metadata: name: kargo-demo namespace: kargo-demo spec: webhookReceivers: - name: q-wh-receiver quay: secretRef: name: q-wh-secret ``` ## Retrieving the Receiver's URL Kargo will generate a hard-to-guess URL from the receiver's configuration. This URL can be obtained using a command such as the following: ```shell kubectl get projectconfigs kargo-demo \ -n kargo-demo \ -o=jsonpath='{.status.webhookReceivers}' ``` ## Registering with Quay.io To configure a single Quay.io repository to notify a receiver when objects (like container images or Helm charts) have been pushed to it: 1. Navigate to `https://quay.io/repository//?tab=settings`, where `` has been replaced with a Quay.io username or organization name and `` has been replaced with the name of a repository belonging to that account and for which you are an administrator. ![Repository Settings](./img/repository-settings.png "Repository Settings") 1. In the Events and Notifications section, click Create Notification. 1. Complete the Create repository notification form. ![Create Repository Notification Form](./img/create-repository-notification.png "Create Repository Notification Form") 1. In the When this event occurs dropdown, select Push to Repository. 1. In the Then issue a notification dropdown, select Webhook POST. 1. Complete the Webhook URL field using the URL [for the webhook receiver](#retrieving-the-receivers-url). 1. Click Create Notification. ![Created](./img/created.png "Created") :::info Verifying delivery of webhook requests can be difficult as Quay.io offers a mechanism for submitting test webhook requests, but offers no way to view the results of such a request. ::: :::info For additional information on configuring Quay.io webhooks, refer directly to the [Quay.io Docs](https://docs.quay.io/guides/notifications.html). ::: --- ## Kargo Event Reference This document contains a complete reference of all Kargo events, including their types and descriptions. :::info These events are always emitted by Kargo to the Kubernetes event log with the keys encoded in the annotations of the event. However, they are primarily consumed by the Pro version of Kargo (such as the [Notification feature](./100-notifications/index.md)) ::: ## Event Fields Many Kargo events share common fields. The following sections describe these common fields, which are referenced in individual event definitions. Each field is given as it appears in the event payload (serialized as JSON) and, if optional, notes whether the type is a pointer when used in expr-lang expressions. In each event definition in [Event Types](#event-types), the included fields are listed under "Payload Includes" with references to the relevant sections. All of the fields described in those events are found at the top level of the event payload unless otherwise noted. ### Common Event Fields These fields are included in all Kargo events: | Field Name | Type | Description | Optional | | ---------- | ------ | ---------------------------------------------- | ----------------- | | `project` | String | The project name the event originated from. | No | | `actor` | String | The user or system that triggered the event. | Yes (is pointer) | | `message` | String | A human-readable message describing the event. | No (may be empty) | | `id` | String | A unique identifier for the event. | No | ### Freight Fields Freight payloads describe the collection of artifacts under evaluation or promotion. | Field Name | Type | Description | Optional | | ------------ | -------------------------- | ------------------------------------------------------------------------------------------------------------------- | ---------------- | | `name` | String | Name of the freight object. | No | | `stageName` | String | Stage associated with the freight when the event fired. | No | | `createTime` | String (RFC3339) | Creation timestamp of the freight object. | No | | `alias` | String | Human-friendly alias assigned to the freight. | Yes (is pointer) | | `commits` | Array\ | Git commits that compose the freight (see [GitCommit fields](#gitcommit-fields)). | Yes | | `images` | Array\ | Container images included in the freight (see [Image fields](#image-fields)). | Yes | | `charts` | Array\ | Helm charts included in the freight (see [Chart fields](#chart-fields)). | Yes | | `artifacts` | Array\ | Additional arbitrary artifacts included in the freight (see [ArtifactReference fields](#artifactreference-fields)). | Yes | #### `GitCommit` Fields | Field Name | Type | Description | Optional | | ----------- | ------ | ---------------------------------------- | -------- | | `repoURL` | String | URL of the Git repository. | Yes | | `id` | String | Commit SHA in the referenced repository. | Yes | | `branch` | String | Branch where the commit was discovered. | Yes | | `tag` | String | Tag that resolved to the commit. | Yes | | `message` | String | Commit message subject line. | Yes | | `author` | String | Author of the commit. | Yes | | `committer` | String | Committer recorded for the commit. | Yes | #### `Image` Fields | Field Name | Type | Description | Optional | | ------------- | -------------------- | ----------------------------------------------- | -------- | | `repoURL` | String | Repository that hosts the container image. | Yes | | `tag` | String | Mutable tag identifying a version of the image. | Yes | | `digest` | String | Immutable digest identifying the image content. | Yes | | `annotations` | Map\ | Arbitrary metadata associated with the image. | Yes | #### `Chart` Fields | Field Name | Type | Description | Optional | | ---------- | ------ | --------------------------------------------------------------------- | -------- | | `repoURL` | String | Helm chart repository URL. | Yes | | `name` | String | Chart name within the repository (empty for OCI-style references). | Yes | | `version` | String | Specific chart version selected for inclusion in the freight payload. | Yes | #### `ArtifactReference` Fields | Field Name | Type | Description | Optional | | ------------------ | -------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | | `artifactType` | String | Unique type of the artifact. | No | | `subscriptionName` | String | Name of the subscription that discovered the artifact. | No | | `version` | String | Version identifies a specific revision of this artifact. | No | | `metadata` | Map\ | Additional metadata associated with the artifact. It is a mostly opaque collection of attributes. "Mostly" because Kargo may understand how to interpret some documented, well-known, top-level keys. Those aside, this metadata is only understood by a corresponding Subscriber implementation that created it. | Yes | ### Freight Verification Fields Freight verification metadata accompanies events emitted while verifying freight. | Field Name | Type | Description | Optional | | ------------------------------ | ---------------- | ------------------------------------------------------------------------------- | ---------------- | | `verificationStartTime` | String (RFC3339) | Timestamp when the verification run began. | Yes (is pointer) | | `verificationFinishTime` | String (RFC3339) | Timestamp when the verification run finished. | Yes (is pointer) | | `analysisRunName` | String | Name of the Argo Rollouts AnalysisRun created for the verification, if present. | Yes (is pointer) | | `analysisTriggeredByPromotion` | String | Name of the promotion that triggered the verification analysis run, if present. | Yes (is pointer) | ### Promotion Fields Promotion payloads describe a promotion resource and the freight it targets. | Field Name | Type | Description | Optional | | -------------- | ----------------------------------------------------- | ----------------------------------------------------- | ---------------- | | `freight` | Object ([Freight fields](#freight-fields)) | Snapshot of the freight referenced by the promotion. | Yes (is pointer) | | `name` | String | Name of the promotion resource. | No | | `stageName` | String | Stage targeted by the promotion. | No | | `createTime` | String (RFC3339) | Creation timestamp of the promotion resource. | No | | `applications` | Array\<[NamespacedName](#applications-entry-fields)\> | Argo CD applications resolved for the promotion step. | Yes | #### Applications Entry Fields Each promotion `applications` entry is a Kubernetes `NamespacedName` tuple. | Field Name | Type | Description | Optional | | ----------- | ------ | ---------------------------------------- | -------- | | `namespace` | String | Namespace that contains the Argo CD app. | No | | `name` | String | Name of the Argo CD application. | No | ## Event Types The complete list of built-in Kargo event types is provided below: - `PromotionCreated` - `PromotionSucceeded` - `PromotionFailed` - `PromotionErrored` - `PromotionAborted` - `FreightApproved` - `FreightVerificationSucceeded` - `FreightVerificationFailed` - `FreightVerificationErrored` - `FreightVerificationAborted` - `FreightVerificationInconclusive` - `FreightVerificationUnknown` Below are the detailed definitions for each event type. ### `PromotionCreated` This event is emitted when a promotion resource is created. **Payload Includes** - [Common event fields](#common-event-fields) - [Promotion fields](#promotion-fields) ### `PromotionSucceeded` This event is emitted when a promotion completes successfully. The payload matches `PromotionCreated` with one additional field. **Payload Includes** - [Common event fields](#common-event-fields) - [Promotion fields](#promotion-fields) Unique to this event: | Field Name | Type | Description | Optional | | --------------------- | ------- | --------------------------------------------------------------------------- | ---------------- | | `verificationPending` | Boolean | Indicates whether post-promotion freight verification is still outstanding. | Yes (is pointer) | ### `PromotionFailed` This event is emitted when a promotion fails, typically because a step or verification did not succeed. **Payload Includes** - [Common event fields](#common-event-fields) - [Promotion fields](#promotion-fields) ### `PromotionErrored` This event is emitted when a promotion encounters an unexpected error. **Payload Includes** - [Common event fields](#common-event-fields) - [Promotion fields](#promotion-fields) ### `PromotionAborted` This event is emitted when a promotion run is aborted before completion. **Payload Includes** - [Common event fields](#common-event-fields) - [Promotion fields](#promotion-fields) ### `FreightApproved` This event is emitted when freight is manually approved for a stage. **Payload Includes** - [Common event fields](#common-event-fields) - [Freight fields](#freight-fields) ### `FreightVerificationSucceeded` This event is emitted when freight verification completes successfully. **Payload Includes** - [Common event fields](#common-event-fields) - [Freight fields](#freight-fields) - [Freight verification fields](#freight-verification-fields) ### `FreightVerificationFailed` This event is emitted when freight verification completes with a failure. **Payload Includes** - [Common event fields](#common-event-fields) - [Freight fields](#freight-fields) - [Freight verification fields](#freight-verification-fields) ### `FreightVerificationErrored` This event is emitted when freight verification encounters an unexpected error. **Payload Includes** - [Common event fields](#common-event-fields) - [Freight fields](#freight-fields) - [Freight verification fields](#freight-verification-fields) ### `FreightVerificationAborted` This event is emitted when freight verification is aborted before completion. **Payload Includes** - [Common event fields](#common-event-fields) - [Freight fields](#freight-fields) - [Freight verification fields](#freight-verification-fields) ### `FreightVerificationInconclusive` This event is emitted when freight verification finishes with an inconclusive result. **Payload Includes** - [Common event fields](#common-event-fields) - [Freight fields](#freight-fields) - [Freight verification fields](#freight-verification-fields) ### `FreightVerificationUnknown` This event is emitted when freight verification ends in an unknown state. **Payload Includes** - [Common event fields](#common-event-fields) - [Freight fields](#freight-fields) - [Freight verification fields](#freight-verification-fields) --- ## Key Concepts :::info This set of features are only available in Kargo on the [Akuity Platform](https://akuity.io/akuity-platform), versions v1.8 and above. ::: This document provides an overview of the key concepts related to Kargo's Notifications feature, explaining the main components involved in sending notifications based on events. ## Channels Channels define the destination and method for sending notifications. They specify where the notifications should be sent, such as to a Slack channel, email address, or other supported platforms. Channels are configured using [`MessageChannel`](../../../20-how-to-guides/20-working-with-projects.md#message-channels) or [`ClusterMessageChannel`](../../../../40-operator-guide/35-cluster-configuration.md#cluster-message-channels) resources. Each channel type may have its own specific configuration options, such as authentication details, recipient information, and message formatting preferences. You can think of channels as a "connection string" of sorts that tells Kargo where to send data. They do not define the message or other information, only the connection details. They are used by [`EventRouter`s](#event-routers) and by the [`send-message` step](../../30-promotion-steps/send-message.md) to send notifications when specific events occur. ## Event Routers At their core, `EventRouter`s are responsible for listening to specific events within Kargo and routing them to the appropriate channels based on defined criteria. They act as the bridge between events and channels, ensuring that notifications are sent to the right destinations when relevant events occur. Due to their design, the same event can be routed to multiple channels or different events can be routed to the same channel without duplicating notification logic. The lack of "notification" in the name is intentional, as `EventRouter`s do not themselves send notifications. Instead, they route events to channels after rendering data from the event, which then handle the actual sending of notifications. This design allows for greater flexibility and reusability, as you can glue together any type of message or data you are interested in without confining it to a specific notification-only format. There is also the possibility (but not guarantee) that in the future we may implement routing of events to other systems beyond just notifications. --- ## Configuring Event Routers [Event Routers](./00-overview.md#event-routers) are used to route events to different destinations based on specified criteria. This document provides guidance on how to configure and use Event Routers in Kargo. :::info Before creating Event Routers, ensure that you or your administrator have set up the necessary [`ClusterMessageChannel`](../../../../40-operator-guide/35-cluster-configuration.md#cluster-message-channels) or [`MessageChannel`](../../../20-how-to-guides/20-working-with-projects.md#message-channels) resources that define the destinations for your notifications. ::: ## Managing Event Routers You can manage Event Routers through the Kargo UI or declaratively via YAML configuration files deployed by Argo CD. To manage it in the UI, go to your Project settings, then in the sidebar, select "Notifications." In this section you can create, update, and delete both your channels and your event routers. :::note Currently, event routers can only be created by editing YAML, which you can do in the UI. We plan to improve this experience in future releases. ::: The simplest form of an event router configuration looks like this: ```yaml kind: EventRouter apiVersion: ee.kargo.akuity.io/v1alpha1 metadata: name: promotion-status namespace: kargo-demo spec: types: - PromotionFailed - PromotionErrored channels: - name: devops-team-slack kind: MessageChannel ``` In this example, the event router named `test-router` is configured to listen for two [event types](../10-event-reference.md#event-types): `PromotionCreated` and `PromotionErrored`. When either of these events occurs, a notification will be sent to the channel named `slack`, which is of kind `MessageChannel` using a default message template defined for the type of event along with the channel type. You can listen for multiple event types and send notifications to multiple channels by adding more entries to the `types` and `channels` lists. :::warning If you're using default message templates, the specified event types _must_ all be of the same "class" (such as Promotion events, Freight events, etc.). Mixing event types from different classes is not supported when using default templates and will error. ::: Additionally, you will likely want to make use of filters to control when notifications are sent. This is done by setting the `when` field in the event router spec, which uses [expr-lang](https://expr-lang.org/) expressions (see [the message formatting docs](./20-message-formatting.md) for more information on available data). The expression must resolve to a boolean value. For example, to only send notifications for promotions targeting a specific stage, you could configure the event router like this: ```yaml kind: EventRouter apiVersion: ee.kargo.akuity.io/v1alpha1 metadata: name: prod-promotions namespace: kargo-demo spec: types: - PromotionCreated - PromotionErrored channels: - name: devops-team-slack kind: MessageChannel when: "event.stageName == 'production'" ``` ## Advanced Configuration ### Message Threading/Grouping With many types of notifications, it can be helpful to group related messages together in a thread or conversation based on items like the freight being promoted or specific items like a tag name or commit ID. This is especially useful for channels like Slack where threaded messages help keep discussions organized. To enable message grouping, you can use the `groupingKey` field in the event router spec. Currently, the `groupingKey` feature is only used by Slack channels, but this will be expanded to other channel types in future releases. A `groupingKey` is a string value that can be a plain string or constructed using expressions enclosed in `${{ }}` which render to a string. The value of the `groupingKey` is used by the message channel to determine which messages should be grouped together. All of the same context is used to evaluate the expression as is used in things like message formatting (see see the [message formatting documentation](./20-message-formatting.md) for more information). For example, to group messages by the freight name and stage being promoted to, you could configure the event router like this: ```yaml kind: EventRouter apiVersion: ee.kargo.akuity.io/v1alpha1 metadata: name: promotion-status namespace: kargo-demo spec: groupingKey: "${{ event.freight.stageName }}-${{ event.freight.name }}" types: - PromotionFailed - PromotionErrored channels: - name: devops-team-slack kind: MessageChannel ``` :::warning Please note that this feature currently behaves in the same way as Argo CD Notifications in that the current state (i.e. which thread is mapped to a grouping key) is stored in memory. This means that if the Kargo controller is restarted, the mapping will be lost and new threads may be created for existing grouping keys. We will be adding persistent storage for this in a future release. ::: ### Custom Templates The default templates and `when` field will likely cover many use cases, but you can also customize the message content and formatting by specifying a custom message body. This allows you to tailor the notifications to your specific needs. The top level fields for configuring a custom message body are: - `output`: The main content of the message for all channels (unless overridden in channel-specific fields). - `encodingType`: The format of the `output` field. Available options include plaintext (an empty string or omitted value), `json`, `yaml`, and `xml`. The output can be customized using [`expr-lang`](https://expr-lang.org/) expressions to include dynamic content based on the event data. These expressions must be enclosed in `${{ }}`. For example, to include the stage name in the message, you could use `${{ event.stageName }}`. For information on available data and formatting options, see the [message formatting documentation](./20-message-formatting.md). :::warning If you are using the top level `output` field, in most cases the output should be a string as each message type can generate a default message from the output in the proper format for that channel. In other words, if you are sending to both SMTP and Slack, the structured data each would expect is different so structured data will need to be provided for each channel type. However, if you are sending to multiple channels of the same type (e.g., multiple Slack channels), you can provide a common structured output that works for all of them. ::: You can also specify channel-specific message bodies by providing `output` fields within the channel configuration list. This allows you to customize the message content for each channel independently. For example: ```yaml kind: EventRouter apiVersion: ee.kargo.akuity.io/v1alpha1 metadata: name: promotion-started namespace: kargo-demo spec: types: - PromotionCreated channels: - name: slack kind: MessageChannel output: "Kargo has kicked off promotion to stage: ${{ event.stageName }}." - name: smtp kind: MessageChannel encodingType: yaml output: | subject: 🚀 Deployment Promotion Started to: email@example.com body: Kargo has kicked off promotion to stage: ${{ event.stageName }}. ``` For information on available data and formatting options, see the [message formatting documentation](./20-message-formatting.md). --- ## Message Formatting ## Overview `EventRouter`s have support for customizing the format of notification messages using [`expr-lang`](https://expr-lang.org/). This allows you to create richly formatted messages tailored to the specific channel you are sending notifications to in addition to the data available from the event payload. ### Using expr-lang Within the `when` field, the entire string is treated as an expression and must evaluate to a boolean. In other fields where expressions are supported, such as the `message` field in the [`send-message`](../../30-promotion-steps/send-message.md) step or the `output` field in an [`EventRouter`](./00-overview.md#event-routers), expressions must be enclosed in `${{ }}`. ### Encoding Types If sending richly formatted messages, the `encodingType` field specifies the format of the message body. Supported values include: - An empty string or omitted value: Plaintext format. The message is treated as plain text and sent verbatim. - `json`: JSON format. The message body is expected to be a valid JSON structure. - `yaml`: YAML format. The message body is expected to be a valid YAML structure. - `xml`: XML format. The message body is expected to be a valid XML structure. ## Event Data Structures For `EventRouter`s and notifications, the event data structure varies depending on the type of event being handled. However, the object passed to expressions does have 3 common top-level fields: - `type`: A string representing the type of event (e.g., `PromotionCreated`). - `event`: An object containing the event-specific data. The structure of this object varies based on the event type. See [Event Reference](../10-event-reference.md) for more details. - `data`: The Kubernetes object associated with the event, if applicable. This is typically the resource that triggered the event such as `Promotion` or `Freight`. It is fetched in its entirety from the Kubernetes API and kept in its raw form. So all fields will be accessible as if you were accessing items in a JSON object (e.g., `data.metadata.name`, `data.spec.stage`, etc.). - `ctx`: Optional. Extra fields injected by the controller, for example `ctx.uiBaseURL` for UI links formatting. ## Available Functions All [built-in functions](https://expr-lang.org/docs/language-definition) provided by `expr-lang` are available for use in expressions. Additionally, the following custom functions are provided specifically for Kargo notifications: - `deref(any)` - Dereferences pointers or references within the event data structure, returning the underlying value. This is useful when printing a value. It is not necessary to use when accessing fields under the pointer. - `escape_json_string(string)` - Escapes special characters in a string to make it safe for inclusion in JSON. It is highly recommended to use this function when adding things like the event message to a JSON formatted message to avoid invalid JSON. ## Message Body Formats ### Slack Slack messages can be formatted using JSON with the following structure: | Key | Type | Description | | ------------ | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------- | | `icon_emoji` | string | An emoji to use as the icon for the message. Overrides `icon_url` if both are set. Should be in the format `:emoji_name:` | | `icon_url` | string | A URL to an image to use as the icon for the message | | `blocks` | array | Rich text message content, represented as Slack "blocks". See [Slack Block Kit documentation](https://docs.slack.dev/block-kit) for more information | | `channel` | string | The Slack channel ID to send the message to. If not set, the channel ID from the channel spec will be used | #### Example Slack Message Body ```json { "icon_emoji": ":rocket:", "blocks": [ { "type": "section", "text": { "type": "mrkdwn", "text": "🚀 A new promotion has been created for stage *${{ event.stageName }}* with freight *${{ event.freight.alias }}*!" } } ], "channel": "C1234567890" } ``` ### SMTP SMTP messages can be formatted using JSON with the following structure: | Key | Type | Description | | --------- | ------- | -------------------------------------------------------------------------------------------------------- | | `subject` | string | The subject line of the email | | `body` | string | The body content of the email | | `to` | array | An array of recipient email addresses | | `html` | boolean | Whether the body should be interpreted as HTML. If `false` or omitted, the body is treated as plain text | #### Example SMTP Message Body ```json { "subject": "🚀 New Promotion Created", "body": "New Promotion CreatedA new promotion has been created for stage ${{ event.stageName }} with freight ${{ event.freight.alias }}!", "to": [ "recipient@example.com" ], "html": true } ``` ### HTTP HTTP messages support an optional structured body. When the `EventRouter` output is plain text, it is sent verbatim as the request body. When the output is structured data (JSON or YAML), it is decoded into the following fields: | Key | Type | Description | | ------------------- | ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | | `body` | any | The request body to send. Strings are sent as-is; other values are serialized as JSON. This can be nested JSON objects or arrays. | | `headers` | map[string]string | Additional HTTP headers to include in the request. These are merged with any headers configured on the channel, with per-message headers taking precedence | | `method` | string | Overrides the HTTP method configured on the channel | | `url` | string | Overrides the URL configured on the channel | | `queryParams` | map[string]string | Additional query parameters to include in the request URL. These are merged with any query parameters configured on the channel | | `successExpression` | string | Overrides the channel-level success expression. An expr-lang expression with access to `response.status`, `response.body`, and `response.headers` | | `failureExpression` | string | Overrides the channel-level failure expression. An expr-lang expression with access to `response.status`, `response.body`, and `response.headers` | #### Example HTTP Message Body ```json { "body": { "text": "🚀 A new promotion has been created for stage ${{ event.stageName }}!" }, "headers": { "X-Custom-Header": "kargo", "Accept": "application/json" }, "successExpression": "response.status >= 200 && ${{ event.freight.name }} == response.body.freightName" } ``` --- ## Notifications :::info This set of features are only available in Kargo on the [Akuity Platform](https://akuity.io/akuity-platform), versions v1.8 and above. ::: Kargo's Notifications feature allows you to configure and send notifications based on specific events that occur within your Kargo projects. This can be useful for keeping team members informed about important actions, such as promotions, approvals, or failures. Notifications are event-driven, meaning they are triggered by specific events rather than being part of a predefined workflow. For sending a notification as part of a promotion workflow, consider using the [`send-message` promotion step](../../30-promotion-steps/send-message.md). --- ## Events Kargo emits Kubernetes events for various actions that occur within the system. These docs cover the various event types and how to work with them. --- ## Kargo User's Guide The Kargo User's Guide is for developers using Kargo to implement flexible and powerful GitOps promotion pipelines. This entire branch of the documentation tree is dedicated to topics such as managing Kargo projects, defining promotion pipelines, and managing credentials and other secrets used by your pipelines -- all from an end-user's perspective. :::info[Not what you were looking for?] If you're an operator responsible for the installation and maintenance of Kargo, you'll find the most relevant topics to that role documented in the [Operator Guide](../40-operator-guide/index.md). If you're primarily a developer and need help installing Kargo into a local Kubernetes cluster just to see what it's all about, the Operator Guide is still a great place to start. If you're a developer interested in contributing to Kargo, you'll find everything you need to get started in the [Contributor Guide](../60-contributor-guide/index.md). ::: --- ## Hacking on Kargo Kargo is implemented in Go. For maximum productivity in your text editor or IDE, it is recommended that you have installed the latest stable releases of Go and applicable editor/IDE extensions, however, this is not strictly required to be successful. ## Running Tests In order to minimize the setup required to apply small changes and to reduce the incidence of tests passing locally, but failing during the continuous integration process due to environmental differences, we've made it easy to execute tests within a container that is maximally similar to those used in CI. To take advantage of this, you only need `make` and [Docker](https://docs.docker.com/engine/install/) (or a Docker-compatible container-runtime). To run all unit tests: ```shell make hack-test-unit ``` :::info If you wish to opt-out of executing the tests within a container (for performance reasons, perhaps), drop the `hack-` prefix from the target: ```shell make test-unit ``` This will require Go to be installed locally. ::: ## Running Linters It is also possible to execute a variety of different linters that perform static code analysis, detect code hygiene issues, assert adherence to project standards, etc. As with unit tests, we've made it easy to execute linters within a container that is maximally similar to those used in CI. To lint Go code only: ```shell make hack-lint-go ``` To lint generated protobuf definitions only: ```shell make hack-lint-proto ``` To lint Helm charts only: ```shell make hack-lint-charts ``` To run _all_ linters with one command: ```shell make hack-lint ``` :::info If you wish to opt-out of executing any or all linters within a container (for performance reasons, perhaps), drop the `hack-` prefix from the desired target. This will require quite a variety of tools to be installed locally, so we do not recommend this if you can avoid it. ::: ## Executing Code Generation Anytime the contents of the `api/` directory have been modified, a code generation process must be manually executed. As with tests and linters, this process is easy to execute within a container, which eliminates the need to install various tools or specific versions thereof: ```shell make hack-codegen ``` :::info If you wish to opt-out of executing code-generation within a container (for performance reasons, perhaps), drop the `hack-` prefix from the target: ```shell make codegen ``` This will require quite a variety of tools to be installed locally, so we do not recommend this if you can avoid it. ::: ## Building the Image To build source into a Docker image that will be tagged as `kargo:dev`, execute the following: ```shell make hack-build ``` :::info There is seldom a need to do this, as the next section will cover a better option for rapidly building and deploying Kargo from source. ::: :::info The [Docker buildx](https://github.com/docker/buildx) machine required by the build process has to be created with the `--driver-opt network=host` option to allow it to access the (temporary) local image registry used for the base image. If you encounter an error during the build process (e.g. `failed to resolve source metadata for localhost:5001/kargo-base:latest-arm64` or `granting entitlement network.host is not allowed by build daemon configuration`), you may need to (re)create the machine using `docker buildx create` with this option set. ::: ## Iterating Quickly This section focuses on the best approaches for gaining rapid feedback on changes you make to Kargo's code base. The fastest path to learning whether changes you have applied work as desired is to execute unit tests as described in previous sections. If the changes you are applying are complex, it can also be advantageous to exercise them, end-to-end, as a user would. Because Kargo is dependent on a Kubernetes cluster, this raises the question of how Kargo can not only be built from source, but also deployed to a live Kubernetes cluster efficiently enough to enable a tight feedback loop as you continue iterating on your changes. The remainder of this section covers the approaches we recommend for enabling this. 1. Launch or re-use an existing local Kubernetes cluster. Any of the following options are viable: If you are a [Docker Desktop](https://www.docker.com/products/docker-desktop/) user, you can follow [these instructions](https://docs.docker.com/desktop/kubernetes/) to enable its built-in Kubernetes support. If it's already enabled, you're ready to go. :::info A specific benefit of this option is that nothing special is required in terms of creating a local image registry connected to the cluster. Additionally, this approach requires no specific port-forwarding rules to be defined. ::: :::info Although this is one of the fastest paths to a local Kubernetes cluster, be aware that Docker Desktop supports only a _single_ Kubernetes cluster. If that cluster reaches a state you are dissatisfied with, resetting it will remove not just Kargo-related resources, but _all_ your workloads and data. ::: [OrbStack](https://orbstack.dev/) is a fast, lightweight, drop-in replacement for Docker Desktop for Mac OS only. You can follow [these instructions](https://docs.docker.com/desktop/kubernetes/) to enable its built-in Kubernetes support. If it's already enabled, you're ready to go. :::info A specific benefit of this option is that nothing special is required in terms of creating a local image registry connected to the cluster. Additionally, this approach requires no specific port-forwarding rules to be defined. ::: :::info Although this is one of the fastest paths to a local Kubernetes cluster, be aware that OrbStack supports only a _single_ Kubernetes cluster. If that cluster reaches a state you are dissatisfied with, resetting it will remove not just Kargo-related resources, but _all_ your workloads and data. ::: If you have any Docker-compatible container runtime installed (including native Docker, Docker Desktop, or OrbStack), you can easily launch a disposable cluster to facilitate Kargo development using [kind](https://kind.sigs.k8s.io/#installation-and-usage). You do not need to install it in advance. The following `make` target will launch a kind cluster with a local image registry wired into it, various port-forwarding rules pre-configured, and Kargo's prerequisites installed: ```shell make hack-kind-up ``` :::info The `hack-kind-up` target will ensure the installation of suitable versions of `kind` and [ctlptl](https://github.com/tilt-dev/ctlptl#how-do-i-install-it) (used for declarative kind configuration) into `hack/bin/`. ::: :::info While this option is a bit more complex than using Docker Desktop or OrbStack directly, it offers the advantage of being fully-disposable. If your cluster reaches a state you are dissatisfied with, you can simply destroy it and launch a new one. ::: If you have any Docker-compatible container runtime installed (including native Docker, Docker Desktop, or OrbStack), you can easily launch a disposable cluster to facilitate Kargo development using [k3d](https://k3d.io). You do not need to install it in advance. The following `make` target will launch a kind cluster with a local image registry wired into it, various port-forwarding rules pre-configured, and Kargo's prerequisites installed: ```shell make hack-k3d-up ``` :::info The `hack-k3d-up` target will ensure the installation of suitable versions of `k3d` and [ctlptl](https://github.com/tilt-dev/ctlptl#how-do-i-install-it) (used for declarative k3d configuration) into `hack/bin/`. ::: :::info While this option is a bit more complex than using Docker Desktop or OrbStack directly, it offers the advantage of being fully-disposable. If your cluster reaches a state you are dissatisfied with, you can simply destroy it and launch a new one. ::: 1. Optional: Configure and start a tunnel for the external webhooks server: If you are working on or testing the external webhooks server, you will benefit from configuring a tunnel from the outside world so that traffic originating from platforms like GitHub, Docker Hub, and others can reach the server. It is easy to accomplish this using [ngrok](https://ngrok.com/). If you wish to do so, you must first [install ngrok](https://ngrok.com/downloads) yourself. With `ngrok` installed, you can conveniently open a tunnel to `localhost:30083` (where the next step will run the external webhooks server) using: ```shell make hack-ngrok ``` Allow this process to run while you are working on or testing the external webhooks server. Interrupt it with `ctrl + c` when you are done. If you have a paid ngrok account that allows you to use a custom domain name for your tunnels, you can specify that domain name using the `KARGO_EXTERNAL_WEBHOOKS_SERVER_HOSTNAME` and (to properly set the protocol scheme) `KARGO_EXTERNAL_WEBHOOKS_SERVER_TLS_TERMINATED_UPSTREAM` environment variables before running the make target: ```shell export KARGO_EXTERNAL_WEBHOOKS_SERVER_HOSTNAME=my-tunnel.ngrok.io export KARGO_EXTERNAL_WEBHOOKS_SERVER_TLS_TERMINATED_UPSTREAM=true make hack-ngrok ``` If the `KARGO_EXTERNAL_WEBHOOKS_SERVER_HOSTNAME` environment variable is undefined, the tunnel will utilize a dynamically-generated subdomain of `ngrok.io`. 1. Build and deploy Kargo from source: [Tilt](https://docs.tilt.dev/#macoslinux) is a convenient tool that builds container images from source and seamlessly deploys them to a local Kubernetes cluster. More importantly, it enables developers to rapidly rebuild and replace running components with the click of a button. You do not need to install it in advance. ```shell make hack-tilt-up ``` :::info The `hack-tilt-up` target will ensure the installation of a suitable versions of `tilt` and `helm` into `hack/bin/`. When run for the first time on a new development cluster, `make hack-tilt-up` will install suitable versions of [cert-manager](https://cert-manager.io/), [Argo CD](https://argoproj.github.io/cd/), and [Argo Rollouts](https://argoproj.github.io/rollouts/) into your cluster using `helm`. The Argo CD dashboard will be exposed at [localhost:30080](https://localhost:30080). The username and password are both `admin`. You may safely ignore any certificate warnings. ::: Tilt will launch a web-based UI running at [http://localhost:10350](http://localhost:10350). Visit this in your web browser to view the build and deployment status of each Kargo component as well as the logs from each component. :::info Tilt is often configured to watch files and automatically rebuild and replace running components when their source code is changed. This is deliberately disabled for Kargo since the Docker image takes long enough to build that it’s better to conserve system resources by only rebuilding when you choose. The web UI makes it easy to identify components whose source has been altered. They can be rebuilt and replaced with a single click. ::: :::info If you specified a custom domain name for a tunnel to the external webhooks server in the previous step by defining a value for the `KARGO_EXTERNAL_WEBHOOKS_SERVER_HOSTNAME` environment variable, you should export the same value for that environment variable before running `make tilt up` as well: ```shell export KARGO_EXTERNAL_WEBHOOKS_SERVER_HOSTNAME=my-tunnel.ngrok.io make hack-tilt-up ``` ::: 1. If necessary, build the CLI from source: ```shell make hack-build-cli ``` This will produce an executable at `bin/kargo--`. You can log in using: ```shell bin/kargo-- login http://localhost:30081 \ --admin \ --password admin \ --insecure-skip-tls-verify ``` 1. If necessary, access the Kargo UI at [localhost:30082](http://localhost:30082). The admin account password is `admin`. You may safely ignore any certificate warnings. 1. When you are done with Tilt, interrupt the running `make hack-tilt-up` process with `ctrl + c`. Components _will remain running in the cluster_, but Tilt will no longer be in control. If Tilt is restarted later, it will retake control of the already-running components. If you wish to undeploy everything Tilt has deployed for you (except for prerequisites), use `make hack-tilt-down`. 1. Clean up your local Kubernetes cluster. Docker Desktop supports only a _single_ Kubernetes cluster. If you are comfortable deleting not just just Kargo-related resources, but _all_ your workloads and data, the cluster can be reset from the Docker Desktop Dashboard. If, instead, you wish to preserve non-Kargo-related workloads and data, you will need to manually uninstall Kargo's prerequisites: ``` make hack-uninstall-prereqs ``` OrbStack supports only a _single_ Kubernetes cluster. If you are comfortable deleting not just just Kargo-related resources, but _all_ your workloads and data, you can destroy the cluster with: ```shell orb delete k8s ``` If, instead, you wish to preserve non-Kargo-related workloads and data, you will need to manually uninstall Kargo's prerequisites: ``` make hack-uninstall-prereqs ``` To destroy the cluster, use: ```shell make hack-kind-down ``` :::info This command deliberately leaves your local image registry running so that if you resume work later, you are doing so with a local registry that’s already primed with most layers of Kargo’s image. If you wish to stop the registry, use: ```shell docker stop kargo-dev-registry ``` To destroy it, use: ```shell docker rm -f kargo-dev-registry ``` ::: To destroy the cluster, use: ```shell make hack-k3d-down ``` :::info This command deliberately leaves your local image registry running so that if you resume work later, you are doing so with a local registry that’s already primed with most layers of Kargo’s image. If you wish to stop the registry, use: ```shell docker stop kargo-dev-registry ``` To destroy it, use: ```shell docker rm -f kargo-dev-registry ``` ::: ## Contributing to Documentation Contributors should ensure that their changes are accompanied by relevant documentation updates. This helps maintain the project's sustainability. Pull requests with corresponding documentation updates are more likely to be merged faster. To make this process smoother, you can refer to [Docusaurus](https://docusaurus.io/docs) for guidance on writing and maintaining docs effectively. ### Previewing Doc Changes Locally After making your changes, preview the documentation locally to ensure everything renders correctly. You can either run it in a container or natively on your system. To build and serve the docs inside a container: ```shell make hack-serve-docs ``` :::info If you wish to opt-out of executing code-generation within a container (for performance reasons, perhaps), drop the `hack-` prefix from the target to run the docs natively on your system: ```shell make serve-docs ``` This will require quite a variety of tools to be installed locally, so we do not recommend this if you can avoid it. ::: --- ## Debugging Kargo From time to time, you may need to debug Kargo. This document outlines options to help you achieve a better understanding of what Kargo (internally) is doing, especially when you are running into issues which are not immediately obvious by just reading the code. As a Kargo user, you may not need to debug Kargo itself, but this document may serve you when a Kargo maintainer asks you to provide additional information about an issue you are facing. ## Enabling pprof endpoints Kargo components can be configured to expose [`pprof` endpoints](https://golang.org/pkg/net/http/pprof/). These endpoints can be used to profile the components when they are running, and can be useful to understand what the components are doing and where they are spending time. To enable the `pprof` endpoint on a component, you can set the `PPROF_BIND_ADDRESS` environment variable to the address where the component should listen for `pprof` requests. For example, to enable the `pprof` endpoint on port `6060` of the controller, you can set the `PPROF_BIND_ADDRESS` environment variable to `:6060`. ```yaml --- apiVersion: apps/v1 kind: Deployment metadata: name: kargo-controller spec: # ...omitted for brevity template: spec: containers: - name: kargo-controller env: - name: PPROF_BIND_ADDRESS value: ":6060" ``` After setting the `PPROF_BIND_ADDRESS` environment variable, the `pprof` endpoints will be available at `http://:6060/debug/pprof/`. ### Collecting a profile To collect a profile, you can port-forward the `pprof` address to your local machine and collect the data from an endpoint of choice. For example, to collect a heap profile, you can run: ```console $ kubectl port-forward -n deployment/ 6060 $ curl -Sk -v http://localhost:6060/debug/pprof/heap > heap.out ``` This will collect a heap profile in the `heap.out` file, which you can then [analyze using `go`](https://go.dev/blog/pprof), or share with a Kargo maintainer. --- ## Signing Commits All commits merged into Kargo's main branch MUST bear a DCO (Developer Certificate of Origin) sign-off. This is a line placed at the end of a commit message containing a contributor’s “signature.” In adding this, the contributor certifies that they have the right to contribute the material in question. Here are the steps to sign your work: 1. Verify the contribution in your commit complies with the [terms of the DCO](https://developercertificate.org/). 1. Add a line like the following to your commit message: ``` Signed-off-by: Joe Smith ``` You MUST use your legal name -- handles or other pseudonyms are not permitted. While you could manually add DCO sign-off to every commit, there is an easier way: 1. Configure your git client appropriately. This is one-time setup. ```shell git config user.name git config user.email ``` If you work on multiple projects that require a DCO sign-off, you can configure your git client to use these settings globally instead of only for Kargo: ```shell git config --global user.name git config --global user.email ``` 1. Use the --signoff or -s (lowercase) flag when making each commit. For example: ```shell git commit --message "" --signoff ``` If you ever make a commit and forget to use the `--signoff` flag, you can amend your commit with this information before pushing: ```shell git commit --amend --signoff ``` 1. You can verify the above worked as expected using `git log`. Your latest commit should look similar to this one: ```shell Author: Joe Smith Date: Thu Feb 2 11:41:15 2018 -0800 Update README Signed-off-by: Joe Smith ``` Notice the `Author` and `Signed-off-by` lines match. If they do not, the PR will be rejected by the automated DCO check. --- ## Contributor Covenant Code of Conduct ## Our Pledge We as members, contributors, and leaders pledge to make participation in our community a harassment-free experience for everyone, regardless of age, body size, visible or invisible disability, ethnicity, sex characteristics, gender identity and expression, level of experience, education, socio-economic status, nationality, personal appearance, race, caste, color, religion, or sexual identity and orientation. We pledge to act and interact in ways that contribute to an open, welcoming, diverse, inclusive, and healthy community. ## Our Standards Examples of behavior that contributes to a positive environment for our community include: * Demonstrating empathy and kindness toward other people * Being respectful of differing opinions, viewpoints, and experiences * Giving and gracefully accepting constructive feedback * Accepting responsibility and apologizing to those affected by our mistakes, and learning from the experience * Focusing on what is best not just for us as individuals, but for the overall community Examples of unacceptable behavior include: * The use of sexualized language or imagery, and sexual attention or advances of any kind * Trolling, insulting or derogatory comments, and personal or political attacks * Public or private harassment * Publishing others' private information, such as a physical or email address, without their explicit permission * Other conduct which could reasonably be considered inappropriate in a professional setting ## Enforcement Responsibilities Community leaders are responsible for clarifying and enforcing our standards of acceptable behavior and will take appropriate and fair corrective action in response to any behavior that they deem inappropriate, threatening, offensive, or harmful. Community leaders have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, and will communicate reasons for moderation decisions when appropriate. ## Scope This Code of Conduct applies within all community spaces, and also applies when an individual is officially representing the community in public spaces. Examples of representing our community include using an official e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event. ## Enforcement Instances of abusive, harassing, or otherwise unacceptable behavior may be reported to the community leaders responsible for enforcement at [kent@akuity.io](mailto:kent@akuity.io). All complaints will be reviewed and investigated promptly and fairly. All community leaders are obligated to respect the privacy and security of the reporter of any incident. ## Enforcement Guidelines Community leaders will follow these Community Impact Guidelines in determining the consequences for any action they deem in violation of this Code of Conduct: ### 1. Correction **Community Impact**: Use of inappropriate language or other behavior deemed unprofessional or unwelcome in the community. **Consequence**: A private, written warning from community leaders, providing clarity around the nature of the violation and an explanation of why the behavior was inappropriate. A public apology may be requested. ### 2. Warning **Community Impact**: A violation through a single incident or series of actions. **Consequence**: A warning with consequences for continued behavior. No interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, for a specified period of time. This includes avoiding interactions in community spaces as well as external channels like social media. Violating these terms may lead to a temporary or permanent ban. ### 3. Temporary Ban **Community Impact**: A serious violation of community standards, including sustained inappropriate behavior. **Consequence**: A temporary ban from any sort of interaction or public communication with the community for a specified period of time. No public or private interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, is allowed during this period. Violating these terms may lead to a permanent ban. ### 4. Permanent Ban **Community Impact**: Demonstrating a pattern of violation of community standards, including sustained inappropriate behavior, harassment of an individual, or aggression toward or disparagement of classes of individuals. **Consequence**: A permanent ban from any sort of public interaction within the community. ## Attribution This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 2.1, available at [https://www.contributor-covenant.org/version/2/1/code_of_conduct.html][v2.1]. Community Impact Guidelines were inspired by [Mozilla's code of conduct enforcement ladder][Mozilla CoC]. For answers to common questions about this code of conduct, see the FAQ at [https://www.contributor-covenant.org/faq][FAQ]. Translations are available at [https://www.contributor-covenant.org/translations][translations]. [homepage]: https://www.contributor-covenant.org [v2.1]: https://www.contributor-covenant.org/version/2/1/code_of_conduct.html [Mozilla CoC]: https://github.com/mozilla/diversity [FAQ]: https://www.contributor-covenant.org/faq [translations]: https://www.contributor-covenant.org/translations --- ## Contributor Guide This guide defines the policies and procedures for contributing to Kargo. Please read it in full before opening issues or pull requests. ## Process Overview Kargo follows an **issue-first contribution model**. Unless authored by a maintainer, all code contributions require a linked issue that has been reviewed and unblocked by maintainers in advance. :::caution Ideas and contributions from the community are always welcome, however, items not appearing on the [roadmap](../100-roadmap.md) are reviewed on a best-effort basis. While we cannot guarantee specific timelines or outcomes, we value and consider every submission. ::: 1. **Open an issue** using the [Bug Report](https://github.com/akuity/kargo/issues/new?template=bug_report.yml) or [Feature Request](https://github.com/akuity/kargo/issues/new?template=feature_request.yml) template. 1. **Wait for maintainer review.** Maintainers will triage the issue, ask clarifying questions, and determine whether the work aligns with the project's priorities and [roadmap](../100-roadmap.md). 1. **Wait for the issue to be unblocked.** When an issue is ready for external contribution, a maintainer will remove all blocking labels. **Do not begin work while any blocking labels are present.** 1. **Open a pull request** that references the unblocked issue using `Closes #` in the PR body. ## Blocking Labels The following labels indicate an issue is **not ready for external contribution**. Pull requests linked to issues carrying any of these labels are automatically converted to a draft until the issue is unblocked. | Label | Meaning | | ----- | ------- | | `kind/proposal` | Feature request under consideration. Not yet unblocked. | | `needs discussion` | Needs further discussion before any work begins. | | `needs research` | Needs investigation or research before work begins. | | `maintainer only` | Reserved for maintainers due to size, complexity, or sensitivity. | | `area/security` | Involves security-sensitive code. Maintainer-coordinated only. | | `size/large` | Large scope. Requires maintainer coordination. | | `size/x-large` | Very large scope. Requires maintainer coordination. | | `size/xx-large` | Extremely large scope. Requires maintainer coordination. | An issue is ready for external contribution when **none** of these labels are present. ### Exceptions A few kinds of contribution are exempt from the issue-first requirement and can be opened as pull requests directly, without a corresponding issue: - **Drive-by fixes.** Pull requests of five or fewer lines (added + removed, combined). This covers typo corrections, comment fixes, and small, obvious bug fixes. - **Documentation-only changes.** Pull requests whose changes are limited to `README.md` and Markdown files anywhere under `docs/`. Exempt pull requests must still adhere to the [Quality Expectations](#quality-expectations) below — the issue requirement is the only thing waived. ## Quality Expectations All pull requests must: - Be authored and reviewed by a human. AI-assisted coding is acceptable, but every line must be understood and verified by the submitter. Submissions that appear to be generated without meaningful human review will be closed. - Include appropriate tests for new and modified code. - Follow existing code conventions and patterns. - Include DCO sign-off on all commits (see [Signing Commits](./30-signing-commits.md)). - Include documentation updates where user-facing behavior has changed. Cryptographic signing of all commits is strongly encouraged. ## Specific Topics --- ## Resources Learn more about Kargo through talks and webinars presented at conferences like KubeCon, ArgoCon, GitOpsCon, etc! ## Kargo GA Webinar 🎉 By Kelsey Hightower & Jesse Suen