Tekton – Tekton

Blog: Tekton Pipelines v1.10.0: Observability, Evolved

Fri, 27 Feb 2026 00:00:00 +0000

We’re excited to announce the release of Tekton Pipelines v1.10.0! The headline for this release is a major infrastructure change: the migration from OpenCensus to OpenTelemetry for all metrics instrumentation.

OpenCensus to OpenTelemetry Migration

OpenCensus has been deprecated in favor of OpenTelemetry, and Tekton Pipelines now follows suit. This release migrates all PipelineRun and TaskRun metrics to OpenTelemetry instruments — histograms, counters, and gauges — and updates the underlying Knative dependency to v1.19.

What Changed

Infrastructure metrics renamed: Go runtime, Workqueue, and K8s Client metrics move from the tekton_pipelines_controller_ prefix to standard OpenTelemetry/Knative namespaces (e.g., kn_workqueue_*, go_*).
New reason label: Duration metrics for PipelineRuns and TaskRuns now include a reason label (e.g., Completed, Succeeded) for more granular breakdown.
Removed metrics: tekton_pipelines_controller_reconcile_count and tekton_pipelines_controller_reconcile_latency have been removed. Use kn_workqueue_process_duration_seconds and kn_workqueue_adds_total instead.
Preserved compatibility: Core metrics like pipelinerun_total, taskrun_total, and taskruns_pod_latency_milliseconds retain their original names and labels.

What You Need to Do

Update your config-observability ConfigMap to use metrics-protocol: prometheus (or grpc/http) instead of the old metrics.backend-destination. If you were already using Prometheus, no changes are needed.
Update your dashboards:
- Replace tekton_pipelines_controller_workqueue_* queries with kn_workqueue_*
- Replace tekton_pipelines_controller_go_* queries with standard go_* metrics
- Account for the new reason label on duration metrics

See PR #9043 for the full migration table and details.

Other Highlights

New Features

SHA-256 support for Git resolver: The Git resolver now validates SHA-256 commit hashes (64 characters), future-proofing for Git v3+ which will use SHA-256 instead of SHA-1. (#9278)

Bug Fixes

Pipeline results from failed tasks: Pipeline-level results now correctly include results from failed, cancelled, and timed-out tasks. Previously, results referencing non-successful task outputs were left as unresolved variable strings. (#9367)
Pipeline param defaults with context variables: Fixed a bug where PipelineRun validation failed when a pipeline parameter’s default value referenced a non-parameter variable like $(context.pipelineRun.name). (#9386)
ResolutionRequest CRD on Kubernetes 1.33+: Removed redundant shortNames from the ResolutionRequest CRD that caused ShortNamesConflict on Kubernetes 1.33+. (#9398)

Get Started

Install or upgrade to v1.10.0:

kubectl apply -f https://infra.tekton.dev/tekton-releases/pipeline/previous/v1.10.0/release.yaml

Check out the full release notes and documentation for more details.

Have questions or feedback? Join us on Tekton Slack or open an issue on GitHub.

Blog: Introducing Tekton Pruner

Thu, 05 Feb 2026 00:00:00 +0000

Tekton Pruner automatically cleans up completed PipelineRuns and TaskRuns based on retention policies you define.

The Problem

Tekton does not delete PipelineRuns and TaskRuns after completion. Over time, this increases etcd storage usage and degrades API performance.

Before: Job-Based Pruner

The Tekton Operator has a job-based pruner configured via TektonConfig:

pruner:
  disabled: false
  schedule: "0 8 * * *"
  startingDeadlineSeconds: 100  # optional
  resources:
    - taskrun
    - pipelinerun
  keep: 3
  # keep-since: 1440
  # NOTE: you can use either "keep" or "keep-since", not both
  prune-per-resource: true

The operator creates a CronJob based on this config:

schedule: Cron expression for when cleanup runs. 0 8 * * * means daily at 8 AM.
resources: Which resources to prune - taskrun, pipelinerun, or both.
keep: Retain the last N runs. With keep: 3, only the 3 most recent runs are kept.
keep-since: Alternative to keep. Retain runs from the last N minutes. keep-since: 1440 keeps runs from the last 24 hours. You can use one or the other, not both.
prune-per-resource: When true, limits apply per pipeline/task name. When false, the limit is global across the namespace.
startingDeadlineSeconds: How long the CronJob can be delayed before it’s considered missed.

You can also override at the namespace level using annotations:

metadata:
  annotations:
    operator.tekton.dev/prune.resources: "taskrun,pipelinerun"
    operator.tekton.dev/prune.keep-since: "7200"
    operator.tekton.dev/prune.keep: "5"

This works, but there are gaps.

First, if you installed Tekton Pipelines with kubectl apply instead of the Operator, you don’t have this pruner.

Second, the batch model creates pressure. Run multiple pipelines a day with a daily cleanup schedule, and you’ve got 500 completed PipelineRuns in etcd by the time the CronJob fires. In high-throughput environments, that’s real load on your API server.

Third, one policy per namespace is limiting. Your nightly test runs don’t need the same retention as production deployments. But with the Operator pruner, they get the same treatment unless you split them into separate namespaces.

Tekton Pruner

Tekton Pruner is event-driven. When a PipelineRun or TaskRun completes, the controller evaluates it against your policies right then. No accumulation between scheduled runs.

It also works without the Operator - install it alongside any Tekton Pipelines deployment.

The configuration model is hierarchical. Set cluster-wide defaults, override per namespace, and use label selectors to apply different policies to different pipelines within the same namespace. Your test pipelines can be cleaned up in minutes while production runs could stick around for a week.

If you’re using Tekton Chains or Tekton Results, don’t worry both add finalizers to PipelineRuns and TaskRuns. Chains adds finalizers (chains.tekton.dev/taskrun for TaskRuns, chains.tekton.dev/pipelinerun for PipelineRuns) that block deletion until signing and attestation storage completes. Results does the same with its own finalizer, holding the resource until it’s persisted. Pruner respects these finalizers, so a run won’t actually be removed until Chains and Results are done with it.

Installation

Via Tekton Operator

If you’re using the Tekton Operator, enable Tekton Pruner through TektonConfig. You need to disable the old job-based pruner first:

apiVersion: operator.tekton.dev/v1alpha1
kind: TektonConfig
metadata:
  name: config
spec:
  pruner:
    disabled: true  # disable the job-based pruner
  tektonpruner:
    disabled: false
    global-config:
      enforcedConfigLevel: global
      ttlSecondsAfterFinished: 3600
      successfulHistoryLimit: 5
      failedHistoryLimit: 3

Standalone Installation

If you installed Tekton Pipelines without the Operator:

export VERSION=0.3.5  # check https://github.com/tektoncd/pruner/releases for latest
kubectl apply -f "https://infra.tekton.dev/tekton-releases/pruner/previous/v${VERSION}/release.yaml"

Verify

kubectl get pods -n tekton-pipelines -l app=tekton-pruner-controller

You should see the controller pod running. The controller handles pruning logic and serves the validating webhook that checks ConfigMaps before they’re applied.

Configuration

ConfigMaps need these labels (since v0.3.0):

labels:
  app.kubernetes.io/part-of: tekton-pruner
  pruner.tekton.dev/config-type: global  # or 'namespace'

Cluster-Wide Defaults

apiVersion: v1
kind: ConfigMap
metadata:
  name: tekton-pruner-default-spec
  namespace: tekton-pipelines
  labels:
    app.kubernetes.io/part-of: tekton-pruner
    pruner.tekton.dev/config-type: global
data:
  global-config: |
    enforcedConfigLevel: global
    ttlSecondsAfterFinished: 3600
    successfulHistoryLimit: 5
    failedHistoryLimit: 3

TTL: Once the time passes after completion, the run is deleted. Doesn’t matter if you’re under the history limit.
History limit: Deletes oldest runs when count exceeds the limit. Only matters if TTL hasn’t already deleted them.

Example: ttlSecondsAfterFinished: 300 and historyLimit: 3. You run a pipeline 5 times.

History limit kicks in → keeps 3, deletes 2 oldest
5 minutes later, TTL expires → all 3 remaining runs deleted

History limit kept 3, but TTL deleted them anyway once 5 minutes passed. If you want runs to persist, increase TTL or use only history limits.

Per-Namespace Config

Set enforcedConfigLevel: namespace to allow namespace-level overrides.

Inline approach - define everything in the global ConfigMap:

data:
  global-config: |
    enforcedConfigLevel: namespace
    ttlSecondsAfterFinished: 3600
    namespaces:
      production:
        ttlSecondsAfterFinished: 604800
        successfulHistoryLimit: 20
      dev:
        ttlSecondsAfterFinished: 1800

Separate ConfigMaps - namespace owners manage their own:

apiVersion: v1
kind: ConfigMap
metadata:
  name: tekton-pruner-namespace-spec
  namespace: my-team
  labels:
    app.kubernetes.io/part-of: tekton-pruner
    pruner.tekton.dev/config-type: namespace
data:
  ns-config: |
    ttlSecondsAfterFinished: 7200
    successfulHistoryLimit: 10

Label-Based Selectors

Different pipelines in the same namespace can have different policies:

data:
  ns-config: |
    pipelineRuns:
      - selector:
        - matchLabels:
            app: critical-service
        ttlSecondsAfterFinished: 604800
        successfulHistoryLimit: 50
      - selector:
        - matchLabels:
            app: test-jobs
        ttlSecondsAfterFinished: 300
        successfulHistoryLimit: 3

Selectors only work in namespace-level ConfigMaps.

What to Watch Out For

System namespaces are off-limits. Don’t create namespace-level ConfigMaps in kube-*, openshift-*, tekton-pipelines, or other tekton-* namespaces.

Webhook validates your config. If your ConfigMap is rejected, check the labels and YAML syntax. The webhook will tell you what’s wrong.

Checking Logs

kubectl logs -n tekton-pipelines -l app=tekton-pruner-controller -f

You’ll see entries when resources are pruned, along with which policy triggered the deletion.

Config Reference

Field	What it does
`enforcedConfigLevel`	`global` applies cluster-wide, `namespace` allows per-ns overrides
`ttlSecondsAfterFinished`	Delete runs older than this many seconds after completion
`successfulHistoryLimit`	Keep this many successful runs per pipeline
`failedHistoryLimit`	Keep this many failed runs per pipeline
`historyLimit`	Used when success/failed limits aren’t set

Blog: Securing HTTP Resolver with Digest Verification

Wed, 04 Feb 2026 00:00:00 +0000

Overview

When fetching remote resources like Tasks and Pipelines from HTTP URLs for your PipelineRun, there’s always a risk that the content could be tampered with during transit or at the source. To address this security concern, the Tekton HTTP resolver now supports optional digest validation, allowing users to verify the integrity of fetched content against a known cryptographic hash.

This feature enables users to specify an expected digest (SHA256 or SHA512) in their PipelineRun definitions. The resolver will then validate the fetched content against this digest before proceeding, ensuring that only verified, untampered resources are executed in your pipelines.

Why Digest Verification Matters

When using the HTTP resolver to fetch remote Tasks or Pipelines for your PipelineRun, several attack vectors could compromise your build process:

Man-in-the-Middle Attacks: An attacker intercepting network traffic could modify the content of fetched resources.
Compromised Source Repositories: If the source repository is compromised, malicious code could be injected into your CI/CD pipeline.
DNS Hijacking: Attackers could redirect your requests to a malicious server serving altered content.

By providing a digest at the time of PipelineRun creation, you establish a cryptographic guarantee that the executed Pipeline or Task matches exactly what you intended to run. This ensures your PipelineRun only executes verified, untampered resources.

How It Works

The HTTP resolver now accepts an optional digest parameter that specifies the expected hash of the fetched content. The digest must be provided in the format <algorithm>:<hash>, where the supported algorithms are:

sha256 - 256-bit hash (64 hexadecimal characters)
sha512 - 512-bit hash (128 hexadecimal characters)

When a digest is provided, the resolver:

Fetches the content from the specified URL
Computes the hash of the fetched content using the specified algorithm
Compares the computed hash against the provided digest
Only returns the content if the hashes match; otherwise, the resolution fails with an error

Enabling the HTTP Resolver

To use the HTTP resolver, ensure that the feature flag is enabled in your Tekton Pipelines installation. The enable-http-resolver flag should be set to true in the resolver’s feature flags ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: resolvers-feature-flags
  namespace: tekton-pipelines-resolvers
data:
  enable-http-resolver: "true"

Usage Examples

Basic Pipeline Resolution with Digest Verification

Here’s an example of using the HTTP resolver with SHA256 digest verification to fetch a Pipeline from a remote repository:

apiVersion: tekton.dev/v1
kind: PipelineRun
metadata:
  name: secure-http-demo
spec:
  pipelineRef:
    resolver: http
    params:
      - name: url
        value: https://raw.githubusercontent.com/tektoncd/catalog/main/pipeline/build-push-gke-deploy/0.1/build-push-gke-deploy.yaml
      - name: digest
        value: sha256:e1a86b942e85ce5558fc737a3b4a82d7425ca392741d20afa3b7fb426e96c66b

Task Resolution with SHA512 Digest

For users requiring stronger hash guarantees, SHA512 is also supported:

apiVersion: tekton.dev/v1
kind: TaskRun
metadata:
  name: secure-task-run
spec:
  taskRef:
    resolver: http
    params:
      - name: url
        value: https://raw.githubusercontent.com/tektoncd-catalog/git-clone/main/task/git-clone/git-clone.yaml
      - name: digest
        value: sha512:a1b2c3d4e5f6...your-128-character-sha512-hash...

Using with Authentication

The digest verification works seamlessly with the HTTP resolver’s authentication features:

apiVersion: tekton.dev/v1
kind: PipelineRun
metadata:
  name: authenticated-secure-demo
spec:
  pipelineRef:
    resolver: http
    params:
      - name: url
        value: https://raw.githubusercontent.com/myorg/private-repo/main/pipeline.yaml
      - name: http-username
        value: git
      - name: http-password-secret
        value: github-token
      - name: http-password-secret-key
        value: token
      - name: digest
        value: sha256:f37cdd0e86d35ac97a65e11c8c2b9af12345678901234567890abcdef1234567

Calculating Digests

Before using digest verification, you need to calculate the hash of your Tekton resource. You can use standard command-line tools available on all major operating systems:

SHA256 Digest

# Calculate sha256 digest
curl -sL https://raw.githubusercontent.com/owner/repo/main/task/task.yaml | sha256sum

SHA512 Digest

# Calculate sha512 digest
curl -sL https://raw.githubusercontent.com/owner/repo/main/task/task.yaml | sha512sum

The output will be a hexadecimal string that you can use as the digest value. Remember to prefix it with the algorithm name (e.g., sha256: or sha512:).

Tip

sha256sum and sha512sum commands are available on all major Linux distributions and macOS. On Windows, you can use PowerShell’s Get-FileHash cmdlet or install tools like Git Bash.

Error Handling

When digest validation fails, the HTTP resolver returns a clear error message to help diagnose the issue:

Error Scenario	Example Error Message
Invalid format	`invalid digest format: sha256`
Unsupported algorithm	`invalid digest algorithm: sha1`
Invalid hash length	`invalid sha256 digest value, expected length: 64, got: 8`
Hash mismatch	`SHA mismatch, expected abc123..., got def456...`

These errors will appear in the PipelineRun or TaskRun status, allowing you to quickly identify and fix any configuration issues.

Best Practices

1. Pin Your Remote Resources

When using the HTTP resolver with digest verification, consider pinning your remote resources to specific commits or tags rather than branches:

# Good: Pinned to a specific commit
- name: url
  value: https://raw.githubusercontent.com/tektoncd/catalog/abc123def/pipeline/my-pipeline.yaml
- name: digest
  value: sha256:e1a86b942e85ce5558fc737a3b4a82d7425ca392741d20afa3b7fb426e96c66b

# Risky: Using a branch that can change
- name: url
  value: https://raw.githubusercontent.com/tektoncd/catalog/main/pipeline/my-pipeline.yaml

2. Automate Digest Updates

Consider automating the process of updating digests when your remote resources change. This can be integrated into your GitOps workflow or CI/CD process.

3. Use SHA256 for Standard Use Cases

SHA256 provides sufficient security for most use cases and produces shorter digest values. Use SHA512 only if your security requirements specifically mandate it.

4. Document Your Digests

Maintain documentation of which digests correspond to which versions of your resources. This helps with auditing and troubleshooting.

Integration with Tekton Chains

This feature complements Tekton Chains for comprehensive pipeline security. While Chains provides attestation and provenance for your build artifacts, digest verification ensures the integrity of the resources used to perform those builds.

Together, they provide end-to-end security for your PipelineRuns:

Digest verification ensures the integrity of your build definitions (Tasks and Pipelines)
Tekton Chains provides signed attestations for your build outputs

Backward Compatibility

The digest parameter is optional, ensuring full backward compatibility with existing configurations. Existing PipelineRuns and TaskRuns that don’t specify a digest will continue to work as before, fetching content without verification.

Conclusion

Digest verification for the HTTP resolver is a significant step toward securing your PipelineRuns. By enabling users to cryptographically verify the integrity of remote resources before execution, this feature helps prevent tampering attacks and ensures that your pipelines execute exactly the code you intend.

We encourage all users who fetch Tasks and Pipelines from HTTP URLs to adopt digest verification as part of their security best practices. The feature is designed to be easy to use while providing strong security guarantees through constant-time comparison and support for industry-standard hashing algorithms.

For more information about the HTTP resolver and its configuration options, refer to the HTTP Resolver documentation.

Have questions or feedback about this feature? Join the conversation on the Tekton Slack or open an issue on GitHub.

Blog: Tekton Pipelines v1.9.0 LTS: Continued Innovation and Stability

Mon, 02 Feb 2026 00:00:00 +0000

We’re excited to announce the release of Tekton Pipelines v1.9.0, our latest Long-Term Support (LTS) release! Since the milestone v1.0.0 release in May 2025, the project has continued to evolve with significant new features, performance improvements, and stability enhancements. This post summarizes the journey from v1.0.0 to v1.9.0, organized by LTS milestones.

Installation

kubectl apply -f https://infra.tekton.dev/releases/pipeline/previous/v1.9.0/release.yaml

v1.0.0 → v1.3.0 LTS (May - August 2025)

The first LTS after v1.0.0 focused on controller resilience and performance.

Features

Exponential backoff retry - Improved handling of transient webhook issues during Pod, TaskRun, and CustomRun creation. Configurable via the wait-exponential-backoff ConfigMap. Documentation
Controller HA improvements - Anti-affinity rules ensure controller replicas are scheduled on different nodes for better availability

PodTemplate param substitution - Enables multi-arch builds with Matrix by allowing param substitution in TaskRunSpecs’ PodTemplate. This lets you target nodes with specific architectures. Documentation

apiVersion: tekton.dev/v1
kind: PipelineRun
metadata:
  name: multi-arch-build
spec:
  pipelineSpec:
    tasks:
    - name: build
      matrix:
        params:
        - name: arch
          value: ["amd64", "arm64"]
      taskSpec:
        steps:
        - name: build
          image: golang:1.21
          script: |
            echo "Building for $(params.arch)"
            GOARCH=$(params.arch) go build -o app-$(params.arch) .            
      # PodTemplate with param substitution to schedule on correct architecture
      podTemplate:
        nodeSelector:
          kubernetes.io/arch: $(params.arch)

Configurable threading - THREADS_PER_CONTROLLER environment variable for tuning controller performance based on cluster size
OOM detection - TaskRuns that fail due to Out-Of-Memory (OOM) conditions now clearly show the termination reason in status

Fixes

Retryable validation errors no longer fail PipelineRuns
PVC cleanup improvements - already-deleted PVCs no longer cause errors
Fixed managed-by annotation propagation to Pods

Breaking Changes

Deprecated metrics removed - Use pipelinerun_total instead of pipelinerun_count, taskrun_total instead of taskrun_count, etc.
linux/arm images dropped - armv5, armv6, armv7 are no longer supported

v1.3.0 LTS → v1.6.0 LTS (August - October 2025)

The second LTS brought major new features for remote resolution and pipeline composition.

Features

Resolvers caching - Automatic caching for bundle, git, and cluster resolvers. Three modes available: always, never, and auto (default, caches only immutable references). Configurable cache size and TTL via ConfigMap.

apiVersion: v1
kind: ConfigMap
metadata:
  name: resolvers-feature-flags
  namespace: tekton-pipelines-resolvers
data:
  enable-bundles-resolver-caching: "true"
  bundles-resolver-cache-ttl: "1h"
  bundles-resolver-cache-size: "100"

Pipelines-in-Pipelines (TEP-0056) - Reference existing Pipelines as tasks within another Pipeline, enabling powerful composition and reuse patterns. Documentation

apiVersion: tekton.dev/v1
kind: Pipeline
metadata:
  name: integration-pipeline
spec:
  tasks:
  - name: run-unit-tests
    taskRef:
      name: unit-test-pipeline
      kind: Pipeline
  - name: run-e2e-tests
    taskRef:
      name: e2e-test-pipeline
      kind: Pipeline
    runAfter:
    - run-unit-tests

managedBy field - Delegate PipelineRun/TaskRun lifecycle control to external controllers for custom orchestration scenarios. Documentation

apiVersion: tekton.dev/v1
kind: PipelineRun
metadata:
  name: externally-managed
spec:
  managedBy: custom-orchestrator
  pipelineRef:
    name: my-pipeline

Concurrent StepActions resolution - Significantly faster TaskRun startup when using multiple remote StepActions

Task timeout overrides - Override individual task timeouts via spec.taskRunSpecs[].timeout. Documentation

apiVersion: tekton.dev/v1
kind: PipelineRun
metadata:
  name: custom-timeouts
spec:
  pipelineRef:
    name: my-pipeline
  taskRunSpecs:
  - pipelineTaskName: slow-task
    timeout: 2h

Quota-aware PVC handling - PipelineRuns wait for quota availability instead of failing immediately
Array values in When expressions - More flexible conditional execution. Documentation
Step displayName - Human-readable names for steps for better observability
ARM64 tested releases - E2E tests now run on ARM64 architecture

Fixes

Fixed signal handling in SidecarLog for Kubernetes-native sidecar functionality
Pods for timed-out TaskRuns are now retained when keep-pod-on-cancel is enabled
Correct step status ordering when using StepActions

v1.6.0 LTS → v1.9.0 LTS (October 2025 - January 2026)

The latest LTS focuses on stability, observability, and pod configuration.

Features

hostUsers field in PodTemplate - Control user namespace isolation for Task pods. Documentation

apiVersion: tekton.dev/v1
kind: TaskRun
metadata:
  name: secure-task
spec:
  podTemplate:
    hostUsers: false
  taskSpec:
    steps:
    - name: run
      image: alpine
      script: echo "Running with user namespace isolation"

Digest validation for HTTP resolver - Ensure integrity of remotely fetched resources by validating SHA256 digests
ServiceAccount inheritance for Affinity Assistants - Better workspace management with proper credentials
Improved error messages - Actual result size now included when exceeding maxResultSize for easier troubleshooting

Fixes

Major performance fix - Resolved issues causing massive invalid status updates that impacted API server load and stability
Parameter resolution - Fixed defaults with object references
Timeout handling - Prevented excessive reconciliation when timeout is disabled
Pod configuration errors - Early detection instead of waiting for timeout
Race conditions - Fixed TaskRun status issues during timeout handling
Sidecar stopping - Fixed 409 conflict errors by using Patch instead of Update
Matrix validation - Prevented panics from invalid result references (v1beta1)

LTS Support Policy

With v1.9.0 being an LTS release, it will receive security and critical bug fixes for an extended period. Users upgrading from previous LTS versions can expect a smooth transition:

From	To	Key Considerations
v1.0.0	v1.3.0 LTS	Update metric dashboards for renamed metrics
v1.3.0 LTS	v1.6.0 LTS	Smooth upgrade, new features opt-in
v1.6.0 LTS	v1.9.0 LTS	Smooth upgrade, stability improvements

Read more about LTS releases and our support policy.

Looking Ahead

The Tekton Pipelines project continues to focus on:

Performance - Reducing reconciliation overhead and improving startup times
User experience - Better error messages, observability, and debugging tools
Resolver improvements - Working towards v2 resolvers with enhanced caching and usability
Kueue integration (TEP-0164) - Native support for Kueue job queueing to enable better resource management and fair-sharing in multi-tenant environments

We’re also making progress on our transition to the Cloud Native Computing Foundation (CNCF), which will provide Tekton with a neutral home and access to a broader ecosystem.

Get Involved

We invite you to try v1.9.0 LTS, provide feedback, and contribute to the project:

Thank you to all the contributors who made these releases possible!

Blog: Tekton Pipelines Reaches 1.0: Stability Today, Innovation Tomorrow

Fri, 23 May 2025 00:00:00 +0000

We’re thrilled to announce the official 1.0 release of the Tekton Pipelines component (tektoncd/pipeline repository) ! While this marks a significant milestone for the project, it’s important to note that the stability you’ve come to rely on with the v1 API has been a reality for over two years. This release solidifies that foundation while paving the way for even more exciting changes in the Tekton ecosystem.

The version 1.0 brings several key updates, reinforcing the project’s commitment to stability and a streamlined user experience. Notably, the StepAction feature has graduated to General Availability (GA), signifying its maturity and readiness for production use. Furthermore, this release includes significant enhancements and bug fixes for the Git resolver, improving its reliability and performance. We’ve also seen the recent introduction and ongoing development of the HTTP resolver, offering greater flexibility in how your pipelines fetch tasks and other resources. You can learn more about resolvers and their capabilities in the Tekton documentation. Finally, the deprecated ClusterTask resource has been removed, encouraging the adoption of the more flexible and namespaced Task resource or the use of the cluster resolver.

The 1.0 release is not the finish line; it’s a significant step forward on an exciting journey and one we could or should have done a while ago. Looking ahead, we’ll be focused on stability, performance and user experience.

Graduate StepAction to the v1 API, ensuring long-term stability and support for this powerful feature.
Working towards a v2 of the resolvers, with a strong focus on improving both usability and performance.
Investing in better observability and metrics to provide users with clearer insights into their Pipeline executions.

Our goal is to make Tekton Pipelines even more intuitive and efficient for developers and operators alike.

Independently of the release, we are also making a significant step in the evolution of the Tekton project as we transition to the Cloud Native Computing Foundation (CNCF). You can follow the progress of this transition in the CNCF TOC issue. Joining the CNCF will provide Tekton with a neutral home and access to a vibrant and supportive ecosystem. This move will foster even greater collaboration, expand our community reach, and ensure the long-term sustainability and growth of Tekton Pipelines for the benefit of all cloud-native practitioners.

We invite you to explore the 1.0 release, engage with our growing community, and contribute to the future of cloud-native CI/CD. The best is yet to come!

Blog: Protect Signing Secrets for Tekton Chains using Confidential Computing

Wed, 23 Apr 2025 00:00:00 +0000

What is Tekton Chains?

Tekton Chains, a part of the Tekton project, provides tools to generate, store, and sign provenance for artifacts built with Tekton Pipelines.

Technically it’s a Kubernetes Custom Resource Definition (CRD) controller that allows you to manage your supply chain security in Tekton. Chains observes Tekton Pipelines and generates provenance for the artifacts built the Pipelines.

The following diagram from the Tekton Chains tutorial shows the Tekton Chains workflow

Src: https://tekton.dev/docs/chains/getting-started-tutorial/

To summarise, here are the steps in Tekton Chains provenance signing

Create signing secret (eg. cosign key-pair - cosign.pub, cosign.key, cosign.password)
Save the signing secret as K8s secret - named signing-secrets in tekton-chains ns
Create the pipelineRun object (execute your pipeline)
Chains controller watches for pipelineRun completion and creates a snapshot
Sign the snapshot using the private part of the signing secret (eg. cosign.key)
As a consumer, verify the provenance by by checking the signature using the public part of the signing secret (eg. cosign.pub)

What is Confidential Computing?

Confidential computing protects your workload and data from unauthorised entities — the host or hypervisor, system administrators, service providers, other virtual machines (VMs) and processes on the host.

Confidential computing protects the data in use, completing the data security triad — securing data at rest, in-transit, and in-use. Only the data owner has access to the data.

This key functionality gives you the additional confidence to run your sensitive workloads in untrusted infrastructure (e.g. third-party data-centers, public cloud) and reap the benefits of scale and cost-efficiency.

A Trusted Execution Environment (TEE) is at the heart of a confidential computing solution. TEEs are secure and isolated environments provided by confidential computing (CC) enabled hardware that prevents unauthorised access or modification of applications and data while in use. You’ll also hear the terms “enclaves” or “secure enclaves”. “TEEs” and “enclaves” are used interchangeably.

So to reap the benefits of confidential computing, your application needs to run inside a TEE (secure enclave) for it to be protected.

How can you leverage Confidential Computing in the Kubernetes ecosystem?

In the Kubernetes world, there are two broad ways for you to use confidential computing:

Confidential Cluster: The entire Kubernetes cluster is running inside a confidential environment. The cluster nodes are confidential VMs, and any workload deployed on these cluster nodes benefits from confidentiality.

Confidential Container: The Kubernetes pod is running inside a confidential environment. This approach is more granular, where confidentiality is available to the workload of your choice.

Both confidential Kubernetes clusters and containers aim to enhance data security and privacy of Kubernetes workloads by leveraging hardware-based encryption and attestations for trust.

Tekton Chains and Confidential Computing

In order to secure Tekton Chains code and the signing secrets from threats by privileged admins, you can leverage confidential computing.

We have used the confidential container (CoCo) approach to secure Tekton chains.

Confidential Containers (CoCo)

Confidential Containers (CoCo) refers to a pod running inside the TEE.

The CNCF Confidential Containers (CoCo) project is leading the effort to standardise confidential computing at the pod level and simplify its consumption in Kubernetes. This enables Kubernetes users to deploy confidential container workloads using familiar workflows and tools without extensive knowledge of the underlying confidential computing technologies.

Confidential Containers (CoCo) building blocks

Following is a high level architecture of a typical CoCo deployment in Kubernetes.

As shown in the diagram, CoCo uses Linux Confidential Virtual Machine (CVM) executing inside a TEE as the foundation. Using CVM as the foundation enables lift-and-shift of existing container workloads.

The container images are downloaded and kept inside the enclave and can be either signed and/or encrypted.

The components inside the CVM, namely image-rs, kata-agent, confidential data hub (CDH), attestation agent (AA) are collectively referred to as the enclave software stack. The enclave software stack is measured, which means that a trusted cryptographic algorithm is used to measure these entities running within the CVM. These measurements can be used for attestation purposes. Access to secrets—including encryption keys to decrypt the container image itself or to unlock a database used by the workload—can be controlled, released only when the enclave is attested to be valid.

Attestation is the process of verifying that an enclave measurement meets some specified criteria set by the workload or data owner (mentioned as Relying party in the diagram). The attestation agent is responsible for initiating attestation and fetching of the secrets from the key broker service (KBS). For detailed information refer to the following documentation.

The CNCF CoCo project provides the following components

Enclave software stack - https://github.com/confidential-containers/guest-components
Relying Party Services - https://github.com/confidential-containers/trustee
CoCo operator - https://github.com/confidential-containers/operator
Trustee operator - https://github.com/confidential-containers/trustee-operator
CoCo enablement for Kata containers local hypervisor - https://github.com/kata-containers/kata-containers
CoCo enablement for Kata containers remote hypervisor - https://github.com/confidential-containers/cloud-api-adaptor

Tekton Chains and Confidential Containers (CoCo)

Recall from Tekton Chains signing workflow, the signing secret is stored as Kubernetes secret accessible to any cluster admin. If you are using third party services, you would want to protect the signing secret even from the cluster admin. CoCo allows you to do that, by ensuring that the signing secret is only available inside the TEE with no access to the cluster admin.

In the following sections we’ll see how the signing secrets can be protected using CoCo.

Setup

For the purpose of this blog, we’ll be using CoCo in AWS leveraging the Kata containers remote hypervisor setup provided via the CoCo cloud-api-adaptor project.

Instructions to set up CoCo on AWS - https://confidentialcontainers.org/docs/examples/aws-simple.

KBS setup details are described in the following blog.

For other environments refer to the CoCo documentation - https://confidentialcontainers.org/docs/getting-started

Tekton Chains Setup

Instructions to install Tekton Chains Installation - https://github.com/tektoncd/chains?tab=readme-ov-file#installation

Signing Secrets for Tekton Chains

We’ll be using the sealed secrets functionality of CoCo. A sealed secret is a way to encapsulate confidential data such that it can be accessed only inside a Trusted Execution Environment (TEE) after verification of the TEE environment. There are two types of sealed secrets in CoCo - envelope and vault.

We’ll be using the vault type. A vault secret is a pointer to a secret that is stored remotely in a KMS. In the CoCo environment this is managed by the Compute attestation operator as shown previously. When deploying a workload as CoCo, the CoCo runtime components inside the TEE will retrieve the actual secret from the remote provider and make it available to the workload. This happens transparently.

Let’s see an example of a vault type sealed secret. The first step is to create a json file containing the location of the secret in a specified format.

An example json file secret.json is shown below. Note the name and the provider. This defines the location of the actual secret secret.

cat secret.json

{
    "version": "0.1.0",
    "type": "vault",
    "name": "kbs:///default/mysecret/secret",
    "provider": "kbs",
    "provider_settings": {},
    "annotations": {}
}

The sealed secret is of the form - sealed.JWS header.JWS body (secret content).signature

For using sealed secrets in K8s, we use the following: sealed.fakejwsheader.base64url(secret content).fakesignature

Continuing with secret.json example, the sealed secret will be the following:

cat secret.json | basenc --base64url -w0

ewogICAgInZlcnNpb24iOiAiMC4xLjAiLAogICAgInR5cGUiOiAidmF1bHQiLAogICAgIm5hbWUiOiAia2JzOi8vL2RlZmF1bHQvbXlzZWNyZXQvc2VjcmV0IiwKICAgICJwcm92aWRlciI6ICJrYnMiLAogICAgInByb3ZpZGVyX3NldHRpbmdzIjoge30sCiAgICAiYW5ub3RhdGlvbnMiOiB7fQp9Cgo=

sealed_secret=sealed.fakejwsheader.ewogICAgInZlcnNpb24iOiAiMC4xLjAiLAogICAgInR5cGUiOiAidmF1bHQiLAogICAgIm5hbWUiOiAia2JzOi8vL2RlZmF1bHQvbXlzZWNyZXQvc2VjcmV0IiwKICAgICJwcm92aWRlciI6ICJrYnMiLAogICAgInByb3ZpZGVyX3NldHRpbmdzIjoge30sCiAgICAiYW5ub3RhdGlvbnMiOiB7fQp9Cgo=.fakesignature

Now let’s see how we can use this approach to protect the signing secrets for Tekton Chains.

As mentioned previously, a CoCo deployment uses a trusted environment to store the secrets and releases it to the workload. The actual secret will be in the trusted environment whereas the sealed secret will be used in the untrusted environment running Tekton Chains.

Generate cosign key-pairs - cosign.pub, cosign.key, cosign.password

cosign generate-key-pair

Store the keys in the KBS. In my setup the cosign are stored under default/cosign-secrets/{cosign.key,cosign.pub,cosign.password}

Create sealed secrets to be used by the Tekton chains controller

`cosign.key.json`

{
    "version": "0.1.0",
    "type": "vault",
    "name": "kbs:///default/cosign-secrets/cosign.key",
    "provider": "kbs",
    "provider_settings": {},
    "annotations": {}
}

Sealed secret
sealed.fakejwsheader.ewogICAgInZlcnNpb24iOiAiMC4xLjAiLAogICAgInR5cGUiOiAidmF1bHQiLAogICAgIm5hbWUiOiAia2JzOi8vL2RlZmF1bHQvY29zaWduLXNlY3JldHMvY29zaWduLmtleSIsCiAgICAicHJvdmlkZXIiOiAia2JzIiwKICAgICJwcm92aWRlcl9zZXR0aW5ncyI6IHt9LAogICAgImFubm90YXRpb25zIjoge30KfQo=.fakesignature

`cosign.pub.json`

{
    "version": "0.1.0",
    "type": "vault",
    "name": "kbs:///default/cosign-secrets/cosign.pub",
    "provider": "kbs",
    "provider_settings": {},
    "annotations": {}
}

Sealed secret
sealed.fakejwsheader.ewogICAgInZlcnNpb24iOiAiMC4xLjAiLAogICAgInR5cGUiOiAidmF1bHQiLAogICAgIm5hbWUiOiAia2JzOi8vL2RlZmF1bHQvY29zaWduLXNlY3JldHMvY29zaWduLnB1YiIsCiAgICAicHJvdmlkZXIiOiAia2JzIiwKICAgICJwcm92aWRlcl9zZXR0aW5ncyI6IHt9LAogICAgImFubm90YXRpb25zIjoge30KfQo=.fakesignature

`cosign.password.json`

{
    "version": "0.1.0",
    "type": "vault",
    "name": "kbs:///default/cosign-secrets/cosign.password",
    "provider": "kbs",
    "provider_settings": {},
    "annotations": {}
}

Sealed secret
sealed.fakejwsheader.ewogICAgInZlcnNpb24iOiAiMC4xLjAiLAogICAgInR5cGUiOiAidmF1bHQiLAogICAgIm5hbWUiOiAia2JzOi8vL2RlZmF1bHQvY29zaWduLXNlY3JldHMvY29zaWduLnBhc3N3b3JkIiwKICAgICJwcm92aWRlciI6ICJrYnMiLAogICAgInByb3ZpZGVyX3NldHRpbmdzIjoge30sCiAgICAiYW5ub3RhdGlvbnMiOiB7fQp9Cg==.fakesignature

The `signing-secrets` looks like this

apiVersion: v1
data:
  cosign.key: c2VhbGVkLmZha2Vqd3NoZWFkZXIuZXdvZ0lDQWdJblpsY25OcGIyNGlPaUFpTUM0eExqQWlMQW9nSUNBZ0luUjVjR1VpT2lBaWRtRjFiSFFpTEFvZ0lDQWdJbTVoYldVaU9pQWlhMkp6T2k4dkwyUmxabUYxYkhRdlkyOXphV2R1TFhObFkzSmxkSE12WTI5emFXZHVMbXRsZVNJc0NpQWdJQ0FpY0hKdmRtbGtaWElpT2lBaWEySnpJaXdLSUNBZ0lDSndjbTkyYVdSbGNsOXpaWFIwYVc1bmN5STZJSHQ5TEFvZ0lDQWdJbUZ1Ym05MFlYUnBiMjV6SWpvZ2UzMEtmUW89LmZha2VzaWduYXR1cmUK
  cosign.password: c2VhbGVkLmZha2Vqd3NoZWFkZXIuZXdvZ0lDQWdJblpsY25OcGIyNGlPaUFpTUM0eExqQWlMQW9nSUNBZ0luUjVjR1VpT2lBaWRtRjFiSFFpTEFvZ0lDQWdJbTVoYldVaU9pQWlhMkp6T2k4dkwyUmxabUYxYkhRdlkyOXphV2R1TFhObFkzSmxkSE12WTI5emFXZHVMbkJoYzNOM2IzSmtJaXdLSUNBZ0lDSndjbTkyYVdSbGNpSTZJQ0pyWW5NaUxBb2dJQ0FnSW5CeWIzWnBaR1Z5WDNObGRIUnBibWR6SWpvZ2UzMHNDaUFnSUNBaVlXNXViM1JoZEdsdmJuTWlPaUI3ZlFwOUNnPT0uZmFrZXNpZ25hdHVyZQo=
  cosign.pub: c2VhbGVkLmZha2Vqd3NoZWFkZXIuZXdvZ0lDQWdJblpsY25OcGIyNGlPaUFpTUM0eExqQWlMQW9nSUNBZ0luUjVjR1VpT2lBaWRtRjFiSFFpTEFvZ0lDQWdJbTVoYldVaU9pQWlhMkp6T2k4dkwyUmxabUYxYkhRdlkyOXphV2R1TFhObFkzSmxkSE12WTI5emFXZHVMbkIxWWlJc0NpQWdJQ0FpY0hKdmRtbGtaWElpT2lBaWEySnpJaXdLSUNBZ0lDSndjbTkyYVdSbGNsOXpaWFIwYVc1bmN5STZJSHQ5TEFvZ0lDQWdJbUZ1Ym05MFlYUnBiMjV6SWpvZ2UzMEtmUW89LmZha2VzaWduYXR1cmUK
immutable: true
kind: Secret
metadata:
  name: signing-secrets
  namespace: tekton-chains
type: Opaque

⚙️ Custom Tekton Chains Deployment

Update the TektonConfig CR:

📖 Tekton Operator Deployment Options

chain:
  options:
    deployments:
      tekton-chains-controller:
        spec:
          template:
            spec:
              containers:
              - name: tekton-chains-controller
                runtimeClassName: kata-remote

🚀 Run the Pipeline

Now you can run your pipeline and let Tekton chains sign the provenance using secrets that are only available inside the confidential environment.

We’ll use the following sample from the getting started guide - https://github.com/tektoncd/chains/blob/main/docs/tutorials/getting-started-tutorial.md

kubectl create -f pipelinerun.yaml

tkn pr list

tkn pr describe

export PR_UID=$(tkn pr describe --last -o jsonpath='{.metadata.uid}')

tkn pr describe --last \
  -o jsonpath="{.metadata.annotations.chains\.tekton\.dev/signature-pipelinerun-$PR_UID}" \
  | base64 -d > metadata.json

✅ Cosign Verification

cosign verify-blob-attestation --insecure-ignore-tlog \
  --key cosign.pub --signature metadata.json \
  --type slsaprovenance --check-claims=false /dev/null

WARNING: Skipping tlog verification is an insecure practice that lacks of transparency and auditability verification for the blob attestation.
Verified OK

🔍 Confidentiality Verification

The following output is from the controller container.

As you can see the actual secrets are available in the volume mount path. Also the secrets are in memory.

cat /sealed/signing-secrets/cosign.pub

-----BEGIN PUBLIC KEY-----
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEnI15trNzbvWtdAfVyNKRnY7/vrAP
vNnJ/gkCnoBNaaBWL/JgbdfQxF8PXMGXuoDvUFuePHwR7OY5xAoEGaVePw==
-----END PUBLIC KEY-----
/ $ cat /sealed/signing-secrets/cosign.key
-----BEGIN ENCRYPTED SIGSTORE PRIVATE KEY-----
eyJrZGYiOnsibmFtZSI6InNjcnlwdCIsInBhcmFtcyI6eyJOIjo2NTUzNiwiciI6
OCwicCI6MX0sInNhbHQiOiJYMVV6d0h3Uy9iUTJNa1FDV2NsUmFZRkdLTnpTTnJz
cy8yOFRyWEw2eEgwPSJ9LCJjaXBoZXIiOnsibmFtZSI6Im5hY2wvc2VjcmV0Ym94
Iiwibm9uY2UiOiI2bjhtSDRna2ZzaGQ3L3pBbTdOVk93WnNycVZVL3FNdCJ9LCJj
aXBoZXJ0ZXh0IjoiZERuaXZWWDRSZnZ6b2tTS2RVaUNRZG5taksxckd6SmlmMmVM
dWxudmo3dzFnbnpMMFdGdnhpRW16MUthc1RscjlhVDg3bFhoVGtqZmpRYU1pL3hT
NHkrRi9RMm5yc0xvcUNLUnNsREdOL016UnlBZjdXQkp4dU9jMGZtdWFiL1ZuM1lJ
aWVDOTBzdjhzZC9TdzBmWi8rVjJrUkEvRStkQk1QZmdUSnFqTzdwSUFoTWNVR01s
YlEyaWxmS01WempUR1hYbk5iODVwUEk2OHc9PSJ9
-----END ENCRYPTED SIGSTORE PRIVATE KEY-----
/ $

uname -a
Linux tekton-chains-controller-5c6c9684-cl4dn 6.11.5-100.fc39.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Oct 22 19:26:45 UTC 2024 x86_64 Linux


# The unsealed secrets are in memory

/ $ df -h
Filesystem                Size      Used Available Use% Mounted on
overlay                   5.9G    166.5M      5.7G   3% /
tmpfs                    64.0M         0     64.0M   0% /dev
tmpfs                     5.9G    166.5M      5.7G   3% /sealed/signing-secrets
tmpfs                     5.9G    166.5M      5.7G   3% /etc/hosts
tmpfs                     5.9G    166.5M      5.7G   3% /dev/termination-log
tmpfs                     5.9G    166.5M      5.7G   3% /etc/hostname
tmpfs                     5.9G    166.5M      5.7G   3% /etc/resolv.conf
shm                      14.7G         0     14.7G   0% /dev/shm
tmpfs                     5.9G    166.5M      5.7G   3% /var/run/sigstore/cosign
tmpfs                     5.9G    166.5M      5.7G   3% /var/run/secrets/kubernetes.io/serviceaccount
tmpfs                    64.0M         0     64.0M   0% /proc/kcore
tmpfs                    64.0M         0     64.0M   0% /proc/keys
tmpfs                    64.0M         0     64.0M   0% /proc/latency_stats
tmpfs                    64.0M         0     64.0M   0% /proc/timer_list

The following screenshot shows the CVM running the container workload - podvm-tekton-chains-controller-.... using an "r6a" instance type which is an AMD SNP confidential compute instance.

References

https://github.com/tektoncd/chains

https://tekton.dev/docs/getting-started/supply-chain-security/

https://github.com/confidential-containers/guest-components/blob/main/confidential-data-hub/docs/SEALED_SECRET.md

https://opensource.googleblog.com/2023/03/getting-to-slsa-level-2-with-tekton-and-tekton-chains.html

https://github.com/tektoncd/operator/pull/2441

Blog: Migration to Github Container Registry

Thu, 03 Apr 2025 00:00:00 +0000

Why and How

Dear Tekton users and contributors, to reduce costs, we’ve migrated all our releases to the free tier on ghcr.io/tektoncd.

All new Tekton releases are exclusively on ghcr.io/tektoncd. Old releases are also now available on ghcr.io/tektoncd.

Please migrate your old releases to ghcr.io immediately since we have limited funding to allocate to gcr.io egress.

To migrate you need to replace gcr.io/tekton-releases with ghcr.io/tektoncd in all your images including all image references in the manifests, not only those that specify the image of the deployment, e.g. in the Tekton pipelines controller deployment:

containers:
- name: tekton-pipelines-controller
  image: ghcr.io/tektoncd/pipeline/controller-...                    # <<== HERE
  args: [
    "-entrypoint-image", "ghcr.io/tektoncd/pipeline/entrypoint-...", # <<== HERE
    "-nop-image", "ghcr.io/tektoncd/pipeline/nop-...",               # <<== HERE
    "-sidecarlogresults-image", "ghcr.io/tektoncd/pipeline/side...", # <<== HERE
    "-workingdirinit-image", "ghcr.io/tektoncd/pipeline/working...", # <<== HERE
    # ...
]

Find all the gcr entries in other components and update them similar to the Tekton pipelines controller.

In general the change should resemble this:

# old
gcr.io/tekton-releases/github/tektoncd/pipeline/cmd/entrypoint:v0.9.2

# new
ghcr.io/tektoncd/github.com/tektoncd/pipeline/cmd/entrypoint:v0.9.2

You could run:

sed -e 's,gcr.io/tekton-releases,ghcr.io/tektoncd,g'

Images relocated

During the migration some images has been moved to a sub folder new location and must be also renamed during the override

Operator job Pruner /tektoncd/dogfooding/tkn becomes /tektoncd/plumbing/tkn

Other images path remains the same.

End of Life Releases

We started enforcing the download of Tekton images from ghcr.io.

Here are the details of what changed:

Tekton LTS releases originally released to gcr.io will remain available on gcr.io until the EOL date.
The same images are available on ghcr.io, so we ask users to please update their manifests and download images from ghcr.io to save egress bandwidth costs.
Any EOL Tekton release will only be available on ghcr.io.
Images on gcr.io will be removed from public access over the next few days.
New releases will only be made on ghcr.io, so no action is required.

Please feel free to reach out to us via Slack or our mailing list if you have any questions.

The original info mails were sent on our mailing list and if you’re in our google group you can read them here.

Blog: Speeding Up Container Image Builds in Tekton Pipelines

Thu, 02 Nov 2023 00:00:00 +0000

Overview

When DevOps or developer teams evaluate the move from a persistent CI server (or a CI server using persistent workers) to a system using non-persistent workers backed by Kubernetes pods, such as Tekton Pipelines, they might also wonder how to speed up container image building using cached layers. While using cached layers is relatively easy when builds always run on the same server where layers can be stored locally, it is more challenging in an environment where the builds are run in non-persistent containers on Kubernetes. This article will show how to use Kaniko caching capabilities to speed up builds in your Tekton Pipeline.

Tekton Pipelines

Tekton is a powerful and flexible, open source, cloud-native framework for creating CI/CD systems, allowing developers to build, test, and deploy across cloud providers and on-premise systems.

Tekton consists of multiple projects. Among them is Tekton Pipelines, which provides the basic building blocks, in the form of Kubernetes CRDs, for creating CI/CD pipelines, whose tasks run as pods managed and orchestrated by Kubernetes.

The main building blocks are:

Step: smaller execution unit of a pipeline, is a Kubernetes container specification and includes the container image and all the info you need to run it, as the command or environment variables. It defines a step of your pipeline as “mvn package” or “docker build”. It is defined inside the Task Kubernetes custom resource.
Task: Kubernetes custom resource, defines a sequence of Steps running in sequential order on a single pod in the same Kubernetes node.
Pipeline: A group of tasks that you can configure to run in parallel or in sequence, it represents your CI/CD pipeline.
TaskRun: Specific instance of a Task, linking it to parameters related to a specific environment or application. (For example: git repository, target container image to build, or target environment to deploy to.)
PipelineRun: Specific instance of a Pipeline associating it to parameters.

You will find more information and tutorials in the Tekton Documentation

Speeding up container image builds on Tekton

Running build steps as containers in Kubernetes has multiple advantages like portability, reusability, and the possibility to leverage all the capabilities of the underlying Kubernetes platform, but since these containers aren’t persistent by nature, optimizing the build execution through caching would need some additional actions compared to what you were used to do in a standalone CI server.

In the following sections, you will see some examples of how to use caching to speed up image builds in Tekton pipelines using Kaniko. Kaniko is an open source tool that can be used to build container images from a Dockerfile. It doesn’t require a Docker daemon to run, which makes it ideal for building images in environments where Docker daemons aren’t available or can’t be run, such as Kubernetes clusters.

To run through the examples you will need a Kubernetes cluster with Tekton Pipelines installed, a git-based Source Code Management (SCM), and a container registry with permissions to push artifacts. I used a GKE cluster on Google Cloud, a GitHub repository, and Artifact Registry, but steps are reproducible on any standard Kubernetes platform, git based SCM, and container registry with null or minimal modifications.

Example code and assets are available in the: tekton-speed-builds folder of the Tekton CD Website repo

To follow along you need to clone the Tekton CD Website repo locally and then copy and activate the tekton-speed-builds folder as a new repo in your personal GitHub account or other git-based SCM if different.

Experiment the default behavior

After you have your copy of the example repository clone it locally and look at its content:

The sample-app folder contains source code, Maven project files, and a Dockerfile for an example Spring Boot Java app. Look at the Dockerfile: it’s multi-staged using the maven base image for packaging your code and the eclipse-temurin:19-jdk-alpine one to run your application:

FROM maven as build
COPY mvnw .
COPY pom.xml .
COPY src src
RUN mvn package -Dmaven.test.skip=true


FROM eclipse-temurin:19-jdk-alpine
COPY --from=build /target/demo-0.0.1-SNAPSHOT.jar app.jar
CMD ["java", "-jar", "app.jar"]

The tekton-assets folders contain resource manifests for the Tekton assets used in this article:
- pipeline.yaml defines a pipeline running 2 Tasks:
  - git-clone.yaml clones the example repo to the source Tekton workspace
  - image-build.yaml runs Kaniko to build a container image from the source code cloned in the workspace and push it to the target container registry
- source-pvc.yaml is the manifest for the source-pvc persistent volume claim that will be used as persistent storage for the source Tekton workspace. The storage class is intentionally not defined to make it portable so it will use whatever is the default on your cluster, you can modify the manifest to use your preferred storage class.
The kaniko-basecache, kaniko-cache, and m2cache folders contain example resources useful to implement the various caching options.

Let’s create the needed Tekton resources and run our pipeline. From your locally cloned repository, apply the resources in the tekton-assets folder to your cluster:

kubectl apply -f tekton-assets

This will create the pvc, tasks, and pipeline described above.

Now let’s start our pipeline manually. Depending on your configuration you may need to specify, using the -s parameter, a Service Account to authenticate to the registry to push the image, check this section of the Tekton docs for more info.

Type the following command to start the pipeline

tkn pipeline start pipeline-clone-and-build --workspace name=source,claimName=source-pvc --showlog

When prompted for the value of the git-url parameter, type your git repository url.

When prompted for the value of the image-name parameter, type your container image url including the repo and image name.

Here’s an example configuration:

➜  ~ tkn pipeline start pipeline-clone-and-build --workspace name=source,claimName=source-pvc --showlog
? Value for param `git-url` of type `string`? https://github.com/ggalloro/tekton-speed-builds
? Value for param `image-name` of type `string`? europe-west1-docker.pkg.dev/galloro-demos/tektoncd/javasample

The pipeline will start and you will see the output of the steps executed.

If your pipeline run is successful the last step of the image-build task will print your image url in the console and you will have your image pushed to your target registry:

[image-build : write-url] europe-west1-docker.pkg.dev/galloro-demos/tektoncd/javasample:457848d

The pipeline execution created 2 TaskRun resources to execute the 2 tasks, type the following command to see the TaskRun information:

tkn tr ls

You will get an output similar to the one below:

NAME                                             STARTED          DURATION   STATUS
pipeline-clone-and-build-run-68hjg-image-build   4 minutes ago    50s        Succeeded
pipeline-clone-and-build-run-68hjg-git-clone     4 minutes ago    11s        Succeeded

Since your build will always run in a new container, if you change just a line in your source code your build will always need to download the base images and the Maven dependencies and can’t leverage any cache, let’s try that.

Change the text in line 25 of the index.html file in sample-app/src/main/resources/templates to “Hello, Slow Builder!”

Commit your change and push it to your remote repository:

git add .
git commit -m "change to index.html"
git push

Start your pipeline again following the same instructions given above.

After the Pipeline run completes look at your TasRuns info again:

NAME                                             STARTED          DURATION   STATUS
pipeline-clone-and-build-run-fnzxj-image-build   55 seconds ago   49s        Succeeded
pipeline-clone-and-build-run-fnzxj-git-clone     1 minute ago     12s        Succeeded
pipeline-clone-and-build-run-68hjg-image-build   4 minutes ago    50s        Succeeded
pipeline-clone-and-build-run-68hjg-git-clone     4 minutes ago    11s        Succeeded

As you can see, even if you change just one line of code, the build needs approximately the same time to execute. In the next sections, we will explore the options to optimize build speed through caching.

Caching layers in a container registry

Kaniko can cache layers created by RUN and COPY commands in a remote repository. Before running a command, Kaniko checks the cache for the layer. If the layer exists in the cache, Kaniko will pull and extract it instead of running the command. If the layer doesn’t exist in the cache, Kaniko will run the command and then push the newly created layer to the cache.

This can help speed up builds in Tekton.

To use Kaniko cache you must add the --cache=true flag to Kaniko in our image-build Task so the build-and-push step reads as follows:

- name: build-and-push
      workingDir: $(workspaces.source.path)/sample-app/
      image: gcr.io/kaniko-project/executor:latest
      args:
        - --dockerfile=$(params.DOCKERFILE)
        - --context=$(params.CONTEXT) # The user does not need to care the workspace and the source.
        - --destination=$(params.IMAGE):$(params.commit)
        - --digest-file=$(results.IMAGE_DIGEST.path)
        - --cache=true
      securityContext:
        runAsUser: 0

To do that you can apply the modified image-build-cache.yaml in the kaniko-cache folder to update the image-build Task:

kubectl apply -f kaniko-cache/image-build-cache.yaml

To leverage Kaniko layer caching capabilities you should also properly configure your Dockerfile to move the directives that are unlikely to change between builds to the top and the ones that are likely to change to the bottom.

Let’s edit the Dockerfile in the sample-app directory to separate the Maven dependency download from the code build so the dependencies are cached in a separate layer from the software artifact and any change to the code won’t cause the dependencies to be downloaded again. Add the RUN mvn dependency:go-offline line to your Dockerfile so it reads as follows:

FROM maven as build
COPY mvnw .
COPY pom.xml .
RUN mvn dependency:go-offline
COPY src src
RUN mvn package -Dmaven.test.skip=true


FROM eclipse-temurin:19-jdk-alpine
COPY --from=build /target/demo-0.0.1-SNAPSHOT.jar app.jar
CMD ["java", "-jar", "app.jar"]

Once you changed your Dockerfile you need to commit and push your changes so your build will use it:

git add .
git commit -m "update to Dockerfile"
git push

Now run your pipeline again as you did before, the first execution output should be similar to the previous run, and all the dependencies will be downloaded.

Let’s look at the TaskRun info. The build duration should be similar to the previous as well (although it might be a bit longer due to the upload of layers to the registry):

NAME                                             STARTED          DURATION   STATUS
pipeline-clone-and-build-run-gsgwx-image-build   1 minute ago     58s        Succeeded
pipeline-clone-and-build-run-gsgwx-git-clone     1 minute ago     17s        Succeeded
pipeline-clone-and-build-run-fnzxj-image-build   55 seconds ago   49s        Succeeded
pipeline-clone-and-build-run-fnzxj-git-clone     1 minute ago     12s        Succeeded
pipeline-clone-and-build-run-68hjg-image-build   4 minutes ago    50s        Succeeded
pipeline-clone-and-build-run-68hjg-git-clone     4 minutes ago    11s        Succeeded

The image layers have been uploaded to your registry, as you can see in the picture below for Artifact Registry (javasample/cache folder):

Now let’s update the code (use “Hello, Fast Builder!” this time), commit your changes, and run the pipeline again following the instructions from the previous section. From the output log, you should note that the bulk of dependencies will not be downloaded again and the cached layer should be used:

The image-build TaskRun execution should be faster compared to the previous execution (42 vs 58 sec in this example):

NAME                                             STARTED         DURATION   STATUS
pipeline-clone-and-build-run-5s87p-image-build   1 minute ago     42s        Succeeded
pipeline-clone-and-build-run-5s87p-git-clone     1 minute ago     17s        Succeeded
pipeline-clone-and-build-run-gsgwx-image-build   7 minutes ago    58s        Succeeded
pipeline-clone-and-build-run-gsgwx-git-clone     7 minutes ago    17s        Succeeded
pipeline-clone-and-build-run-fnzxj-image-build   18 minutes ago   49s        Succeeded
pipeline-clone-and-build-run-fnzxj-git-clone     18 minutes ago   12s        Succeeded
pipeline-clone-and-build-run-68hjg-image-build   21 minutes ago   50s        Succeeded
pipeline-clone-and-build-run-68hjg-git-clone     22 minutes ago   11s        Succeeded

You have experienced how enabling caching layers to a container registry can improve your build speed on Tekton, the effective speed gain would depend on various factors such as project configuration and connection to your registry.

Caching base images on a persistent disk

In addition to cache layers on your container registry, Kaniko can use a persistent volume to store cached base images. By default, Kaniko will look for cached base images in the /cache folder but you can customize that with the --cache-dir flag.

The persistent volume must be populated before usage, Kaniko provides an image for doing that: gcr.io/kaniko-project/warmer

Let’s add a volume mount mapping the /cache path to a PVC named basechache-pvc to our build-image Task:

- name: build-and-push
      workingDir: $(workspaces.source.path)/sample-app/
      image: gcr.io/kaniko-project/executor:latest
      args:
        - --dockerfile=$(params.DOCKERFILE)
        - --context=$(params.CONTEXT) # The user does not need to care the workspace and the source.
        - --destination=$(params.IMAGE):$(params.commit)
        - --digest-file=$(results.IMAGE_DIGEST.path)
        - --cache=true
      # kaniko assumes it is running as root, which means this example fails on platforms
      # that default to run containers as random uid (like OpenShift). Adding this securityContext
      # makes it explicit that it needs to run as root.
      securityContext:
        runAsUser: 0
      volumeMounts:
        - name: basecache
          mountPath: /cache
    - name: write-url
      image: docker.io/library/bash:5.1.4@sha256:b208215a4655538be652b2769d82e576bc4d0a2bb132144c060efc5be8c3f5d6
      script: |
        set -e
        image="$(params.IMAGE):$(params.commit)"
        echo -n "${image}" | tee "$(results.IMAGE_URL.path)"
  volumes:
    - name: basecache
      persistentVolumeClaim:
        claimName: basecache-pvc

To do that you can apply the modified image-build-basecache.yaml in the kaniko-basecache folder to update the image-build Task:

kubectl apply -f kaniko-basecache/image-build-basecache.yaml

Now let’s create the basecache-pvc PVC:

kubectl apply -f kaniko-basecache/pvc-basecache.yaml

Then, run the kaniko-warmer pod to populate the disk:

kubectl apply -f kaniko-basecache/kaniko-warmer.yaml

Check the kaniko-warmer pod logs:

kubectl logs -f kaniko-warmer

Until you get the output confirming that the 2 base images have been downloaded

INFO[0000] Retrieving image manifest maven              
INFO[0000] Retrieving image maven from registry index.docker.io 
INFO[0004] Retrieving image manifest eclipse-temurin:19-jdk-alpine 
INFO[0004] Retrieving image eclipse-temurin:19-jdk-alpine from registry index.docker.io

When done you can make a change to your code again (“Hello, Base Builder!”) commit it and start your pipeline again as done in the previous tests.

This execution will leverage cached layers as the previous one and, in addition to that, will use the base images from your basecache-pvc persistent volume, as you can see in the output:

Even in this case, the speed improvement will depend on various factors such as how big your base images are, storage performance, and your registry connection.

Caching dependencies on a persistent disk

Another option to speed up builds is to cache dependencies locally. In our example Maven project we can leverage the fact that by default, maven caches the dependency modules into $HOME/.m2 directory. This possibility depends on the language and build tools you are using.

In this example, we will add another persistent volume to the build-image task and map it to the /root/.m2 folder to persist the Maven cache between builds.

Let’s apply the modified image-build-m2cache.yaml in the m2cache folder to update the image-build Task:

kubectl apply -f m2cache/image-build-m2cache.yaml

Now let’s create the m2cache-pvc PVC:

kubectl apply -f m2cache/pvc-m2cache.yaml

Since we don’t need to cache Maven dependencies in an image layer anymore, let’s remove the RUN mvn dependency:go-offline line from our Dockerfile and revert to its original structure. Commit and push your changes after that.

Now run your pipeline again as you did before, Maven will download your dependencies again when the mvn package is run since your previously cached layer is not referenced anymore in your Dockerfile. Duration should be similar to our initial test.

Make a change to your code again (“Hello, Very Fast Builder”), commit it, and start your pipeline again as done in the previous tests.

This execution will leverage cached layers as the previous one and, in addition to that, will use the base images from your basecache-pvc persistent volume and cached dependencies.

Even in this case, if you look at your TaskRun list, you should see a speed improvement compared to the first run:

NAME                                             STARTED          DURATION   STATUS
pipeline-clone-and-build-run-lk8x2-image-build   1 minute ago     46s        Succeeded
pipeline-clone-and-build-run-lk8x2-git-clone     1 minute ago     14s        Succeeded
pipeline-clone-and-build-run-w7sd5-image-build   3 minutes ago    59s        Succeeded
pipeline-clone-and-build-run-w7sd5-git-clone     3 minutes ago    12s        Succeeded

Summary

You saw multiple options to leverage different types of caching to speed up image builds in Tekton using Kaniko:

How to cache layers in your container registry
How to use cached base images in a persistent volume
How to host your local dependencies cache on a persistent volume

Now, you are ready to speed up your build in Tekton !

Note

This blog was originally posted in the Continuous Delivery Foundation Blog

Blog: Distributed Traces for Testing with Tekton Pipelines and Tracetest

Tue, 15 Aug 2023 00:00:00 +0000

Tekton is an open-source framework for creating efficient CI/CD systems. This empowers developers to seamlessly construct, test, and deploy applications across various cloud environments and on-premise setups.

Tracetest, an open-source testing tool that uses OpenTelemetry traces for testing, offers a sophisticated test harness for distributed cloud-native apps. It empowers users to test their apps by harnessing data from distributed traces produced by OpenTelemetry. This enables creating test specs and assertions that validate whether an application aligns with the intended behavior, as defined by pre-established test parameters.

Why use distributed traces for testing?

The rationale behind integrating Tracetest with Tekton is compelling. Tracetest uses pre-existing OpenTelemetry instrumentation to execute assertions against every part of an HTTP transaction.

Combining Tracetest and Tekton enables adding trace-based testing into CI/CD pipelines within your Kubernetes cluster. This integration not only furnishes the capability to initiate planned test cycles and synthetic testing but also preserves the core tenets of trace-based testing. It empowers exhaustive and in-depth assertions with trace data.

Infrastructure Overview

The following is a high level sequence diagram on how Tekton and Tracetest interact with the different pieces of the system.

1. Install Tekton Pipelines, Triggers, and Dashboard

kubectl apply --filename \
https://storage.googleapis.com/tekton-releases/pipeline/latest/release.yaml

kubectl apply --filename \
https://storage.googleapis.com/tekton-releases/triggers/latest/release.yaml

kubectl apply --filename \
https://storage.googleapis.com/tekton-releases/triggers/latest/interceptors.yaml

kubectl apply --filename \
https://storage.googleapis.com/tekton-releases/dashboard/latest/release.yaml

2. Install Tracetest CLI

Install Tracetest CLI by following these instructions for your OS.

brew install kubeshop/tracetest/tracetest

curl -L https://raw.githubusercontent.com/kubeshop/tracetest/main/install-cli.sh | bash

choco source add --name=kubeshop_repo --source=https://chocolatey.kubeshop.io/chocolatey ; choco install tracetest

3. Install Tracetest in Your Kubernetes Cluster

tracetest server install

[Output]
How do you want to run TraceTest? [type to search]:
  Using Docker Compose
> Using Kubernetes

Select Using Kubernetes.

[Output]
Do you have OpenTelemetry based tracing already set up, or would you like us to install a demo tracing environment and app? [type to search]:
  I have a tracing environment already. Just install Tracetest
> Just learning tracing! Install Tracetest, OpenTelemetry Collector and the sample app.

Select Just learning tracing! Install Tracetest, OpenTelemetry Collector and the sample app..

Confirm that Tracetest is running:

kubectl get all -n tracetest

[Output]
NAME                                  READY   STATUS    RESTARTS        AGE
pod/otel-collector-7f4d87489f-vp6zn   1/1     Running   0               5m41s
pod/tracetest-78b9c84c57-t4prx        1/1     Running   3 (4m15s ago)   5m29s
pod/tracetest-postgresql-0            1/1     Running   0               5m42s

NAME                              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)               AGE
service/otel-collector            ClusterIP   10.96.173.226   <none>        4317/TCP              5m46s
service/tracetest                 ClusterIP   10.96.248.146   <none>        11633/TCP,4317/TCP   5m42s
service/tracetest-postgresql      ClusterIP   10.96.155.147   <none>        5432/TCP              5m42s
service/tracetest-postgresql-hl   ClusterIP   None            <none>        5432/TCP              5m42s

NAME                             READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/otel-collector   1/1     1            1           5m46s
deployment.apps/tracetest        1/1     1            1           5m42s

NAME                                        DESIRED   CURRENT   READY   AGE
replicaset.apps/otel-collector-7f4d87489f   1         1         1       5m46s
replicaset.apps/tracetest-78b9c84c57        1         1         1       5m42s

NAME                                    READY   AGE
statefulset.apps/tracetest-postgresql   1/1     5m42s

By default, Tracetest is installed in the tracetest namespace.

To explore the Tracetest Web UI, run the command:

kubectl --kubeconfig <path-to-your-home>/.kube/config --context <your-cluster-context> --namespace tracetest port-forward svc/tracetest 11633

4. Create a Test in Tracetest

Start by clicking Create > Create New Test > HTTP Request > Next > Choose Example (dropdown) > Pokeshop - List (generates a sample test from the Tracetest demo) > Next > URL is prefilled with http://demo-pokemon-api.demo/pokemon?take=20&skip=0 > Create & Run.

This will trigger the test and display a distributed trace in the Trace tab to run assertions against.

Proceed to add a test spec to assert all database queries return within 500 ms. Click the Test tab and proceed to click the Add Test Spec button.

In the span selector make sure to add this selector:

span[tracetest.span.type="database"]

In the assertion field add:

attr:tracetest.span.duration < 500ms

Save the test spec and publish the test.

The database spans that are returning in less than 500ms are labeled in green.

This is an example of a trace-based test that asserts against every single part of an HTTP transaction, including all interactions with the database.

Let’s introduce how Tekton makes it possible to run this test as part of your CI/CD pipeline.

5. Create a Task in Tekton

Click the Automate tab.

This contains both a YAML definition for the test run and a guide how to run the test with the Tracetest CLI.

Save this into a file called test-api.yaml:

# test-api.yaml

type: Test
spec:
  id: L7wr5xeVR
  name: Pokeshop - List
  description: Get a Pokemon
  trigger:
    type: http
    httpRequest:
      method: GET
      url: http://demo-pokemon-api.demo/pokemon?take=20&skip=0
      headers:
      - key: Content-Type
        value: application/json
  specs:
  - selector: span[tracetest.span.type="database"]
    name: Database queries less than 500ms
    assertions:
    - attr:tracetest.span.duration < 500ms

Note that you’ll use this CLI command to run the test:

tracetest run test --file test-api.yaml --required-gates test-specs --output pretty

It set’s the required gates to pass the test by only validating the test specs.

Create another YAML file, name it install-and-run-tracetest.yaml. This contains the Tekton Task definition.

# install-and-run-tracetest.yaml

apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
  name: install-and-run-tracetest
spec:
  steps:
    - name: create-test-files
      image: ubuntu
      script: |
        #!/usr/bin/env bash
        cat <<EOF >/workspace/test-api.yaml
        type: Test
        spec:
          id: L7wr5xeVR
          name: Pokeshop - List
          description: Get a Pokemon
          trigger:
            type: http
            httpRequest:
              method: GET
              url: http://demo-pokemon-api.demo/pokemon?take=20&skip=0
              headers:
              - key: Content-Type
                value: application/json
          specs:
          - selector: span[tracetest.span.type="database"]
            name: Database queries less than 500ms
            assertions:
            - attr:tracetest.span.duration < 500ms
        EOF        
      volumeMounts:
      - name: custom
        mountPath: /workspace
    - name: install-and-run-tracetest
      image: kubeshop/tracetest:v0.13.3
      # The official Tracetest image comes with the Tracetest CLI installed
      script: |
        # Configure and Run Tracetest CLI
        tracetest configure -g --endpoint http://tracetest.tracetest.svc.cluster.local:11633/
        tracetest run test --file /workspace/test-api.yaml --required-gates test-specs --output pretty        
      volumeMounts:
      - name: custom
        mountPath: /workspace
  volumes:
  - name: custom
    emptyDir: {}

kubectl apply -f ./install-and-run-tracetest.yaml

Make sure to use the Tracetest service as the endpoint for your tracetest configure command. This may vary depending on your installation.

http://tracetest.tracetest.svc.cluster.local:11633/

6. Run the Tracetest Trace-based Test in Tekton with a TaskRun

Finally, to run the test, create a TaskRun.

Create a file called install-and-run-tracetest-run.yaml.

# install-and-run-tracetest-run.yaml

apiVersion: tekton.dev/v1beta1
kind: TaskRun
metadata:
  name: install-and-run-tracetest-run
spec:
  taskRef:
    name: install-and-run-tracetest

kubectl apply -f ./install-and-run-tracetest-run.yaml

Here’s how to check the logs:

kubectl logs --selector=tekton.dev/taskRun=install-and-run-tracetest-run

You can also trigger a Task with the Tekton CLI.

tkn task start install-and-run-tracetest

[Output]
TaskRun started: install-and-run-tracetest-run-xmhfg

In order to track the TaskRun progress run:

tkn taskrun logs install-and-run-tracetest-run-gccjk -f -n default

[Output]
[install-and-run-tracetest] ✔ Pokeshop - List (http://tracetest.tracetest.svc.cluster.local:11633/test/RUkKQ_aVR/run/3/test) - trace id: 0549641531d3221ded696f2fd3b20ce6
[install-and-run-tracetest] 	✔ Database queries less than 500 ms
[install-and-run-tracetest]

To preview which tasks failed or succeeded, use this command:

tkn taskrun list

[Output]
NAME                                  STARTED          DURATION   STATUS
install-and-run-tracetest-run         3 minutes ago    23s        Succeeded
install-and-run-tracetest-run-nmptn   7 minutes ago    33s        Failed
install-and-run-tracetest-run-bhf7v   20 minutes ago   23s        Succeeded
install-and-run-tracetest-run-wm8bj   21 minutes ago   22s        Succeeded
install-and-run-tracetest-run-dbrbt   23 minutes ago   24s        Failed

7. Trigger Trace-based Tests with an EventListener

By using Tektons’s triggers, you can trigger tests via an eventlistener.

Create a TriggerTemplate and TriggerBinding

# install-and-run-tracetest-trigger-binding.yaml

apiVersion: triggers.tekton.dev/v1beta1
kind: TriggerTemplate
metadata:
  name: install-and-run-tracetest-template
spec:
  resourcetemplates:
  - apiVersion: tekton.dev/v1beta1
    kind: TaskRun
    metadata:
      generateName: install-and-run-tracetest-run-
    spec:
      taskRef:
        name: install-and-run-tracetest

# install-and-run-tracetest-trigger-template.yaml

apiVersion: triggers.tekton.dev/v1beta1
kind: TriggerBinding
metadata:
  name: install-and-run-tracetest-binding
spec:
  params:
  - name: run
    value: $(body.run)

kubectl apply -f install-and-run-tracetest-trigger-binding.yaml
kubectl apply -f install-and-run-tracetest-trigger-template.yaml

Create an EventListener

# install-and-run-tracetest-event-listener.yaml

apiVersion: triggers.tekton.dev/v1beta1
kind: EventListener
metadata:
  name: install-and-run-tracetest-event-listener
spec:
  serviceAccountName: tekton-robot
  triggers:
    - name: install-and-run-tracetest-trigger 
      bindings:
      - ref: install-and-run-tracetest-binding
      template:
        ref: install-and-run-tracetest-template

The EventListener requires a service account to run. To create the service account for this example create a file named tekton-robot-rbac.yaml and add the following:

# tekton-robot-rbac.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: tekton-robot
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: triggers-example-eventlistener-binding
subjects:
- kind: ServiceAccount
  name: tekton-robot
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: tekton-triggers-eventlistener-roles
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: triggers-example-eventlistener-clusterbinding
subjects:
- kind: ServiceAccount
  name: tekton-robot
  namespace: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: tekton-triggers-eventlistener-clusterroles

kubectl apply -f tekton-robot-rbac.yaml
kubectl apply -f install-and-run-tracetest-event-listener.yaml

Enable port forwarding.

kubectl port-forward service/el-install-and-run-tracetest-event-listener 8080

Hitting the port forwarded endpoint will trigger the task.

curl -v \
   -H 'content-Type: application/json' \
   -d '{"run":true}' \
   http://localhost:8080

Checking the taskruns will confirm this.

tkn taskrun list

[Output]
NAME                                  STARTED          DURATION   STATUS
install-and-run-tracetest-run-69zrz   4 seconds ago    ---        Running(Pending)

Finally, check the logs:

tkn taskrun logs -f install-and-run-tracetest-run-69zrz

[Output]
[install-and-run-tracetest] ✔ Pokeshop - List (http://tracetest.tracetest.svc.cluster.local:11633/test/RUkKQ_aVR/run/5/test)
[install-and-run-tracetest] 	✔ Database queries less than 500 ms

8. Preview Trace-based Tests with Tekton Dashboard

Start by port forwarding the Tekton Dashboard.

kubectl --namespace tekton-pipelines port-forward svc/tekton-dashboard 9097:9097

Open it up in your browser at http://localhost:9097. Navigate to the TaskRuns. Open the TaskRun you ran above.

This lets you easily preview the Tracetest test runs by copying the link to the Tracetest instance. However, because you’re using an internal service for Tracetest, make sure to use the port forwarded link to access the Tracetest Dashboard.

Replace:

http://tracetest.tracetest.svc.cluster.local:11633/test/RUkKQ_aVR/run/4/test

With:

http://localhost:11633/test/RUkKQ_aVR/run/4/test

Alternatively, you can use the Deep Link feature to trigger a new test run directly via the browser like this:

http://localhost:11633/test/RUkKQ_aVR/run?

Conclusion

To sum up, the partnership between Tekton and Tracetest offers a powerful approach to testing Kubernetes with distributed tracing. Tekton provides a framework for building efficient CI/CD pipelines. Tracetest utilizes OpenTelemetry traces for testing, making it a valuable tool for testing cloud-native apps.

By combining Tracetest and Tekton, developers gain the ability to integrate trace-based testing into Kubernetes clusters. This integration allows for scheduled test cycles, synthetic tests, and thorough assertions using trace data. This cohesive approach ensures that applications align with intended behavior, improving overall reliability and functionality. As cloud-native apps continue to evolve, this integration showcases how to enhance testing as well.

Next Steps

To explore more options that Tracetest gives you, check out the docs to learn more!

If you like what the Tekton and Tracetest communities are doing, please leave a ⭐️ in GitHub.

Blog: Getting To SLSA Level 2 with Tekton and Tekton Chains

Wed, 19 Apr 2023 00:00:00 +0000

Overview

As application developers, we achieve amazing results quickly by leveraging a rich ecosystem of freely available libraries, modules and frameworks that provide ready-to-use capabilities and abstract away from underlying complexity. This is so foundational to how we work that we’ll nonchalantly build and publish an app that pulls in hundreds of dependencies without even thinking about it. And it’s only fairly recently, in the wake of some very high profile and high impact compromises, that we’ve started to reckon with the fact that this wonderful ecosystem is also a security quagmire. All of the dependencies that feed into your build make up your software supply chain, and supply chains need to be secured. In this post, we’ll show how an increasingly popular open source CI/CD system, Tekton, implements the OpenSSF SLSA framework to provide you with supply chain security guarantees.

Software Supply Chain Security

A software supply chain is anything that goes into or affects your code from development, through your CI/CD pipeline, until it gets deployed into production. Increasingly, the software supply chain has become a vector for attacks. The recent Log4j, SolarWinds, Kaseya, and Codecov hacks highlight vulnerable surface areas exposed by an insecure software supply chain.

Between 2020 and 2021, there has been a 650% Surge in OSS supply chain attacks, and Gartner projects that 45% of organizations worldwide will have experienced software supply chain attacks by 2025.

Supply Chains Levels for Software Artifacts (SLSA)

The Supply chain Levels for Software Artifacts (SLSA) framework is a check-list of controls to prevent tampering, improve integrity, and increase security in the packages and infrastructure used by projects, businesses or enterprises. SLSA formalizes criteria around software supply chain integrity to help the industry and open source ecosystem secure the software development life cycle at all stages.

As part of the framework, SLSA has multiple levels of assurances. These levels contain industry-recognized best practices to create four levels of increasing assurance.

SLSA provides a set of requirements that needs to be met for an artifact to be considered for a particular SLSA level.

Tekton + Tekton Chains

Tekton is a powerful and flexible, open source, cloud-native framework for creating CI/CD systems, allowing developers to build, test, and deploy across cloud providers and on-premise systems. Tekton consists of several subprojects which are relevant to SLSA:

Pipelines: A system that allows one to define a pipeline of CI/CD tasks and have it be orchestrated by the Tekton controller.
Chains: A standalone system which observes Pipelines and generates provenance for the artifacts built by Pipelines. Tekton build processes are defined as tasks and pipelines. A Task is a collection of Steps that are defined and arranged in a specific order of execution as part of a continuous integration flow.

A Pipeline is a collection of Tasks defined and arranged in a specific order of execution as part of a continuous integration flow.

A TaskRun is an instantiation of a Task with specific inputs, outputs and execution parameters while a PipelineRun is an instantiation of a Pipeline.

A Task/Pipeline can define a set of Results. TaskRuns and PipelineRuns create Results as defined in the Task/Pipeline. Results are used to communicate to Tekton Chains run specifics like the uri and the digest of the built artifact.

Tasks, Pipelines, TaskRuns and PipelineRuns are defined through yaml files. The entire build is defined by the set of yaml files which define Tekton Tasks, Pipelines, TaskRuns and PipelineRuns. These yaml files can be checked in as code and run directly from the code repository.

Getting to SLSA L1: Automation + Provenance

For an artifact to be SLSA L1 compliant it should satisfy the following:

Scripted build: All build steps are fully defined in some sort of “build script”. The only manual command, if any, is to invoke the build script.
Provenance: The provenance is available to the consumer in a format that the consumer accepts. The format SHOULD be in-toto SLSA Provenance, but another format MAY be used if both producer and consumer agree and it meets all the other requirements.

Tekton Tasks, TaskRuns, Pipelines and PipelineRuns are specified in yaml files. These yaml files can be considered as scripts and can even be checked in into a code repository. These could also be run from code repositories. Tekton Chains provides a way to generate provenance in in-toto SLSA format. As such, Tekton can easily make builds which satisfy the SLSA L1 requirements.

Note

The linked example demonstrates SLSA level 2 on Google Cloud, but the same principles are applicable to any build environment.

Let’s follow through with an example, which has the following files:

setup.sh: Sets up Google cloud to run an instance of the build specified in pipeline_run.yaml. It also installs Tekton Pipeline and Tekton Chains. In the production environment, this would be run once to set up the environment and all builds would use the same environment.
pipeline_run.yaml: This file is the actual build file that is run by Tekton Pipelines. The build here first clones a Github repo, builds the container specified in the source and uploads it to a Docker repository.

The build script pipeline.yaml is the definition of the script while pipeline_run.yaml defines an instance of the build. It provides instance specific parameters for the build. Though both pipeline_run.yaml and pipeline.yaml are in source control for this example, the build definition is in pipeline.yaml and as such pipeline.yaml being in source control would satisfy the requirement of a source controlled build script.

kubectl create -f
    https://raw.githubusercontent.com/tektoncd/chains/main/docs/vendor/gcp/slsa-2/pipeline_run.yaml

Tekton Chains for Provenance Generation

Provenance is metadata about how an artifact was built, including the build process, top-level source, and dependencies. Knowing the provenance allows software consumers to make risk-based security decisions.

Tekton Chains observes TaskRuns and PipelineRuns in a Kubernetes cluster. Once the runs are done, Chains collects information (provenance) about the Run or the build process and the artifact created by the Run. It signs the provenance and stores the signed provenance. The provenance generated for the example build complies to the SLSA provenance schema and is explained further below.

Note that every step of the build has been recorded and can be reconstructed by following the steps in the provenance.

Next Steps: CI/CD @ SLSA L2

SLSA requires that for a build to be SLSA L2 compliant it should satisfy the following

Every change to the source is tracked in a version control system
All build steps were fully defined in some sort of “build script”. The only manual command, if any, was to invoke the build script.
All build steps ran using some build service, not on a developer’s workstation.
The provenance is available to the consumer in a format that the consumer accepts. The format SHOULD be in-toto SLSA Provenance, but another format MAY be used if both producer and consumer agree and it meets all the other requirements.
The provenance’s authenticity and integrity can be verified by the consumer. This SHOULD be through a digital signature from a private key accessible only to the service generating the provenance.
The data in the provenance MUST be obtained from the build service (either because the generator is the build service or because the provenance generator reads the data directly from the build service). Regular users of the service MUST NOT be able to inject or alter the contents.

Every change to the source is tracked in a version control system

Tekton does not explicitly enforce that the source is version controlled. Tekton users can enforce that the source is version controlled by writing an appropriate Task which will check for version control. The source should also be communicated by Tekton Pipelines to Tekton Chains through a result variable that is suffixed with -ARTIFACT_INPUTS.

All build steps were fully defined in some sort of “build script”. The only manual command, if any, was to invoke the build script.

This is a requirement for SLSA L1 as well and as explained above, Tekton provides a way to script the build through yaml files. The build is defined as a Pipeline (or Task) which can be saved as a yaml file and submitted into source control. The build instance which is defined as a PipelineRun (or TaskRun) can resolve the Pipeline (or Task) yaml from source control and use it for the current instance of the build.

All build steps ran using some build service, not on a developer’s workstation.

Tekton can be hosted on a cloud provider or on a hosted Kubernetes cluster and run as a build service. The build scripts can be submitted into source control (like GitHub) and Tekton can read the scripts directly from source control.

Provenance should be available

This is a requirement for SLSA L1 and as explained above Tekton Chains provides build provenance.

Provenance should be signed and Authenticated

As can be seen in the example, Tekton Chains creates and signs the build provenance. The signature can be verified anytime to ensure that the provenance has not been tampered after the build and the provenance is really created by the build process that claims to have built it. The signing is done according to the SLSA specification using the DSSE format.

Tekton Chains creates the provenance and signs it using a secure private key. Chains then uploads the signed provenance to a user-specified location, one of which is Google Cloud’s Container Analysis, which implements the open standard Grafeas API for storing provenance.

Provenance should be generated by a Service

Note that the provenance in the example is generated by the Tekton Chains service and it cannot be modified after it has been generated, which is guaranteed by the signature.

SLSA requirements for the contents of the provenance, for the build to be considered L2.

All images below are extracted from the provenance of the example build. These can be verified by re-running the example.

Identifies artifact: The provenance MUST identify the output artifact via at least one cryptographic hash. The subject field in the SLSA provenance captures the location of the built artifact and the cryptographic hash associated with it. To be able to capture the artifact, Tekton Pipelines should populate the result variable -ARTIFACT_OUTPUTS with the location and the digest of the artifact.

Identifies builder: The provenance identifies the entity that performed the build and generated the provenance. The builder.id field captures the builder that built the artifact.

Identifies build instructions: The provenance identifies the top-level instructions used to execute the build. In our example, the build script is in source control. Recording the repo, the path in the repo and the commit hash will uniquely identify the build instructions used to build the artifact.

Identifies source code: The provenance identifies the repository origin(s) for the source code used in the build. The materials field records all the dependencies used to build the artifact, one of which is the source code. In the example the source used is in a GitHub repo, and as such the repo name and the commit hash will uniquely identify the source code.

Conclusion

SLSA aims to secure the software supply chain by providing guidelines on how the software build should be done. Tekton pipelines and Tekton chains implement those guidelines and help in securing the software supply chain.

Note

This blog post was first published on the Google Open Source Blog.

Blog: Meet with Tekton Community at cdCon + GitOpsCon 2023

Fri, 14 Apr 2023 00:00:00 +0000

The Continuous Delivery Foundation (CDF) is happy to host its fourth flagship event, cdCon, taking place on May 8–9, 2023 in Vancouver, Canada as cdCon + GitOpsCon, co-organized with the Cloud Native Computing Foundation (CNCF), making it the must-attend event for anyone who is involved in CD, DevOps, and GitOps.

By combining the two events, cdCon + GitOpsCon aims to bring their communities together to collaborate and build the future of GitOps and CD. The program committees reviewed more than 250 submissions and selected over 60 sessions that will span topics from technical challenges and deep dives, to end-user stories and introductory content.

The cdCon + GitOpsCon program contains sessions from the most widely used CI/CD and GitOps technologies. The Tekton Community will be there with project updates and various talks from our community members and users. Here are some of the sessions you don’t want to miss.

Operating Tekton with Secure Defaults - Christie Wilson Warwick & Wendy Dembowski, Google
Tekton Project Update and Roadmap - Andrea Frittoli, IBM
Save Your Pipelines (and Clusters!) with Tekton Results - Adam Kaplan, Red Hat & Dibyo Mukherjee, Google
Tekton Supply Chain Security - Billy Lynch, Chainguard

In addition to these sessions, Andrea Frittoli of IBM, a member of the Tekton Governance Committee and maintainer will take part in the Graduated Projects Keynote Panel, discussing Tekton Community’s experiences with graduation and sharing his thoughts on why graduation matters for the community and users of Tekton.

We look forward to an awesome event and we hope to see you there!

Blog: Tekton Graduation

Wed, 26 Oct 2022 00:00:00 +0000

We’re very happy to announce that Tekton has reached graduated status within the Continuous Delivery Foundation (CDF). The CDF Technical Oversight Committee (TOC) conducted public voting to decide on the graduation status for Tekton, and the result was unanimously positive. The Tekton community is very proud of the results of the vote and will continue working to make Tekton better and safer for its users.

In this blog post I will explore a bit of the history of Tekton, its “road to graduation” and what this milestone means for the project.

Early Days

The Tekton project has its roots in Knative, where it was initially called “Knative Build” and later “Knative Pipeline”. The project was spun off in August 2018, when it got it’s current name and a new home on GitHub as “tektoncd/pipeline”. Here’s one of the project’s very first commits:

commit 49d2316d71e8c315e0a8fd76008bc2920f56b3c3
Author: Christie Wilson <[email protected]>
Date:   Fri Aug 31 17:15:35 2018 -0700

Add pipeline strawman example

@dlorenc @ImJasonH @tejal29 @aaron-prindle and I have been working on a
strawman proposal for adding a Pipeline CRD and also for possibly
envolving the Build CRD into a slightly more generic Task CRD.

This PR demonstrates some paper prototype examples of what it could look
like to define pipelines using the CRDs described in the README.

A few months later, in March 2019, Tekton was donated to the newly formed Continuos Delivery Foundation (CDF).

Growing the community

Since then the project has thrived thanks to a rich community of contributors. One of earlier repositories to be created was the community one, where the project documented its governance, code of conduct and contributing guidelines.

At the end of 2019 Tekton the number of repositories had grown to eleven. A regular monthly release cadence was established - Pipeline had 9 releases and in March 2020 the beta version of the Tekton Pipeline API was released. During this time, we also implemented the project’s release automation using Tekton itself.

The “results” and “chains” project were started in spring 2020, and in August the Tekton Hub was officially launched.

The Tekton community kept growing thanks to Tekton adopters and end-users contributing features, ideas and issues.

Contributions and contributors to Tekton Projects over time. Source devstats.

Focus on security

One year later, it’s now 2021, Tekton had matured considerably, while remaining true to its nature of having a small footprint and giving users full flexibility in how they setup their CI/CD system through Tekton.

This very flexibility has enabled Tekton to become the base for the implementation of more opinionated services on top, ranging from open source projects, and cloud services as well as end-user platforms for DevOps services.

It was then prime time to focus more thoroughly on security! In July 2021 the Tekton Vulnerability Team was formed. Tekton pipelines release v0.29 was the first one to be signed through Tekton chains, with the provenance Rekor UUID included in the release notes.

End user adoption shows how Tekton, with the help of Tekton Chains, can help solve software supply chain security challenges.

Very excited to see SolarWinds Trebuchet really exploring the full potential of Tekton!! 😻 #SupplyChainSecurityCon #KubeCon https://t.co/IFJKvED1q7 pic.twitter.com/bS1UuBGuxh
— tektoncd (@tektoncd) October 11, 2021

In March 2022, thanks to the the sponsorship of the CDF, Tekton completed an independent security audit.

Later that year, Tekton achieved the OpenSSF Best Practices badges for its six core components.

Long Term Support

In October 2022 the Tekton community defined its policy for long term support (LTS):

The Tekton project maintains four release branches for each project, created one every three months, which results in a overall support window of approximately one year for each of these releases.

These releases are called LTS, and throughout the support period, patch releases may be created to resolve:

CVEs (under the advisement of the Tekton Vulnerability Team)
dependency issues (including base image updates)
critical core component issues

The community-wide policy can be extended by each project. Pipeline continuous with its monthly cadence, and it selects four releases a year for LTS.

Tekton Pipeline Releases and their support window.

Graduation

The Graduated Stage for projects under the CD Foundation umbrella is when they have reached their growth goals and are now on a sustaining cycle of development, maintenance, and long-term support. Graduated Stage projects are used commonly in enterprise production environments and have large, well-established project communities.

More specifically, the Technical Oversight Committee (TOC) defines graduation as a set of requirements about project maturity, best practices, security stance and adoption.

Tekton is the second project to graduate from the CD Foundation, after Jenkins. All the details about the twelve graduation criteria be found in the graduation proposal.

What’s Next

Graduation is a great milestone for the project and a great responsibility too. The community will continue to work on improving Tekton, while keeping the project stable and secure.

We are working on several security features like trusted resources and trusted workloads, as well as releasing the v1 version of the API.

Acknowledgments

The graduation has been the result of the years long work of the Tekton community. Congratulations and thank you to all the Tekton contributors who made it possible!

Thank you for the the CDF for hosting the project and supporting it, especially through the security audit. Thank you to the TOC and its chair Oleg for their support and sponsorship as well as to the CDF team, Fatih, Jesse and Roxanne for their great work with marketing and organizing the press release.

Tekton – Tekton

Blog: Tekton Pipelines v1.10.0: Observability, Evolved

OpenCensus to OpenTelemetry Migration

What Changed

What You Need to Do

Other Highlights

New Features

Bug Fixes

Get Started

Blog: Introducing Tekton Pruner

The Problem

Before: Job-Based Pruner

Tekton Pruner

Installation

Via Tekton Operator

Standalone Installation

Verify

Configuration

Cluster-Wide Defaults

Per-Namespace Config

Label-Based Selectors

What to Watch Out For

Checking Logs

Config Reference

Links

Blog: Securing HTTP Resolver with Digest Verification

Overview

Why Digest Verification Matters

How It Works

Enabling the HTTP Resolver

Usage Examples

Basic Pipeline Resolution with Digest Verification

Task Resolution with SHA512 Digest

Using with Authentication

Calculating Digests

SHA256 Digest

SHA512 Digest

Tip

Error Handling

Best Practices

1. Pin Your Remote Resources

2. Automate Digest Updates

3. Use SHA256 for Standard Use Cases

4. Document Your Digests

Integration with Tekton Chains

Backward Compatibility

Conclusion

Blog: Tekton Pipelines v1.9.0 LTS: Continued Innovation and Stability

Installation

v1.0.0 → v1.3.0 LTS (May - August 2025)

Features

Fixes

Breaking Changes

v1.3.0 LTS → v1.6.0 LTS (August - October 2025)

Features

Fixes

v1.6.0 LTS → v1.9.0 LTS (October 2025 - January 2026)

Features

Fixes

LTS Support Policy

Looking Ahead

Get Involved

Blog: Tekton Pipelines Reaches 1.0: Stability Today, Innovation Tomorrow

Blog: Protect Signing Secrets for Tekton Chains using Confidential Computing

What is Tekton Chains?

What is Confidential Computing?

How can you leverage Confidential Computing in the Kubernetes ecosystem?

Tekton Chains and Confidential Computing

Confidential Containers (CoCo)

Confidential Containers (CoCo) building blocks

Tekton Chains and Confidential Containers (CoCo)

Setup

Tekton Chains Setup

Signing Secrets for Tekton Chains

cosign.key.json

cosign.pub.json

cosign.password.json

The signing-secrets looks like this

⚙️ Custom Tekton Chains Deployment

🚀 Run the Pipeline

`cosign.key.json`

`cosign.pub.json`

`cosign.password.json`

The `signing-secrets` looks like this