PipeCD – PipeCD

Blog: What is new in pipedv1 (plugin-arch piped)

Tue, 02 Sep 2025 00:00:00 +0000

Since the alpha release from Jun 2025, plugin-arch piped (aka. pipedv1) is closer to first offical release, we are working on v1.0.0-rc0 and will be released in the coming days.

In this article I would like to share some improvements that have been fixed in the new version pipedv1 compared to its predecessor piped.

Overview

While developing pipedv1, we always considered ensuring backward compatibility, meaning that the features available in piped will be mostly preserved in pipedv1. This ensures seamlessness between using piped and pipedv1, reducing the risk of problems when switching to pipedv1.

Changes in pipedv1 are of 2 types:

Performance improvements or addressing some features requested in pipedv0 but could not be met due to technical reasons.
Removal of some features that are obsolete or no longer prioritized

Fundamental changes like pipedv1 being able to support different platforms and making build pipeline deployment more powerful by turning stage executors into plugins have been described in this blog. This post will focus on smaller changes related to the actual usage of pipecd.

New in pipedv1

`plugins` configurations

From pipedv1, plugin binaries will be the real executor that perform pipeline’s stages execution. Plugin configuration is placed in 2 locations: piped config and application config.

The piped config is where basic plugin configurations are stored, such as where to load the plugin binary or some configurations that can be shared between applications.

apiVersion: pipecd.dev/v1beta1
kind: Piped
spec:
  projectID: dev
  pipedID: xxx
  ...
  plugins:
    - name: kubernetes
      port: 7001
      url: https://github.com/pipe-cd/pipecd/releases/download/link-to-plugin
      deployTargets:
        - name: local
          config:
            kubectlVersion: 1.32.4
        - name: remote
          config:
            kubectlVersion: 1.33.0
    - name: wait
      port: 7002
      url: https://github.com/pipe-cd/pipecd/releases/download/link-to-plugin

NOTE: platformProviders and cloudProviders in piped config will not be available in pipedv1 config. Instead, the deployTargets configuration in plugin config will determine where to deploy your application resources.

Meanwhile, the application config is where plugin configurations are stored specifically for the deployment of that application.

apiVersion: pipecd.dev/v1beta1
kind: Application
spec:
  name: test-app
  labels:
    env: local
  planner:
    alwaysUsePipeline: true
  pipeline:
    - name: K8S-SYNC
  plugins:
    kubernetes:
      input:
        kubectlVersion: 1.33.0
      quickSync:
        prune: true

Pipedv1 automatically identifies plugins based on the stage defined in the pipeline configuration. Which means below application config is valid as well

apiVersion: pipecd.dev/v1beta1
kind: Application
spec:
  name: test-app
  labels:
    env: local
  pipeline:
    - name: K8S-SYNC

For applications config without a pipeline definition, plugins config is required to help piped infer the plugin needed while performing deployment

apiVersion: pipecd.dev/v1beta1
kind: Application
spec:
  name: test-k8s-app
  labels:
    env: local
  plugins:
    kubernetes:

NOTE: kind: XXXApp is deprecated in pipedv1, kind: Application will be used for all applications in app configuration file for pipedv1 managed application.

Migrating application configs to the new format will be covered in detail in another blog. You can learn more about the migration flow for now please check out the migration issue.

Pipeline planning and configuration

Creating pipelines based on different deployment strategies for various specific applications is a strong point that makes PipeCD stand out. With pipedv1, this feature is further enhanced by solving some old limitations in the current piped.

Since pipedv1, you can create a pipeline consisting of only one SYNC stage.

apiVersion: pipecd.dev/v1beta1
kind: Application
spec:
  name: test-app
  pipeline:
    stages:
      - name: K8S_SYNC
        with:
          ...

Or optionally, create a pipeline that includes the SYNC stage combined with other common stages, for example.

apiVersion: pipecd.dev/v1beta1
kind: Application
spec:
  name: test-app
  pipeline:
    stages:
      - name: WAIT
        with:
          duration: 30s
      - name: K8S_SYNC
      - name: WAIT
        with:
          duration: 30s

Stages configurations

Regarding stages (units of the pipecd pipeline), a feature that has been requested by many users is skipOn, which allows skipping a stage in the application deployment pipeline under certain conditions.

In the current version of piped, skipOn is only supported in limited ways for some stages (like Analysis Stage - ref: docs), in pipedv1, all stages support this feature.

apiVersion: pipecd.dev/v1beta1
kind: Application
spec:
  name: test-app
  pipeline:
    stages:
      - name: WAIT
        skipOn:
          paths:
          - '*/canary'
        with:
          duration: 30s

An unstable feature in piped is stage timeout, in pipedv1 version, all stages also support timeout settings. The default stage timeout will be 6 hours.

apiVersion: pipecd.dev/v1beta1
kind: Application
spec:
  name: test-app
  pipeline:
    stages:
      - name: WAIT_APPROVAL
        timeout: 1h # Default is 6h
        with:
          approvers:
          - khanhtc1202

Deprecated / Dis-supported features

“Weird” flag `--enable-default-kubernetes-cloud-provider` in piped execution

A long-time “weird” flag for piped binary execution if you don’t use PipeCD for a Kubernetes application is --enable-default-kubernetes-cloud-provider, which is basically legacy due to the first implementation of PipeCD focusing on supporting only Kubernetes. The default value is false, but if you set that flag to true, piped execution requires connecting to the Kubernetes cluster even before any application deployment is actually triggered.

In pipedv1, nothing like that flag exists in piped execution because pipedv1 supports whatever platform its plugins support equally.

Kubernetes templating feature

Related to Kubernetes support features of PipeCD. Currently, piped supports the Helm Git Remote Chart feature (ref piped config helm chart repository), which pulls the chart file stored directly on Git to the local and builds/templates as for a Local Chart.

With the emergence and standardization of OCI, storing and sharing Helm Charts has become easier via the Helm registry. Therefore, in pipedv1, we decided to stop supporting this feature.

Analysis stage query templating feature

Current piped support building queries used while evaluating metrics with deployment-specific data to be embedded in the analysis template (ref: analysis templating docs).

From pipedv1, along with built-in and custom args are supported with placeholders as {{ .App }} and {{ .AppCustomArgs }} respectively.

Kubernetes-specific built-in args like {{ .K8s.Namespace }} will be marked as deprecated and unsupported after several releases. The corresponding usage for the Kubernetes Namespace use case is changed to {{ .AppCustomArgs.k8sNamespace }}.

Summary

Here are some changes you might notice when switching to pipedv1. We ensure a certain level of backward compatibility between piped and pipedv1. The improvements are all aimed at making pipedv1 support more platforms and making it easier to build pipelines based on plugins.

The official documentation for pipedv1 is still being prepared, and the experimental release of pipedv1 will be available on the official pipecd repo release tab, along with some built-in plugins in the next few days.

Thanks for your attention, cheer 🍻

Blog: Plugin architecture Piped alpha version has been released

Mon, 16 Jun 2025 00:00:00 +0000

It has been months from the Piped plugin architecture blog, and after over a year of development, we happy to announce that the alpha version of Piped is ready 🎉

In this blog, we will show you how to run the plugin-arch piped with some built in plugins developed by the maintainers team.

If you want to know more about the plugin-arch piped internal, please read the pipedv1 README in pipecd repo.

Prerequisites

kubectl and a k8s cluster (They are not required if you won’t use the kubernetes plugin)

1. Setup Control Plane

Run a Control Plane that your piped will connect to. If you want to run a Control Plane locally, see How to run Control Plane locally.
- The Control Plane version must be v0.52.0 or later.
Generate a new piped key/ID.

2.1. Access the Control Plane console.

2.2. Go to the piped list page. (https://{console-address}/settings/piped)

2.4. Add a new piped via the +ADD button.

2.5. Copy the generated piped ID and base64 encoded key.

2. Run plugin-arched piped

Download piped binary.

# OS="darwin" or "linux" CPU_ARCH="arm64" or "amd64"
curl -Lo ./piped_kubecon_jp_2025 https://github.com/pipe-cd/pipecd/releases/download/kubecon-jp-2025/piped_kubecon_jp_2025_{OS}_{CPU_ARCH}

chmod +x ./piped_kubecon_jp_2025

Create a piped config file like the following.

apiVersion: pipecd.dev/v1beta1
kind: Piped
spec:
  apiAddress: {CONTROL_PLANE_API_ADDRESS} # like "localhost:8080"
  projectID: {PROJECT_ID}
  pipedID: {PIPED_ID}
  pipedKeyData: {BASE64_ENCODED_PIPED_KEY} # or use pipedKeyFile
  repositories:
    - repoID: repo1
      remote: https://github.com/your-account/your-repo
      branch: xxx
  # See https://pipecd.dev/docs/user-guide/managing-piped/configuration-reference/ for details of above.
  # platformProviders is not necessary.

  plugins:
    - name: kubernetes
      port: 7001 # Any unused port
      url: https://github.com/pipe-cd/pipecd/releases/download/kubecon-jp-2025/plugin_kubernetes_kubecon_jp_2025_darwin_arm64 # choose binary on the release for your own OS and CPU arch
      deployTargets:
        - name: cluster1
          config: 
            masterURL: https://127.0.0.1:61337   # shown by kubectl cluster-info
            kubeConfigPath: /path/to/kubeconfig
            kubectlVersion: 1.33.0
    - name: wait
      port: 7002 # Any unused port
      url: https://github.com/pipe-cd/pipecd/releases/download/kubecon-jp-2025/plugin_wait_kubecon_jp_2025_darwin_arm64 # choose binary on the release for your own OS and CPU arch

    - name: example-stage
      port: 7003 # Any unused port
      url: https://github.com/pipe-cd/community-plugins/releases/download/kubecon-jp-2025/plugin_example-stage_kubecon_jp_2025_darwin_arm64 # choose binary on the release for your own OS and CPU arch
      config:
        - commonMessage: "[common message]"

Run piped

./piped_kubecon_jp_2025 piped --config-file=/path/to/piped-config.yaml --tools-dir=/tmp/piped-bin

If your Control Plane runs on local, add --insecure=true to the command to skip TLS certificate checks.

3. Deploy an application

Let’s create application with kubernetes plugin.

Create an app.pipecd.yaml like the following.

apiVersion: pipecd.dev/v1beta1
kind: Application
spec:
  name: canary
  labels:
    env: example
    team: product
  pipeline:
    stages:
      # Deploy the workloads of CANARY variant. In this case, the number of
      # workload replicas of CANARY variant is 10% of the replicas number of PRIMARY variant.
      - name: K8S_CANARY_ROLLOUT
        with:
          replicas: 10%
      # Wait 10 seconds before going to the next stage.
      - name: WAIT
        with:
          duration: 10s
      # Update the workload of PRIMARY variant to the new version.
      - name: K8S_PRIMARY_ROLLOUT
      # Destroy all workloads of CANARY variant.
      - name: K8S_CANARY_CLEAN

Create Resources in the same dir as app.pipecd.yaml.

deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: canary
  labels:
    app: canary
spec:
  replicas: 2
  revisionHistoryLimit: 2
  selector:
    matchLabels:
      app: canary
      pipecd.dev/variant: primary
  template:
    metadata:
      labels:
        app: canary
        pipecd.dev/variant: primary
    spec:
      containers:
      - name: helloworld
        image: ghcr.io/pipe-cd/helloworld:v0.32.0
        args:
          - server
        ports:
        - containerPort: 9085

service.yaml

apiVersion: v1
kind: Service
metadata:
  name: canary
spec:
  selector:
    app: canary
  ports:
    - protocol: TCP
      port: 9085
      targetPort: 9085

Push the app.pipecd.yaml and the resources to your remote repository.
On the Control Plane console, register the application via PIPED V1 ADD FROM SUGGESTIONS tab.

And more

Another hot news, we want to share you this time. As part of PipeCD team activity at KubeCon Japan 2025, the piped community plugins repository is officially opened 🎉

If you’re interested, join us, discuss pipecd at KubeCon Japan and fire some PRs to community-plugins repo. We are happy to help you build your desired deployment pipeline and plugins.

Let’s build and connect!

Blog: MCP Server for PipeCD Docs Has Been Released

Mon, 26 May 2025 00:00:00 +0000

Introduction

Finding the right information in the official PipeCD docs can sometimes be challenging due to its volume. To address this, we have developed the “PipeCD Docs Local MCP Server,” which allows you to perform full-text searches on the official documentation locally.
This server automatically fetches the PipeCD docs from GitHub and provides simple APIs for full-text search via the MCP protocol.

The repository is here: https://github.com/pipe-cd/docs-mcp-server

Example

For example, when I ask Cursor how to configure PipeCD for ECS, Cursor will search docs via the MCP server like this:

Usage

See README for details.

After preparing npm config to fetch the package from GitHub Packages registry, you can simply run the server by npx @pipe-cd/docs-mcp-server@latest.

For example, for Cursor editor, add the following to your mcp.json:

{
  "mcpServers": {
    "pipe-cd.docs-mcp-server": {
      "type": "stdio",
      "command": "npx",
      "args": [
        "@pipe-cd/docs-mcp-server@latest"
      ]
    }
  }
}

MCP Tools

The MCP Server provides two tools for now:

search_docs: Performs full-text search with keywords.
read_docs: Retrieves the content of a specified page.

Development Highlights

Efficient cloning with sparse checkout
Only the necessary directory (docs/content/en) is cloned to speed up processing.
Simple Search Logic
The current search logic is simple: it prioritizes title matches, followed by content matches.

Conclusion

We hope this tool helps PipeCD users to configure or find information more efficiently. Contributions, including bug reports and feature requests are welcome via Issues.

Blog: What Happens before Deployment? GitOps Mechanism

Mon, 28 Apr 2025 00:00:00 +0000

This article explains the process before a deployment begins in PipeCD. Since PipeCD is pull-based, there are various processes that take place before a deployment starts.

Probably, you have felt “What is Pending??” at least once:

The target audience is those who want to:

troubleshoot when “deployment doesn’t start as expected…”
understand GitOps mechanisms and implementation
contribute to PipeCD

This article is based on PipeCD v0.51.2 (latest version at time of writing). The specifications in the plugin version are slightly different.

This article omits discussion of Deployment Chain (due to complexity).

Overview

The state transitions before deployment begins are shown in the diagram below.

stateDiagram-v2
  Not_Started --> PENDING: [1] Trigger
  PENDING --> PLANNED: [2] Plan (Determine pipeline)
  PLANNED --> RUNNING: [3] Schedule (Execute stages)

Phases [1] through [3] are executed asynchronously by separate components (goroutines). This mechanism allows deployment to resume even if Piped crashes during the process.

1. Trigger

PipeCD initiates application deployment through four types of triggers:

[A] Detecting changes in Git (main method)
- Includes integration from CI through EventWatcher
[B] Manual start from UI/pipectl
[C] Automatic deployment on drift detection (requires configuration)
[D] Deployment Chain (details omitted)

The above four points can be configured trigger in app.pipecd.yaml. Only [C] is disabled by default.

[A] Detecting changes in Git

This occurs when the following two conditions are met after fetching the latest commit from the Git repository:

There are new commits that haven’t been deployed
There is at least one change in files at the same level or below app.pipecd.yaml

For 2., the detection scope can be adjusted using paths/ignores settings. This allows detection of changes in shared template files located in different directories, for example.

[B] Manual trigger from UI/pipectl

This is a push-style pattern.

From the UI, you can start it on the Application page. This is convenient for cases like “redeploying after a deployment failure due to environment issues rather than application issues.”

You can also start it from pipectl using the following command. This is useful for cases like “direct deployment from CI (exceptionally) or workflows.”

pipectl application sync --app-id={app-id}

[C] Automatic deployment on drift detection (requires configuration)

This occurs when there’s a difference between the Git definition (=desired state) and the actual environment. PipeCD has a separate mechanism called Drift Detection that continuously detects drifts.

This trigger is disabled by default, and manual repair is the basic approach after drift detection. This trigger is for those who want to strictly treat Git as the Single Source of Truth.

Deployment is triggered when an application meets all four of the following conditions:

“Automatic deployment on drift detection” is enabled
- Set the trigger.onOutOfSync.disabled item to false (default is true)
Drift has occurred (status is OutOfSync)
Not currently deploying
A specified time has passed since the last deployment (default is 5 minutes)
- This prevents the scenario where “deployment completes but the drift detection mechanism is delayed, causing false drift detection and deployment”
- Can be configured with the trigger.onOutOfSync.minWindow item

https://pipecd.dev/docs-v0.51.x/user-guide/configuration-reference/#onoutofsync

Trigger code

See the trigger package, trigger.go is the entry point and determiner.go determines whether to deploy or not.

Personally, reading the trigger package significantly improves understanding of PipeCD’s behavior.

https://github.com/pipe-cd/pipecd/blob/v0.51.2/pkg/app/piped/trigger/

2. Plan: Determine Pipeline

This phase determines “how to deploy using which pipeline.” It mainly does two things:

Determine whether to use Quick Sync or Pipeline Sync
Determine stages

The planner instance runs as a goroutine in a 1:1 relationship with Deployment, allowing multiple deployments to be processed concurrently. Also, piped is implemented to prevent multiple deployments from running simultaneously for the same application.

2.1. Determining Quick Sync or Pipeline Sync

What are Quick Sync and Pipeline Sync?

PipeCD has two deployment strategies:

Quick Sync
- Executes deployment with a single stage (K8S_SYNC or ECS_SYNC, etc.)
- Optimal for cases where “only Git synchronization is needed, no analysis required”
Pipeline Sync
- Executes deployment according to a defined pipeline
- Optimal for cases requiring staged deployment like Canary deployment

Strategy determination logic

Quick Sync is selected when any of the following conditions are met:

[A] User manually starts deployment from UI selecting “Quick Sync”
[B] No pipeline is configured in app.pipecd.yaml
[C] (k8s only) Conditions in app.pipecd.yaml’s planner.commitMatcher.quickSync are met
[D] (except Terraform) No successful deployment exists for that application
[E] (k8s only) No changes in workload or config (details)

Pipeline Sync is selected when any of the following conditions are met:

[A] User manually starts deployment from UI selecting “Pipeline Sync”
[B] planner.alwaysUsePipeline is set to true in app.pipecd.yaml (default is false)
[C] (k8s only) Conditions in app.pipecd.yaml’s planner.commitMatcher.pipeline are met

The detailed decision flowchart is like this:

flowchart TD
    Start[Start] --> UI{Triggered from UI and selected 'Auto'?}

    UI -->|Quick Sync selected| QuickSync
    UI -->|Not from UI or selected 'Auto'| Pipeline{Pipeline configured in app.pipecd.yaml?} 
    UI -->|Pipeline Sync selected| PipelineSync

    Pipeline-->|No| QuickSync
    Pipeline-->|Yes| CommitMatcherPipe{**k8s only**: meets commitMatcher.pipeline conditions in app.pipecd.yaml?}

    CommitMatcherPipe-->|No / non-k8s | CommitMatcherQuick{**k8s only**: meets commitMatcher.quickSync conditions in app.pipecd.yaml?}
    CommitMatcherPipe-->|Yes| PipelineSync

    CommitMatcherQuick-->|Yes| QuickSync
    CommitMatcherQuick-->|No / non-k8s | Always{alwaysUsePipeline=true in app.pipecd.yaml?}

    Always-->|No| Succeeded{Successful deployment exists?}
    Always-->|Yes| PipelineSync

    Succeeded-->|No| QuickSync
    Succeeded-->|Yes| Varied{Platform-specific determination logic}

    Varied --> QuickSync
    Varied --> PipelineSync

2.2. Determining Stages

There’s not much noteworthy here.

For Quick Sync:
- Since there’s no user-defined pipeline, generates stages like K8S_SYNC for K8s or ECS_SYNC for ECS
For Pipeline Sync:
- Generates stages according to the user-defined pipeline

Planner code

The core logic of Plan is implemented per platform in the planner package. The controller package controls its execution.

https://github.com/pipe-cd/pipecd/blob/v0.51.2/pkg/app/piped/planner/kubernetes/kubernetes.go

3. Schedule: Execute Pipeline

The scheduler component manages pipeline progression. It mainly does two things:

Execute each stage of the pipeline
Execute rollback on failure

Like the planner, the scheduler instance runs as a goroutine in a 1:1 relationship with Deployment, allowing multiple deployments to be processed concurrently.

3.1. Executing Each Stage of the Pipeline

Executes each stage sequentially based on the pipeline generated in the Plan phase. Specific processing content varies by platform.

3.2. Executing Rollback on Failure

If any stage fails, executes the rollback stage. Like 3.1., the rollback stage name and processing content vary by platform.

Scheduler code

The Run() and executeStage() methods in scheduler.go of the controller package are important.

https://github.com/pipe-cd/pipecd/blob/v0.51.2/pkg/app/piped/controller/scheduler.go#L191-L661

The specific processing logic of each stage is in the executor package. The executor package describes “execute this process in this stage,” making it relatively easy to understand.

Appendix: What is the controller?

This section is intended for those interested in code reading.

The controller is the manager of the planner and scheduler. Users don’t need to be aware of it.

The controller package, which contains the planner and scheduler, includes code related to deployment flow control. The controller.go is the entry point, and the controller runs continuously from Piped startup.

The controller performs the following every 10 seconds:

Create planner/scheduler for new deployments if needed
Delete completed planner/scheduler
Notify planner/scheduler when a user cancels deployment

Conclusion

While PipeCD has many other features and code, the trigger and controller form its core. I hope this helps find which code to get hints when you think “it would be nice if PipeCD had a feature like xxx.”

Blog: Introduction to EventWatcher: Connecting CI to PipeCD

Mon, 28 Apr 2025 00:00:00 +0000

This article explains the basics of “EventWatcher”, a crucial feature in practical PipeCD usage.

This article is intended for those who:

“Don’t know how to integrate CI with PipeCD”
“Have found the EventWatcher feature but can’t grasp how it works”

This article is based on PipeCD v0.51.2 (latest at the time of writing).

Background: Why EventWatcher is needed

Basically, PipeCD is a CD tool that continuously deploys specified config/manifests.

Deployment typically requires manifest changes. So, how can we “deploy using a new image (etc.) after CI completion”? It’s annoying to update the manifest repo manually each time.

This is where the EventWatcher feature comes into play!

What is EventWatcher?

EventWatcher is a feature in PipeCD that seamlessly connects CI and CD. It updates the manifest repo based on events from CI, triggering CD.

https://pipecd.dev/docs-v0.51.x/user-guide/event-watcher/

Mechanism

The overall picture of EventWatcher looks like the diagram below.

The EventWatcher feature itself handles steps 4-6:

1-3. Develop new app code and store the results (container images, etc.) in a container registry
4. Publish an event in CI to pass the new image URI to PipeCD
5. Piped detects the event
6. Update the manifest repo using the data provided by the event (image URI, etc.)
7-8. Automatic deployment occurs through the standard deployment flow of PipeCD

Usage

You need to setup three areas:

1. Piped Configuration

Which manifest repo events to handle
Commit permissions for the manifest repo

https://pipecd.dev/docs-v0.51.x/user-guide/managing-piped/configuring-event-watcher/

2. app.pipecd.yaml Configuration

Which events to handle for the Application (matcher)
What actions to take for matched events (handler)

(Example) Configuration to update the spec.template.spec.containers[0].image field in deployment.yaml when the helloworld-image-update event is triggered:

apiVersion: pipecd.dev/v1beta1
kind: KubernetesApp
spec:
  name: helloworld
  eventWatcher:
    - matcher: # Which events to handle
        name: helloworld-image-update
      handler: # What actions to take
        type: GIT_UPDATE
        config:
          replacements:
            - file: deployment.yaml
              yamlField: $.spec.template.spec.containers[0].image

Full configuration reference: https://pipecd.dev/docs-v0.51.x/user-guide/configuration-reference/#eventwatcher

3. Event Triggering via pipectl or GitHub Actions

Event name (set in app.pipecd.yaml as matcher.name)
New value (image URI, etc.)

(Example) Command to trigger an event named helloworld-image-update to change the image URI to ghcr.io/xxx/helloworld:v0.2.0:

pipectl event register \
    --address={CONTROL_PLANE_API_ADDRESS} \
    --api-key={API_KEY} \
    --name=helloworld-image-update \
    --data=ghcr.io/xxx/helloworld:v0.2.0

If you use GitHub Actions for your CI, using PipeCD’s official actions-event-register is recommended (configurations are the same):

https://github.com/marketplace/actions/pipecd-register-event

For example, triggering an event named helloworld-image-update to change the image URI to ghcr.io/xxx/helloworld:v0.2.0:

      - uses: pipe-cd/[email protected]
        with:
          api-address: ${{ secrets.API_ADDRESS }}
          api-key: ${{ secrets.API_KEY }}
          event-name: helloworld-image-update
          data: ghcr.io/xxx/helloworld:v0.2.0

You can also handle events differently by environment using the --labels option, even for events with the same name.

Advanced: `--contexts` Option

Adding --contexts to pipectl event register allows you to include various information in the manifest repo commit. For example, passing the “commit hash and URL of the app repo that triggered the event” makes it easier to track “which app repo changes caused this event” in the manifest repo. This is particularly useful when deployments occur frequently.

https://pipecd.dev/docs-v0.51.x/user-guide/event-watcher/#optional-using-contexts

Appendix: Code Reading

The EventWatcher code is consolidated in one file (though it’s quite long). Remembering that “EventWatcher runs continuously as a goroutine from Piped startup” makes it slightly easier to read.

https://github.com/pipe-cd/pipecd/blob/v0.50.2/pkg/app/piped/eventwatcher/eventwatcher.go

Conclusion

Using EventWatcher enables seamless integration between CI and CD. There are various advanced applications, and it would be great to see more examples being shared.

Additionally, Deployment Traces feature enhances the integration of CI and PipeCD. For more details, please see this article:

https://pipecd.dev/blog/2024/12/26/the-gap-between-ci/cd-and-how-pipecd-appeared-to-fulfill-the-gap/

Blog: The Gap Between CI/CD and How PipeCD Appeared to FulFill the Gap

Thu, 26 Dec 2024 00:00:00 +0000

PipeCD became a CNCF Sandbox last year and has only just begun to attract the attention of the community, but it has a long history of development since its inception, more than four years ago.

In this article, I would like to share the fundamental CICD challenges that the PipeCD development and operations team at CyberAgent identified, how we solved them in the current version of PipeCD, and our future direction.

Note: The content may be change in the future since it’s still under development.

GitOps and CICD

Although there are many different definitions and understandings, there are some commonalities when it comes to related concepts such as GitOps and CICD:

CI/CD separation is necessary
CI belongs to the application source code scope, and the output of CI is the input of CD. CD belongs to the manifest source code scope, and the input of CD is the output of CI
CD tools perform tasks based on GitOps principles

EventWatcher as the bridge between CI and CD

Since the first version, which didn’t have much user feedback, the development team has been pushing testing and using dogfooding techniques as a way to detect shortcomings in PipeCD’s functionality. Since these early days, the problem of effectively linking CI and CD has emerged.

source: https://www.weave.works/blog/what-is-gitops-really

A common way to bypass the “immutability firewall” between CI and CD without paying too much attention to what CI is being used for is to rely on an artifact (container image) that is the common element between CI and CD. This approach is used by many popular CDs today, such as ArgoCD and FluxCD.

In addition to advantages such as a generic interface (as it only depends on the container image itself), this approach also has limitations, such as:

It needs to support many different container registries (Docker Hub, ECR, GCR, ACR, Harbor, etc.)
It can only support container images as the artifacts, and not other artifacts such as Helm Charts, Terraform modules, Lambda source zips, etc.
As the number of container images to be monitored increases, performance-related issues arise

Because PipeCD’s most important feature is to support various platforms, being able to only support container images as artifacts is not suitable for PipeCD. Therefore, we decided to develop a new solution to solve this problem: EventWatcher.

Instead of choosing a container image as the interface between CI and CD, PipeCD directly uses events sent from CI to PipeCD as triggers for the CD pipeline.

An event is considered a command to PipeCD (similar to triggering a deployment from the UI). It contains information such as:

The event definition name and event label, are used to determine which application deployments should be triggered when piped processes this event
Information about what data is used by piped when processing the event
Information specifying paths to fields in the manifest file

For example, suppose you want to use PipeCD EventWatcher feature to auto trigger deployment for your Kubernetes application. You first need to update your application configuration manifest like this


apiVersion: pipecd.dev/v1beta1
kind: KubernetesApp
spec:
  name: helloworld
  eventWatcher:
    - matcher:
        name: event-test
      handler:
        type: GIT_UPDATE
        config:
          replacements:
            - file: deployment.yaml
              yamlField: $.spec.template.spec.containers[0].image

Then, in the last step of your CI pipeline, you need to use pipectl to send an event to PipeCD controlplane like this

$ pipectl event register --name=event-test --data=ghcr.io/pipecd/helloworld:v0.49.0

The piped which manage your Kubernetes application will be notified by the event, it will create a commit to your application manifest repo, which contains

  spec:
    template:
      spec:
        containers:
        - name: helloworld
-         image: ghcr.io/pipecd/helloworld:v0.48.0
+         image: ghcr.io/pipecd/helloworld:v0.49.0

And the piped will trigger a new deployment based on the changed commit.

EventWatcher helps you avoid performance issues as the number of container images you monitor grows. It supports a wide variety of artifact types (Helm charts, Terraform modules, Lambda source zips, etc.) while easily ensuring compatibility with a wide range of CIs.

The issue with EventWatcher

EventWatcher solves the above problems, but it also has certain limitations, although they are not directly due to the design of EventWatcher.

Due to security requirements, projects usually do not store source code and manifests in the same repository, but separate them into two repositories.

A common request is that when users look at a pipeline deployment on PipeCD, how can they know what source code (code repo) changes are being deployed in this deployment?

If you use container images as an interface between CI and CD, you can take advantage of the LABEL feature of containers to store information about source code commit hashes, branches, etc. The OCI specification also has some predefined LABELs that you can use without defining additional keys.

LABEL org.opencontainers.image.source="https://github.com/pipe-cd/pipecd"
LABEL org.opencontainers.image.revision="0123456789abcdefg"

However, although this approach is similar to the previous one, it is only suitable for artifacts that are container images, and at the same time, scanning container images to get information about commit hashes and displaying it in the UI incurs additional processing overhead.

At this point, we realized a strength in the EventWatcher design: sending events from CI to CD.

The essence of this problem is that when viewing a deployment, you need to know information in the code repository that is only relevant to the manifest repository. In the approach of sending events between CI and CD, you add the necessary information, such as the commit hash and PR URL, as part of the event. When it is piped to create a commit change to the manifest repository, this information is used to link the deployment with the commit trigger and commit the code change.

We use the git trailer feature to add this information to the manifest change commit. And finally, wrap up it as --context flag of the pipectl command.

pipectl event register --name=event-test --data=ghcr.io/pipecd/helloworld:v0.49.0 \
--context=Source-Commit-Hash= 6512b1f3fcba7342c11,Source-URL=https://github.com/pipe-cd/pipecd/commit/6512b1f3fcba7342c11,PipeCD-Official-Docs=https://pipecd.dev/

If you’re using GitHub as your manifest repo remote storage, you can see it on GitHub UI like this

With this feature released under PipeCD v0.49.3, users now can have a link from CD to CI for tracking back which application code change causes this deployment in CD pipeline.

It’s good, but it’s not enough

After the release of the EventWatcher update, the community provided positive feedback about resolving the connection issues between CI and CD. However, we notice some limitations when using PipeCD in very complex projects with many microservices in the same repository.

Each event sent to PipeCD to trigger a deployment can actually be shared to trigger multiple deployments. This is obvious, for example, changes related to protos or common shared libraries, such as logging and metrics libraries, lead to multiple deployments at once.

This becomes problematic when a development team includes many people working on the same repository and the number of changes affecting many services is large enough: PipeCD’s current UI deployment list makes it difficult to determine exactly which deployments were triggered by which code change.

Naively, we could just add a root_cause_commit_hash or codebase_commit_hash field to PipeCD’s deployment model and use this information to filter deployments triggered by the same repository code change in the PipeCD UI.

However, if users need to know more than just commit hash, this small change doesn’t reliably achieve the goal. What users really want is a link from the code repository to the manifest repository, that the devs (who know the code change), can use to track which deployments shift those changes to the clusters.

So we looked back at the design of PipeCD and EventWatcher and took the thinking a step further.

We decided to add the concept of Deployment Trace to PipeCD, using the code repository commit hash as the root key. For each code repository commit hash, a Deployment Trace contains the following information:

commit_hash: The commit hash of the code repository is used as the key
code repository information: Includes code change author, URL, branches, commit messages, authors, etc.
deployments: A list of deployments triggered by the key commit_hash. Each deployment contains the complete information of the PipeCD deployment (commit, path, pipeline in manifest repository, etc.)

This allows users to easily follow the information from the code repository -> manifest repository -> deployment without searching for information step by step. From the perspective of an application engineer making changes to the code repository, it is easy to see which deployment was triggered by a commit, what that deployment was, and how that deployment impacts the service.

We are summarizing this GitHub issue on the design and implementation of deployment tracing. Feel free to add any idea on the issue.

Conclusion

In this article, we shared the issues we encountered when using PipeCD in a highly complex project, how we solved them through the design and implementation of EventWatcher, and our next steps with deployment tracing.

The deployment tracing feature is planned and will be implemented as soon as possible. It is expected to be released around March 2025. The PipeCD project is an open-source project; anyone can participate in the development of the project, and we welcome any contributions from the community.

If you have any questions or opinions, feel free to talk in CNCF Slack > #pipecd or PipeCD Community Meeting.

Blog: Overview of the Plan for Pluginnable PipeCD

Thu, 28 Nov 2024 00:00:00 +0000

In this article, we introduce the overview of the “Pluginnable” architecture of PipeCD, a significant progress currently under development.

Note: The content may be change in the future since it’s still under development.

Summary

In Pluginnable PipeCD, Plugins execute deployments instead of Piped’s core.
Anyone can develop Plugins and you can use Plugins developed by others.
It will enable users to deploy to more various platform with more various Stages.
- platform example: Cloudflare, Azure, and AWS CloudFormation/CDK
- stage example: Automated Analysis by New Relic or Amazon CloudWatch

What’s Pluginnable PipeCD?

Role of Plugins

In Pluginnable PipeCD, Plugins are actors who execute deployments on behalf of a Piped.

Current: A Piped deploys to each platform by itself

In Pluginnable PipeCD: Plugins deploy to each platform

Piped’s core will control deployment flows.

Where Plugins come from?

Plugins are fetched from outside a Piped. They are not PipeCD built-in code.

Plugin’s binary is placed in GitHub or something, and a Piped loads it on launch.

Configuration would be like this:

# Piped's config YAML
apiVersion: pipecd.dev/v1beta1
kind: Piped
spec:
    plugin: # [New Area]
       - name: k8s_plugin
         # [HERE] (The URL format might be changed)
         sourceURL: https://github.com/xxxx/k8s-plugin
         ...

The key point is that you can develop your own plugins, or use plugins developed by others.

Addiitonally, deployments to Kubernetes, CloudRun, Terraform, ECS, and Lambda, which are currently supported by PipeCD, will be supported as official plugins.

How does a Plugin run?

A Plugin runs as a gRPC server. On a Piped starts, the Piped core will load the binary of each specified plugin and launch it as a gRPC server. While a deployment, a Plugin communicate with Piped core.

How to develop a Plugin?

Anyone can develop a Plugin freely. We will pubilsh a guide on how to develop a Plugin in the future.

Developing a Plugin will not be so tough. Here I introduce somes reasons:

The main task is implementing what you want to execute in each stage.
- Piped’s core will control deployment flows and handle git. You don’t need to deeply understand the mechanism of GitOps.
- To be exact, if you want Plan Preview or Drift Detection for the plugin, you need to implement them.
Plugins can be also written by languages other than Golang.
You don’t need to put your plugin in PipeCD official repository.
- You can determine review policy and release timing as you want.

Advantages of Pluginnable PipeCD

There are three big advantages. Thanks to them, users will not need to manage additional CD tools when adding a kind of platform or customizing behaviors.

1. Users can deploy to platforms other than K8s/CloudRun/Terraform/ECS/Lambda

Currently, PipeCD supports deploying to Kubernetes, CloudRun, Terraform, ECS, and Lambda.

Plugins enable you to deploy to the other platforms, for example, Cloudflare, Azure, and Amazon CloudFormation/CDK.

2. Users can choose behaviors as they want

Example 1: Deploy in a way different from built-in.
Example 2: For Kubernetes (or any platform), deploy simply for some Applications with Plugin-A, and deploy in a complicated way for the other Applications with Plugin-B.

3. Users can develop and use custom Stages

Example 1: Analyze by New Relic or Amazon CloudWatch in the ANALYSIS stage.
Example 2: Run a Kubernetes Job or a Lambda function to execute a job.

These effects will be more powerful as more Plugins come out since you can use plugins made by others.

Schedule overview

Pluginnable PipeCD will be released around Feburary 2025.

The Plugin development guide will be published after March 2025. After that, new Plugins will come out.

For the current PipeCD users, please migrate to the Pluginnable version. New features will be added to the Pluginnable version. We will share the details of migration ways later. We’ve been making efforts to minimize the amount of the migration tasks. The main tasks will be modifying Piped configs and Application configs.

Conclusion

Pluginnable PipeCD is a big step toward the PipeCD’s vision - “The One CD for All {applications, platforms, operations}”. The community energy will be more important.

To see the development progress, visit the RFC or the meta Issue.

If you have any questions or opinions, feel free to talk in CNCF Slack > #pipecd or PipeCD Community Meeting.

Blog: Performance improvement in Git operations on PipeCD v0.48.9

Wed, 11 Sep 2024 00:00:00 +0000

We have implemented performance improvements for Git operations in PipeCD v0.48.9, which was released on September 10, 2024. In this post, we will explain how we enhanced the performance of Git operations in PipeCD and when you can expect to see these improvements in your daily use.

Background

The PipeCD agent (piped) maintains a git repository cache in the local filesystem to accelerate Git operations. When piped needs to clone a repository, it first checks if the repository is already cloned in the cache. If the repository is already cloned, piped will fetch the latest changes from the remote repository. If the repository is not cloned, piped will clone it from the remote repository. Piped uses sync.Mutex to ensure that only one goroutine can access the cache at a time.

Problem

When multiple deployments are triggered simultaneously within a single repository, piped waits for other goroutines to finish their Git operations because sync.Mutex ensures that only one goroutine can access the cache at a time. This creates a bottleneck in Git operations, causing delays in deployments.

Solution

We have improved the performance of Git operations by implementing singleflight.Group from the golang.org/x/sync/singleflight package. The singleflight.Group is a package for duplicate function call suppression and caching. It provides a mechanism to suppress duplicate function calls, meaning that if multiple goroutines call the same function with the same key, only one call will be made to the function. The singleflight.Group also provides a caching mechanism, where the result of the function call is cached and returned to other goroutines that call the same function with the same key. We use this singleflight.Group to ensure that only one goroutine can clone the repository from the remote repository, while other goroutines wait for the result of the function call. This makes Git operations faster and more efficient, preventing delays in deployments.

Conclusion

We have enhanced the performance of Git operations in PipeCD v0.48.9 by implementing singleflight.Group from the golang.org/x/sync/singleflight package. This improvement makes Git operations faster and more efficient, preventing delays in deployments even when multiple deployments are triggered simultaneously within a single repository. You can expect to experience these improvements in your daily use of PipeCD v0.48.9.

Blog: PipeCD at CNCF KubeCon Hong Kong 2024

Fri, 09 Aug 2024 00:00:00 +0000

KubeCon Hong Kong 2024 is only two weeks away! PipeCD team is excited to announce upcoming activities at KubeCon. Let’s come to meet each other and share insights and knowledge 🙌

PipeCD’s session with Karmada

We will have a presentation as below.

Title: Unlocking Scalability and Simplifying Multi-Cloud Management with Karmada and PipeCD
Speakers: @khanhtc1202(from PipeCD) and Hongcai Ren(from Karmada)
DateTime: Thursday August 22, 2024 11:50 - 12:25 HKT
Venue: Level 1 | Hung Hom Room 7

PipeCD’s booth

We will also be around the project booth. Let’s come and discuss!

Booth information:

Project: PipeCDi
Kiosk #: TBA
Kiosk Shift: TBA
Location: TBA

That’s all! Have a great KubeCon 🙌

Blog: PipeCD at CNCF KubeCon EU 2024

Sat, 16 Mar 2024 00:00:00 +0000

KubeCon EU 2024 is only four more weeks away! PipeCD team is excited to announce upcoming activities at KubeCon. Let’s come to meet each other and share insights and knowledge. We’re excited to see you there 🙌

PipeCD’s presentations

We will have two presentations at KubeCon EU 2024.

On Tuesday, @khanhtc1202 will give Effective centric CD pipeline: Toward PipeCD v1.0 in the form of a lightning talk.

On Thursday, developer team host a workshop as 🚨 Contribfest: PipeCD Contribfest, hope to see you guys that time with 💻.

PipeCD’s booth

PipeCD team also be around the project booth, let’s come and discuss.

Booth information

Project: PipeCD
Kiosk #: PP16-A
Kiosk Shift: Part-time - AM (Fri)
Location: Solutions Showcase | Level 7.2 | Hall 7.2 | Project Pavilion

That’s all! Have a great KubeCon 🙌

Blog: Control Plane on local by docker-compose

Thu, 14 Mar 2024 00:00:00 +0000

Currently, you can deploy and operate a PipeCD Control Plane on a Kubernetes cluster or on Amazon ECS. However, some developers would like to build a Control Plane more easily for introduction or development. This blog shows you how to install a PipeCD Control Plane on local machine easily.

This blog is for those who would like to:

begin using PipeCD and experiment its features instantly, including Control Plane and Piped

Architecture

The general architecture of PipeCD Control Plane is as below.

Note: See Architecture Overview doc for details. In this blog, you will build a Control Plane by these containers:

Server: pipecd server
Ops: pipecd ops
Cache: Redis
Data Store: MySQL
File Store: MinIO

Installation of Control Plane

Get the demo codes from https://github.com/pipe-cd/demo/blob/main/control_plane/docker-compose/
Execute the below command for docker-compose.yaml you got in [1.].

docker-compose up

After executing the above command, you will see logs like below if success.

pipecd-server-1   | successfully loaded control-plane configuration
pipecd-server-1   | successfully connected to file store
pipecd-server-1   | successfully connected to data store
pipecd-server-1   | grpc server will be run without tls
pipecd-server-1   | grpc server will be run without tls
pipecd-server-1   | grpc server is running on [::]:9080
pipecd-server-1   | grpc server is running on [::]:9083
pipecd-server-1   | grpc server will be run without tls
pipecd-server-1   | admin server is running on 9085
pipecd-server-1   | grpc server is running on [::]:9081
pipecd-server-1   | start running http server on :9082

To confirm: Access the console of the Control Plane

Access http://localhost:8080 on your web browser.
Enter a value as below and CONTINUE.
- Project Name: control-plane-local
Enter values as below and LOGIN.
- Username: hello-pipecd
- Password: hello-pipecd
You will see the applications page. Success!

Clean up

To clean up the Control Plane, execute the below command.

docker-compose down

Note: By following commands instead of above one, you can keep data such as Piped or applications on the Control Plane even after restarting/updating the server component.

# Restart only the server component.
docker-compose rm -fsv pipecd-server
docker-compose up pipecd-server

Blog: How to set up a local development environment

Thu, 15 Jan 2026 00:00:00 +0000

Why do you need a Local Development Environment?

If you plan to contribute code, fix bugs, or add new plugins to PipeCD, you should run the control plane and Piped agent directly from the source code. This local development setup lets you test your Go changes live before submitting a pull request.

This guide walks through building and running the components from source so you can iterate quickly and confidently.

Estimated setup time: 20–30 minutes

Prerequisites

Make sure you have the following installed and configured:

Go (v1.21 or higher)
Docker
kubectl
Kind (required to create the local cluster)
Node.js and Yarn (required to build the web UI)
A fork of the PipeCD repository

To contribute to PipeCD, we recommend setting up a fork:

Fork pipe-cd/pipecd on GitHub.
Clone your fork locally:

git clone https://github.com/<YOUR_USERNAME>/pipecd.git
cd pipecd

1. Prepare and Start the Local Cluster

This step prepares a local Kubernetes cluster and container registry required for running the PipeCD control plane locally.

Update dependencies

Run these commands to ensure your local Go modules and web dependencies are up to date. Starting the environment may fail if these are outdated.

make update/go-deps
make update/web-deps

Start local registry and cluster

A helper command starts a local kind cluster and a container registry. This command also automatically creates the pipecd namespace where the components will run.

make up/local-cluster

After the cluster starts, export the kubeconfig to ensure kubectl can communicate with it:

kind export kubeconfig --name pipecd

2. Run the PipeCD Control Plane (from source)

The control plane provides the web UI, API, and metadata storage. Running it from source ensures you are testing your latest changes.

Start the control plane

This command compiles the Go code, builds the web assets, and starts the control plane server locally.

make run/pipecd

Access the UI

Once the control plane is running, forward the port to access the UI from your browser. Open a new terminal and run:

kubectl port-forward -n pipecd svc/pipecd 8080

Then open your browser:

URL: http://localhost:8080?project=quickstart
Username: hello-pipecd
Password: hello-pipecd

3. Configure and Run the Piped Agent (from source)

The Piped agent connects the control plane to your local Kubernetes cluster. You will run this agent from source as well.

Register Piped in the UI

Go to Settings → Piped (or open http://localhost:8080/settings/piped?project=quickstart).
Click + ADD, give it a name (for example, dev), and save.
Crucial step: Copy the generated Piped ID and Base64 encoded key immediately.

Create the Piped configuration

Create a file named piped-config.yaml in your repo root:

apiVersion: pipecd.dev/v1beta1
kind: Piped
spec:
  projectID: quickstart
  # Replace here with your piped ID.
  pipedID: <COPIED_PIPED_ID>
  # Base64 encoded string of the piped private key.
  # Replace here with your piped base64 key.
  pipedKeyData: <COPIED_ENCODED_PIPED_KEY>
  apiAddress: localhost:8080
  repositories:
  - repoId: example
    remote: [email protected]:pipe-cd/examples.git
    branch: master
  syncInterval: 1m
  platformProviders:
  - name: example-kubernetes
    type: KUBERNETES
    config:
      # Replace here with your kubeconfig absolute file path.
      kubeConfigPath: /path/to/.kube/config

Run Piped from source

Use your local code and the config file you just created:

make run/piped CONFIG_FILE=piped-config.yaml INSECURE=true

Cleaning Up

To stop and remove the local cluster and registry when you are done, run:

make down/local-cluster

Next Steps

Once Piped starts and shows as connected in the UI, your local development environment is ready. You can now build and test your local changes.

Blog: Terraform Provider for PipeCD is Out!

Thu, 27 Jul 2023 00:00:00 +0000

Now PipeCD resources can be managed by Terraform🎉

terraform-provider-pipecd, which is a Terraform provider for PipeCD, has been released! Thanks to @arabian9ts, @kurochan, and @sivchari for creating this provider!

Until now, there were two methods for adding or updating piped and applications in PipeCD: manually accessing the Control Plane through a browser or using the provided CLI, pipectl. While these methods were intuitive and easy to understand, they had the drawback of being prone to operational errors as the number of managed piped and applications increased. Even incorporating pipectl into shell scripts resulted in a procedural approach. It’s not very cool to have a tool that provides GitOps capabilities but can’t be managed using GitOps principles, right?

The newly released terraform-provider-pipecd addresses these issues. With this provider, you can easily deploy PipeCD resources using the declarative syntax of Terraform.

Using terraform-provider-pipecd, you can enjoy the following benefits:

Easy deployment: It simplifies the process of deploying PipeCD.
Declarative code management: By managing it as code, you can track changes and reviews through pull requests.
Integration with continuous deployment: When combined with CI/CD tools, it enables automation of configuration testing and the addition or modification of PipeCD resources.

Currently, the PipeCD resources that can be managed with terraform-provider-pipecd are:

Piped Agent
Application

In the future, based on community demand, more resources that can be managed might be added. Please try out the provider and provide feedback on Github.

How to Use

While this blog post doesn’t delve into detailed usage, it provides an example of registering a CloudRun Application with PipeCD using HCL code:

// Setup the provider
terraform {
  required_providers {
    pipecd = {
      source  = "pipe-cd/pipecd"
      version = "0.1.0"
    }
  }
  required_version = ">= 1.4"
}

// Declaration of CloudRun Application
resource "pipecd_application" "main" {
  kind              = "CLOUDRUN"
  name              = "example-application"
  description       = "This is the simple application"
  platform_provider = "cloudrun-inproject"
  piped_id          = "your-piped-id"
  git = {
    repository_id = "examples"
    remote        = "[email protected]:pipe-cd/examples.git"
    branch        = "master"
    path          = "cloudrun/simple"
    filename      = "app.pipecd.yaml"
  }
}

With this simple code, you can define the configuration information for an application. Then, just use the terraform command to perform an apply.

For more details, check out the documentation and the example in the repository.

Please watch our Community Meeting #2. @arabian9ts, a maintainer of terraform-provider-pipecd explaned how to add an application step by step.

Born in the Community

This project was originally developed by @arabian9ts, @kurochan, and @sivchari to make the PipeCD’s management better, and later donated to the pipe-cd organization. Many thanks for their contribution! (And they stay to be the maintainer of the projects to improve it more. How amazing is it?)

PipeCD is supported by the power of such community efforts. We encourage everyone to join the PipeCD community and participate.

Blog: CNCF Sandbox admission 🎉

Fri, 19 May 2023 00:00:00 +0000

Today the whole PipeCD team is overjoyed because the PipeCD project has just been accepted into the CNCF Sandbox program!!! 🥳

This is a great achievement for the whole team and is the beginning of our journey into CNCF. Also, many thanks to the CNCF TOC (Technical Oversight Committee) for the feedback they provided during the project evaluation.

The CNCF Sandbox admission is going to be the cherry on the top of the project milestone, and hope this accelerates further development of the PipeCD project and community.

We are in the middle of the onboarding process, and everyone is so excited about that.

Stay tuned for more news and happy PipeCD-ing!!!

Blog: PipeCD best practice 02 - control plane on ECS

Tue, 07 Feb 2023 00:00:00 +0000

This blog is a part of PipeCD best practice series, a guideline for you to operate your own PipeCD. Currently, you can deploy and operate the PipeCD control plane on a Kubernetes cluster easily, but some developers that would like to introduce PipeCD can not prepare Kubernetes environments. If you have the same problem, this blog is for you. We will show you how to deploy the PipeCD control plane on Amazon ECS.

Architecture

Note: Please refer to architecture-overview docs for definitions of PipeCD components such as server, ops, cache, datastore and filestore.

Following the above graph for PipeCD control plane runs on Amazon ECS, we have to prepare these next components

Secrets Manager

You should put config files (control-plane-config.yaml, envoy-config.yaml) on Secrets Manager because config files contain some credentials such as database passwords. The examples of config files for ECS are here. Please edit these files according to your environment.

aws secretsmanager create-secret --name control-plane-config \
--description "Configuration of control plane" \
--secret-string `base64 control-plane-config.yaml`
aws secretsmanager create-secret --name envoy-config \
--description "Configuration of control plane" \
--secret-string `base64 envoy-config.yaml`

You should also put encryption key

aws secretsmanager create-secret --name encryption-key \
--description "Encryption key for control plane" \
--secret-string `openssl rand 64 | base64`

RDS(datastore)

It is possible to use RDS as a datastore. Edit your configuration file for the control plane according to your RDS setting.

  datastore:
    type: MYSQL
    config: 
        url: root:password@tcp(endpoint_of_rds:3306) 
        database: quickstart

Redis(cache)

It is possible to use Redis as a cache. Note the endpoint of Redis for the task definition.

S3(filestore)

It is possible to use S3 as a filestore. The filestore contains state files that describe your secure infrastructure, so make sure to make the bucket private. Only allow pipecd-server(ECS) to access this bucket.

ECS

You need to create two different services for pipecd-server and a pipecd-ops because they have the same ports and different permissions. The pipecd-server can be accessed by external clients such as piped or web clients, so this service includes the pipe-cd gateway and this service must be connected to the application loadbalancer. The pipecd-ops can only be accessed by admin users, so this service must only be accessed via SSM session manager. ECS agent sets config files as environment variables in container from secrets manager. Create configuration files from environment variables in the container as below.

echo $ENVOY_CONFIG; echo $ENVOY_CONFIG | base64 -d >> envoy-config.yaml

Note: Attach IAM policy to get secrets from Secrets Manager to task execution role.

Note: If you manage both of RDS and ECS by terraform, you can rewite datastore endpoint in the configuration file. sed -i -e s/pipecd-mysql/${var.db_instance_address}/ control-plane-config.yaml;

Note: Attach IAM policy to access filestore to task role.

task definitions examples (using terraform variables)

gateway and server

{
name  = "pipecd-gateway"
image = var.gateway_image_url
portMappings = [
  {
    hostPort      = 9090
    containerPort = 9090
    protocol      = "tcp"
  }
]
essential = false
command = [
  "/bin/sh -c 'echo $ENVOY_CONFIG; echo $ENVOY_CONFIG | base64 -d >> envoy-config.yaml; envoy -c envoy-config.yaml;'"
]
entrypoint = [
  "sh",
  "-c"
]
secrets = [
  {
    "name" : "ENVOY_CONFIG",
    "valueFrom" : "arn:aws:secretsmanager:${data.aws_region.current.id}:${data.aws_caller_identity.self.account_id}:secret:${var.envoy_config_secret}"
  },
]
},
{
name  = "pipecd-server"
image = var.server_image_url
portMappings = [
]
command = [
  "/bin/sh -c 'echo $CONTROL_PLANE_CONFIG; echo $CONTROL_PLANE_CONFIG | base64 -d >> control-plane-config.yaml; sed -i -e s/pipecd-mysql/${var.db_instance_address}/ control-plane-config.yaml; echo $ENCRYPTION_KEY >> encryption-key; pipecd server --insecure-cookie=true --cache-address=${var.redis_host}:6379 --config-file=control-plane-config.yaml --enable-grpc-reflection=false --encryption-key-file=encryption-key --log-encoding=humanize --metrics=true;'"
]
entrypoint = [
  "sh",
  "-c"
]
secrets = [
  {
    "name" : "ENCRYPTION_KEY",
    "valueFrom" : "arn:aws:secretsmanager:${data.aws_region.current.id}:${data.aws_caller_identity.self.account_id}:secret:${var.encryption_key_secret}"
  },
  {
    "name" : "CONTROL_PLANE_CONFIG",
    "valueFrom" : "arn:aws:secretsmanager:${data.aws_region.current.id}:${data.aws_caller_identity.self.account_id}:secret:${var.control_plane_config_secret}"
  },
]
},

{
name  = "pipecd-ops"
image = var.ops_image_url
portMappings = [
  {
    "name" : "http",
    "protocol" : "tcp",
    "containerPort" : 9082,
    "appProtocol" : "http"
  },
  {
    "name" : "http",
    "protocol" : "tcp",
    "containerPort" : 9085,
    "appProtocol" : "http"
  },
]
command = [
  "/bin/sh -c 'echo $CONTROL_PLANE_CONFIG; echo $CONTROL_PLANE_CONFIG | base64 -d >> control-plane-config.yaml; sed -i -e s/pipecd-mysql/${var.db_instance_address}/ control-plane-config.yaml; echo $ENCRYPTION_KEY >> encryption-key; pipecd ops --cache-address=${var.redis_host}:6379 --config-file=control-plane-config.yaml --log-encoding=humanize --metrics=true;'"
]
entrypoint = [
  "sh",
  "-c"
]
secrets = [
  {
    "name" : "ENCRYPTION_KEY",
    "valueFrom" : "arn:aws:secretsmanager:${data.aws_region.current.id}:${data.aws_caller_identity.self.account_id}:secret:${var.encryption_key_secret}"
  },
  {
    "name" : "CONTROL_PLANE_CONFIG",
    "valueFrom" : "arn:aws:secretsmanager:${data.aws_region.current.id}:${data.aws_caller_identity.self.account_id}:secret:${var.control_plane_config_secret}"
  },
]
},

ALB

You must prepare two target groups for both HTTP and gRPC. Make two hosts and listner rules as below. Listner protocol should be HTTPS becuase it uses gRPC.

Terraform example

PipeCD gives control-plane-aws-ecs-terraform-demo, which we use Terraform to prepare PipeCD controlplane components and install it on air.

Prepare

Prepare SSL certificate Prepare your domain and SSL certificate from AWS certificate manager.
Create a s3 bucket for terraform backend Write bucket name to 00-main.tf

terraform {
  backend "s3" {
    bucket  = "example-pipecd-control-plane-tfstate" #your bucket name for terraform backend
    region  = "ap-northeast-1"
    key     = "tfstate"
    profile = "pipecd-control-planeg-terraform" #your profile
  }
  required_providers {
    aws = {
      version = "~> 3.34.0"
    }
  }
}

Edit variables.tf for your project

//export
locals {
  alb = {
    certificate_arn = ""
  }
  
  redis = {
    node_type = "cache.t2.micro"
  }

  rds = {
    node_type = "db.t3.micro"
  }

  ecs = {
    memory = "1024"
    cpu = "512"
  }
}

Create a S3 bucket for filestore and write the bucket name for it to control-plane-config.yaml and variables.tf

apiVersion: "pipecd.dev/v1beta1"
kind: ControlPlane
spec:
  datastore:
    type: MYSQL
    config:
      url: sample:test@tcp(pipecd-mysql:3306)
      database: quickstart
  filestore:
    type: S3
    config: # edit here
        bucket: example-pipecd-control-plane-filestore 
        region: ap-northeast-1
  projects:
    - id: quickstart
        staticAdmin:
          username: hello-pipecd
          passwordHash: "$2a$10$ye96mUqUqTnjUqgwQJbJzel/LJibRhUnmzyypACkvrTSnQpVFZ7qK" # bcrypt value of "hello-pipecd"

//export
locals {
  s3 = { # These must be unique in the world.
    filestore_bucket = "${local.project}-filestore" # edit here
  }
}

Write config of RDS for datastore to control-plane-config.yaml Note: Do not edit hostname (pipecd-mysql) because it will be edited autimaticaly by terraform.

apiVersion: "pipecd.dev/v1beta1"
kind: ControlPlane
spec:
  datastore:
    type: MYSQL
    config: # edit here
      url: sample:test@tcp(pipecd-mysql:3306)
      database: quickstart
  filestore:
    type: S3
    config: 
      bucket: example-pipecd-control-plane-filestore 
      region: ap-northeast-1
  projects:
    - id: quickstart
      staticAdmin:
        username: hello-pipecd
        passwordHash: "$2a$10$ye96mUqUqTnjUqgwQJbJzel/LJibRhUnmzyypACkvrTSnQpVFZ7qK" # bcrypt value of "hello-pipecd"

Put an encryption key and config file in Secrets Manager and write the path to variables.tf

locals {
  sm = {
    control_plane_config_secret = ""
    envoy_config_secret         = ""
    encryption_key_secret       = ""
  }
}

Deploy

terraform apply

You can login pipecd-ops via ECS exec.

aws ssm start-session --target ecs:${CLUSTER}_${TASK_ID}_${CONTAINER_ID} --document-name AWS-StartPortForwardingSession --parameters '{"portNumber":["9082"],"localPortNumber":["19082"]}'

That’s all! Now you have your own PipeCD controlplane and it’s ready to go!!

Blog: May 2022 update

Wed, 01 Jun 2022 00:00:00 +0000

Published by the PipeCD dev team every few months, this update will provide you with news and updates about the project! Please click here if you want to see the last status update.

What’s changed

It’s been a while from the last update about the development of our PipeCD project, but you can be assured that our PipeCD development is going well. Since the last report, PipeCD team has introduced 5 major releases (from v0.27.0 to the latest v0.32.3) with several UI improvements and some interesting features that I can’t wait to show you now. This blog post walks you through some notable changes. For all other changes, please check out each release note.

Cloud Run live state

Until now, you may notice that only application of kind Kubernetes has a part in its application detail page named live-state which shows you the situation of components of the current running PipeCD application. From version v0.27.0, we made that feature available for applications of kind Cloud Run too.

Make analysis stage skippable

One of the outstanding features PipeCD has is Automated deployment analysis (ADA), which helps you evaluate the impact of the current deployment right middle its running. Though it’s necessary to have that kind of stage for the stability of your service, sometimes you may want to skip the stage for some kind of quick/hot fixes. In that case, this feature is for you.

Enable to load Helm chart from OCI registry

With the release of Helm version v3.8.0, Helm is able to store and work with charts in container registries, as an alternative to Helm repositories. PipeCD adopts that feature right a way and for now, you’re able to fetch Helm chart from wherever OCI registries and use it to deploy your services. You can find more about how to use this feature in detail in this docs.

The very first touchable FileDB

⚠️ DEPRECATED NOTICE: FileDB has been discontinued as of PipeCD v0.53.0 due to maintenance complexity and reliability concerns. PipeCD now exclusively supports Firestore and MySQL as datastore backends.

It’s been a while since the first time we made a tweet about how PipeCD aims to remove the database as its dependencies to make the installation easier and compact, today we want to make the first achievement on the road to #nodatabase world.
For now, you can have a quick shot of using the new FileDB “database” of PipeCD with the quickstart example easily on your local machine by changing the quickstart/control-plane-values.yaml file contains config.data section to something like below:

    apiVersion: "pipecd.dev/v1beta1"
    kind: ControlPlane
    spec:
      datastore:
        type: FILEDB  # ⚠️ DEPRECATED: Use FIRESTORE or MYSQL instead
      filestore:
        type: MINIO
        config:
          endpoint: http://pipecd-minio:9000
          bucket: quickstart
          accessKeyFile: /etc/pipecd-secret/minio-access-key
          secretKeyFile: /etc/pipecd-secret/minio-secret-key
          autoCreateBucket: true
      projects:
        - id: quickstart
          staticAdmin:
            username: hello-pipecd
            passwordHash: "$2a$10$ye96mUqUqTnjUqgwQJbJzel/LJibRhUnmzyypACkvrTSnQpVFZ7qK" # bcrypt value of "hello-pipecd"

Please follow the development guideline or quickstart guideline if you don’t know how to start/install PipeCD control plane on your local machine.

The PipeCD play is available

For all users who want to have a glance at how and what PipeCD can give you in real-life usecases, this PipeCD play environment is for you. We have a blog about that live demo, feel free to check it from here.

And many other UI improvements are updated to the PipeCD web console, let’s check it out on our play environment.

We’re opening more and more

PipeCD is an OSS project! And we want to make not just the source code open, but the whole deployment process should be opened as well in order to get more contribution and support from the open community. We moved almost all of our CI flow to the open Github Actions platform, and as the result, everyone who makes contributions to our PipeCD repository can easily follow/access the CI stages via Github Actions UI. At the time I wrote this blog, PipeCD CI flow had been opened and can be accessed via this link.

What are next

The team continues actively working on improving the PipeCD product. Besides fixing the reported issues, enhancing the existing features, here are some new features you may catch for few next releases:

Piped management via web console: restartable and configuration checkable.
RBAC for PipeCD resources based on the Label mechanism

If you have any features want to request or find out a problem, please let us know by creating issues to the pipe-cd/pipecd repository.

Follow us on Twitter to keep track of all the latest news: https://twitter.com/pipecd_dev

Happy PipeCD-ing 👋

Blog: The PipeCD play environment is here!!!

Tue, 12 Apr 2022 00:00:00 +0000

Good news, PipeCD team’s happy to bring you a place where you can have a look at the PipeCD platform in use. We call it the play environment - https://play.pipecd.dev.

PipeCD play environment view

With this live demo, you can now have a look at the PipeCD platform in use without self preparing quickstart or such. It’s way easier (and faster) to make a try by clicking around and seeing how you can get when using PipeCD as the CD platform on your own.

How to access

The PipeCD play environment console is available at https://play.pipecd.dev. After following the link, you will get the login page as below.

Type play to the input box as the project name to login and click to Continue to go to the account sign in page. Followed by LOGIN WITH GITHUB.

Then, that’s it. Feel free to click around and see what PipeCD can bring to you in real-life use.

Some pages you may feel interest

Application detail page - show the application's state and info

Deployment detail page - show the deployment's stages and its log

You can also have a look at PlanPreview feature example, via the play project application configuration repository named examples at PR #108 comment.

Plan preview - give the early feedback by showing the changes which will be applied on PR merged

Notes for the PipeCD play environment

Since the limitation of the resources, you can only log in with the Viewer role. This means you can only click around and see PipeCD team prepared examples, triggering new deployments or creating new resources are disabled.
Currently, only applications of kinds: KUBERNETES, CLOUDRUN and TERRAFORM are prepared and available to see on the play environment console. We will add example applications of other kinds (LAMBDA, ECS, etc.) later.
The application configuration files for those examples on the play environment are published at pipe-cd/examples repository. Those configuration files are real-life useable configurations, but only on our cluster, since the encrypted credentials placed in those files are ours. You can replace those and use on your own.

Happy PipeCD-ing 👋

Blog: February 2022 update

Thu, 10 Feb 2022 00:00:00 +0000

Published by the PipeCD dev team every month, this update will provide you with news and updates about the project! Please click here if you want to see the last status update.

What’s changed

Since the last report, PipeCD team has introduced 4 releases (v0.24.0, v0.24.5, v0.25.0, v0.25.1). This blog post walks you through their notable changes. For all other changes, please check out each release note.

Introducing Label mechanism

Label concept has been introduced. Labels are key/value pairs that are attached to PipeCD resources such as application, deployment… As a developer, you can use labels to add identifying attributes to them. We believe that supporting the Label will give us more flexibility on grouping, filtering as well as managing access control on PipeCD resources.

Application labels can be specified via the labels field in the application configuration file while deployment labels are inherited from its application.

apiVersion: pipecd.dev/v1beta1
kind: KubernetesApp
spec:
  name: example
  labels:
    env: prod
    team: product

From v0.24.0, application labels are shown on the application list and detail page. All filter forms also have a new label box to allow filtering resources by their labels.

Environment and Deployment Configuration File were deprecated

Since the Label was introduced, the Environment becomes a subset of the Label concept. So we decided to deprecate the Environment concept from v0.24.0. Even though it is still displayed on the web console, it will be removed completely from the next release. So make sure to switch to using Label instead.

The deployment configuration file (.pipe.yaml) has been deprecated as well. Please use the application configuration file instead, it has the same format but with some new fields such as name, description, and labels. By using this new config file, all application information can be stored and managed in your Git repository. The new config file also gives PipeCD the ability to detect unregistered applications in Git repository to suggest users to add on the control plane.

Basically, a deployment configuration file can be migrated to be an application configuration file just by adding the name field.

# Old deployment configuration file
apiVersion: pipecd.dev/v1beta1
kind: KubernetesApp
spec:
  ...

# New application configuration file
apiVersion: pipecd.dev/v1beta1
kind: KubernetesApp
spec:
  name: example # Add this field.
  labels:
    env: prod
    team: product
  ...

If you are having many applications, you can use this pipectl command to migrate a bunch of your files seamlessly.

Event list page on web console

The event concept was designed to help external services such as CI systems to be able to interact with PipeCD. For instance, you can use the Event Watcher feature to trigger a new deployment of a given application when a new version of its container image or Helm chart or Terraform package has been just published.

Checking the status of each sent event could be helpful for the developer while troubleshooting, so from v0.25.0 that page was added to PipeCD’s web console.

Automated configuration drift detection for Cloud Run application

Configuration Drift is a phenomenon where running resources of service become more and more different from the definitions in Git as time goes on, due to manual ad-hoc changes and updates. As PipeCD is using Git as a single source of truth, all application resources and infrastructure changes should be done by making a pull request to Git. Whenever a configuration drift occurs it should be notified to the developers and be fixed.

PipeCD includes the Configuration Drift Detection feature, which automatically detects the configuration drift and shows the result in the application details web page as well as sends the notifications to the developers.

Before v0.25.0, this feature could only be used for Kubernetes apps, but now we’re happy to announce that Cloud Run apps can also have that feature.

What are next

The team continues actively working on improving the PipeCD product. Besides fixing the reported issues, enhancing the existing features, here are some new features the team is currently working on:

Realtime application state for Cloud Run application
RBAC for PipeCD resources based on the Label mechanism
Enable running control-plane without database. It means file storage (such as GCS, S3, Minio) can be used for both data store and file store.

If you have any features want to request or find out a problem, please let us know by creating issues to the pipe-cd/pipecd repository.

Follow us on Twitter to keep track of all the latest news: https://twitter.com/pipecd_dev

Blog: PipeCD best practice 01 - operate your own PipeCD cluster

Wed, 29 Dec 2021 00:00:00 +0000

PipeCD is an open source project, you can freely use the released versions of PipeCD to create and operate a continuous delivery system for your service or company. Quickstart docs - a complete guide for you to install required components of PipeCD and deploy a simple application via PipeCD is a good point to get started. However, for the sake of the simplicity of that tutorial, some points to keep in mind when operating a PipeCD cluster have been omitted. You will review the PipeCD architecture again and get some tips on how to operate a PipeCD cluster in this post.

Again, review the PipeCD architecture

Component Architecture

Note: Please refer to concepts docs for definitions of PipeCD components such as Control Plane, Piped, Application, etc.

At a glance, PipeCD is composed of 2 components, the Control Plane and the Piped(s). As the figure above, depending on your task/role, you need to work with different components of PipeCD.

As a operator - platform team, most of your time working with PipeCD is to operate the PipeCD Control Plane. You may also need to cooperate with the product team, helping them install Piped(s) to their applications running cluster depending on your company team’s structure. The space surrounded by a green border shows the operators working area.
As a developer - product team, most of your time working with PipeCD is to deploy, manage and observe your applications via application configuration files and PipeCD web console. You may also need to cooperate with the platform team to install Piped(s) to your applications running cluster if necessary. The space surrounded by a blue border shows the developers working area.

Tips to operate PipeCD cluster

Only a single Control Plane needs to be installed and will be operated by the operators (platform team). The installed Control Plane doesn’t have to be in the same cluster as your applications running cluster or the Piped installed cluster. Just make sure the Piped(s) can connect to the Control Plane via outbound requests is enough.
Developers from the product team may need to self-install piped into their applications running cluster if the platform team does not manage those credentials.

Once the Piped - the runner is installed successfully, developers only need to care about app configuration - which defines the pipeline so that piped(s) can use them to deploy your applications. Interaction with the applications is mainly done via the PipeCD web console and the configuration files stored on Git.

As an operator, PipeCD control-plane and piped(s) are what you should care about. As a developer, you should care about piped which installed in your applications running cluster and your applications’ PipeCD configurations.

It is possible to use a single piped to manage deployments of all the applications to the clusters other than the one in which piped is being installed. However, as security best practices and avoid generating cluster credentials, we highly recommend running each Piped inside each cluster and that Piped will only manage the applications on that cluster. In case you want to deploy applications that are required to be installed across clusters/regions/environments, and want to follow this rule of thumb, please refer to the deployment-chain feature.
Credentials used by piped(s) while deploying your applications should be managed by the developers of the production team themselves. You can refer to the secret management feature supported by PipeCD for this purpose.

Blog: January 2022 update

Wed, 05 Jan 2022 00:00:00 +0000

Published by the PipeCD dev team every month, this update will provide you with news and updates about the project! Please click here if you want to see the last status update.

Happy New Year

First of all, PipeCD team would like to wish you all a very Happy New Year. 🥳

2021 has been a great year for both PipeCD project and PipeCD team. Many features have been introduced by more than 2000 pull requests from 33 contributors. We would like to thank all those wonderful contributors.

Stepping into 2022, PipeCD team looks forward to contributing even more to make PipeCD project better and more useful for many users in our OSS community.

What’s changed

Since the last report, PipeCD team has introduced 2 releases (v0.22.0, v0.23.0). These releases bring many updates to help PipeCD becomes more stable as well as introduce some interesting features. In this blog post, we will recap those new features. For all other changes, please check out each release note to see.

Deployment chain

As you know, each application in PipeCD contains a group of resources that are managed together and be deployed to a single cloud provider such as a Kubernetes cluster. The application works independently of each other since there is no connection between them.

But in some use cases, you might want to build a more complex deployment flow where the connection between applications is required. That feature has been requested from a number of PipeCD users.

By v0.23.0, PipeCD team has brought that feature to reality. It will allow you to roll out applications to multiple clusters/regions gradually or to promote applications across environments. Those deployment strategies can be done simply by specifying a chain of applications in the postSync field of the first application.

apiVersion: pipecd.dev/v1beta1
kind: KubernetesApp
spec:
  ...
  postSync:
    chain:
      applications:
        - name: second-app
          kind: KUBERNETES
        - name: third-app
          kind: CLOUDRUN

Currently, in-chain applications can be specified only by name, kind but finding applications by label will also be supported in the near future as well. Since PipeCD is supporting multiple application kinds (such as Kubernetes, Terraform, CloudRun…) so applications from different kinds are allowed to be included in a chain.

For more details, please check out its documentation page.

Environment is no longer required and will be replaced by Label

Environment was changed to be optional while registering Piped or Application. You still can use it from the web console or specify the environment for an application via the envName field in the application configuration file, but we are planning to completely remove the environment concept in the near future. As an alternative, a new Label concept has been introduced, and Environment can be imagined as a particular label.

An application can contain one or more labels in its application configuration file as below:

apiVersion: pipecd.dev/v1beta1
kind: TerraformApp
spec:
  name: simple-app
  envName: production
  labels:
    #env: production
    service: backend
  ...

Filtering application or deployment by the label is being implemeted currently. We hope that it will be shipped in the next release.

Simplifying the way of registering an application

Before v0.23.0, while registering a new application user has to configure the application information manually via a registration form. For now, most of the information can be specified in the application configuration file. And Piped agents automatically find the un-registered applications to suggest to users. It means that all you have to do is just a few clicks on the web console.

Picking from the suggested application list to register

Please note that to be suggested, the application configuration file must be suffixed by .pipecd.yaml. And the default name of that file has been changed to app.pipecd.yaml instead of .pipe.yaml as before.

Showing the connection status of Pipeds

The connection status of Piped agents to the control plane is shown on the Settings page in real-time. This gives the operator a quick look at the current status of a particular Piped agent is.

More controls on triggering the deployment

By default, when a new merged pull request touches an application, a new deployment for that application will be triggered to execute the sync process.

From v0.22.0, a new trigger field has been added to the application configuration file to allow users to customize the triggering condition. For example, using onOutOfSync to enable the ability to attempt to resolve OUT_OF_SYNC state whenever a configuration drift has been detected.

The deployment triggering of each application can be configured with the following fields:

onCommit: controls triggering new deployment when newly added Git commits touched the application.
onCommand: controls triggering new deployment when received a new SYNC command from the web console or pipectl
onOutOfSync: controls triggering new deployment when application is at OUT_OF_SYNC state. Enabling this will force Piped to always attempt to keep the application as synced as possible.
onChain: controls triggering new deployment when the application is counted as a node of a deployment chain. (this configuration is added from v0.23.0)

apiVersion: pipecd.dev/v1beta1
kind: KubernetesApp
spec:
  trigger:
    onCommit:
      disabled: false
    onCommand:
      disabled: false
    onOutOfSync:
      disabled: true
    onChain:
      disabled: true

What are next

The team continues actively working on improving the PipeCD product. Besides fixing the reported issues, enhancing the existing features, here are some new features the team is currently working on:

Reduce the maintenance cost of the control plane by supporting using file storage (such as GCS, S3, Minio) for both data store and file store. It means no database is required to run the control plane.
Automated configuration drift detection for Cloud Run application

If you have any features want to request or find out a problem, please let us know by creating issues to the pipe-cd/pipecd repository.

Follow us on Twitter to keep track of all the latest news: https://twitter.com/pipecd_dev

Blog: November 2021 update

Mon, 01 Nov 2021 00:00:00 +0000

Published by the PipeCD dev team every month, this update will provide you with news and updates about the project! Please click here if you want to see the last status update.

What’s changed

Last month, PipeCD team has introduced 4 releases (v0.20.0, v0.20.1, v0.20.2, v0.21.0). Those releases bring many updates to help PipeCD becomes more stable as well as introduce some interesting features. In this blog post, we will recap those new features. For all other changes, please check out each release note to see.

Lambda Deployment

Before v0.20.0, PipeCD only supports deploying an AWS Lambda function by using container image as the input. It means that to run a Lambda function, you have to prepare its container image by building the source code and storing its image in a container registry. That preparation is normally considered as a responsibility of a CI system. That way is straightforward and has been highly recommended recently in the Cloud Native development process.

But to provide more flexibility to the developers, especially those are still not able to apply container technology into their development process. Starting from v0.20.0 PipeCD introduces two new ways to deploy an AWS Lambda function: deploying directly from source code stored in Git, deploying a Zip packing code stored in AWS S3.

# Deploy Lambda function directly from source code stored in a Git repository.
apiVersion: pipecd.dev/v1beta1
kind: LambdaFunction
spec:
  name: SimpleFunction
  source:
    git: [email protected]:username/lambda-function-code.git
    ref: v1.0.0
    path: hello-world

# Deploy Lambda function by a Zip packing source code stored in S3.
apiVersion: pipecd.dev/v1beta1
kind: LambdaFunction
spec:
  name: SimpleFunction
  s3Bucket: pipecd-sample-lambda
  s3Key: pipecd-sample-src
  s3ObjectVersion: 1pTK9_v0Kd7I8Sk4n6abzCL

# Deploy Lambda function by using a container image.
apiVersion: pipecd.dev/v1beta1
kind: LambdaFunction
spec:
  name: SimpleFunction
  image: ecr.ap-northeast-1.amazonaws.com/lambda-test:v0.0.1

Check out its documentation for more details.

Automated Deployment Analysis (ADA)

ADA is a good way to automate the process of evaluating the impact of a deployment during its lifetime to help building a robust release process. Any application can be configured to analyze its deployments based on its metrics data, log messages, or results from directly emitting requests to the application.

Until now, PipeCD has been providing a way to allow the developer to configure a list of query and threshold to analyze metrics data of a running deployment. Any query that returns a result exceeding the given threshold will cause the deployment to be failed, and a rollback stage will be executed if needed. Although this way is relatively straightforward, there is a challenge you can see here; it isn’t easy to find an appropriate threshold value to use since software is constantly changing time over time that making those values could be changed dynamically. Therefore, some new analysis strategies PREVIOUS, CANARY_BASELINE, CANARY_PRIMARY have been introduced since v0.20.0. They allow analyzing without requiring any threshold values.

Sum up, you can choose one of the following four strategies which fits your use case:

PREVIOUS: A method to compare the time series of metrics data between the running and the last successful deployment
CANARY_BASELINE: A method to compare the time series of metrics data between the Canary and Baseline variants
CANARY_PRIMARY: A method to compare the time series of metrics data between the Canary and Primary variants
THRESHOLD: A method to check the given queries againts the given thresholds

# ADA by comparing the time series of metrics data between the running and the last successful deployment.
- name: ANALYSIS
  with:
    duration: 30m
    metrics:
    - strategy: PREVIOUS
      provider: my-prometheus
      deviation: HIGH
      interval: 5m
      query: sum (rate(http_requests_total{status=~"5.*"}[5m])) / sum (rate(http_requests_total[5m]))

# ADA by comparing the time series of metrics data between the Canary and Baseline variants.
- name: ANALYSIS
  with:
    duration: 30m
    metrics:
    - strategy: CANARY_BASELINE
      provider: my-prometheus
      deviation: HIGH
      interval: 5m
      query: sum (rate(http_requests_total{job="foo-{{ .Variant.Name }}", status=~"5.*"}[5m])) / sum (rate(http_requests_total{job="foo-{{ .Variant.Name }}"}[5m]))

# ADA by checking the given queries againts the given thresholds.
- name: ANALYSIS
  with:
    duration: 30m
    metrics:
    - strategy: THRESHOLD
      provider: my-prometheus
      interval: 5m
      expected:
        max: 0.01
      query: sum (rate(http_requests_total{status=~"5.*"}[5m])) / sum (rate(http_requests_total[5m]))

Check out its documentation for more details.

Notification

Any PipeCD event (deployment triggered, planned, analysis completed…) can be configured to send to external services like Slack or a Webhook service. While forwarding those events to a chat service helps developers have a quick and convenient way to know the deployment’s current status, forwarding to a Webhook service may be useful for triggering other related tasks.

Until now, only Slack has been supported. From v0.21.0 you can configure your Pipeds to send its events to external services via Webhook. See here to know how to configure.

Besides that, events sent to Slack can be configured to mention one or multiple Slack users. It would be helpful to prevent developers from missing important events.

apiVersion: pipecd.dev/v1beta1
kind: KubernetesApp
spec:
  pipeline:
    stages:
      - name: K8S_CANARY_ROLLOUT
      - name: WAIT_APPROVAL
      - name: K8S_PRIMARY_ROLLOUT
      - name: K8S_CANARY_CLEAN
  notification:
    mentions:
      - event: DEPLOYMENT_WAIT_APPROVAL
        slack:
          - slack-user-id-1
          - slack-user-id-2
      - event: "*"                      # Specifying "*" means mentioning the given users for all events.
        slack:
          - slack-user-id-3

Piped Operation

As you know, PipeCD is designed with control-plane and agent model (see component architecture diagram). Piped is a single binary agent you run in your cluster, your local network to handle the deployment tasks related to that cluster. It can be run as a Pod inside a Kubernetes cluster, as a process in a virtual machine, or even on your local machine. Piped is stateless so that it can be restarted without worrying about data loss. That helps the product team reduce the cost of maintaining their Pipeds. You can see those installation guides to know how to install Piped on each environment.

From v0.20.0, managing Piped has become even easier than ever. With Remote-Upgrade feature, you can restart the currently running Piped with another version from the web console. And with Remote-Config feature, you can enforce your Piped to use the latest config data just by updating its config file stored in a remote location such as a Git repository.

Select a list of Pipeds to upgrade from Settings page

Talks

Two members from PipeCD team did talk about PipeCD at some local events in Japan.

2021/10/08 - Introducing PipeCD as a unified CD system for both infrastructure and application

Cloud Native Developers JP, by @nghialv (Youtube video in Japanese)
2021/10/29 - How to deploy AWS application with GitOps style by PipeCD

JAWS-UG コンテナ支部, by @khanhtc (Youtube video in Japanese)

What are next

The team continues actively working on improving PipeCD product. Besides fixing the reported issues, enhancing the existing features, here are some new features the team is currently working on:

Deployment Chain - Allow rolling out to multiple clusters gradually or promoting across environments
Resource Tagging - A better way to group and filter PipeCD resources such as applications, deployments

If you have any features want to request or find out a problem, please let us know by creating issues to the pipe-cd/pipecd repository.

Follow us on Twitter to keep track of all the latest news: https://twitter.com/pipecd_dev

Blog: Migrate Control Plane datastore to MySQL

Thu, 01 Apr 2021 00:00:00 +0000

Since PipeCD release v0.9.8 which introduces MySQL as PipeCD control-plane datastore, we plan to drop the support for MongoDB datastore in the near future. Consider the supports of cloud providers (GCP, AWS, Azure, etc), MySQL has a higher status over MongoDB is one of the most reasons for this support-dropping conclusion.

If you are using MongoDB as your PipeCD control-plane datastore, the following guide is manipulation to migrate your datastore to MySQL.

Before start

You need to install the pipectl version v0.9.10-1-4ab28c0 command-line tool.

Validate your installed pipectl command

$ pipectl datastore -h
Manage control-plane datastore resource.

Usage:
  pipectl datastore [command]

Available Commands:
  migrate     Migrate data to MySQL datastore.

Migration

Step by step guide for migration

1. Prepare your MySQL datastore instance.

Note:

MySQL v8.0 or later is required.
To enable pipectl to migrate data to the new MySQL datastore, your new MySQL data instance has to be connected from pipectl running environment.
The current implementation of pipectl datastore subcommand connects directly to datastore, you do not need to authenticate your pipectl binary with API Key as for other subcommands. Just make sure pipectl running environment is in the same network with your MySQL instance is enough.
Your MySQL instance has to be initialized with the going to be used database, make sure you create it before move to the next step.

For example, if you use docker to create your new MySQL instance, the command should be

$ docker run -d \
    --name test-db \
    -e MYSQL_ROOT_PASSWORD=XXX \
    -e MYSQL_DATABASE=*database-name* \
    mysql:8.0

2. Stop PipeCD control-plane

In case your PipeCD control-plane is installed by helm, simply run helm uninstall command would help.

3. Migrate using `pipectl`

Migrate using the following command (replace upstream-data-src, downstream-data-src and database with your corresponding values)

$ pipectl datastore migrate \
    --upstream-data-src="mongodb://127.0.0.1:27017/quickstart" \
    --downstream-data-src="root:test@tcp(127.0.0.1:3306)" \
    --database=quickstart

Note:

Make sure your data-srcs value are formatted as same as the above example.
If you want to migrate only specific data models (not all at once), use the --models flag as follow --models=Application,Project. (Use pipectl datastore migrate -h to get the list of migratable models)

4. Start PipeCD control-plane with the new configuration

Your new control-plane’s configuration should be updated as follow:

 apiVersion: "pipecd.dev/v1beta1"
 kind: ControlPlane
 spec:
   datastore:
-    type: MONGODB
+    type: MYSQL
     config:
-      url: mongodb://127.0.0.1:27017/quickstart
+      url: root:test@tcp(127.0.0.1:3306)
       database: quickstart
...

See ConfigurationReference for the full configuration.

Restart PipeCD control-plane as same as you start it before, your PipeCD should be ready 🚀.

Blog: Announcing PipeCD

Tue, 06 Oct 2020 00:00:00 +0000

Today we are excited to announce the open-source availability of PipeCD: a continuous delivery system for declarative Kubernetes, Serverless, and Infrastructure applications. PipeCD aims to provide a unified CD solution for multiple application kinds on multi-cloud that empowers engineers to deploy faster with more confidence. It is also available as a GitOps tool that enables doing deployment operations by pull request on Git.

Deployment Details Page

Background

As one of our Developer Productivity team’s missions, we aim to empower engineers to deploy their services faster, more frequently with reliability. Martin Fowler, in his book Continuous Delivery, points out that “The biggest risk to any software effort is that you end up building something that isn’t useful. The earlier and more frequently you get working software in front of real users, the quicker you get feedback to find out how valuable it really is.”

Recently, with the popularity of cloud services and the container technology, engineers have even more options in choosing the infrastructure model, the cloud services, which are most suitable for their team’s requirements. At CyberAgent, we have a large number of services from many teams where each team can have a different infrastructure model and a different cloud service. Some big projects also deploy their services on multi-cloud. This diversification leads to a problem that we are facing, lacking a robust CD system for all teams.

So we decided to create a new CD system that provides a unified interface for many application kinds to improve the developer experience.

Key Features

While designing PipeCD, we focused on the following 4 key features with the aim of creating a CD system that provides a good experience for both developers and operators.

Visibility

Visibility is one of the most requested factors we received when surveying engineers in our company. Visibility is the ability to see what’s going on in the cluster, the ability to see how each component of the application had been deployed, the ability to know quickly why the deployment was failed. Visibility for a team leader is the ability to know the delivery performance of the team and what metrics should be improved. With PipeCD, we always strive to maximize the visibility for engineers, operators as well as team leaders. Currently, it includes:

Deployment pipeline UI shows clarify what is happening
Separate logs viewer for each individual deployment
Realtime visualization of application component and state
Deployment notifications to slack, webhook endpoints
Insights show the delivery performance

In addition, the entire state of the service is managed through Git, so you can view the whole state of the cluster and all audit logs provided by Git.

Automation

Automation reduces or removes repetitive overhead of frequent releases. So maximizing automation helps to minimize human error during the deployment process, as well as reduce the amount of work engineers need to do. PipeCD has the following automated functionalities:

Automated deployment analysis based on metrics, logs, emitted requests
Automatically roll back to the previous state as soon as analysis or a pipeline stage fails
Automatically detect configuration drift to notify and render the changes
Automatically watch and detect the new container images to deploy

Secure

The CD system often carries a lot of credentials needed to access the cluster and the necessary services of the teams. Ensuring the safety of the teams is always on our top priority. So while designing the PipeCD, we decided not to store those credentials in a central place. Instead of that, all user’s credentials always stay inside their clusters.

Support single sign-on and role-based access control
Credentials are not exposed outside the cluster and not saved in the control-plane
Piped makes only outbound requests and can run inside a restricted network

Multi-provider & Multi-Tenancy

Multi-provider means supporting multiple cloud services, multiple container registries, multiple monitoring services for doing deployment analysis. You can use PipeCD to deploy your Kubernetes applications, CloudRun, AWS Lambda application and Terraform application. It also supports doing progressive delivery with canary and blue-green strategy.

Easy to operate multi-cluster, multi-tenancy by separating control-plane and piped
Support multiple analysis providers including Prometheus, Datadog, Stackdriver, and more
Support multiple application kinds on multi-cloud including Kubernetes, Terraform, Cloud Run, AWS Lambda

While designing PipeCD, we simplified its architecture by minimizing the number of components, so you do not have to install many things to enable all features. In addition, PipeCD also supports storing data in several fully-managed services to minimize the operating cost.

Currently, we have completed the basic features and many of the features are in the alpha status. We are working hard to increase the stability and planning to release a stable version in the next months.

Community

PipeCD team hopes to receive a warm welcome and feedback from the open-source community. We value every contribution and invite you to join us on GitHub, Slack and Twitter.

Visit our website and documentation at https://pipecd.dev
Check out the code at https://github.com/pipe-cd/pipecd or explore the examples
Join us on Slack @cloud-native/pipecd to chat with other developers
Follow us on Twitter @pipecd_dev to get the latest news

PipeCD team is hiring engineers/interns to work on PipeCD. Please contact us if you are interested.

Thanks

Finally, we would like to thank the existing open-source CD projects like Spinnaker, FluxCD, ArgoCD… PipeCD has been built on many great ideas from those great projects. PipeCD team would also like to thank CyberAgent’s engineers and collaborators from other companies, who have sent us so much valuable feedback throughout the development process to this day.

PipeCD – PipeCD

Blog: What is new in pipedv1 (plugin-arch piped)

Overview

New in pipedv1

plugins configurations

Pipeline planning and configuration

Stages configurations

Deprecated / Dis-supported features

“Weird” flag --enable-default-kubernetes-cloud-provider in piped execution

Kubernetes templating feature

Analysis stage query templating feature

Summary

Blog: Plugin architecture Piped alpha version has been released

Prerequisites

1. Setup Control Plane

2. Run plugin-arched piped

3. Deploy an application

See also

And more

Blog: MCP Server for PipeCD Docs Has Been Released

Introduction

Example

Usage

MCP Tools

Development Highlights

Conclusion

Blog: What Happens before Deployment? GitOps Mechanism

Overview

1. Trigger

[A] Detecting changes in Git

[B] Manual trigger from UI/pipectl

[C] Automatic deployment on drift detection (requires configuration)

Trigger code

2. Plan: Determine Pipeline

2.1. Determining Quick Sync or Pipeline Sync

What are Quick Sync and Pipeline Sync?

Strategy determination logic

2.2. Determining Stages

Planner code

3. Schedule: Execute Pipeline

3.1. Executing Each Stage of the Pipeline

3.2. Executing Rollback on Failure

Scheduler code

Appendix: What is the controller?

Conclusion

Blog: Introduction to EventWatcher: Connecting CI to PipeCD

Background: Why EventWatcher is needed

What is EventWatcher?

Mechanism

Usage

1. Piped Configuration

2. app.pipecd.yaml Configuration

3. Event Triggering via pipectl or GitHub Actions

Advanced: --contexts Option

Appendix: Code Reading

Conclusion

Blog: The Gap Between CI/CD and How PipeCD Appeared to FulFill the Gap

GitOps and CICD

EventWatcher as the bridge between CI and CD

The issue with EventWatcher

It’s good, but it’s not enough

Conclusion

Blog: Overview of the Plan for Pluginnable PipeCD

Summary

What’s Pluginnable PipeCD?

Role of Plugins

Where Plugins come from?

How does a Plugin run?

How to develop a Plugin?

Advantages of Pluginnable PipeCD

1. Users can deploy to platforms other than K8s/CloudRun/Terraform/ECS/Lambda

2. Users can choose behaviors as they want

3. Users can develop and use custom Stages

Schedule overview

Conclusion

Blog: Performance improvement in Git operations on PipeCD v0.48.9

Background

Problem

Solution

Conclusion

`plugins` configurations

“Weird” flag `--enable-default-kubernetes-cloud-provider` in piped execution

Advanced: `--contexts` Option