DEV Community: DevStream

App-Centric Configuration: Adding More Value to Your Life

Tiexin Guo — Wed, 30 Nov 2022 01:55:06 +0000

DevStream's Story

About nine months ago, DevStream was first publicly released. Since then, it has evolved a lot. If this is the first time you have come across DevStream, maybe read this blog for a quick overview.

In the current incarnation (v0.9), every single DevOps tool (including integrations of tools and CI/CD pipeline setup for apps) is treated as a "Tool", which is a DevStream concept. So, if you would like to define a DevOps platform that does the following:

repository scaffolding
continuous integration w/ GitHub Actions
install Argo CD as the tool for continuous deployment
continuous deployment w/ Argo CD

Your DevStream tools.yaml config file would look similar to this:

tools:
- name: repo-scaffolding
  instanceID: myapp
  options:
    destinationRepo:
      owner: [[ githubUser ]]
      repo: [[ app ]]
      branch: main
      repoType: github
    sourceRepo:
      org: [[ githubUser ]]
      repo: dtm-scaffolding-python
      repoType: github
    vars:
      ImageRepo: [[ dockerUser ]]/[[ app ]]
      AppName: [[ app ]]
- name: githubactions-python
  instanceID: default
  dependsOn: [ repo-scaffolding.myapp ]
  options:
    owner: [[ githubUser ]]
    repo:  [[ app ]]
    language:
      name: python
    branch: main
    docker:
      registry:
        type: dockerhub
        username: [[ dockerUser ]]
        repository: [[ app ]]
- name: helm-installer
  instanceID: argocd
- name: argocdapp
  instanceID: default
  dependsOn: [ "argocd.default", "githubactions-python.default" ]
  options:
    app:
      name: [[ app ]]
      namespace: argocd
    destination:
      server: https://kubernetes.default.svc
      namespace: default
    source:
      valuefile: values.yaml
      path: helm/[[ app ]]
      repoURL: ${{repo-scaffolding.myapp.outputs.repoURL}}

A quick explanation - what DevStream does with this config is the following:

install Argo CD;
repository scaffolding for myapp;
CI/CD for myapp.

It looks nice and works like a charm.

However, suppose you have more than one app to worry about (more often than not, that would be the case in the microservice era). In that case, things start to get a bit complicated: you'd have to repeat the githubactions-golang and argocdapp sections as many times as the number of apps you manage.

That is to say, to manage ten apps/microservices, your DevStream YAML file may grow to well past 300 lines of config.

Today, we are happy to announce that it's no longer the case. We've simplified it. By a big margin. With the release of v0.10.

Read on.

Adding Value to Your Life

A big config is definitely harder to read and maintain, and we don't like that. That's why we decided to solve this problem in the first place.

However, we waited to get right into it.

Instead, we started to think about our market positioning and value proposition. What is DevStream anyway? What problems does it solve? Why would other engineers want to use it instead of building stuff manually or purchasing one-stop DevOps platforms?

If we could precisely answer these questions, we would know how to improve it to the next level.

The discussion went on and on. The whole team, including our CEO, discussed multiple days. We've put tens of hours of thought and debate into it, until we figured out where exactly DevStream can add value to your life:

installing DevOps tools (but that's not the point at all)
integrating DevOps tools and building engineering platforms (now we're talking)
application lifecycle/software development lifecycle management:
- repository bootstrapping
- continuous integration pipelines, integrating best practices, typical stages, and steps into it
- continuous deployment pipelines, integrating helm chart/Dockerfile best practices
- ...

Lightbulb: Apps? Apps!

Since many values DevStream could bring are around applications/microservices and their lifecycle management, why not simply build configurations based on that?

We call this new thought "app-centric" configuration, and let's get right into it:

Backward Compatibility

First things first, the previous "tools" config still works. We wouldn't want to break that. That is to say, if you copy-paste the first code snippet from the beginning of this article, it's supposed to work without any problem, just like previous versions.

(Much) Shorter, Simpler, Easier, While Achieving the Same

You'd be surprised that the config below does exactly the same thing as the code snippet at the very beginning of this article.

apps:
- name: myapp
  spec:
    language: python
    framework: django
  repo:
    url: github.com/devstream-io/myapp
  repoTemplate:
    url: github.com/devstream-io/dtm-scaffolding-python
  ci:
    - type: githubactions
  cd:
    - type: argocdapp

Is this magic? How'd we do that?

First, we created a new abstraction that is called "apps". Each app corresponds to a real-world application or microservice that you manage.

Secondly, we simplified the configurations as much as possible with five small sections:

spec: language, framework related to the application, which is shared information with other sections
repo: where to create/bootstrap the repository for the app
repoTemplate: the template used to bootstrap the app's repo
ci: setting up continuous integration for this app
cd: setting up continuous deployment for this app

Last but not least, more "defaults" and "best practices" are integrated, by default, into DevStream, so that you don't have to override most of the configs.

Neat, right? I know.

One File to Rule Them All

Putting It All Together:

config:
  state:
  backend: local
  options:
    stateFile: devstream.state

tools:
- name: helm-installer
  instanceID: argocd

apps:
- name: myapp1
  spec:
    language: python
    framework: django
  repo:
    url: github.com/devstream-io/myapp1
  repoTemplate:
    url: github.com/devstream-io/dtm-scaffolding-python
  ci:
  - type: githubactions
  cd:
  - type: argocdapp
- name: myapp2
  spec:
    language: golang
    framework: gin
  repo:
    url: github.com/devstream-io/myapp2
  repoTemplate:
    url: github.com/devstream-io/dtm-scaffolding-golang
  ci:
  - type: githubactions
  cd:
  - type: argocdapp

I hope you enjoy this new feature. Have fun experimenting tools together with apps!

DevStream 0.9.0 Release

Tiexin Guo — Fri, 16 Sep 2022 01:09:23 +0000

Official Releases for Different Platforms

Major Changes since Last Release

This version focuses on two things:

refactoring
making the Jenkins/GitLab/Harbor/Java combination prod-ready

Here we highlighted a few changes in this release:

Core

helm-type plugins now support local charts
there is a new "force" flag to dtm destroy command, thanks to @csonezp
k8s backend is supported for storing state

Plugins

GitHub Actions for Python is enhanced
Jenkins/GitLab/Harbor plugins are greatly enhanced, including documentation, features, default values, etc.
ArgoCD plugin now treat namespace in an "upsert" manner
new plugin: harbor
added many default options for each plugin

Docs

Install dtm with asdf in quickstart, thanks to @zhenyuanlau
GitLab + Jenkins + Harbor On Premise Toolchain best practice (in Mandarin)
There is a new document (in both English and Chinese) detailing how to setup a local development Environment for Golang/K8s related projects (not specific to DevStream)

See the full changelog here.

DevStream Certification Program Introduction

Tiexin Guo — Thu, 21 Jul 2022 08:12:37 +0000

Hello, and welcome to another DevStream blog post!

In today's post, DevStream is happy to announce the new certification program for our contributors!

0 Background

For veteran contributors dating back to the beginning of this year, I'm sure you still remember our original certification, which is a well-designed digital image that certifies your efforts in the open-source community.

As beautiful as it is, there are some drawbacks with a certification in the form of digital images. To name a few:

The image is pretty big, making it hard to share on social media platforms.
Since it's an image, it's not so easy to embed it in your profile page, for example, LinkedIn profile, GitHub profile, or even inside your CV.
It's not easy for others to verify the authenticity of the certification.
Others don't know much about what you did to get that achievement, what skills you already mastered.

1 Introducing the New Digital Certification Program

With those pain points in mind, we sought out a new form of credential/certification system that would address those very drawbacks, and it wasn't long before we got ourselves an answer: Credly.

TL;DR: Credly helps issue digital badges.

OK, that was brutally brief, isn't it? So, let's read the long version:

2 The Credly Story

Credly was founded to help people connect their verified abilities to opportunities. And, we strive to bring equity and access to every member of the current and future workforce. After leading the transformation in how people learn and connect online, our team turned its sights on bringing innovation to the outputs of meaningful learning experiences: the credential itself.

In 2018, Credly acquired the Acclaim credential business from Pearson and became the most comprehensive global solution for recognizing skills, capabilities, and achievements. Pearson invested in Credly’s growth and in 2022, acquired Credly to provide powerful solutions in the global workforce learning and talent market.

Credly is leading the digital credential movement, making talent more visible and opportunity more accessible.

Credly works with leading organizations that share our focus on unleashing the potential of the workforce with digital credentials.

3 Why Did DevStream Go with Credly

No brainer: it seems 95% of the top IT certifications are issued through the Credly network.

Plus, I got to know Credly because AWS uses Credly for its certifications. Have a quick look at this beautiful wall of AWS certifications you can get:

Checkout my profile on Credly:

4 Wait a Minute. So I Got My Certs. Where Can I Show Them Off?

GitHub profile:

LinkedIn profile:

Tweet all you want, and DevStream will try our best to retreat your tweets!

Spoil alert: DevStream is redesigning our community page on our official website. I know, I know, it looks ugly; for now. It will be pretty, soon. You have my words. Contributors and their certifications will show up on the community page. Think of it as a "wall of fame" of DevStream. How cool is that!

5 Where Do We Go From Here

Earn

We will issue a digital badge (and we will try to automate the process) when you earn a DevStream certification. Digital badges are active as long as your certification is valid.

Claim

You’ll get an email from Credly’s platform with instructions for claiming your badge.

Use Credly’s platform sharing features to post on your social media newsfeeds and add your digital badge to your professional profile, as mentioned above.

DevStream Joins CNCF Sandbox

Tiexin Guo — Thu, 16 Jun 2022 03:54:59 +0000

On Jun 14, 2022, in the CNCF TOC Meeting, DevStream passed the technical oversight committee's review, and joined CNCF sandbox!

Background

DevStream dated back in Oct 2021 and released the first minor version v0.1.0 in Feb 2022. Now, DevStream is a big family with 26 contributors and hundreds of users in our user group. We've attracted some attention, too. Now, we are on our way to 400 GitHub stars, which is all organic growth!

Thank You

Here, I, Tiexin Guo, PMC Chair of DevStream, representing our whole team, would like to say a heartfelt "thank you" to the community.

Joining the CNCF wouldn't be remotely possible if it weren't for you guys: all the users, contributors, and organization members.

Here in the DevStream community, we often ask ourselves this question: what is DevStream? What is DevStream's core value? What defines DevStream? An open-source DevOps toolchain manager? Setting up your DevOps platform at the click of a button within 5 minutes?

While those are all true, they don't define DevStream.

You define DevStream. You are DevStream. DevStream wouldn't be a thing without our passionate community.

The CNCF TOC Review

Oh, by the way, we didn't just pass the TOC review, or meet the bare minimum requirements. It's proud and fair to say that we passed with flying colors.

Normally, the CNCF TOC meeting is held every two months, and they review a bunch of projects in that one-hour meeting. Some projects are scrutinized; it's not uncommon to see the TOC carefully review a project for about 10 minutes.

We passed within 2 minutes and 10 seconds. Yep, that's not a typo :) See those glorious two minutes here on youtube. Or, you can choose to watch the whole video if you would like to know more about how CNCF TOC meeting works:

The review of DevStream starts from 41:48.

The result is officially documented here; @amye has also made a PR already to have DevStream listed.

What We Did Right

This honor belongs to the community, and here I'd like to give back to the community a short list of actions that we think we did right from the beginning and accelerated everything:

good documentation
good roadmap
some main contributors
consistent contributions from the community
community meetings
a slack channel (where the contributors communicate asynchronously)

In the doc, it's also critical to have some form of developer guide that lowers the bar and reduces the barriers for new contributors. After all, open-source is all about community.

Summary

In the next blog post, I will try to write a short FAQ which I summarized from CNCF TOC meetings. I will try my best to answer questions like:

What exactly will the TOC meeting review?
What aspects are important?

Stay tuned. See you in the next one!

A Brief Introduction to Code Review: Everything You Want to Know

Tiexin Guo — Mon, 06 Jun 2022 05:09:03 +0000

This is Tiexin Guo, DevStream PMC Chair (an open-source DevOps project with an enthusiastic community.)

And today, we are going to talk about code review.

Specifically, we are going to talk about:

why do we need to do code reviews
how to review;
how to do code review on GitHub
how to review for open-source projects

But if you want to get acquainted with some best practices for reviewing code of any kind (closed-source projects, on GitLab, etc.,) please do read on, because the principles and methodologies are universal.

Without further adieu, let's get started.

1 Why Do We Need to Do Code Review?

I know you might already have a question in your head even before reading the title of this post: why do we need to do a code review?

An excellent question, indeed. Because once you feed the code into a compiler, the result will always be the same. Why on earth do we need to bother reviewing code and writing clean code, anyway?

Because code is for humans to read, not for machines to run.

"Clean or not" makes no difference to a machine, but it makes all the difference for humankind. Cleaner code is much easier to organize, refactor, add new features into, debug, to ...

Adding a new feature into a bad codebase could easily cost 100x the time compared to a clean codebase. If it's possible at all.

So, the answer is easy: code review is for code quality.

2 Why Not Using Tools and Automation for Code Quality?

There are already a bunch of tools or third-party services which can help you to evaluate the quality of a piece of code (don't ask me how I knew this.)

For example, GitHub already showed how many lines of code changes there are in a PR. There are tools to tell you what is a "big" function, a "large" file; how many "big" functions and "large" files you've got in your repo; their percentage; etc.

But hey, you can invent a million metrics, and they are still only "indicators" of code quality; they don't "define" good code.

For example, in my previous company, we used an in-house tool that would check if the length of a function exceeds 50 lines of code. Why 50? why not 40, or 60? If it's 51, is it "clean"? If not, how about I delete a necessary empty line intended for separating two blocks? How about I delete a necessary comment?

No metric is perfect, and there are a bunch of ways to hack it and bypass it.

The only golden standard of code quality is for humans to review. Because code is meant to be read by humans.

3 Code Review: What Are the Standards?

Once we've figured out why we need to do it, we need to answer this not-so-technical question: what kind of code needs to be refactored? What kind of code needs to be rewritten? What can be merged? Anyway, what standard should I use?

I know what you are thinking, and you are right: for code with obvious bugs and errors, you can't merge'em. This isn't in the scope of our discussion.

How about code that runs properly, does what needs to be done, but has a few "defects," like "missing necessary comments," or "not easy to read/understand?"

3.1 New Contributors

I'd like to make it clear: for new contributors, it's absolutely fine if they didn't meet your standard (in the beginning.)

For new contributors, it's key to let them feel they are welcome, and there isn't much that hinders their contribution. Easy to understand: if it's extremely difficult to contribute, they won't. So, the thing is: don't let new contributors feel frustrated. We need to make the process as welcoming as possible for newcomers. Nothing is more important than letting new contributors feel easy and welcome.

But, this isn't to say that we tolerate "dirty" code. If a new contributor made a commit that isn't perfect to your standard, guiding them and helping them to learn is the best way, and we should do it encouragingly. They will learn and next time they won't make the same mistakes again. We can even lead by examples, and give them sample code snippets, or even create another improvement commit and let them know.

3.2 Senior Members

For senior contributors and members, we intend to keep the quality high.

It's not enough if a PR merely does what it's supposed to. The readability should be fine, too. Evidently, a quick review reduces the time of code review and we could release a new feature quickly, but in the long run, careless reviews would give us a run for our money: as code quality decreases, it would take longer and longer to do even the most simple task.

To sum up, we should:

keep it simple and welcoming for new contributors;
keep the bar high for ourselves, core developers, and community members.

4 Working in an Open-Source Project

Last month, 3 more contributors joined the DevStream community. Circled (or to be precise, rectangulared) in red:

Now, we have seven active members in total. Plus, we've also got c.a. 20 external contributors. Yep, that's not a typo; you read it right: we've got more than 20 contributors, already.

As a leader in an open-source project, it's not easy managing an open-source project, coding, reviewing all those pull requests (PR) from the community every day, AND still having a life (don't ask me how I knew this.)

I guess this isn't uncommon for many project leads. Yes, you need to manage priorities properly and get things done, but there is a line when you simply have too many things to do and can't finish them all.

Time to have a "reviewers" group and start "group" reviews! Simple answer, isn't it?

But how?

What's the process? What's the standard? Any tricks to speed it up while keeping the quality high in the air? A million questions to be answered.

OK, let's try to tackle some of the important ones in this blog post.

5 Taking up a Code Review Task

Let's do a code review, now.

Before start reviewing anything, let's accept a quest first.

Why do we need to do this? I'm glad you asked.

The open-source community works asynchronously. That is to say, they don't do too much live communication, like phone calls, video meetings, or even offline, face-to-face meetings. Instead, they rely heavily on messages (by using instant message tools, but probably not reading them instantly) and emails. You need to let other people know that you are on this task so that others won't do the same thing.

Even if you are not in an open-source project, for example, in a closed-source, internal project with a team sitting together around the table, it's also nice to let others know that "hey guys, I'll work on this, please don't do duplicated work."

OK, then how to mark a task as yours?

If you are already the reviewer of a PR, the "Reviewers" section will show your avatar, and there will be some hint in the yellow bar saying that "This pull request is waiting on your review." There is also a green button "Add your review." Click this button to start a reviewing process.

Then how to become a "reviewer"?

Easy question (and not easy at the same time:)

Since you are a reviewer, you have permission to click the gear "⚙️" icon to assign a reviewer (for example, yourself.) But if you are not a legit reviewer yet, you should become a reviewer first.

6 GitHub Code Review Walk-Through

Click the "Add your review" button and we will be directed to the Code Review page:

Here, we have a few features worth exploring:

the "file tree" on the left
file filter
the "Viewed" checkbox for each file
the "..." button

Once you've gotten familiar with all of those quirks and features, your code review life quality will be greatly improved.

To read more details about these, check out the official doc here. Explore on. I'll wait for you to finish that doc first.

For simple code changes, it's more than enough to view the code diff on the code review web page. Example:

We could easily tell if this change makes sense or not (hint: No. Missing comma. Try to find it :) If it's fixed, we can approve it:

7 Reviewing a PR Locally

For more complicated PRs, if the author also didn't include any screenshot of the test they had run (see this example here,） it's better to check out the code locally and review it with the help of an IDE.

Take this PR for example: pr #641. To download this PR locally, let's execute:

git fetch upstream pull/641/head:pr641
git checkout pr641

Results:

Then we simply start our IDE and read the code there:

In IDE, it's much easier to spot possible mistakes (like missing a comma.) And, you can also build, test, and run it locally to see if it passes the unit tests and even integration tests.

You might think this is redundant because a PR might have triggered a CI process already:

But hey, not all CI workflows are born equally. For example, not all CI workflows would run integration tests. So, testing a PR locally is still necessary for bigger PRs and for more complicated changes, to do a quick regression.

So far, you can fully understand what a PR has changed.

8 Committing to Somebody Else's PR

A general rule of thumb: don't.

If you want to do it very badly, rethink your intention. Ask yourself: do you have to do this?

Normally, it's not recommended to commit to other people's work directly. It's not respectful to the original author, and it's complicated (especially if the original author needs to add another commit.)

But, rules are to be broken. Like every other single rule in the world, there are exceptions to this rule.

Taking this PR for an example:

It's from a new contributor. We don't want to make "contributing to the project" too hard, which would shut the door and scare too many potential contributors away.

Plus, this PR isn't simple enough to get it 100% right. It seems easy, but to make it perfect, you'd need to know all dtm sub-commands, which certainly is a high bar to cross for a first-time contributor.

That's why we added another commit to improve his work (important: and also let him know about it,) and included the test result:

How to do this exactly?

First, make a local commit.

Make changes, then commit:

git add xxx
git commit -m "bug: yyy"

Then, we need to find the PR's source branch.

We can find out from which branch this PR comes from:

Now we see the "fork" info, like:

Then we can add a new remote:

git remote add himku [email protected]:himku/devstream.git

remote: himku([email protected]">[email protected]:himku/devstream.git)
branch: fix-issue-559

Then we push to it:

git push himku HEAD:fix-issue-559

Cool, right? But again: rethink before you want to do this; because more often than not, you don't necessarily want to do it.

9 Summary

Using AWS S3 to Store DevStream State

Tiexin Guo — Mon, 30 May 2022 09:38:25 +0000

In our latest release v0.6.0, using AWS S3 to store DevStream state is supported.

In this blog, we are going to demonstrate the usage of AWS S3 to store DevStream's state files.

Terminology

State: if you don't already know about DevStream state, please read this doc and this blog helps, too.

Backend: where to actually store the state file. It can be either local or s3 at the moment.

Configuration

In the main config file, we have a section to configure the state. Currently, local and S3 are supported.

Local example:

varFile: variables-gitops.yaml

toolFile: tools-gitops.yaml

state:
  backend: local
  options:
    stateFile: devstream.state

S3 example:

varFile: variables-gitops.yaml

toolFile: tools-gitops.yaml

state:
  backend: s3
  options:
    bucket: devstream-remote-state
    region: ap-southeast-1
    key: devstream.state

Config File Examples

In this demo, we use the following configs:

config.yaml:

varFile: variables-gitops.yaml

toolFile: tools-gitops.yaml

state:
  backend: s3
  options:
    bucket: devstream-test-remote-state
    region: ap-southeast-1
    key: devstream.state

variables-gitops.yaml:

githubUsername: IronCore864
repoName: dtm-test-go
defaultBranch: main

dockerhubUsername: ironcore864

argocdNameSpace: argocd
argocdDeployTimeout: 5m

tools-gitops.yaml:

tools:
- name: github-repo-scaffolding-golang
  instanceID: default
  options:
    owner: [[ githubUsername ]]
    org: ""
    repo: [[ repoName ]]
    branch: [[ defaultBranch ]]
    image_repo: [[ dockerhubUsername ]]/[[ repoName ]]

Getting Started

Before reading on, now is a good time to check if you have configured your AWS related environment variables correctly or not.

For macOS/Linux users, do:

export AWS_ACCESS_KEY_ID=ID_HERE
export AWS_SECRET_ACCESS_KEY=SECRET_HERE
export AWS_DEFAULT_REGION=REGION_HERE

For more information, see the official document here.

Apply

Then let's run dtm apply:

tiexin@mbp ~/work/devstream-io/devstream $ ./dtm apply
2022-05-30 17:07:59 ℹ [INFO]  Apply started.
2022-05-30 17:07:59 ℹ [INFO]  Using dir <.devstream> to store plugins.
2022-05-30 17:07:59 ℹ [INFO]  Using s3 backend. Bucket: devstream-test-remote-state, region: ap-southeast-1, key: devstream.state.
2022-05-30 17:08:00 ℹ [INFO]  Tool (github-repo-scaffolding-golang/default) found in config but doesn't exist in the state, will be created.
Continue? [y/n]
Enter a value (Default is n): y

2022-05-30 17:08:08 ℹ [INFO]  Start executing the plan.
2022-05-30 17:08:08 ℹ [INFO]  Changes count: 1.
2022-05-30 17:08:08 ℹ [INFO]  -------------------- [  Processing progress: 1/1.  ] --------------------
2022-05-30 17:08:08 ℹ [INFO]  Processing: (github-repo-scaffolding-golang/default) -> Create ...
2022-05-30 17:08:12 ℹ [INFO]  The repo dtm-test-go has been created.
2022-05-30 17:08:29 ✔ [SUCCESS]  Tool (github-repo-scaffolding-golang/default) Create done.
2022-05-30 17:08:29 ℹ [INFO]  -------------------- [  Processing done.  ] --------------------
2022-05-30 17:08:29 ✔ [SUCCESS]  All plugins applied successfully.
2022-05-30 17:08:29 ✔ [SUCCESS]  Apply finished.

As we can see from the output, the S3 backend is used, and it also shows the bucket and key you are using, and in which region this bucket lives.

Checking the State File

After apply, let's download the state file from S3 and check it out:

tiexin@mbp ~/work/devstream-io/devstream $ aws s3 cp s3://devstream-test-remote-state/devstream.state .

And if we open the downloaded file, we will see something similar to the following:

github-repo-scaffolding-golang_default:
  name: github-repo-scaffolding-golang
  instanceid: default
  dependson: []
  options:
    branch: main
    image_repo: ironcore864/dtm-test-go
    org: ""
    owner: IronCore864
    repo: dtm-test-go
  resource:
    org: ""
    outputs:
      org: ""
      owner: IronCore864
      repo: dtm-test-go
      repoURL: https://github.com/IronCore864/dtm-test-go.git
    owner: IronCore864
    repoName: dtm-test-go

which is exactly the same as if we were using the local backend to store state.

Like this quick tutorial? Then I suggest to read more of our latest DevOps blogs:

DevStream v0.6.0 Release

Tiexin Guo — Wed, 25 May 2022 07:47:33 +0000

Demo

Talk is cheap. Show me the demo:

Official Releases for Different Platforms

Major Changes

Plugins now are released to an AWS S3 bucket. dtm init also downloads plugins from S3.
AWS S3 can be used to store DevStream state.

New Contributors

@aeinrw made their first contribution in https://github.com/devstream-io/devstream/pull/456
@lyleshaw made their first contribution in https://github.com/devstream-io/devstream/pull/463
@aFlyBird0 made their first contribution in https://github.com/devstream-io/devstream/pull/479
@wzymumon made their first contribution in https://github.com/devstream-io/devstream/pull/477
@iyear made their first contribution in https://github.com/devstream-io/devstream/pull/498
@SongDunYu made their first contribution in https://github.com/devstream-io/devstream/pull/540
@shubham-cmyk made their first contribution in https://github.com/devstream-io/devstream/pull/536
@yclchuxue made their first contribution in https://github.com/devstream-io/devstream/pull/553
@24sama made their first contribution in https://github.com/devstream-io/devstream/pull/562

Congratulations, and keep up the good work!

Full Changelog: https://github.com/devstream-io/devstream/compare/v0.5.0...v0.6.0

For more details, see the GitHub release page here.

On DevOps: 1. What It Is

Tiexin Guo — Thu, 12 May 2022 02:23:28 +0000

Note: opinions are my own. They don't represent my current company or any previous companies I've worked for.

Note 2: recently, my colleague published a blog post (in Chinese, here's the link if you are interested). I like this post, but I'd like to share more with you on the topic of DevOps. So I decided to publish a miniseries (7 episodes planned) on DevOps. Here's the first article.

Note 3: this article is originally published on Jan 14, 2021, on Medium in my DevOps Chat publication named 4th Coffee. See it here if you are interested. This post is edited again on May 10th, 2022.

Background

I’ve been working as a DevOps engineer since 2016, and as of today in 2022, I'm still working on it (as the PMC of DevStream). In 2021, I was lucky enough to join AWS as a Senior (L6) DevOps (already left, though). But this article doesn’t represent AWS’s view; nor does it represent my current corporate view. It’s my personal opinion, which I learned and formulated over the years of hands-on experience in projects.

In the beginning of 2021, I had the chance to talk on the DevOps topic — what it is, why we need it, and how to measure if we are doing it well or not. So I thought I should take that opportunity, form my thoughts into words, and share them with the community.

In my professional career, I’ve worked in different businesses, including university research, search engine, government, internet, enterprise software, mobile, marketing business intelligence, logistics, automotive, pharmaceutical, insurance, consulting, etc.; and I’ve worked in different types of companies and projects: state-owned company, local internet giant, global IT enterprise, foreign start-ups, global traditional organizations, etc. And I have to say that I have learned and grown a lot during this great journey both as a person and as an engineer (full stack developer and DevOps engineer). I also attended a few conferences and meet-ups and even gave some talks in them, but I have never got the chance to summarize what I do and share the knowledge with a bigger audience, and that’s the reason I wanted to start this series.

What is DevOps?

Let Me Tell You a Story First.

I have interviewed over 200 engineers in the past 3 years, and sometimes I’d like to ask this question. It’s probably meaningless; I only asked it because I want to see what it means to everybody.

And in all the answers I got, it seems most people always start with this sentence: “that’s a good question.” Then the person starts to explain his thoughts.

I’m not a native English speaker, but over the years, I’ve been gradually learning the fact that “that’s a good question” equals “I don’t know.”

It’s a meaningless filler phrase to fill the awkward silence during conversations, and it buys you some time to organize your thought before you answer the question.

Only on infrequent occasions would you hear this sentence because you really had asked an excellent question.

I have also interviewed (as an interviewee) many companies over the last 5 years, both start-ups and large corporations. I also got asked about this quite often, at least more often than I had anticipated. Sometimes, during the conversation, you started to realize that the person or the company who asked this question didn’t really know the answer either; they asked this because they wanted to learn from you and see what new ideas you can bring to the team.

Why Is It So Hard to Define DevOps?

Why is this question hard to answer?

Because, in my opinion, DevOps is a lot.

Let’s Try to Figure It out

Let me quote some definitions first before we discuss DevOps.

Here I quote three definitions, and they are from Wikipedia (because it’s like an internet dictionary), AWS (because we all hear that AWS does good DevOps), and Atlassian (because as a DevOps, we probably use their tools a lot so they might know a thing or two):

Wikipedia:

DevOps is a set of practices that combines software development (Dev) and IT operations (Ops). It aims to shorten the systems development life cycle and provide continuous delivery with high software quality.[1][2] DevOps is complementary with agile software development; several DevOps aspects came from Agile methodology.

(I know, there are a lot of links. You don't have to read all of them, but it's recommended.)

AWS:

DevOps is the combination of cultural philosophies, practices, and tools that increases an organization’s ability to deliver applications and services at high velocity: evolving and improving products at a faster pace than organizations using traditional software development and infrastructure management processes. This speed enables organizations to better serve their customers and compete more effectively in the market.

Atlassian:

DevOps is a set of practices that works to automate and integrate the processes between software development and IT teams, so they can build, test, and release software faster and more reliably.

It seems there are some similarities, which are:

practices
velocity

And there are differences, of course, and let’s see them one by one:

A Combination of Dev and Ops

Wikipedia says it’s a combination of Dev and Ops.

This basically says nothing because even if I knew nothing about this subject, I had already known from the word itself that it combines Dev and Ops.

An excellent question here is: is DevOps really Dev + Ops?

Yes, and No.

Yes, because you would need the skillsets from both Dev and Ops.

No, because simply putting two teams together wouldn’t get you there.

Cultural Philosophies

What AWS’s definition has and others don’t have, is the “cultural philosophies,” and I like it.

Let’s say you have the tools and skillsets from both Dev and Ops. Would that be enough to enable you to achieve higher velocity?

Probably not.

The fact is, you already have Dev and Ops teams and the corresponding skillsets in your org. Then you pick up a new CI tool that is famous in the DevOps world. Would you be able to deliver faster already?

I don’t think so.

This is because how we do things is more important than what we have, and by definition, “how we do things” is the cultural part.

Culture

Culture is how human societies behave. What the customs are, what the norms are.

For example:

What is right, what is wrong?
What is acceptable, and what isn’t?
How do you learn?
And more importantly, how do you encourage the team to learn?

Culture can be highly technical. For example: how do you do code review? What types of code are acceptable, and what are not?

And it can also be highly nontechnical. For example: what standards do we use for hiring and promoting?

DevOps Culture

Then, what exactly is the DevOps culture?

You probably see this image (or something very similar to this) a lot. And to be honest, the DevOps culture is hidden in this picture:

You don’t separate Dev and Ops. You have a shared understanding and shared responsibility. You take up the responsibility as a whole through collaboration and communication. And this is the cornerstone of the DevOps culture.

Is DevOps a role?

With the culture part explained, it’s easy to answer this question.

Yes, but no.

Yes, because companies create job titles like this.

No, because companies that created the job titles like this only put Ops in the DevOps team in many cases.

It’s not really a job role. It’s more of a culture, methodologies, and mindset to do things. Putting teams together and throwing in some new tools won’t give you DevOps.

What is Required to Achieve the DevOps Culture?

Because we have shared understanding and shared responsibilities, traditional Dev needs to learn from Ops, and traditional Ops need to learn from Dev.

Because we have new practices and tools, everybody needs constant learning about methodologies and new technologies.

So, how to achieve a DevOps culture? At this point, this question can answer itself:

DevOps requires a “growth and continuous learning mindset” to succeed.

How to Enable the Growth and Learning Mindset

Gartner, the globally leading research and advisory company, predicted in 2019 that by 2022, 75% of DevOps initiatives would fail to meet their expectations; this number grows to 90% by 2023.

Reason: limitations of management approaches used by leadership

Leadership support is one of the critical elements of its success.

Motivation

Let me tell you another story that you probably will relate to:

I’ve worked in many start-up companies, and it seems many of those start-ups share a “start-up culture”: free breakfast, and free beer in the fridge.

The idea behind this is simple: by creating a more welcoming environment, they think employees will be more motivated.

Do I love free beer? Of course.

Will this really motivate me? No.

When I search for my next job position, I will not use “free beer in the fridge” as a standard. People who get paid at least 10k per month probably wouldn’t care if they can get 2 dollars worth of beer.

Having a free beer in the fridge won’t motivate your team to learn and grow continuously.

What Motivates the Team to Grow

Technical people need 3 things to be motivated, and they are:

Mastery
Autonomy
Purpose

The reasons are easy to understand:

When you have comprehensive knowledge or skill in a subject and have great accomplishments, you will be more motivated. It’s like that if you get more praise for your work in the school, you will be more determined to do it better next time.

The team learns fast and wants to do greater things. If they are constantly blocked and slowed down, especially by leadership, they lose their motivation. It’s like driving a Porsche sportscar at 50km/h. You have the best tool, but you can’t speed up. You wouldn’t enjoy driving like that, and you lose interest in driving over time.

The team needs to know why they are doing something. Because without a purpose, everything is just garbage.

Leadership

Once we figured out the culture, and what motivates the team to implement the culture, it’s easy to figure out what we should seek in leadership: we need necessary behavioral characteristics that empower and encourage the team, rather than technical skillsets and expertise, for example, if the leader is a professional in the cloud or 12-factor app or what have you.

Leadership should encourage the team to learn fast, try fast, and fail fast because by embracing failure as a learning opportunity, you learn the fastest way.

Leadership should empower the team, and give the team enough autonomy to make decisions free of second-guessing. I’ve seen exactly the opposite cases very often, and the leadership doesn't even know they are discouraging the team.

Leadership should provide clear goals and key metrics as part of the motivation for the team’s continuous learning and growth mindset.

IT Enables Business

If you go back to the beginning where I quoted AWS’s definition of DevOps, you will see that AWS mentions something that neither Wikipedia nor Atlassian mentioned:

This speed enables organizations to better serve their customers and compete more effectively in the market.

Yes, all definitions mentioned DevOps' benefit being velocity, but only AWS explained why you need this velocity: to serve your customer.

I like it because, without this purpose, DevOps is just an empty gesture.

In modern days, more and more organizations start to realize that IT architecture not only serves the business because it’s a must-have but also enables more possibilities for new businesses and growth. And that’s where the DevOps’s benefit will bring you to.

Summary

After this long discussion, it’s safe to try to make a definition of DevOps:

DevOps is a combination of:

a shared responsibility model
continuous growth and learning mindset
skillsets from both Dev and Ops
best practices
modern toolchains

that increases velocity to deliver faster, get customer feedback faster, and adapt your product and service faster to serve your customers better.

In the next article or a few articles, I will discuss the benefits that DevOps brings.

Getting Started with minikube

Daniel Hu — Sat, 23 Apr 2022 15:22:39 +0000

What’s up guys, this is Daniel! Yeah, this English blog is WRITTEN by me, not TRANSLATED by me! It’s my first English blog in my whole life. If you are a Chinese reader, maybe you’ve ever seen my Chinese blogs before. Right, I wrote a lot of blogs, but only in Chinese. Believe it or not, English isn’t my strong suit. But I want to have a try today and I’ll do my best. Let’s get started.

What We Will Talk about Today

If you ask me how to create a local Kubernetes cluster, I'll tell you Kind, minikube, and Kubeadm are all good choices. If you ask me which tool is the most recommended for beginners, my answer must be minikube or Kind. minikube is more powerful than Kind, and Kind is easier than minikube.

In my previous article, I walked you guys through "How to Create a Local Kubernetes Cluster from the Ground Up" with Kind. If you are interested, here's a quick link for you:

Creating a Local Kubernetes Cluster from the Ground Up

You might ask why that article is written by Tiexin Guo, not me. Of course! As I said in the beginning, this blog you are reading now is my first English blog. So, technically, the one above isn’t “my” article, but I did write the Chinese version of it:

从零开始快速搭建本地 Kubernetes 测试环节

Both minikube and Kind are the most well-known and popular choices to run a Kubernetes environment on a local computer. Since I've shown you how to use Kind, I might as well introduce minikube to youadorable today.

Quick Start with `minikube`

minikube is a command-line tool like Kind that you can use to create and configure Kubernetes clusters locally on your computer.

How to Install `minikube`

If your computer is the same as mine, Mac with m1 chip, you can install the latest minikube stable release in 2 steps:

curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-darwin-arm64
sudo install minikube-darwin-arm64 /usr/local/bin/minikube

If you are using Homebrew, just one command:

brew install minikube

If your computer is of other platforms or architectures, see the “Installation” section in minikube start for more details.

`minikube start`

Like the section title we used here, to start using minikube, run minikube start:

$ minikube start
😄  minikube v1.25.2 on Darwin 12.2.1 (arm64)
👎  Unable to pick a default driver. Here is what was considered, in preference order:
    ▪ docker: Not healthy: "docker version --format {{.Server.Os}}-{{.Server.Version}}" exit status 1: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
    ▪ docker: Suggestion: Start the Docker service <https://minikube.sigs.k8s.io/docs/drivers/docker/>
💡  Alternatively you could install one of these drivers:
    ▪ hyperkit: Not installed: exec: "hyperkit": executable file not found in $PATH
    ▪ parallels: Not installed: exec: "prlctl": executable file not found in $PATH
    ▪ vmware: Not installed: exec: "docker-machine-driver-vmware": executable file not found in $PATH
    ▪ virtualbox: Not installed: unable to find VBoxManage in $PATH
    ▪ podman: Not installed: exec: "podman": executable file not found in $PATH

❌  Exiting due to DRV_DOCKER_NOT_RUNNING: Found docker, but the docker service isn't running. Try restarting the docker service.

Ok, it doesn’t matter; we need to install Docker. In my previous article How to Create a Local Kubernetes Cluster from the Ground Up, I introduced how to "Install Docker".

After the Docker is installed, what we'll need in total?

A not too bad computer;
Internet connections;
Container or virtual machine hypervisor, such as: Docker, Hyper-V, KVM, VirtualBox, or VMware Workstation ...

Let's try to run minikube start again. This command is time-consuming the first time you use it, especially if the network speed is low. Feel free to have a cup of coffee then come back to see the result.

2000 years later...

Well Done! 0 error, 0 warning! The logs are beautiful!

$ minikube start
😄  minikube v1.25.2 on Darwin 12.2.1 (arm64)
✨  Automatically selected the docker driver
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
💾  Downloading Kubernetes v1.23.3 preload ...
    > preloaded-images-k8s-v17-v1...: 419.07 MiB / 419.07 MiB  100.00% 8.54 MiB
    > index.docker.io/kicbase/sta...: 28.23 MiB / 343.12 MiB  8.23% 1.34 MiB p/
    > index.docker.io/kicbase/sta...: 343.12 MiB / 343.12 MiB  100.00% 3.77 MiB
❗  minikube was unable to download gcr.io/k8s-minikube/kicbase:v0.0.30, but successfully downloaded docker.io/kicbase/stable:v0.0.30 as a fallback image
🔥  Creating docker container (CPUs=2, Memory=3885MB) ...
🐳  Preparing Kubernetes v1.23.3 on Docker 20.10.12 ...
    ▪ kubelet.housekeeping-interval=5m
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass

❗  /usr/local/bin/kubectl is version 1.21.5, which may have incompatibilites with Kubernetes 1.23.3.
    ▪ Want kubectl v1.23.3? Try 'minikube kubectl -- get pods -A'
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

Access Your Shiny Kubernetes Cluster

Do you have kubectl installed? If yes, you can now use it to access your shiny Kubernetes cluster:

$ kubectl get pod -A
NAMESPACE     NAME                               READY   STATUS    RESTARTS        AGE
kube-system   coredns-64897985d-hdmqd            1/1     Running   0               3m10s
kube-system   etcd-minikube                      1/1     Running   0               3m24s
kube-system   kube-apiserver-minikube            1/1     Running   0               3m25s
kube-system   kube-controller-manager-minikube   1/1     Running   0               3m25s
kube-system   kube-proxy-nkzmn                   1/1     Running   0               3m10s
kube-system   kube-scheduler-minikube            1/1     Running   0               3m23s
kube-system   storage-provisioner                1/1     Running   1 (2m40s ago)   3m22s

Alternatively, minikube can download the appropriate version of kubectl with the command minikube kubectl:

minikube kubectl -- get pod -A
    > kubectl.sha256: 64 B / 64 B [--------------------------] 100.00% ? p/s 0s
    > kubectl: 52.82 MiB / 52.82 MiB [--------------] 100.00% 6.04 MiB p/s 8.9s
NAMESPACE     NAME                               READY   STATUS    RESTARTS       AGE
kube-system   coredns-64897985d-hdmqd            1/1     Running   0              4m32s
kube-system   etcd-minikube                      1/1     Running   0              4m46s
kube-system   kube-apiserver-minikube            1/1     Running   0              4m47s
kube-system   kube-controller-manager-minikube   1/1     Running   0              4m47s
kube-system   kube-proxy-nkzmn                   1/1     Running   0              4m32s
kube-system   kube-scheduler-minikube            1/1     Running   0              4m45s
kube-system   storage-provisioner                1/1     Running   1 (4m2s ago)   4m44s

Cooooool!

Are you thinking this command is too complicated? Use the config below and you'll be happy:

alias kubectl="minikube kubectl --"

I'm used to command-line to access the Kubernetes cluster. Yes, I never use the Kubernetes Dashboard. But if you like to have some insight into your cluster state with a Dashboard, minikube bundles it for you:

$ minikube dashboard
🔌  Enabling dashboard ...
    ▪ Using image kubernetesui/dashboard:v2.3.1
    ▪ Using image kubernetesui/metrics-scraper:v1.0.7
🤔  Verifying dashboard health ...
🚀  Launching proxy ...
🤔  Verifying proxy health ...
🎉  Opening http://127.0.0.1:51801/api/v1/namespaces/kubernetes-dashboard/services/http:kubernetes-dashboard:/proxy/ in your default browser...

At this time, your browser will automatically open a new tab, and the Dashboard will soon show up:

Deploy an Application in Your Shiny Kubernetes Cluster

Let’s create a sample deployment with the image preslavmihaylov/kubehelloworld:latest and expose it on port 3000.

The minikube start told us to use the k8s.gcr.io/echoserver:1.4 to launch the hello-minikube application, but it'll fail on a Mac with m1 chip. See this issue for more details.

$ kubectl create deployment sample-deploy --image=preslavmihaylov/kubehelloworld:latest
deployment.apps/sample-deploy created
$ kubectl expose deployment sample-deploy --type=NodePort --port=3000
service/sample-deploy exposed
$ kubectl get pod
NAME                             READY   STATUS    RESTARTS   AGE
sample-deploy-795887588d-flkgb   1/1     Running   0          50s
$ kubectl get svc
NAME            TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE
kubernetes      ClusterIP   10.96.0.1      <none>        443/TCP          66m
sample-deploy   NodePort    10.107.59.38   <none>        3000:32631/TCP   43s

Then we can use kubectl to forward the port:

$ kubectl port-forward service/sample-deploy 3000:3000
Forwarding from 127.0.0.1:3000 -> 3000
Forwarding from [::1]:3000 -> 3000

Open the browser, visit localhost:3000, and "Hello World!" will soon show up.

Another way to access this service is to use the minikube service sample-deploy command and minikube will launch a browser page for you with the right url:port address.

And we can use the command minikube tunnel to use the LoadBalancer type Service. Here is an example:

$ kubectl delete service sample-deploy
service "sample-deploy" deleted
$ kubectl expose deployment sample-deploy --type=LoadBalancer --port=3000
service/sample-deploy exposed

In another terminal window, start the tunnel to create a routable IP with the command minikube tunnel:

$ minikube tunnel
✅  Tunnel successfully started

📌  NOTE: Please do not close this terminal as this process must stay alive for the tunnel to be accessible ...

🏃  Starting tunnel for service sample-deploy.

Now you can use the EXTERNAL-IP to access the Service:

$ kubectl get svc
NAME            TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
kubernetes      ClusterIP      10.96.0.1       <none>        443/TCP          86m
sample-deploy   LoadBalancer   10.96.205.167   127.0.0.1     3000:31945/TCP   28s
$ curl 127.0.0.1:3000
Hello World!

How to Manage the Cluster

I'll show you some commands here:

minikube pause # Pause Kubernetes cluster without impacting any deployed resources.
minikube unpause # Unpause the Paused Kubernetes cluster.
minikube stop # Remember the minikube start? Need I say more?
minikube config set memory 20480 # Change the default memory limit (requires a restart).
minikube addons list # Browse the easily installed Kubernetes addons catalog.
minikube start -p aged --kubernetes-version=v1.16.1 # Create a second cluster running an older Kubernetes version.
minikube delete --all # Yeah, delete all, delete anything, delete all clusters.

That's all?

No! But if you want to know them all, I'll show you the command docs here.

How to Push Images into a `minikube` Cluster

There are about 8 ways you can use to push images into a minikube cluster. And I'll show you the most useful two ways today.

Push Images with the `cache` Command

minikube cache add [flags] [options] # Add an image to local cache.

Cache? Yes, the image will be cached by minikube and automatically pushed into all clusters, not like the Kind method: kind load docker-image <IMAGE> [IMAGE...] [flags].

Let's see how many sub-commands cache has:

$ minikube cache -h
Add, delete, or push a local image into minikube

Available Commands:
  add         Add an image to local cache.
  delete      Delete an image from the local cache.
  list        List all available images from the local cache.
  reload      reload cached images.

Easy, isn't it?

The only sub-command we need to talk about might be reload. If your image changed after you had cached it, you need to execute minikube cache reload [flags] [options]

E.g.:

minikube cache add alpine:latest # Cache alpine:latest for all clusters.
minikube cache reload alpine:latest # Reload alpine:latest if it's changed.
minikube cache list # Display what images we have added to the cache.
minikube cache delete alpine:latest # Just delete alpine:latest

Pushing Directly to the In-Cluster Docker Daemon

If the image we want to use inside the minikube cluster is built locally instead of pulled from a registry, we can reuse the Docker daemon inside the minikube cluster. Yeah, we don’t have to build the image on our host machine and cache it; we can just build inside the same docker daemon inside the minikube cluster.

$ docker ps
CONTAINER ID   IMAGE                    COMMAND                  CREATED       STATUS       PORTS                                                                                                                                  NAMES
170fe8252660   kicbase/stable:v0.0.30   "/usr/local/bin/entr…"   6 hours ago   Up 6 hours   127.0.0.1:50981->22/tcp, 127.0.0.1:50982->2376/tcp, 127.0.0.1:50984->5000/tcp, 127.0.0.1:50985->8443/tcp, 127.0.0.1:50983->32443/tcp   minikube

$ eval $(minikube docker-env)
$ docker ps
CONTAINER ID   IMAGE                            COMMAND                  CREATED       STATUS       PORTS     NAMES
de4fedaf4ef0   preslavmihaylov/kubehelloworld   "docker-entrypoint.s…"   5 hours ago   Up 5 hours             k8s_kubehelloworld_sample-deploy-795887588d-flkgb_default_2e6b194a-bdc7-44cc-b455-91959a4206a9_0
d45474ce1888   k8s.gcr.io/pause:3.6             "/pause"                 5 hours ago   Up 5 hours             k8s_POD_sample-deploy-795887588d-flkgb_default_2e6b194a-bdc7-44cc-b455-91959a4206a9_0
...

What happened?

After the command eval $(minikube docker-env) is executed, docker accesses the docker daemon inside the minikube cluster! What's the secret with minikube docker-env command? Let's dig into it:

$ minikube docker-env
export DOCKER_TLS_VERIFY="1"
export DOCKER_HOST="tcp://127.0.0.1:50982"
export DOCKER_CERT_PATH="/Users/danielhu/.minikube/certs"
export MINIKUBE_ACTIVE_DOCKERD="minikube"

# To point your shell to minikube's docker-daemon, run:
# eval $(minikube -p minikube docker-env)

I don't think I need to explain more about what happened. And we can build against the Docker inside minikube now. Just execute a command like docker build -t my-image:latest ., my-image:latest is instantly accessible to the Kubernetes cluster.

How to …

Oops, there are many how-tos we could talk about, but I believe that you can start using minikube now. Maybe I'll talk about more how-tos next time. If you want to dive into minikube now, go ahead and study the official documentation.

Summary

Cool, I finished writing this blog, It's like opening Pandora's box; I love this feeling. Of course, I know this blog is at the primary-school-student level, but I believe I can do better and better.

Don’t forget to give a Like, Comment, and Subscribe! I’ll see you in the next article!

Creating a Local Kubernetes Cluster from the Ground Up

Tiexin Guo — Mon, 18 Apr 2022 08:00:25 +0000

From the ground up? Yep, from the ground up!

Overview

Creating a Kubernetes cluster can be tricky.

There are multiple tools designed just for that job. There are even companies that provide "installing K8s" as a service. And, to create a production-ready Kubernetes cluster with all the best practices in mind requires detailed designing and planning.

So, the scope of this article isn't to help you to create a "production-ready" cluster.

However, after reading this article, you should be able to create a local "testing" Kubernetes cluster, and for the developing and testing of DevStream, it should more than suffice.

Even for a local testing cluster, there are multiple tools you can choose from. For example, there is minikube, and there is kind. kind is a tool for running local Kubernetes clusters using Docker container “nodes”. kind was primarily designed for testing Kubernetes.

In this article, we are going with kind. We are not opinionated; we are not saying that kind is better than minikube or vice versa; we are merely choosing a tool to get the job done. If you are more familiar with other tools, it's completely fine!

This article uses macOS as our local development environment. If you are using Windows or Linux, you can still read this post to get a general idea and achieve the same.

Install Docker

Docker works in a way using Linux's Namespace and Cgroup. It's quite easy to install Docker on Linux. On macOS and Windows, Docker runs with virtualization. However, we do not need to worry too much detail here, because it's quite simple to download and run Docker Desktop.

Go to https://www.docker.com/products/docker-desktop, find the correct version of Docker Desktop (Intel/amd64, or M1/arm64):

Double click on the Docker.dmg file, and we see the installation interface like the following:

Simply drag "Docker" to our "Applications," and within a few seconds, it's done! We can start it from the Launchpad:

Wait a few seconds and we can see the starting page:

Click the “gear" ⚙️ icon to change settings about Docker Desktop. For example, if we need to run a lot of containers, we might need to increase the memory. Here, we changed the memory to 4.00 GB:

Remember to "Apply & Restart" to ensure the changes are effective.

Introduction to `kind`

kind (Kubernetes-in-docker) uses a Docker container as a "node" to deploy Kubernetes. It's mainly used for testing Kubernetes itself.

kind is simple, containing a command-line tool named kind and a Docker image which has Kubernetes and systemd. kind uses Docker on the host machine to create a container, which runs systemd, which in turn runs the container runtime, kubelet, and other Kubernetes components. So, we end up with a whole Kubernetes cluster in one container.

Note that although in the explanation above, the Cluster is only a single node cluster, it's possible to create a multi-node Kubernetes cluster.

Creating a Kubernetes Cluster with Kind at the Click of a Button

clone DevStream's repo: https://github.com/devstream-io/devstream
cd devstream; make e2e-up

That's it.

If you check out the Makefile, you will see it actually runs a shell script that runs kind to create the cluster.

However, you wouldn't be satisfied if we end this article right here, right now, would you.

So, let's have a deep dive into kind. Fasten your seat belt, because we are gonna fly!

Creating a Kubernetes Cluster with Kind at Two Clicks of a Button

On GitHub, we can find the latest release of kind: https://github.com/kubernetes-sigs/kind/releases.

Choose relatively new versions, and install:

# method 1: download pre-compiled binary
cd /tmp
curl -Lo ./kind https://github.com/kubernetes-sigs/kind/releases/download/v0.12.0/kind-darwin-arm64
chmod +x ./kind
sudo mv kind /usr/local/bin/

# method 2: go get and compile locally
go get sigs.k8s.io/[email protected]

We can also download the Docker image beforehand. Here we choose v1.22 of Kubernetes:

kindest/node:v1.22.0@sha256:b8bda84bb3a190e6e028b1760d277454a72267a5454b57db34437c34a588d047

Create cluster:

kind create cluster --image=kindest/node:v1.22.0 --name=dev

Sample output:

Creating cluster "dev" ...
 ✓ Ensuring node image (kindest/node:v1.22.0) 🖼
 ✓ Preparing nodes 📦
 ✓ Writing configuration 📜
 ✓ Starting control-plane 🕹️
 ✓ Installing CNI 🔌
 ✓ Installing StorageClass 💾
Set kubectl context to "kind-dev"
You can now use your cluster with:

kubectl cluster-info --context kind-dev

Have a question, bug, or feature request? Let us know! https://kind.sigs.k8s.io/#community 🙂

Follow the hints above, next, let's run kubectl cluster-info --context kind-dev to switch the context and make sure you are in the right Kubernetes context.

$ kubectl get node
NAME                STATUS   ROLES                  AGE    VERSION
dev-control-plane   Ready    control-plane,master   7m4s   v1.22.0

$ kubectl get pod -n kube-system
NAME                                     READY    STATUS     RESTARTS      AGE
coredns-78fcd69978-hch75                  1/1     Running       0          10m
coredns-78fcd69978-ztqn4                  1/1     Running       0          10m
etcd-dev-control-plane                    1/1     Running       0          10m
kindnet-l8qxq                             1/1     Running       0          10m
kube-apiserver-dev-control-plane          1/1     Running       0          10m
kube-controller-manager-dev-control-plane 1/1     Running       0          10m
kube-proxy-mzfgc                          1/1     Running       0          10m
kube-scheduler-dev-control-plane          1/1     Running       0          10m

Now we have a local cluster for testing and developing Kubernetes.

Creating a Kubernetes Cluster with Kind at Three Clicks of a Button

A minimum highly-available Kubernetes cluster is composed of 3 nodes. In this section, let's see how to create a multi-node, highly-available cluster locally using kind.

`kind` Cluster Config

We can pass a config file to kind by using the --config parameter. Let's have a look at a kind config file:

# this config file contains all config fields with comments
# NOTE: this is not a particularly useful config file
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
# patch the generated kubeadm config with some extra settings
kubeadmConfigPatches:
- |
  apiVersion: kubelet.config.k8s.io/v1beta1
  kind: KubeletConfiguration
  evictionHard:
    nodefs.available: "0%"
# patch it further using a JSON 6902 patch
kubeadmConfigPatchesJSON6902:
- group: kubeadm.k8s.io
  version: v1beta2
  kind: ClusterConfiguration
  patch: |
    - op: add
      path: /apiServer/certSANs/-
      value: my-hostname
# 1 control plane node and 3 workers
nodes:
# the control plane node config
- role: control-plane
# the three workers
- role: worker
- role: worker
- role: worker

We can see that the config has two sections: the upper part being kubeadm related stuff and the lower part being nodes related settings. Apparently, the "nodes" part is where we are going to edit to achieve a multi-node cluster.

1 Control Plane Node, 3 Worker Nodes Cluster

Let's create a config named multi-node-config.yaml with the following content:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
- role: worker
- role: worker

Then run:

$ kind create cluster --config multi-node-config.yaml \
 --image=kindest/node:v1.22.0 --name=dev4

We will get some output similar to the previous single-node cluster output, except for the "Joining worker nodes" part:

Creating cluster "dev4" ...
 ✓ Ensuring node image (kindest/node:v1.22.0) 🖼
 ✓ Preparing nodes 📦 📦 📦 📦
 ✓ Writing configuration 📜
 ✓ Starting control-plane 🕹️
 ✓ Installing CNI 🔌
 ✓ Installing StorageClass 💾
 ✓ Joining worker nodes 🚜
Set kubectl context to "kind-dev4"
You can now use your cluster with:

kubectl cluster-info --context kind-dev4

Thanks for using kind! 😊

Let's run the following command to check out our new cluster:

$ kubectl cluster-info --context kind-dev4
Kubernetes control plane is running at https://127.0.0.1:51851
CoreDNS is running at https://127.0.0.1:51851/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
$ kubectl get node
NAME                 STATUS   ROLES                  AGE     VERSION
dev4-control-plane Ready   control-plane,master 3m28s   v1.22.0
dev4-worker          Ready    <none>                 2m54s   v1.22.0
dev4-worker2         Ready    <none>                 2m54s   v1.22.0
dev4-worker3         Ready    <none>                 2m54s   v1.22.0

From the result above, we can see that this cluster has 1 control plane node and 3 worker nodes.

3 Control Plane Nodes, 3 Worker Nodes, Highly Available Cluster

Note: "Highly available" here only means that we have three control plane nodes. It's not strictly "highly available" because apparently, the three control plane nodes are actually on the same host, so when the host is down, everything is gone.

Prepare the ha-config.yaml with the following content:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: control-plane
- role: control-plane
- role: worker
- role: worker
- role: worker

Run:

$ kind create cluster --config ha-config.yaml \
 --image=kindest/node:v1.22.0 --name=dev6

We can see familiar outputs, with the exception being "Configuring the external load balancer” and “Joining more control-plane nodes":

Creating cluster "dev6" ...
 ✓ Ensuring node image (kindest/node:v1.22.0) 🖼
 ✓ Preparing nodes 📦 📦 📦 📦 📦 📦
 ✓ Configuring the external load balancer ⚖️
 ✓ Writing configuration 📜
 ✓ Starting control-plane 🕹️
 ✓ Installing CNI 🔌
 ✓ Installing StorageClass 💾
 ✓ Joining more control-plane nodes 🎮
 ✓ Joining worker nodes 🚜
Set kubectl context to "kind-dev6"
You can now use your cluster with:

kubectl cluster-info --context kind-dev6

Have a nice day! 👋

Some fun facts:

the number of boxes after "Preparing nodes" equals the number of nodes
the final greeting message is different: it was "Thanks for using kind! 😊" previously and now it's "Have a nice day! 👋"

Heck, those kind developers sure went some extra miles to enhance the user experience!

Check the cluster:

$ kubectl cluster-info --context kind-dev6
Kubernetes control plane is running at https://127.0.0.1:52937
CoreDNS is running at https://127.0.0.1:52937/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
$ kubectl get node
NAME                  STATUS   ROLES                  AGE     VERSION
dev6-control-plane    Ready    control-plane,master   8m19s   v1.22.0
dev6-control-plane2   Ready    control-plane,master   7m46s   v1.22.0
dev6-control-plane3   Ready    control-plane,master   7m20s   v1.22.0
dev6-worker           Ready    <none>                 7m      v1.22.0
dev6-worker2          Ready    <none>                 7m      v1.22.0
dev6-worker3          Ready    <none>                 7m      v1.22.0

We can see this cluster has 3 control plane nodes.

So far, we have mastered how to use kind to create a multi-node Kubernetes cluster locally.

Advanced `kind` Features

Now that we know how to create clusters using kind, let's have a look at some advanced operations which could help you better use the clusters.

Port Mapping

Imagine you are running an Nginx container listening on port 8080 in a kind cluster but you wish the outside world (outside of the cluster) to access the Nginx port. To achieve this, we can add the extraPortMappings configuration:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  extraPortMappings:
  - containerPort: 8080
    hostPort: 8080
    listenAddress: "0.0.0.0"
    protocol: tcp

In this way, the port on the pod 8080 is mapped to the port on the host.

Expose `kube-apiserver`

Sometimes we want to install Kubernetes with kind on one host, but access the cluster from another host. By default, the kube-apiserver installed by kind listens on 127.0.0.1 (with a random port.) To make the kind cluster accessible from another host, we need to make kube-apiserver listen on a network interface (for example, eth0.)

In the config file, we add networking.apiServerAddress. The IP is your local nic's IP:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
  apiServerAddress: "192.168.39.1"

Enable Feature Gates

Feature gates are a set of key=value pairs that describe Kubernetes features that are only available in Alpha, Beta or GA stage.

If we want to try some of those features, we can enable Feature Gates. In the config file, use:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
featureGates:
  FeatureGateName: true

Importing Images

Because the Kubernetes cluster is in fact running in a Docker container, by default, it can't access Docker images that are on the host. However, we can import those images from hosts to the Kubernetes cluster created by kind:

# import my-image:v1
kind load docker-image my-image:v1 --name dev
# import my-image.tar
kind load image-archive my-image.tar --name dev

With this method, when we are building a new Docker image on the host which we want to run in a kind Kubernetes cluster, we can:

docker build -t my-image:v1 ./my-image-dir
kind load docker-image my-image:v1
kubectl apply -f my-image.yaml

How to see the images that are available in the kind Kubernetes cluster? Easy:

docker exec -it dev-control-plane crictl images

dev-control-plane is the name of the kind cluster.

You can also use crictl -h to see all the supported commands. For example, we can delete an image by using crictl rmi <image_name>.

Summary

Have fun playing with Kubernetes locally!

If you like this article, please like, comment, and subscribe. See you in the next article!

Dagger (the CI/CD Tool, not the Knife) In-Depth: Everything You Need to Know (as of Apr 2022)

Tiexin Guo — Tue, 12 Apr 2022 01:38:48 +0000

1 What is Dagger?

TL;DR: Dagger runs your CI/CD pipelines locally in a Docker container, and can run the container in any CI environment (as long as that CI can run a container, of course.)

Do you want the long answer to this million-dollar question? It's hard to answer, honestly. News is calling it a "DevOps platform"; the VC that funded Dagger even called it a "DevOps operating system."

But, in fact, Dagger is neither of those things.

Before we can answer what Dagger is, let's have a look at it in-depth:

2 Quirks and Features

2.1 BuildKit

In Dagger, the configuration is executed in BuildKit, which is the execution engine at the heart of Docker.

BuildKit was developed as part of the Moby project, the latter of which is an open framework to assemble specialized container systems without reinventing the wheel by Docker. Basically, it's a toolkit for converting source code to build artifacts in an efficient, expressive, and repeatable manner. It was announced in 2017 and began shipping with Docker Engine in 2018’s version 18.09.

2.2 CUE

Unlike most popular CI systems out there, you don't write YAML in Dagger; you write CUE.

I feel you because I didn't know what's CUE either. Turned out, CUE is an open-source data validation language and inference engine with its roots in logic programming.

It aims to simplify tasks involving defining and using data. It's actually a superset of JSON, so users familiar with JSON should feel comfortable with it already and can get started quickly. It also has got built-in auto-formatting (yay.)

Although the language is not a general-purpose programming language, it has many applications, such as data validation, data templating, configuration (that's probably why Dagger decided to use it in the first place), querying, code generation, and even scripting.

2.3 Wait a Minute

Since it's already reusing Docker's parts for configuration execution, why not reuse Docker's other part, Dockerfile, for configuration?

What's the purpose of using another language just for the configuration?

Solomon Hykes, the founder of Dagger, actually answered exact this question on their official Discord channel:

We needed a modern declarative language with a type system, a package manager, native yaml and json interop, a formal spec, and a standalone community not locked to one tool.

Also Dockerfiles are specific to build, but Dagger is more general-purpose automation

There was no way at all Dockerfile could support our requirements (speaking as one of the original authors of the Dockerfile syntax)

3 Enough Tech Spec. What Does Dagger Do?

3.1 What Dagger Isn't

First, let me tell you what Dagger isn't by quoting the official documentation:

Dagger does not replace your CI: it improves it by adding a portable development layer on top of it.

OK, so it's not yet another CI (or CD, for that matter).

Dagger didn't even try to replace your existing CI, at all. But rather, it improves your CI, by adding a wrapper layer.

I know the term "wrapper" doesn't sound fancy, so let's call it by its official reference, and that is "portable development layer".

3.2 So, Just a Wrapper?

Disappointed? Don't conclude too quick; follow me. First, let's look at some other DevOps/cloud related examples:

Think of Terraform. You've got multiple environments to manage. Even with reuseable modules and roles, you still have duplicated code across envs. Then comes Terragrunt, which is a (thin) wrapper that provides extra tools for keeping your configurations simple without repeating yourself.
Think of AWS CDK. It actually is a wrapper layer on top of CloudFormation, which lets you use your familiar programming languages to define and provision AWS cloud infrastructure, so that you don't have to deal with CloudFormation's non-human-readable configurations. Of course, your code still converts to a format that CloudFormation understands, and your infrastructure is still managed by CloudFormation; AWS CDK doesn't really interact with AWS APIs directly. That's why it's only a wrapper layer on top of CloudFormation.
Think of CDKTF (CDK for Terraform); it's no different than AWS CDK, perhaps because CDKTF is inspired by AWS CDK and also uses AWS's jsii library to be polyglot. It's a wrapper layer on top of Terraform that translates your code into Terraform HCL so that you don't have to learn HCL. But in essence, your infrastructure is still managed by Terraform HCL, not your code directly. So, it's yet another wrapper.

You must have already figured out where I am going with this, and you are right: yes, Dagger is no different. It is a wrapper.

But, of course, the wrapper has to do something to be useful, right? Then what exactly does Dagger do? What exactly does Dagger wrap? Good Questions.

3.3 What Dagger Can Do

In any CI system, you define some steps and actions in a certain format (YAML, most likely) and run it in your CI system. For example, in Jenkins, maybe you will write some groovy file. In GitHub Actions, you write some YAML with multiple steps.

Basically, Dagger runs those "steps and actions" in a Docker container. Then where do you run the Dagger docker container itself? Great question: you can either run it locally (because you can install Docker desktop, right?) or in your existing CI (since most CIs can run a docker container.)

If you think about it: Dagger doesn't wrap your CI pipelines or systems. It wraps those detailed steps and actions into a Docker container and still runs in your existing CI. It's like writing a big Dockerfile, and when you run the container, it does git clone, source code static scan, test, build, artifact upload, and what have you.

4 What Dagger Really Is

Yes, Dagger is a wrapper, that part is true.

But, it doesn't wrap CI systems; it wraps your pipeline steps and actions into a container (you have to rewrite those steps and actions in Dagger's syntax, though), and the wrapped result can run in another CI (as long as that CI can run a container.)

In this sense, Dagger is yet another CI, except that CI runs in a container and most CI systems happen to be able to run containers.

5 Benefits

I think there are 3 major advantages of using Dagger

5.1 Local Development

Firstly, there is no need to install any dependencies specific to this application, because Dagger manages all the intermediate steps and transient dependencies inside the Docker container.

This might not be an advantage if we are talking about CI, but it is an advantage if we are talking about local development.

Think of Go where you have to install modules or think of Nodejs where you might even need to switch Node versions then do an NPM install. Now you can do all of those inside a container and only get the final result to your local laptop.

5.2 "On-Premise" CI

You can run your pipeline locally now since you can easily have the Docker desktop up and running locally.

I'm not sure if this is a solid requirement, though. Maybe it is? Because we all have powerful laptops now; why waste money on some CI systems when you can run them locally?

The idea of running it anywhere as long as Docker is available is intriguing, though. If you don't want to buy CI as a service, you can run Dagger in your own infrastructure.

5.3 Migrate to Another CI

Since your "steps and actions" now are running in a container, you can run it elsewhere, in another CI system.

Should you need to migrate to another CI, you do not need to rewrite your CI steps anymore. For example, you don't want to use your company's old Jenkins instance anymore, but you are already using Dagger with Jenkins, and now you want to give GitHub Actions a try.

It's worth noting that in this scenario, there are still two things to do:

If you are using Jenkins now, and want to migrate those Jenkins pipelines into Dagger, you need to do it manually. The cost is the same (if not more) as rewriting your whole pipeline in GitHub Actions' syntax.
You will still need to learn about your new CI system: how a job is triggered, the syntax, etc.

Here you can see Dagger does provide a solution to avoid CI lock-in (to some extent.) But it is not a game changing solution that could resolve your flexibility concern.

For a DevOps engineer, a DevOps toolchain that could accommodate different needs and priorities from different teams is more than appealing. That being said, each component is modular and pluggable, and you could free yourself from tedious work like launching, integrating, and managing all these pieces.

Ideally, you could define your desired DevOps tools in a single human-readable YAML config file, and with one single command, you have your whole DevOps toolchain and SDLC workflow set up or changed.

If you are intrigued by the simplicity of "DevOps toolchain as code", don't hesitate to check out DevStream here.

6 Should I use Dagger Now?

Is it promising? Maybe. Should I start using it now? My answer is No. Three reasons:

The Dagger project itself still uses GitHub Actions. Why? Probably because it has limitations and can't do everything you can achieve with GitHub Actions.
You probably won't change your CI system every 6 months. If you only change it one time in 4 years, why bother adding that wrapper?
Dagger is only recently announced. It hasn't supported a whole lot of CI systems yet. Maybe the CI you want to switch to doesn't support it yet.

Like, comment, subscribe. See you in the next article!

Creating a DevStream (dtm) Plugin for Anything

Tiexin Guo — Tue, 29 Mar 2022 09:27:10 +0000

Yes, the title of this post isn't bluffing: you can actually create a plugin for just about anything that takes your fancy.

In my previous article, I walked you guys through DevStream's codebase.

If you haven't read it yet, here's a quick link for you:

https://blog.dtm.dev/post/2022-03/01-hello-world/.

In this blog, we will walk you through the steps of creating a DevStream plugin from scratch with an example.

What is DevStream

DevStream is an amazing tool that lets you install, update, manage, and integrate your DevOps tools quickly and flexibly.

Not to brag, but with DevStream, you can have your own DevOps toolchain that is specifically tailored to your need up and running in 5 minutes.

Don't believe it? Check out our docs and follow the quickstart guide!

Existing Plugins

At the moment of publishing this article, we already support the following tools:

Trello (including integration with GitHub)
Jira (integration with GitHub)
GitHub Repository bootstrapping (for Go)
GitHub Actions (for Go, Python, and Nodejs)
GitLab CI (for Go)
Jenkins (installation)
ArgoCD
ArgoCD Application (the deployment of your apps)
Prometheus + Grafana
DevLake
OpenLDAP

We aim to have 50 plugins at the end of 2022!

Check out our README for the latest status.

Why Would I Want to Create a DevStream Plugin

Wait. YOU ALREADY HAVE TONS OF PLUGINS! Why on earth would I want to create yet another one?

I agree with you.

Can I get a show of hands, who here has made a DevStream plugin before?

Very few, if any, I guess.

However (I know it's just a fancy "but"), there are, in fact, things that you want to build a plugin for:

Maybe you are building a nice and shiny DevOps tool, and you want to be able to set it up quickly without any hassle. You would have to write some automation scripts for it anyway, right? DevStream got you covered.
Maybe you even need to integrate your tool with other tools to get the most out of it and you really don't want to reinvent a lot of wheels just to manage those boring integrations. Again, DevStream got you covered.
Maybe you have both internal tools and open-source tools whose installation and integration need to be automated. The open-source tools are fine, but how to manage those internal ones and integrate them? DevStream got you covered.

Or, maybe you just want to learn Go's plugin, become a contributor, join our community and earn your certification (maybe a small present, too, who knows.) No problem.

No matter what your intention is and what thing you want to achieve, DevStream got you covered.

So hang tight, let's get started.

Design: A "Local File" Plugin

In this example, let's build something dum but simple, just to show you the process of creating a plugin.

We are creating a "local file" plugin. You specify the name and the content, and the plugin will create a local file for you. Let's decide how we are going to use this plugin:

tools:
- name: my-file
  plugin: localfile
  options:
    filename: foo.txt
    content: "hello, world"

Basically:

the name of the plugin: localfile
options of the plugin:
- filename
- content

The plugin will create a file with desired content according to this piece of config.

Scaffolding Automagically

First, let's clone the DevStream repo and generate some scaffolding code:

git clone [email protected]:merico-dev/stream.git
cd stream
# builds dtm locally to make sure it's using the same dependencies as your new plugin
make build-core
./dtm develop create-plugin --name=localfile

There will be some useful output to guide you through the whole process.

Now, if we do a git status, we can see some new stuff are already created automagically:

$ git status
On branch main
Your branch is up to date with 'origin/main'.

Untracked files:
  (use "git add <file>..." to include in what will be committed)
    cmd/plugin/localfile/
    docs/plugins/localfile.md
    internal/pkg/plugin/localfile/

Let's have a quick recap of the directory structure:

cmd/

cmd/plugin/localfile/main.go is the main entrance of your plugin. But here you don't need to do anything; nothing.

dtm has already generated the code for you, including the interfaces that you must implement.

docs/

docs/plugins/localfile.md is the automatically generated documentation.

Yep, I know dtm is automagic, but it can't read your mind. I'm afraid that you will have to write your own doc.

But hey, at least here you get a reminder that you need to create a doc :)

internal/pkg/

internal/pkg/plugin/localfile has your plugin's main logic.

Here, we need to:

define your input parameters (options);
implement the validation of the input parameters;
implement four mandatory interfaces.

Core Concepts

Config/State/Resource

Before explaining interfaces and implementing them, let's have a look at how DevStream actually works:

Config is a list of tools, each of which has a name, plugin, options, etc.
State is a map containing each tool's name, plugin, options, etc. It's used to store the result of dtm's last action.
Resource is the tool that the plugin created. The Read interface returns a description of that resource, which should be the same as the State if nothing has been changed after dtm's last action.

DevStream decides what to do based on your Config, the State, and the Resource's status. See the flowchart below:

Interfaces

A quick recap: each DevStream plugin must satisfy the following four interfaces:

Create
Read
Update
Delete

Return values:

Create and Update return two values (map[string]interface{}, error). The first return value is considered as the State, which will be stored in DevStream's state file.
Read's first return value is a description of the Resource, which should be the same as the State if nothing has changed.
Delete returns (true, nil) if there is no error; otherwise it returns (false, error).

Coding

Input Options/Validation

Now let's open internal/pkg/plugin/localfile/options.go and add options according to our design in the previous section.

The internal/pkg/plugin/localfile/options.go should look like:

package localfile

// Options is the struct for configurations of the localfile plugin.
type Options struct {
    Filename string
    Content  string
}

Let's also implement the validation function of the input options.

Open internal/pkg/plugin/localfile/validate.go and change the logic to verify the options. It should look like this:

package localfile

import "fmt"

// validate validates the options provided by the core.
func validate(options *Options) []error {
    res := make([]error, 0)

    if options.Filename == "" {
        res = append(res, fmt.Errorf("filename does not exist"))
    }

    return res
}

Create()

We want to create the file based on the input options. So, let's do just that in the file internal/pkg/plugin/localfile/create.go.

OK, talk is cheap, show me the code:

package localfile

import (
    "fmt"

    "github.com/mitchellh/mapstructure"

    "github.com/merico-dev/stream/pkg/util/log"
)

func Create(options map[string]interface{}) (map[string]interface{}, error) {
    var opts Options
    if err := mapstructure.Decode(options, &opts); err != nil {
        return nil, err
    }

    if errs := validate(&opts); len(errs) != 0 {
        for _, e := range errs {
            log.Errorf("Options error: %s.", e)
        }
        return nil, fmt.Errorf("opts are illegal")
    }

    err := writefile(opts.Filename, opts.Content)
    if err != nil {
        return nil, err
    }

    status := map[string]interface{}{
        "filename": opts.Filename,
        "content":  opts.Content,
    }

    return status, nil
}

If you compare the code above to the code generated by dtm develop, you will see that we basically did nothing here. We only filled some blanks which are marked by dtm's comment. The same is true for all the other interfaces.

A small tip: here, we can put the function writefile in internal/pkg/plugin/localfile/localfile.go, so that the code is better organized.

The localfile.go will look like this:

package localfile

import "os"

func writefile(filename, content string) error {
    f, err := os.Create(filename)
    if err != nil {
        return err
    }

    defer f.Close()

    _, err = f.WriteString(content)
    if err != nil {
        return err
    }

    return nil
}

Read()

Let's also implement the Read interface.

The logic is simple: we want to try to see if the file exists or not, and if yes, what's the filename and content.

internal/pkg/plugin/localfile/read.go:

package localfile

import (
    "fmt"
    "os"
    "strings"

    "github.com/mitchellh/mapstructure"

    "github.com/merico-dev/stream/pkg/util/log"
)

func Read(options map[string]interface{}) (map[string]interface{}, error) {
    var opts Options
    if err := mapstructure.Decode(options, &opts); err != nil {
        return nil, err
    }

    if errs := validate(&opts); len(errs) != 0 {
        for _, e := range errs {
            log.Errorf("Options error: %s.", e)
        }
        return nil, fmt.Errorf("opts are illegal")
    }

    content, err := os.ReadFile(opts.Filename)
    if err != nil {
        if strings.Contains(err.Error(), "no such file or directory") {
            return nil, nil
        }
        return nil, err
    }

    status := map[string]interface{}{
        "filename": opts.Filename,
        "content":  string(content),
    }

    return status, nil
}

On the return value:

If the file doesn't exist, return nil, no error, because it means the resource hasn't been created yet (or deleted by somebody)
If the file exists, return the "status" of it and no error.
Otherwise, return nil and the error.

Update()

Update will be triggered if Read returns a different result than what's recorded in the State.

For the implementation, since we are updating a file, it's the same as Create, so we can actually reuse it here without duplicated code:

internal/pkg/plugin/localfile/update.go:

package localfile

func Update(options map[string]interface{}) (map[string]interface{}, error) {
    return Create(options)
}

Delete()

Last but not least, let's implement the Delete interface.

internal/pkg/plugin/localfile/delete.go:

package localfile

import (
    "fmt"
    "os"

    "github.com/mitchellh/mapstructure"

    "github.com/merico-dev/stream/pkg/util/log"
)

func Delete(options map[string]interface{}) (bool, error) {
    var opts Options
    if err := mapstructure.Decode(options, &opts); err != nil {
        return false, err
    }

    if errs := validate(&opts); len(errs) != 0 {
        for _, e := range errs {
            log.Errorf("Options error: %s.", e)
        }
        return false, fmt.Errorf("opts are illegal")
    }

    err := os.Remove(opts.Filename)
    if err != nil {
        return false, err
    }

    return true, nil
}

Just delete the file; nothing to look at here.

Build

Now that we finished coding, let's build our new plugin. Our Makefile now supports building a specific plugin only:

make build-plugin.localfile

Test

First, let's create a config file config-localfile-test.yaml:

tools:
- name: my-file
  plugin: localfile
  options:
    filename: foo.txt
    content: "hello, world"

Now, let's apply:

$ ./dtm -y apply -f config-localfile-test.yaml
2022-03-29 11:25:17 ℹ [INFO]  Apply started.
2022-03-29 11:25:17 ℹ [INFO]  Using dir <.devstream> to store plugins.
2022-03-29 11:25:17 ℹ [INFO]  Tool my-file (localfile) found in config but doesn't exist in the state, will be created.
2022-03-29 11:25:17 ℹ [INFO]  Start executing the plan.
2022-03-29 11:25:17 ℹ [INFO]  Changes count: 1.
2022-03-29 11:25:17 ℹ [INFO]  -------------------- [  Processing progress: 1/1.  ] --------------------
2022-03-29 11:25:17 ℹ [INFO]  Processing: my-file (localfile) -> Create ...
2022-03-29 11:25:17 ✔ [SUCCESS]  Plugin my-file(localfile) Create done.
2022-03-29 11:25:17 ℹ [INFO]  -------------------- [  Processing done.  ] --------------------
2022-03-29 11:25:17 ✔ [SUCCESS]  All plugins applied successfully.
2022-03-29 11:25:17 ✔ [SUCCESS]  Apply finished.

As we can see, the plugin has created our file successfully. To verify, you can open the "foo.txt" file to have a look.

If we apply again, nothing should happen, since the file is already created with the desired content:

$ ./dtm -y apply -f config-localfile-test.yaml
2022-03-29 11:30:43 ℹ [INFO]  Apply started.
2022-03-29 11:30:43 ℹ [INFO]  Using dir <.devstream> to store plugins.
2022-03-29 11:30:45 ℹ [INFO]  No changes done since last apply. There is nothing to do.
2022-03-29 11:30:45 ✔ [SUCCESS]  Apply finished.

But, what if somebody changed the content of "foo.txt"? Let's experiment:

$ echo "changed" > foo.txt
$ ./dtm -y apply -f config-localfile-test.yaml
2022-03-29 11:26:40 ℹ [INFO]  Apply started.
2022-03-29 11:26:40 ℹ [INFO]  Using dir <.devstream> to store plugins.
2022-03-29 11:26:40 ℹ [INFO]  Tool my-file (localfile) drifted from the state, will be updated.
2022-03-29 11:26:40 ℹ [INFO]  Start executing the plan.
2022-03-29 11:26:40 ℹ [INFO]  Changes count: 1.
2022-03-29 11:26:40 ℹ [INFO]  -------------------- [  Processing progress: 1/1.  ] --------------------
2022-03-29 11:26:40 ℹ [INFO]  Processing: my-file (localfile) -> Update ...
2022-03-29 11:26:40 ✔ [SUCCESS]  Plugin my-file(localfile) Update done.
2022-03-29 11:26:40 ℹ [INFO]  -------------------- [  Processing done.  ] --------------------
2022-03-29 11:26:40 ✔ [SUCCESS]  All plugins applied successfully.
2022-03-29 11:26:40 ✔ [SUCCESS]  Apply finished.

We can see that the file has drifted from the state, so it will be updated. After the operation, the file's content is changed back to what we defined in our config.

Heck, let's even try deleting this file and see what happens:

$ rm foo.txt
$ ./dtm -y apply -f config-localfile-test.yaml
2022-03-29 11:27:33 ℹ [INFO]  Apply started.
2022-03-29 11:27:33 ℹ [INFO]  Using dir <.devstream> to store plugins.
2022-03-29 11:27:33 ℹ [INFO]  Tool my-file (localfile) state found but it seems the tool isn't created, will be created.
2022-03-29 11:27:33 ℹ [INFO]  Start executing the plan.
2022-03-29 11:27:33 ℹ [INFO]  Changes count: 1.
2022-03-29 11:27:33 ℹ [INFO]  -------------------- [  Processing progress: 1/1.  ] --------------------
2022-03-29 11:27:33 ℹ [INFO]  Processing: my-file (localfile) -> Create ...
2022-03-29 11:27:33 ✔ [SUCCESS]  Plugin my-file(localfile) Create done.
2022-03-29 11:27:33 ℹ [INFO]  -------------------- [  Processing done.  ] --------------------
2022-03-29 11:27:33 ✔ [SUCCESS]  All plugins applied successfully.
2022-03-29 11:27:33 ✔ [SUCCESS]  Apply finished.

Apparently, DevStream thought the file should exist according to the state, but it doesn't, so it is created again.

Last, let's try to delete this file with dtm:

$ ./dtm -y delete -f config-localfile-test.yaml
2022-03-29 11:32:24 ℹ [INFO]  Delete started.
2022-03-29 11:32:24 ℹ [INFO]  Using dir <.devstream> to store plugins.
2022-03-29 11:32:24 ℹ [INFO]  Tool my-file (localfile) will be deleted.
2022-03-29 11:32:24 ℹ [INFO]  Start executing the plan.
2022-03-29 11:32:24 ℹ [INFO]  Changes count: 1.
2022-03-29 11:32:24 ℹ [INFO]  -------------------- [  Processing progress: 1/1.  ] --------------------
2022-03-29 11:32:24 ℹ [INFO]  Processing: my-file (localfile) -> Delete ...
2022-03-29 11:32:24 ℹ [INFO]  Prepare to delete 'my-file_localfile' from States.
2022-03-29 11:32:24 ✔ [SUCCESS]  Plugin my-file (localfile) delete done.
2022-03-29 11:32:24 ℹ [INFO]  -------------------- [  Processing done.  ] --------------------
2022-03-29 11:32:24 ✔ [SUCCESS]  All plugins deleted successfully.
2022-03-29 11:32:24 ✔ [SUCCESS]  Delete finished.

Hooray!

Summary

I'm sorry that I put the "TL;DR" at the end; it's only because I want you to have a global feeling of DevStream. But for advanced developers, here's a "TL;DR" (or use it as a checklist) for you:

Run dtm develop create-plugin --name=your-plugin.
Do a git status, then:
- find the generated files
- edit/code at the places where there is already a comment mark
Pay special attention to the return value of the interfaces. Figure out the relationship between State and Resource. What the interfaces return decides how this plugin will behave.
Last but not least, update the documentation.

If you enjoy reading this post, please like, comment, and subscribe! I will see you next time.

DEV Community: DevStream

App-Centric Configuration: Adding More Value to Your Life

DevStream's Story

Adding Value to Your Life

Lightbulb: Apps? Apps!

Backward Compatibility

(Much) Shorter, Simpler, Easier, While Achieving the Same

One File to Rule Them All

DevStream 0.9.0 Release

Official Releases for Different Platforms

Major Changes since Last Release

Core

Plugins

Docs

More

DevStream Certification Program Introduction

0 Background

1 Introducing the New Digital Certification Program

2 The Credly Story

3 Why Did DevStream Go with Credly

4 Wait a Minute. So I Got My Certs. Where Can I Show Them Off?

5 Where Do We Go From Here

Earn

Claim

Share

DevStream Joins CNCF Sandbox

Background

Thank You

The CNCF TOC Review

What We Did Right

Summary

A Brief Introduction to Code Review: Everything You Want to Know

1 Why Do We Need to Do Code Review?

2 Why Not Using Tools and Automation for Code Quality?

3 Code Review: What Are the Standards?

3.1 New Contributors

3.2 Senior Members

4 Working in an Open-Source Project

5 Taking up a Code Review Task

6 GitHub Code Review Walk-Through

7 Reviewing a PR Locally

8 Committing to Somebody Else's PR

9 Summary

Using AWS S3 to Store DevStream State

Terminology

Configuration

Config File Examples

Getting Started

Apply

Checking the State File

DevStream v0.6.0 Release

Demo

Official Releases for Different Platforms

Major Changes

New Contributors

More

On DevOps: 1. What It Is

Background

What is DevOps?

Why Is It So Hard to Define DevOps?

Let’s Try to Figure It out

A Combination of Dev and Ops

Cultural Philosophies

Culture

DevOps Culture

Is DevOps a role?

What is Required to Achieve the DevOps Culture?

How to Enable the Growth and Learning Mindset

Motivation

What Motivates the Team to Grow

Leadership

IT Enables Business

Summary

Getting Started with minikube

What We Will Talk about Today

Quick Start with minikube

How to Install minikube

minikube start

Access Your Shiny Kubernetes Cluster

Deploy an Application in Your Shiny Kubernetes Cluster

Quick Start with `minikube`

How to Install `minikube`

`minikube start`

How to Push Images into a `minikube` Cluster

Push Images with the `cache` Command

Introduction to `kind`

`kind` Cluster Config

Advanced `kind` Features

Expose `kube-apiserver`