DEV Community: Raul Piraces Alastuey

Take ownership of your Twitter data, set-up your own Twitter updated archive in GitHub

Raul Piraces Alastuey — Mon, 02 Jan 2023 15:53:45 +0000

Due to the "recent" acquisition of Twitter and all changes that came after that, lots of users started considering other alternatives such as Mastodon, Tumblr, Nostr and many others...

But what happens to the generated data over the years we have been making in this platform, what do we make with them? Discarding them? My thoughts on this are... why not just keeping it safe for the future?

In this case it is where Tweetback come into action. An awesome project by Zach Leatherman to take ownership of your Twitter data in a public archive you can self-host (for example in GitHub).

Overview

Thanks to Zach Leatherman and its awesome project Tweetback, we can self host our own Twitter archive and keeping it up-to-date if necessary (just like I have done with mine).

The process to make your own archive is simple. Just follow the steps below.

Getting your Twitter archive in first place

This part is important to initialize our archive.

Twitter allows you to export a backup of all of your data you have in this platform.

As they explain in their help center, log in into your Twitter account (Web client) and:

Click More in the main navigation menu to the left of your timeline.
Select Settings and privacy.
Select "Your Account" under Settings.
Click on "Download an archive of your data".

After clicking there, click in the button that says to request your archive. Then it will change the status to "Requesting archive".

After taking this action, Twitter will collect your data and send you an email with a link to a .zip file with all your exported data. This process, can take up to 48 hours... so let's relax and keep with the process.

Note: don't forget to download the .zip file as soon as you receive the email and save it somewhere (as the links expires shortly).

Getting the Tweetback code and setting it up

In my case I just forked the repository but you can clone it and upload it to other site if you want.

Note that you will need to have Node.js installed (and Git of course).

Then follow this steps:

Clone the repo (or your fork):

git clone https://github.com/tweetback/tweetback.git

Note: if you choose creating a fork, the process of updating the repository for possible future changes from the base repository its easier with the "sync fork" option that GitHub provides.

Enter to the repository main folder with the terminal and perform a npm install (this will generate a package-lock.json file you will keep).
Following the instructions of Tweetback, open the zip file you downloaded from Twitter, and extract the data/tweets.js file from it into the database folder in your Tweetback repository folder.
Edit this file to change the first line window.YTD.tweet.part0 to module.exports (a simple replace) so the first part of the file will look as follows:

module.exports = [
{
  "tweet" : {
...

Note: if you have lots of tweets, the process can take a while...

On the root folder of the repository run with the terminal the command npm run import which will perform the first initialization of our tweets database (going through the tweets.js file).
At the end, you will see some output regarding counts and other info (but no errors).
After importing the tweets, we will be making some changes to the _data/metadata.js file, which will customize some page settings for our archive. We will modify the options as follows (leave the others without modifications):

Now we are ready with our archive! Take a look at how it will be by running npm run start at the root folder and accessing the local URL that the process outputs to the terminal.
If everything looks good for you, commit and push your changes to the repository.

(Optional) Set the subpath where your page will reside

You can skip this step if your twitter archive will be in the root of a domain/subdomain (such as tweets.piraces.dev), otherwise we will have to set the subpath where it will be served (for example if hosting in GitHub pages such as piraces.github.io/tweets).

To do this, we will need to do some changes to the eleventy.config.js file as follows (at the end before the closing }):

...
eleventyConfig.addPlugin(EleventyHtmlBasePlugin);
return {pathPrefix: "/twitter/"}};

Note: change the pathPrefix accordingly to the path you want.

(Optional) Configuring a subdomain to point to our GitHub Pages deployment

In this tutorial we will be using GitHub Pages to host our twitter archive, so since in my case I have the domain piraces.dev managed by Cloudflare, I will create a subdomain called tweets.piraces.dev to point to the GitHub pages deployment.

To do this we will only have to create a CNAME DNS record with the value of the subdomain (tweets in this case) pointing to your GitHub domain that will be in the format HANDLE.github.io where HANDLE will be your handle/username in GitHub (piraces.github.io in my case).

The picture above shows how to set it for Cloudflare, but it will be very similar in other providers.

Deploying our archive using GitHub pages

Once our repo has all the changes mentioned above, we will have to configure GitHub pages for our repository.

To do so, we must follow this steps:

Create an empty branch named gh-pages (where the deployed page will be). From your terminal:

git switch --orphan gh-pages
git commit --allow-empty -m "Initial commit on orphan branch"
git push -u origin gh-pages

Go to the repository page in GitHub and click "Settings" and there "Pages" from the left toolbar.
In the "Pages" screen, we will be setting the options as follows:

Setup a workflow to publish the page on every push to main branch or manually. To do so, we must go to the "Actions" tab in the GitHub repository page, and then select "New workflow". From this page, we select "Skip this and set up a workflow yourself ->", then a editor appears and we can setup the workflow. You can use the following code (adapting some fields) to complete yours:

name: Publish page

on:
  push:
    branches: [ "main" ] # Change this to the name of the main branch of your repository
  workflow_dispatch: # Setting this enables the possibility to run the workflow manually

jobs:
  build:
      runs-on: ubuntu-latest
      steps:
      - uses: actions/checkout@v3
      - name: Use Node.js 18.x
        uses: actions/setup-node@v3
        with:
            node-version: 18.x
      - run: npm ci # Note that you must commit the package-lock.json to the repo and quit it from the .gitignore file
      - run: npm run build
      - name: Deploy to Github Pages
        uses: crazy-max/ghaction-github-pages@v3
        with:
          # Build directory to deploy
          build_dir: _site
          # Write the given domain name to the CNAME file
          fqdn: tweets.piraces.dev # Change this to your configured subdomain or completely delete the line if not using it.
          jekyll: false
          keep_history: true # If you don't want to preserve a commit history, set to false to perform a git push --force
          env:
            GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN  }} # This secret is available by default

When finishing editing, click on "Start commit" and commit the workflow to the main branch. The workflow will start running after the commit, you can check the status from the "Actions tab" in the GitHub repository page.
If the workflow shows green (all steps are OK), go and check your URL and see your Twitter archive 🎉 (note, if something goes wrong with your configured subdomain, please revisit step 3).

(Optional) Setting an scheduled action to update our archive

If you may still using Twitter in the future, consider modifying the workflow created in the previous step to fetch all new tweets using the Twitter API.

To do so, you will need a Twitter bearer token. You will only be allowed to obtain a Twitter bearer token if you are a "Twitter developer" 😢, so you will have to fill some forms and apply to become one.

To start the process to become a "Twitter developer" visit developer.twitter.com, sign-in and start the process... It may be an annoying process with manually reviews.

Once you are a "Twitter developer", proceed to create an app and then getting its bearer token. Documentation about getting the bearer token can be found in the official developer documentation of Twitter.

When you have everything set-up correctly, copy the bearer token that will look like something like AAAA... (like random characters). Now go to the repository with your terminal and try to perform the update process for yourself locally (before modifying the remote workflow) to make sure everything goes smoothly.

When you are in the repository base path with your terminal, run the following command (where AAAAAA... is your actual complete bearer token):

TWITTER_BEARER_TOKEN=AAAAAA... npm run fetch-new-data

This will fetch any tweets that are not in the database because you made them later.

After fetching new tweets, you will need to rebuild your site with:

npm run build

Take a look to the updated archive by running it locally with npm run start and accessing the URL that the process outputs to the terminal.

If everything looks OK to you, we are ready to modify our GitHub workflow to automate this!

Follow then this steps:

Go to your repository GitHub page, select the "Actions" tab, select your workflow in the left navbar, locate the latest execution of the workflow and click on "View workflow file". After this, select the edit icon and start modifying your workflow.

Note: you can also modify the workflow in your local environment accessing the folder .github/workflows/ and editing the workflow.

We will be adding a new step after - run: npm ci to run the fetching process of new tweets. To do so, first you may need to add the Twitter bearer token as a secret to your actions (since you don't want it to be publicly visible). If you don't know how to do so, please follow the steps in the GitHub official documentation.
After declaring the secret, add a step after - run: npm ci like the following (assuming your secret is named TWITTER_BEARER_TOKEN, if not, change it accordingly):

...
- run: npm run fetch-new-data
  env:
    TWITTER_BEARER_TOKEN: ${{ secrets.TWITTER_BEARER_TOKEN }} # Change the secret name if different

In order to do this periodically, I decided to make my workflow run every night. To do so, you will only have to add a schedule entry in the on entry with a cron rule (adjust it if you want to your requirements):

name: Publish page

on:
  push:
    branches: [ "main" ] # Change this to the name of the main branch of your repository
  schedule:
    - cron: "30 0 * * *" # This will trigger the workflow every day at 00:30 UTC
  workflow_dispatch: # Setting this enables the possibility to run the workflow manually

Your workflow now will look like this (with minimal differences):

name: Fetch new data & publish page

on:
  push:
      branches: [ "main" ]
  schedule:
      - cron: "30 0 * * *"
  workflow_dispatch:

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v3
    - name: Use Node.js 18.x
      uses: actions/setup-node@v3
      with:
          node-version: 18.x
    - run: npm ci
    - run: npm run fetch-new-data
      env:
        TWITTER_BEARER_TOKEN: ${{ secrets.TWITTER_BEARER_TOKEN }} # Here the secret TWITTER_BEARER_TOKEN
    - run: npm run build
    - name: Deploy to Github Pages
      uses: crazy-max/ghaction-github-pages@v3
      with:
        # Build directory to deploy
        build_dir: _site
        # Write the given domain name to the CNAME file
        fqdn: tweets.piraces.dev
        jekyll: false
        keep_history: true
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} # Here the secret GITHUB_TOKEN

Save your workflow, commit and push it.
Checkout the execution process in the "Actions" tab of your GitHub repository to check everything goes OK.
You have now an "always updated" Twitter archive to be there for you and the people that may want to check it completely unattended! 🚀

Note: if you have any doubts regarding code/workflows you can always check my Twitter archive implementation: GitHub - piraces/twitter_archive.

(Optional) Adding your archive to tweetback-canonical (list of hosted Twitter backups)

tweetback-canonical is a package to resolve twitter URLs to new canonically hosted twitter backups, made by Zach Leatherman too.

Simple follow the process in their README.md to add the URL of your Twitter archive to the mapping.js file.

The process its very simple:

Create a fork of the tweetback-canonical repository.
Modify the mapping.js file adding your Twitter archive public URL to the end of the object, in the format:

"TWITTER_HANDLE": "FULL_URL_TO_TWITTER_ARCHIVE",

Note: your twitter handle must go without the @ symbol and you must end the line with a comma.

Commit and push your changes with a commit message with starts with mapping:.
Open a PR to the main branch of the official repository and wait for approval and merge.
You are done with all! Congrats! 🎉

Conclusion

We have seen in detail how to use the awesome Tweetback project to build and self-host our own Twitter archive/backup. Whether you may, or may not, want to get off Twitter, I think it's a great approach to get the ownership of your data anyways.

Doing this you are not exposed to lose the investment you put in generating those tweets if something bad happens (hopefully not 🙏).

Don't forget to report any issues you find to the official repository and thanks the author (Zach) for the awesome work!

Happy deploy! 🎉🎉

OpenPGP identity proof

Raul Piraces Alastuey — Thu, 17 Nov 2022 20:04:42 +0000

This is an OpenPGP proof that connects my OpenPGP key to this dev.to account. For details check out https://keyoxide.org/guides/openpgp-proofs

[Verifying my OpenPGP key: openpgp4fpr:84BF523F3F3EFA760C3E2C0C0C1A484B87269CD7]

How to use the Google One Tap solution in your webapp to authenticate users

Raul Piraces Alastuey — Thu, 06 Jan 2022 17:18:01 +0000

If you are reading this, I think it is a very high chance that you have already seen one of the components shown in the image above. Some popular sites like Quora, have implemented this "Google One Tap experience" and also the traditional "Sign-in with Google" button. Maybe you are wondering how does it work or how they have implemented this experience, so I am going to explain this in this post.

Introduction

Google Identity Services its defined by Google as: "our new cross-platform sign-in SDK for Web and Android apps, supporting and streamlining multiple types of credentials".

In our case, and keeping the target to web apps, they provide to us "seamless sign-in and sign-up flows", such as:

"Sign In With Google" button: personalized and customizable sign-up or sign-in button to our websites.
"One-tap sign-up": allows sign up new users with just one tap of a button, no interruptions and users can end with a secure "password-less" account on your site protected by their Google Account.
"Automatic sign-in": allows sign users automatically when they return to our site on any device or browser ( even if the session expires ).

This three points is what I will mainly cover in this post.

What we can do with this and for what

As shown in the main overview page of "Sign In with Google", basically this processes will allow/help us to quickly and easily manage users authentication and sign-in to out website. Users will sign-in, provide consent and we will securely receive their profile information.

What can be useful for?

Pre-populate new accounts with the received information.
SSO with Google Accounts without making the user re-entering passwords or usernames in other sites.
Protect comments, voting or forms from abuse.
Ease the user experience in your site by automatically sign in them when they return (or let them do so with simple click).
Many other things that you can imagine...

If you would like to test it or integrate it with any of your websites, then... let's get started!

Getting started

You only need basic HTML and JS knowledge to get started. All this setup process is done with JS libraries and HTML to place the buttons/modals.

Note: you may want to know the supported browsers and platforms before continuing.

The first thing we will do is to setup a Google API client ID and configure your consent screen. This is a pretty straightforward process and it is very well explained and documented here.

When you are ready with these and have the necessary data, you will have to load the client library in your webpage. This can be achieved with a little script tag like the following:

<script src="https://accounts.google.com/gsi/client" async defer></script>

This script tag can be included in the head tag of your website or in the body after all your content.

Note that we are including the keywords async and defer to optimize the page loading speed. It is important too to review your CSP to allow this script source and connection. Otherwise, you can download the source and serve it by yourself (I do not recommend this and I have not tested this approach).

After doing this, we are already loading the client library to support the "Sign in With Google" experience.

To finalize the setup, we will have to choose the components for our pages, which are basically a "Sign in" button or the "Google One Tap UX".

What Google recommends is to add the "Sign in" button on our main login pages and the "Google One Tap" to all pages in our sites and enable its "Automatic sign-in option".

In this case we are including the "Google One Tap UX" but adding the "Sign in" button is a straightforward process and also implemented in the demo webapp we will see later. If you want to know about adding the "Sign in" button checkout this links:

Understanding the Google One Tap experience

When configured, the login widget appears as a popup, and it will prompt the users to sign in or sign up with the existing Google account when any of these conditions are met:

They are already logged in with their Google account.
They are already logged in with their Google account in Chrome browser.

If you want to make users sign in when this conditions are not met, then you should display a simple "Sign In With Google button".

There are some points to know on how and when the Google One Tap experience can change:

The process may show a pop-up window if we are using an unsupported browser or the dialog is covered with other content (as a security measure).
Users can opt out of One Tap if they disable the "Google Account sign-in prompts" in their account settings. In this case One Tap will not display.
The dialog of One Tap has an exponential cooldown when it is closed by the user. The dialog then won't display in the same browser or the last website visited for a period of time. When more times the user closes the prompt, the more time the prompt will not show.
On mobile devices, One Tap will close automatically after a short time if the user do not interact with it.

After understanding the "flow", lets get the Google One Tap to display in our page.

Displaying Google One Tap

In order to display Google One tap we can choose between JS and HTML. In my case I prefer to use JS and handle the logic there, but choose whatever fits you the most.

It is also important to choose if we will want to handle the response in the client side or in the server side. I will cover both in this post.

Handling the response in client side

When handling the response in client side, a JS function will be triggered when the flow is completed. We can achieve this in HTML+JS or only JS as said above.

In JS (see comments inside for more information):

function handleCredentialResponse(response) {
  // Here we can do whatever process with the response we want
  // Note that response.credential is a JWT ID token
  console.log("Encoded JWT ID token: " + response.credential);
}
window.onload = function () {
  google.accounts.id.initialize({
    client_id: "YOUR_GOOGLE_CLIENT_ID", // Replace with your Google Client ID
    callback: handleCredentialResponse // We choose to handle the callback in client side, so we include a reference to a function that will handle the response
  });
  // You can skip the next instruction if you don't want to show the "Sign-in" button
  google.accounts.id.renderButton(
    document.getElementById("buttonDiv"), // Ensure the element exist and it is a div to display correcctly
    { theme: "outline", size: "large" }  // Customization attributes
  );
  google.accounts.id.prompt(); // Display the One Tap dialog
}

In HTML:

<script>
function handleCredentialResponse(response) {
  // Here we can do whatever process with the response we want
  // Note that response.credential is a JWT ID token
  console.log("Encoded JWT ID token: " + response.credential);
}
</script>
<div id="g_id_onload"
     data-client_id="YOUR_GOOGLE_CLIENT_ID"
     data-callback="handleCredentialResponse"
     data-your_own_param_1_to_login="any_value"
     data-your_own_param_2_to_login="any_value">
</div>

Including this HTML code in our pages will make Google One Tap prompt when the client library of Google loads. We will have to replace YOUR_GOOGLE_CLIENT_ID with our client id and the data-callback attribute value with our JS function which Google will trigger to perform a callback (this a client-side approach). Additionally, we can specify custom data as the data-your_own_param_1_to_login and data-your_own_param_2_to_login attributes, which will be sent to the callback function.

Handling the response in server side

When handling the response in server side, we will receive a call to an endpoint when the flow is completed. We can achieve this in pure HTML or pure JS as said above.

In JS (see comments inside for more information):

window.onload = function () {
  google.accounts.id.initialize({
    client_id: "YOUR_GOOGLE_CLIENT_ID", // Replace with your Google Client ID
    login_uri: "https://your.domain/your_login_endpoint" // We choose to handle the callback in server side, so we include a reference to a endpoint that will handle the response
  });
  // You can skip the next instruction if you don't want to show the "Sign-in" button
  google.accounts.id.renderButton(
    document.getElementById("buttonDiv"), // Ensure the element exist and it is a div to display correcctly
    { theme: "outline", size: "large" }  // Customization attributes
  );
  google.accounts.id.prompt(); // Display the One Tap dialog
}

In HTML:

<div id="g_id_onload"
     data-client_id="YOUR_GOOGLE_CLIENT_ID"
     data-login_uri="https://your.domain/your_login_endpoint"
     data-your_own_param_1_to_login="any_value"
     data-your_own_param_2_to_login="any_value">
</div>

Including this HTML code in our pages will make Google One Tap prompt when the client library of Google loads. We will have to replace YOUR_GOOGLE_CLIENT_ID with our client id and the data-login_uri attribute value with our login endpoint which Google will use to perform a callback (this a server-side approach). Additionally, we can specify custom data as the data-your_own_param_1_to_login and data-your_own_param_2_to_login attributes, which will be sent to the callback endpoint.

Customizing the experience

The Google One Tap experience can be customized in several ways. Here are some things you can do to customize it.

Automatic sign-in and sign-out

To do this, we only have to add the HTML attribute data-auto_select with a value of true to our HTML code as the following snippet:

<div id="g_id_onload"
     data-client_id="YOUR_GOOGLE_CLIENT_ID"
     data-auto_select="true"
     data-login_uri="https://your.domain/your_login_endpoint">
</div>

Note that the user must first be signed-in to their Google account and have previously granted consent to share their account profile with your app in order to automatic sign-in to work.

For sign out , we must take into account that we can enter a dead-loop UX if the user is redirected with a page with Google One Tap activated after the sign out. To avoid this, put the class g_id_signout on the button that performs the sign out of the user (this will prohibit auto-selection after a user signs out). This can be achieved as follows:

<div class="g_id_signout">Sign Out</div>

Also in JS:

const button = document.getElementById('signout_button');
button.onclick = () => {
  google.accounts.id.disableAutoSelect();
}

Change the sign-in context

We can change the wording in the prompt indicating a context, in order to match our use case. We can use the following context with its corresponding wording:

Context	Wording
signin	"Sign in with Google"
signup	"Sign up with Google"
use	"Use with Google"

To specify the context, just add the HTML attribute data-context with a value included in the table above. For example:

<div id="g_id_onload"
     data-client_id="YOUR_GOOGLE_CLIENT_ID"
     data-login_uri="https://your.domain/your_login_endpoint"
     data-context="use">
</div>

Position of the prompt

By default, the prompt is always shown in the top-right corner of desktop web browser windows. To display it inside a container we can indicate the data-prompt_parent_id HTML attribute indicating an HTML element id which will be its container. Also we can establish styles to this element. For example:

<div id="g_id_onload"
     data-client_id="YOUR_GOOGLE_CLIENT_ID"
     data-login_uri="https://your.domain/your_login_endpoint"
     data-context="use">
</div>

Close One Tap when tapping outside

By default, it will close when tapping outside the prompt. If you want this behavior to change, use the data-cancel_on_tap_outside HTML attribute and set it to false. For example:

<div id="g_id_onload"
     data-client_id="YOUR_GOOGLE_CLIENT_ID"
     data-login_uri="https://your.domain/your_login_endpoint"
     data-cancel_on_tap_outside="false">
</div>

Managing the state with cookies

If you are using cookies, then you can toggle the One Tap display status if a cookie is present or not. To do so, use the data-skip_prompt_cookie HTML attribute and set it to the cookie name you want to use to toggle the display.

It is simple:

If the cookie is not set or the value is empty, Google One Tap will behave normally.
If the cookie is set and the value not is empty, Google One Tap will not display.

Example code snippet:

<div id="g_id_onload"
     data-client_id="YOUR_GOOGLE_CLIENT_ID"
     data-login_uri="https://your.domain/your_login_endpoint"
     data-skip_prompt_cookie="sid">
</div>

A simple demo

For this blog post, I have developed a simple frameworkless static site with a live demo you can try by yourself. Only HTML, CSS and JS are used... all is client side.

In the demo, a popup for Google One Tap will be shown in the top right corner and once you give consent to the application, it will be auto authenticate yourself in subsequent visits to the demo. There is also a "Sign in with Google" button in the case you dismiss the modal (to trigger the authentication), this button behaves similar to the modal, pre-selecting a Google account you are already authenticated with (this can be customized).

Here is a sample view of what happens when you authenticate:

Note: The picture and possibly sensitive data has been removed from the image, but will be shown to yourself if you try it.

The following scopes are requested when you use the demo webapp:

API	Scope	Description
-	.../auth/userinfo.email	See your primary Google Account email address
-	.../auth/userinfo.profile	See your personal info, including any personal info you've made publicly available
-	openid	Associate you with your personal info on Google

Don't worry about the data or consent you are giving to this sample app, this data never leaves the browser. Trust me, I am not interested in collecting data in any way or do something with it. To do so, I would have to deploy some kind of service somewhere and I don't want and neither have time and motivation to do so. This is not an "excusatio non petita, accusatio manifesta", I only want to be clear with the treatment of your data. I invite you to check the source code for the demo.

In other case do not try the demo app 😢.

Going further into details

The are other "advanced topics" that this post does not cover and you may be interested to learn. So here is a set of different topics and links to go further:

Hope they are useful for you!

Conclusion

As shown and explained in this post, I consider Google One Tap a great, quick and easy "flow" to authenticate your users in a webpage or app.

It can be incorporated as a requirement to comment in your blog, vote in a poll, pre-populate an account information if the user is signing up in your app and lots of other things your imagination could think of.

What do you think? Would you use it?

Happy coding! 🎉🎉

Improving your code for style, quality, maintainability, design... with Roslyn Analyzers

Raul Piraces Alastuey — Mon, 31 May 2021 06:18:01 +0000

Roslyn Analyzers analyze your code for style, quality and maintainability, design and other issues.

I stumbled upon Roslyn Analyzers while contributing to an issue to the Microsoft Bicep repository, where I found a BannedSymbols.txt file where it appeared that System.Console.Write and System.Console.WriteLine where being targeted and pointing to not use them for logging purposes.

That triggered my interest, as I tried to put a simple Console.WriteLine statement and an alert similar as the image above appeared in Visual Studio.

I thought that these kind of "custom rules" combined with the csproj TreatWarningsAsErrors option (<TreatWarningsAsErrors>true</TreatWarningsAsErrors>) could be a very great solution to maintain dotnet projects code quality, maintainability, design and style in a nice way. In my opinion, more useful and necessary in OSS projects or projects with lots of people working on it.

Using BannedApiAnalyzers in a dotnet project

Using BannedApiAnalyzers in a dotnet project is easy:

First of all, install the Microsoft.CodeAnalysis.BannedApiAnalyzers NuGet in the project you want to use this feature.
Place a BannedSymbols.txt file in the project and mark to include it in the project. For example modifying the csproj:

<ItemGroup>
  <AdditionalFiles Include="BannedSymbols.txt" />
</ItemGroup>

Or with Visual Studio, specifying the file properties:

Include your own custom rules to ban a symbol with the following format (description text is optional and will be displayed as description in diagnostics):

{Documentation Comment ID string for the symbol}[;Description Text]

That's all! The fields in the BannedSymbols.txt file will be processed and mark as warnings the use of the specified banned symbols. These warnings are of type RS0030, RS0031 or RS0035. More info can be found in the roslyn analyzers repo.

Take into account that we could use a BannedSymbols.txt file per project or a Solution wide one, including the same BannedSymbols.txt file in all projects.

How to specify the rules

As explained above, the entries in BannedSymbols.txt must have the following format:

{Documentation Comment ID string for the symbol}[;Description Text]

For details on the ID string format, they recommend to take a look at "Documentation Comments" docs.

Nevertheless, we have a awesome example in the "How to use Microsoft.CodeAnalysis.BannedApiAnalyzers" docs.

Taking this example, considering the following code:

namespace N
{
    class BannedType
    {
        public BannedType() {}

        public int BannedMethod() {}

        public void BannedMethod(int i) {}

        public void BannedMethod<T>(T t) {}

        public void BannedMethod<T>(Func<T> f) {}

        public string BannedField;

        public string BannedProperty { get; }

        public event EventHandler BannedEvent;
    }

    class BannedType<T>
    {
    }
}

We can ban different symbols regarding the code above, taking a look to the following table:

Symbol in Source	Sample Entry in BannedSymbols.txt
class BannedType	T:N.BannedType;Don't use BannedType
class BannedType	T:N.BannedType`1;Don't use BannedType
BannedType()	M:N.BannedType.#ctor
int BannedMethod()	M:N.BannedType.BannedMethod
void BannedMethod(int i)	M:N.BannedType.BannedMethod(System.Int32);Don't use BannedMethod
void BannedMethod(T t)	M:N.BannedType.BannedMethod`1(``0)
void BannedMethod(Func f)	M:N.BannedType.BannedMethod`1(System.Func{``0})
string BannedField	F:N.BannedType.BannedField
string BannedProperty { get; }	P:N.BannedType.BannedProperty
event EventHandler BannedEvent;	E:N.BannedType.BannedEvent

One of the main caveats which I found when using this feature and banning symbols was the use of wildcards... For example if you want to ban all System.Console.Write methods in a project, you must specify all variants of the methods, as you can see in the Project Bicep example.

I made a demo project where I played around in a project with these banning tools, so you can watch how to use them and you can try things:

piraces / BannedApiAnalyzersDemo

A basic demo on how to use BannedApiAnalyzers in a .NET Core project

How this work when working with IDEs or without them

In Visual Studio, this analyzer works out-of-the-box, as you can see in the following image:

And if we enable the TreatWarningsAsErrors option:

Regarding other IDEs, JetBrains Rider also works out-of-the-box with this analyzer:

Regardless of the IDE, the dotnet CLI will show the warnings (or errors) when building or running the project, which is also awesome:

More on Roslyn Analyzers

Microsoft created a set of analyzers called Microsoft.CodeAnalysis.FxCopAnalyzers (which is now deprecated) that contains the most important "FxCop" rules from static code analysis, converted to Roslyn analyzers. These analyzers check your code for security, performance, and design issues, among others. Check out how to use them.

These analyzers have been consolidated in different packages.

The BannedApiAnalyzers is one of them, but there are others also as useful as this one:

Microsoft.CodeAnalysis.NetAnalyzers: Included by default for .NET 5+. For earlier targets see this.
Microsoft.CodeAnalysis.PublicApiAnalyzers: Helps library authors monitor changes to their public APIs (more info).

Check out the Roslyn Analyzers repository for more information.

Conclusion

I found this analyzer very useful and I personally will make use of it in my projects to improve the code quality and maintainability.

I think Roslyn analyzers are very powerful and can provide very useful features in our dotnet projects. It is worth trying it out in my opinion.

What do you think?

Happy coding! 🎉🎉

Awesome console apps in C#

Raul Piraces Alastuey — Wed, 09 Dec 2020 00:00:00 +0000

This post is part of C# Advent Calendar 2020.

It has been a while and console apps are still a thing (maybe more used by developers), but they are used everyday.
For example, as a developer, I use npm or dotnet command line programs to do neccessary tasks in my job or in my free time experimenting.
These type of tools have been since the begining of times, but also evolving, since the terminal support for different colors, rich language, and other feautures has been improved.

We now see some programs representing progress bars, spinners, and some other features.
But how can we do it like this programs?

How to make the console more "attractive"?

I remember executing console apps that shows a 'FIGlet' as a banner, introducing the main execution for the program, and then printing some output in an ordered way that could be easily understood, but maybe not familiar for some users and/or 'difficult to understand'.

A sample FIGlet from the program dnschef

I feel this kind of tools intuitive and fancy in some way, and I like to use them too. But, I think we need to understand that terminal is also evolving, and the more intuitive and easier we could make this console apps, the better for our possible audience (our target users).

The are almost support in almost every language and libraries (like Rich for Python) that we can use to improve the final user experience.

In the case of dotnet, two of the most heared libraries/toolkit (at least for me) that will make our console apps great are:

Spectre.Console a .NET 5/.NET Standard 2.0 library, heavilly based in the Rich library for Python.
gui.cs a simple toolkit for buiding console GUI apps for .NET, .NET Core, and Mono that works on Windows, Mac, and Linux/Unix.

It is also support for doing great things in console in the dotnet framework but first lets see what this libraries can do for us.

Let's get with them!

Spectre.Console

As said above, Spectre.Console is a .NET 5/.NET Standard 2.0 library that makes it easier to create beautiful, cross platform, console applications (heavily inspired by the excellent Rich library for Python).

Spectre.Console, has support for tables, grids, panels and markup language.
Has also support for the most common SRG parameters when styling and support for up to 24-bit colors in terminal (the library will also downgrade the colors depending of the current terminal).

A picture showing the main feautres of Spectre.Console

The documentation it is very well written and full of samples to get started with every feature of the library.

With this library, we are capable to improve data output to the user, better progress indicators and better prompting for user decisions.

On my way doing some research of this library, I developed a console app as a demo of the use of this library:

piraces / ConsoleCoin

Simple console app to track CryptoCurrencies prices

Example console app with Spectre.Console

I felt the library easy to understand and use, with great support and documentation, it is easy to improve some of our console apps using it. Give it a try!

Gui.cs

gui.cs, its a simple (but complete and full of features) toolkit for buiding console GUI apps for .NET, .NET Core, and Mono that works on Windows, Mac, and Linux/Unix. Made by Miguel de Icaza, actually working in Microsoft.

In this case this toolkit I feel like it is oriented for more complex console apps trying to expose some UI for its users.

It contains out of the box features and support for various controls for building text UIs: buttons, checkboxes, dialogs, labels, menus, progress bars...

But that is not all! It is also cross platform, supports keyboard and mouse input, flexible layout, clipboard support, and others...

Features of gui.cs

I have tried this library but not all its features, and as far I can tell is that is a very awesome library with you could build almost any kind of app I can think of (replacing the user interface for a text based one).

Other ways...

For sure, the Console class included in the .NET API, expose lots of properties, methods and events to improve the console experience.

The libraries shown above are useful for the most of apps in order to not reinvent the wheel. But, in case you feel like you do not need them or you want to implement it better on your own. There is always the possibility of doing awesome thing with the dotnet framework itself alone.

Conclusion

Console apps exist and will be there for a long time, they improve, and also the terminal they run in. Trying to make them more user friendly and more "understandable", its for sure a very good bet for get our users comfortable using our tools/apps.

In my case, I love executing almost everything I can in the terminal. I love console tools and apps, and I also love how they make the most out of the terminal they run in to give you the most.

Hope this post will be useful for your console tools/apps!

This post is part of the C# Advent Calendar 2020, thanks for letting me participate with my humble post!

Happy Coding! Happy Advent! Happy Christmas!

Happy Advent and happy christmas!!

Performing static code analysis of your Kubernetes object definitions with a Github Action

Raul Piraces Alastuey — Mon, 14 Sep 2020 18:28:36 +0000

Nowadays, Kubernetes is one of the most popular and loved platforms used by the community, to run and orchestrate container workloads.

It is present in several open-source and private projects as the base of its infrastructure. They trust to use Kubernetes as a platform.

The problem

The world of Kubernetes can be sometimes very complex and if we make use of it, we need to ensure we are doing it right.

SwiftOnSecurity

@swiftonsecurity

One time I tried to explain Kubernetes to someone.
Then we both didn't understand it.

15:40 PM - 06 Aug 2019

478 2817

There are lots of tools that can ease that work for us to simplify the process to get Kubernetes working correctly as our platform.

As you may know, the Kubernetes workloads are most commonly defined as YAML formatted documents. Sometimes, it is rather hard to express constraints or relationships between manifests files.

What can be done to help ease the process of using Kubernetes correctly?

There are existing tools to integrate static checking allowing catching errors and policy violations closer to the development lifecycle.

These tools gives us the guarantee around the validity and safety of the resource definitions is improved, therefore you can trust that production workloads are following best practices (which is a must nowadays).

As developers, we need to guarantee the validity and safety of our Kubernetes manifests to ensure the security and integrity of our production environments.

Here comes when the tool kube-score can help us.

Kube-score is an open-source tool that performs Kubernetes object analysis with recommendations for improved reliability and security.

How can we introduce kube-score in our CI/CD processes?

GitHub Actions to the rescue!

My Workflow

My developed GitHub action (kube-score check), allows GitHub users to execute kube-score in their workflows along other actions and guarantee the validity and safety of their Kubernetes manifests.

It is very simple to use, you only have to perform a checkout of the project repository and execute the action, which takes as input an array of manifests (or directories with manifests) which you want to validate and test (wildcards supported):

Read the full Readme of the project to see all options.

Submission Category:

Maintainer Must-Haves / DIY Deployments

Yaml File or Link to Code

Please take a look at the action repository, and give any feedback if you want!

Contributions are welcomed too!

piraces / kube-score-ga

Github action to execute kube-score with selected manifests (YAML, Helm or Kustomize)

kube-score Github Action

This action executes kube-score with selected manifests (with support for YAML, Helm or Kustomize manifests).

Features

💻 Compatible with Windows, Linux and Darwin Operating Systems.

🏗 Supported architectures: ARMv6, ARM64, x64.

📂 Multiple folders and files supported within one run of the action (with wildcards support).

🔢 All versions of kube-score can be selected and used.

⚡ Support for caching kube-score tool to improve speed in subsequent runs.

Inputs

`kube-score-version`

(Optional): The version of kube-score to use. Defaults to the latest available.

`manifests-folders`

Required: An array of relative paths containing manifests to analyze with kube-score (separated with commas). It is mandatory to establish a wildcard for the files or the concrete filename.

Example: ./manifests/*.yml,./other/manifests/*.yml

`ignore-exit-code`

(Optional): Will ignore the exit code provided by kube-score, will always pass the check. This could be useful in case of using the action in an information way.

…

View on GitHub

See the YAML file for the action: action.yml

Additional Resources / Info

In the links below, you can see the GitHub action running with different forks of popular projects, such as the Application Gateway Ingress Controller (AGIC) of Azure, or the examples provided in the official repo of Kubernetes.

Take a look and see how AGIC Custom Resource Definitions (CRDs), are passing all validations and recommendations (as is a ready to use in production controller).

See the GitHub Action in use:
- Kubernetes/examples repository (fork): https://github.com/piraces/examples/actions
- Azure/application-gateway-kubernetes-ingress repository (fork): https://github.com/piraces/application-gateway-kubernetes-ingress/actions
GitHub Repository: https://github.com/piraces/kube-score-ga
GitHub Marketplace: https://github.com/marketplace/actions/kube-score-check

Getting notified on the latest releases of your day-to-day tech stack

Raul Piraces Alastuey — Sat, 04 Jul 2020 15:30:00 +0000

Nowadays there are lots of technologies and dependencies involved in our day-to-day projects, therefore making it more difficult to know them all and to track their changes.

New releases of tools, software, and software development related technologies happen almost every day. This can be sometimes kind of frustrating. You can be working in a brand new project using the latest available versions, and in the process of developing the project, versions bump and new features and fixes are available. This don’t have to be bad indeed… These new versions can improve our product/project and solve some problems for us, and we know we have to update them someday and I personally think it’s better to bump version by version than bumping from one version to another with several versions in the middle (which may result in a complicated process).

Then… How to get up-to-date with all of these? Can we manage to get notified every time a technology we use is updated? Sure we can do it!

Approaches

There are different approaches to accomplish this. We are going to review some of the most used ones and the ones that I prefer.

Using GitHub notifications

Nowadays with the Open Source initiative, the majority of projects we use to develop every day are open sourced and hosted in platforms like GitHub, GitLab and others (for example: Node.js, Deno, Vue.js, Microsoft projects…).

This projects usually manage their releases using GitHub releases, so we can use the built-in GitHub notifications to get notified on every release of the projects we want to track in a very simple way (just tap on watch and select releases):

This way new releases will appear in our feed of notifications in our profile of GitHub. Depending in our notifications configuration, we can achieve to get this notifications in our mail too.

Simple.

Nevertheless, I found this process a little bit confusing sometimes and I missed some of the new releases of my used projects. This can be because, in my case, the GitHub notifications feed was plenty of notifications of mentions, PRs, reviews, and others. So, I tried to change this and organize myself better, moving this notifications to a RSS feed.

GitHub release notifications in a RSS feed

GitHub provides an RSS feed for releases for every project, which follows the following URL structure:

https://github.com/{USER}/{PROJECT}/releases.atom

For example:

https://github.com/nodejs/node/releases.atom

View the raw feed

Knowing this, we can use our favorite RSS reader/client to group RSS feeds of different releases and use it to read and be notified for the latest releases.In my case, I use Feedly, to make a feed and group different sources naming and organizing them:

In this way, I personally check the feed every day if new releases have been published or if some versions have went out of preview to an stable version.Also note, that it’s always important to check the changelog usually provided with each release, before attempting to use the last version (there could be breaking changes or deprecated functionalities you’ll need to manage).

Not on GitHub? No problem…

There could be some projects that are not open sourced, or using another Version Control platform like GitLab. In the case of GitLab, we can subscribe to release notifications like GitHub and manage them to get the notifications in our mail easily as explained with GitHub above:

What if a project is not open sourced or not in any of the mentioned platforms?

Well… There are also plenty of options to keep track of them. Usually largely used projects tend to have a Twitter account, a blog, or other resource we can watch and keep up-to-date.

In this case, to make all notifications came to the same place we could use some mechanism of automation to watch for us blog updates or twitter accounts and notify us on new releases.

Some tools like IFTTT, provides us simple yet powerful workflows to automate this kind of things. For example, we can set up a workflow to get new releases information from a blog (via RSS) to our mail:

Other options could be to watch a Twitter account and send it to our mail, set up an RSS feed of a Twitter account updates which contains certain words (like “release”), and other workflows you could imagine.

Personally, I have not found a project whose releases I couldn’t track.

Conclusion

There are plenty of options to get up-to-date with your tech stack or favorite tooling. Simply use the most comfortable to you, or the one you find more productive.

I’m sure this process will help you out to keep you up-to-date and be a more informed and organized developer.

Do you use another way to keep up-to-date with new versions and releases?

Happy coding! 🎉🎉

Automating Docker container base image updates with Watchtower

Raul Piraces Alastuey — Sun, 10 May 2020 20:00:00 +0000

In the case you are using Docker for your developments, projects, apps… Normally you use base images to start with. These base images are (normally) frequently updated, and in some cases we are going to want to keep up to date with a tag of a base image. In this cases, Watchtower is a great devops tool that is going to manage this cases for us with very little hassle.

For example, lets suppose having a very simple a Python project containerized with the official Python 3 base image.Our docker file references the base image with the 3 tag:

FROM python:3

ADD my_awesome_script.py /

RUN pip install numpy

CMD ["python", "./my_awesome_script.py"]

If we build the image and deploy the container, it’s going to keep the latest base image of Python 3 at the moment of building it, but we want this base image to keep up to date to benefit from all changes and improvements being made in the different versions of Python 3.

In this cases Watchtower will help us!

Overview

With Watchtower we can update the running version of our containerized apps automatically. In the case of using our own base images, by simple pushing a new image to the Docker Hub or our own image registry.

Watchtower will pull down the new image, shutdown gracefully the existing container and restarting it with the same options that were used when initially deployed.

This can be achieved as simple as running the following command in the same machine our container is deployed:

docker run -d \
    --name watchtower \
    -v /var/run/docker.sock:/var/run/docker.sock \
    containrrr/watchtower

Watchtower will then start monitoring our running Docker containers and watch for changes to the images that those containers where originally started from. If a change its detected, Watchtower will automatically restart the container using the new image.

In the previous python example, Watchtower will pull the latest python base image every few minutes, and compare it to the one used to run the container. If a change its detected, it will stop/remove the container and restart it with the same options.

Note : Since Watchtower code needs to interact with the Docker API in order to monitor the running containers, we need to mount /var/run/docker.sock with the -v flag when running it.

Features

Watchtower can be used as simply as the example above, but it has several features and options that make it a great tool in very different scenarios.

Let’s see them.

Configuring the Watchtower execution

There are many arguments we can use to customize the behavior of Watchtower.Some of the most interesting ones are:

--cleanup: remove old images after updating.
--host: the Docker daemon socket to connect to.
--include-stopped: will also update created and exited containers.
--revive-stopped: with the --include-stopped argument, it will start the containers after updating them.
--interval: poll interval in seconds.
--label-enable: to only update containers with the label com.centurylinklabs.watchtower.enable set to true.
--monitor-only: only monitor, not update.
--no-restart: useful if using an external system manages the containers.
--run-once: run an update attempt one time and then exit.
--schedule: allows us to specify a Cron expression which sets when to check for updates.

Container selection

As we have seen, Watchtower will watch all containers and try to update them. However, in most cases we will need to specify which containers should be updated.

We can control this for example when running the Watchtower container, specifying the name of the containers to watch:

docker run -d \
    --name watchtower \
    -v /var/run/docker.sock:/var/run/docker.sock \
    containrrr/watchtower mycontainer myothercontainer

But the recommended way is to set the label com.centurylinklabs.watchtower.enable as true or false in our containers depending if we want Watchtower to update them or not. This can be achieved inserting the label in the Dockerfile or when running the container with the --label flag.

Note : the mentioned label with a value of true is only needed if we start Watchtower with the --label-enable flag. This will only update containers with that flag set as true, otherwise it is not needed as it watch all containers by default (which don’t have the label set as false). Useful to reverse the default behavior of Watchtower.

Notifications

Watchtower is able to send notifications when containers are updated. The types of notifications to send are set by passing a comma-separated list of values to the --notifications option or the WATCHTOWER_NOTIFICATIONS environment variable, which has the following valid values: email, slack, msteams and gotify.

We can also specify the log level for notifications using the --notifications-level option or the WATCHTOWER_NOTIFICATIONS_LEVEL environment variable, with the following values: panic, fatal, error, warn, info, debug.

In order to configure notifications for the different services, take a look at the official guide, which everything is covered in detail.

Using private Docker registries

In the case of using private Docker registries, we need to supply Watchtower the authentication credentials for the registry with the environment variables REPO_USER and REPO_PASS. Another way to do this is to mount the host’s docker config file into the container (at the root of the filesystem /).

Example:

docker run -d \
  --name watchtower \
  -e REPO_USER=username \
  -e REPO_PASS=password \
  -v /var/run/docker.sock:/var/run/docker.sock \
  containrrr/watchtower container_to_watch --debug

More features!

There are other features (such as linked containers, remote hosts, secure connections…) not covered in this blog post that can be useful in some cases too, please take a look to the Watchtower project page to see them all in detail.

Conclusion

We have seen how to automate the update process of Docker images in our machines using a simple yet powerful tool.This tool can be used in our development or production environments to simplify the process of keeping images of containers up-to-date.

I have also seen this tool being used in personal environments, such as Raspberry ones with docker containers where in that case are used for entertainment purposes (like Plex) and its very useful too!

Personally, I am using this tool and I am very happy with the results. What about you? Give it a try!

Happy deployments! 🎉🎉

Replacing self-hosted agents with ephemeral pipelines agents in Azure DevOps

Raul Piraces Alastuey — Sun, 15 Mar 2020 17:00:00 +0000

Schema of sample architecture using ephemeral agents

If you have Azure Resources that aren’t exposed on the internet but only accessible via a private network, you can’t use Microsoft-hosted agents because they can’t connect to the private network. Therefore, we need to maintain a pool (or several pools) of self-hosted agents, with the associated costs and effort to maintain that pool(s).

In this case it is where Ephemeral pipelines agents come into action.

Overview

Ephemeral pipelines agents come to eliminate the need to use and maintain pools of self-hosted agents for deployment purposes.This type of agents are capable to deploy to private azure resources. The process is not very complex. Ephemeral pipelines agents run in an Azure Container Instance (ACI) with access to the private network where the other Azure resources are. The agents are created to run a pipeline job and then deleted to avoid extra costs and resources consumption.

This way, we can deploy to private Azure resources without having to expose them on the internet or having to maintain self-hosted agents on the same (or with access) virtual network (with their associated costs and cons).

The agent that runs in the ACI its only a basic Docker image with the necessary dependencies to run the agent and to be capable to deploy to our private resources. For example, a base Ubuntu image with the necessary dependencies and the deploy agent installed.

TL;DR: The purpose of this task is to create a short-lived Azure Pipelines Agent to run a deploy in a private virtual network so we can deploy to Azure Resources that are not internet accessible.

⚠️ Important: This approach (and task) is currently in preview and has known issues and limitations. Please take this into account before proceeding to use this process.

How does the process works

Only three steps/requirements:

One docker image that can run a deploy agent in a container (needed to provision agents).
Provision one ephemeral pipeline agent to run the deployment job. This process can be done using this task. This task provisions, configures and registers an agent on an ACI using the docker image mentioned in the first step.
The container runs one pipeline job, and then it unregisters the agent and deletes the container (it self destructs).

So considering we have setup a docker image that can run a deploy agent in a container, our pipeline will look like this:

Example pipeline with ephemeral agents

As you can see, two jobs are needed:

The first one provisions the agent in one pool to run the deployment job.
The second one (depending on the first being finished correctly), runs the deployment job. It runs on the agent that the first job has provisioned.

Tutorial

In this post I will guide you through a simple tutorial on how to deploy assets on a container in an Azure Storage account in a private virtual network, using Ephemeral pipeline agents.

So let’s get started!

Requirements:

A virtual network with a security group.
A dedicated subnet in the virtual network to run the ephemeral agents.
The agent must run in the same Azure location as the virtual network.
All the created subnets must share the same security group.

Overview of the resources

For this tutorial I have created two resource groups:

One for the virtual network and the security group.
Another one for the Azure Container Registry (ACR) and the storage we will deploy to.

Resource group for the VNet and security group

Resource group for the ACR and Storage account

Also, the Azure Storage account should be in the private network, with a configuration similar to this one:

Azure Storage Account VNet configuration

If you need it, the main repo of ephemeral agents has sample scripts on how to deploy these resources.

Pushing the base image for ephemeral agents

Once finished configuring our resources in Azure, we will need the base image to use with the ephemeral agents.In this case I have used the Agent Images available in the GitHub repo, specifically the Ubuntu one. We can use the pipeline in the GitHub repo to deploy our Agent image in the ACR (using an ACR Service connection).

After that, we should have one repository with one image in our ACR:

ACR with sample Agent Image based on Ubuntu

Setting up the pipeline in Azure DevOps

After creating the needed resources and pushing the Agent Image to the ACR, we are ready to create our pipeline and use ephemeral agents to deploy.

First of all, we will need to create an additional agent pool in our Project, and grant access to all pipelines.

Permissions:

For this process we will need to give specific permissions in order to allow the pipeline to register agents in the new pool, running jobs in these agents and unregister them.

In order to register the agent a token with sufficient permissions on the Azure DevOps Organization the agent is going to be registered is needed.

You can use two different types of tokens: a Personal Access Token or an OAuth token (recommended).

If using an Personal Access Token (PAT), it requires the Agent Pools Read & Manage scope and that the user who owns the PAT has administration privileges on the agent pool you intend to register the agent.
If using an OAuth token, the best approach is to use System.AccessToken which are short lived, dynamic and automatically managed by the system.

If using the OAuth token, you need to met the following conditions:

The timeout for the job that creates the agent only expires after the deploy job in the agent is finished (because the token is used to register and unregister the agent in the pool).
The account Project Collection Build Service (YOUR_ORG) needs to have administration permissions on the pool (granted at the organization level, not at team project level).

In this example, permissions on the pool (for the OAuth token approach) will look like the following:

Pool permissions for OAuth token

Additional considerations:

You must register an Azure Resource provider for the namespace Microsoft.ContainerInstance in order to create container instances for the agents. This can be done easily opening a Powershell instance in the Azure portal and executing the following command:

Register-AzureRmResourceProvider -ProviderNamespace 'Microsoft.ContainerInstance'

This will register the needed Azure Resource provider.

Service connections:

We will need a Service connection using a Service principal, which has to have access to the VNet resource group and the ACR and Storage Account resource group. We can set it up in the project settings in Azure DevOps, granting permissions to one resource group and then going to the Azure Portal to grant permissions to the another one. Make sure to grant permissions to all pipelines.

Also, as mentioned above we will need an ACR Service Connection to our ACR to run the ephemeral agents in the ACI from the Agent Image in the ACR (granting permissions to all pipelines).

Pipeline definition:

In this part of the process two pipelines are created for my sample project, one for creating the base image for the agents in the ACR (which we have previously created), and another one for the main deploy process of our project.

Pipelines defined in our projects

The main deploy pipeline used in this tutorial is the same defined in the GitHub repo sample. As we have seen in the previous stages, we have to define two jobs: one to provision the agent and another one to perform the deploy job.

Configure the variables in the sample pipeline to reference your resources correctly and then try to run it:

Pipeline preparing the agent in the first job

If we have configured everything correctly, the pipeline will succeed:

Final pipeline result

If we inspect our created agent pool, we will see the executed job but no agents registered in the pool (which is the main purpose of this process). This is because the agent has unregistered itself when the deployment job finished.

Pool executed job

Final result:

After the deploy pipeline execution, we can check how we successfully have deployed our assets to the Azure Storage container connected to the private virtual network:

Deployed assets on Azure Storage container inside the VNet

Conclusion

We have seen how ephemeral pipelines agents works, how to replace self-hosted agents with ephemeral ones and the most important part: how to reduce maintenance costs of having our own self-hosted agents pool(s).

It is important to emphasize that this process its currently in preview, so maybe some things just do not work out of the box, but I personally think its an excellent approach to avoid having self-hosted agents in the mentioned situations.

Happy deploy! 🎉🎉

Sources:

Understanding and managing peer dependencies in your project

Raul Piraces Alastuey — Sun, 19 Jan 2020 12:00:00 +0000

Have you ever seen any similar warning to the ones in the image above?

That is completely normal.We usually install several libraries alongside our frameworks of choice in our projects. Each library and framework, has its own internal dependencies and other defined as peerDependencies.

What are peer dependencies?

Well… a good definition can be found in this Stack Overflow answer:

peerDependencies are for dependencies that are exposed to (and expected to be used by) the consuming code, as opposed to “private” dependencies that are not exposed, and are only an implementation detail.

Libraries and modules declare in their own package.json their own internal dependencies and peerDependencies.These peer dependencies are dependencies that are exposed to the developer using the module, consuming the code. The module is telling us that it is a dependency we should take care about and install it. The module may expose an interface where the peer dependency is used, and then we should use a compatible version of the peer dependency to use the code in order to ensure a correct behavior with no errors.

For example some module, lets say module-a, has a peer dependency with module-b.If we install module-a in our project, yarn or npm will warn us about a peer dependency on module-b that we should install.

If we do not use module-b in our project, the solution is pretty simple:

Install the module as a dependency in the project. Depending if the module is for developing, testing, building our project or not; we should install it as a dev dependency or dependency correspondingly.

But what about if we are already using that module-b in our project?

Conflicts…

Following the example above, if we have compatible versions of module-b then there is no problem.For example, the module establish a peer dependency with module-b on version 2.0 and we are using the same version in our package.json (or declares a range of versions which our dependency is in).

If we use another version not the same of the peer dependency or not in the declared range, then npm or yarn will warn us about a conflict in peer dependencies in the installation process.

Then the approach should be adapt our dependency on module-b to a compatible version with the version used by the module-a:

In some scenarios this can involve changing our code, because changing to that module version has breaking changes.
In other scenarios, these can be simply solved without changing our code, only changing the dependency version.

How do I solve this?

There are some public projects of npm packages that solves this for us automatically, but I would personally not recommend these. Using them could cause us more problems than we previously have. Also, doing it manually as explained in the cases above, allows us to have more knowledge about our dependencies in our project and how the dependencies work.

The manual process involves tinkering with the package manager, installing/removing packages, adjusting versions and in some cases changing our code (as explained above).

With this two approaches choose the one that you think fits you the most.

Conclusion

Sometimes is usual to ignore these warnings, but warnings are something we should care about.Knowing how peer dependencies work, we can get rid of these warnings, ensuring our project will work fine with all external dependencies.

It can be a little tricky to solve all warnings in certain scenarios, where it may imply changing our code and how we use certain modules, but for sure it is worth it. This makes our project more stable, resilient and ensures no unexpected errors when using some modules.

Happy codding!

My way on automated dependency updates

Raul Piraces Alastuey — Sun, 15 Dec 2019 18:00:00 +0000

In order to keep my dependencies updated in all my personal projects automatically, I use Dependabot in combination with GitHub Actions, Travis CI and Netlify deploy previews.

Let’s break into each step/tool of my workflow:

Dependabot

Dependabot is the responsible of checking for package updates, open new PRs for the updates, and merge them automatically.

It has many options in order to adapt it to any project or desired behavior. In my case, I have set it to Auto bump all package versions specified in the package.json, scheduled daily and without any filters. I also set it to add a ‘dependencies’ label to each opened PR (in order to difference them from other PRs).

There are other options regarding scheduling, pull request, merge and rate limiting options. These are configured as following for my personal repositories:

In my case, I have scheduled to run every day in the morning (except for weekends).

Regarding the PR options, I have specified to automatically rebase PRs if they have conflicts, which it is very useful when there are several PRs opened and merging one, generates conflicts with other.

I have also check the options to use directory branch names and to include security advisory details because I think they are useful and informative.

To configure Dependabot automatic merge, we have several options, filters and others.

Regarding automatic merge, I have set to treat PR approval as a request to merge (which I automate in the next section with a GitHub action).

I have also set to create a merge commit if the PR is merged by Dependabot to keep track of it.

The are other options for automatic PR merging, like enabling auto-merging to be enabled on projects. Then, Dependabot will auto-merge PR if there are no conflicts and checks for PRs passes (in projects with this option enabled).

The only reason I have included the additional step of the GitHub action is to give more flexibility to my workflow in the future , so I can add custom logic for each project and customize when the PRs should be merged (since by configuration, Dependabot PRs depend on my approval, which the GitHub action gives). But the action, could not be necessary at all.

GitHub Actions

In my approach, I use GitHub actions to give the necessary approval to Dependabot PRs which is needed by Dependabot to auto-merge them (Dependabot treat the approval as a request to merge the PR).

Actually I approve every Dependabot PR without any additional logic. Nevertheless, I think this step could bring me more flexibility to add custom logic on when a specific Dependabot PR should be merged.

For example, if I have an specific library which I want to freeze (not upgrading it), or any other related logic, I can add this to the Github Action.

The base action I use for this right now is hmarr/auto-approve-action (v2).

Travis CI

I have set up Travis CI on my projects to check commits and provide additional PR checks to the Dependabot PRs.

In Travis CI, unit tests, the linter and other checks are run to check everything work as expected. Nevertheless, I consider this is not enough testing for the automated PRs… So I am considering to add e2e tests in the future (most probably using Cypress).

If this check runs ok, Dependabot can continue with the automatic merging… but there is another additional check: Netlify.

Netlify

My automated dependency updates projects are deployed on Netlify, so I added the option on Netlify to make a preview of the deploy and check if it does not fail to deploy and it looks and works as expected.

This is only an additional check to make sure the automation is enough covered.

Workflow

After explaining each check and tool, let’s get into the main workflow and what happens when Dependabot triggers.

My current automated workflow has the following steps:

Dependabot checks Daily for available updates for packages listed in each package.json of my projects (currently I am only using it in JS projects, but it support other languages).
- Whenever an update is available, it opens a new pull request to the project including the corresponding changes, estimated compatibility and the package changelog.
- This pull request triggers all checks for pull requests, which includes one GitHub Action that approves the PR in my name (hmarr/auto-approve-action), Travis CI checks to check the update does not break anything and a Netlify deploy action to preview the deploy and test there is no problem when deploying.
Once all checks have successfully passed , Dependabot automatically merges the PR to master which triggers the deploy to production (rebasing the branch if necessary).
If one or more checks fail , Dependabot comment on the PR explaining it won’t merge the PR since some CI checks failed. Then we should analyze the problems and manually fix them to upgrade the package. Nevertheless, if we think it can be something related to the package itself, or it is not worth it to update, we can just close the PR or tell Dependabot (using the options available in the Dependabot PR description in the PR comments) to not upgrade that package or wait to other version.

This is the final result:

Daily automated dependencies updates for my main personal projects, simple and without fear of breaking them.

How do you do in order to manage your dependencies their potential security vulnerabilities?

Happy codding!

The NTP Pool project: How to use and contribute

Raul Piraces Alastuey — Sat, 14 Sep 2019 14:27:03 +0000

Introduction

The NTP Pool Project is a large virtual farm of servers that offers NTP service for anyone. The project consists of a DNS system that balances the load of millions of queries of time synchronization for devices all across the world (tablets, smartphones, computers, routers...). Also vendors like Ubuntu, use this service for all its clients (it's also commonly used in many other Linux distros). The actual goal is to provide real and accurate time synchronization to this devices, thanks to the great number of servers, using them to divide the load along the pool.

Actually, the pool has around 4000 servers in different stratums, syncing time across them and offering time synchronization services.

These servers are in different stratums following the NTP hierarchy:

Hierarchy of an NTP system

To know more about different stratums and the characteristics of each stratum servers you can take a look here.

Using the pool

If you are interested in using this awesome service, please follow the instructions on the official page: How do I use pool.ntp.org?

Which is basically changing your NTP servers to point to the servers in the pool:

server 0.pool.ntp.org
server 1.pool.ntp.org
server 2.pool.ntp.org
server 3.pool.ntp.org

The 0, 1, 2 and 3.pool.ntp.org names point to a random set of servers that will change every hour

Contributing

"Individually, we are one drop. Together, we are an ocean."
-- Ryunosuke Satoro

Contributing is very easy, so if you feel like wanting to contribute, simply follow the next steps.

Can I contribute?

To contribute you only need a static IP and permanent public network access. If you met these requirements, try and follow the tutorial to contribute to the project!

Note: please read carefully the official page on joining the pool, before you proceed (https://www.pool.ntp.org/en/join.html)

Setting up the server

The following instructions are for Linux distributions, specifically for Ubuntu/Debian, but the package and the configuration file are almost the same across all distros. Also, we are going to configure a stratum 3 server, but the instructions are almost the same for a stratum 4 or 2 server.

1. Install the NTP daemon

Simply execute the install command:
sudo apt-get install ntp

2. Choosing static NTP Servers

First we must search for stratum 2 servers to sync with.
It is important to choose at least 4 stratum 2 servers, that are geographically near to our server (at least in the same country).

You can search for servers in this official list: http://support.ntp.org/bin/view/Servers/StratumTwoTimeServers

If you want to configure an stratum 2 server, take some servers from the Stratum 1 list:
http://support.ntp.org/bin/view/Servers/StratumOneTimeServers

Note: choose only 'OpenAccess' servers unless you’ve received approval to choose other type of server.

Once we have the servers, we have to know its IPv4 and IPv6 addresses. Sometimes the IP address it is informed in the list, but if it is not or you want to confirm the IP, we can extract this info by executing a basic dig command. For example:

dig 6.ntp.snails.email ANY

Which produces the following output:

;; ANSWER SECTION:
6.ntp.snails.email. 5999    IN  A   139.162.170.219
6.ntp.snails.email. 5999    IN  AAAA 2a01:7e01::f03c:91ff:fe8b:e9e0

3. Configure the NTP daemon

Once we have all IP addresses of the servers we chose, we are ready to configure the NTP daemon.

Modify the configuration file for NTP, usually located in /etc/ntp.conf. Delete all lines starting with the pool word that are there by default, and then paste line by line the servers in the format:

server ntp_server_1 iburst

Where 'ntp_server_1' is IP address of the server (one address per line).

Also, make sure that your configuration file has a driftfile and the noquery option is present in the restrict lines of the config file.

The driftfile option, helps to achieve a stable and accurate time, storing the frequency offset and the required frequency to remain in synchronization with correct time.

The noquery option does not allow management queries, which is useful to prevent attacks or being vulnerable to queries that could modify the state of the server.

The final config file will be something like this:

driftfile /var/lib/ntp/ntp.drift

server ntp_server_1 iburst
server ntp_server_2 iburst
server ntp_server_3 iburst
server ntp_server_4 iburst
server ntp_server_5 iburst

# By default, exchange time with everybody, but don't allow configuration.
restrict -4 default kod notrap nomodify nopeer noquery limited
restrict -6 default kod notrap nomodify nopeer noquery limited

# Local users may interrogate the ntp server more closely.
restrict 127.0.0.1
restrict ::1

The iburst option following the server, is there because of the NTP Pool recommendations. With this option, if the server is unreachable, this will send eight packets instead of one packet (only the first time).

4. Restarting the NTP daemon and testing the config

Once configured, we simply have to restart the service to load the new configuration:

sudo systemctl restart ntp

After restarting the service, wait around 5 minutes until it stabilizes the time sources, make sure the port 123 (UDP) is open, and then test it. You can also test the service using the command ntpq -p or from other server using ntpdate -q SERVER_IP.

5. Add the server to the NTP Pool

This is the final step. Our server is running and configured correctly, so let's add it to the pool.

Go to ntppool.org, and click in 'Manage Servers'.
Sign up (or login if you already have an account).
Write the hostname of your NTP server or one of its IPv4 / IPv6 static addresses.
Submit it!

If you have also an IPv6 address then submit it too when adding the servers.

Once done, your server must appear in the 'Your servers' list, where you can adjust the net speed of your server to match the requirements.

Finally, the current score of your server will be improving over time. Initially is possible that your server will have a negative or less than 10 points in score, which means it will be excluded from the pool. But, let the service sync the time from sources and it will be added to the pool.

6. Congrats!

Your server is part of the pool and is helping syncing time for several devices.

Thank you for contributing!! 🎉

Originally posted in my personal blog: https://piraces.dev/posts/the-ntp-pool-project-how-to-use-and-contribute/