In April, we recently found out that our spending on GCP services, where we host our projects and applications, is significant. In total, we spent $4,327.45 on GCP during that month. Our client requested us to review these expenses and identify the services that are consuming a large portion of our budget. After investigating our billing cycle for that month, we identified the following services as the main contributors to our expenses:

• Cloud functions
• BigQuery
• CloudSQL

Billing report graph for April:

We identified several action items to reduce our total spending on these services. After implementing all of them, we successfully reduced our costs by approximately $3000 per month.

Billing report graph for November:

However, since we continuously develop projects and plan to upgrade the architecture of our main application to accommodate 3x-5x more users, there is a possibility that our spending may increase in the future. To ensure we monitor our spending on GCP services, we have implemented a system to track our monthly expenses. We have set a specific budget limit and established different levels of alerts to notify us when we approach that limit.

As a solution, we introduced a new micro-service for tracking and monitoring the spending amount at the GCP service. This microservice elevates an alert whenever the overall budget surpasses or meets predefined thresholds. Doing this lets you stay on track with your budget and take action if spending gets out of hand.

Ways to get the GCP budget Alerts

It can be configured for billing accounts and projects, allowing customization based on a specified spending limit or the previous month’s expenditures. These alerts are activated when your spending exceeds a predefined percentage of the budget.

We can receive notifications conveniently through these services according to our preferences.

• E-mail.
• Cloud Pub/Sub events.

Upon reaching the threshold, the emails are sent to billing administrators and the billing account users. It’s important to note that the email template is fixed and cannot undergo customization. On the other hand, Cloud Pub/Sub events offer flexibility by serving as alert triggers, enabling the transmission of notifications to Slack. Unlike the fixed email template, the Slack message is customizable and can be adjusted as per your requirements.

I know you are thinking what is a Cloud Pub/Sub event?  

Cloud Pub/Sub event, must be a bit mysterious at first, but it’s essentially a way to receive updates efficiently. Think of it as a secret messenger silently carrying important messages about your budget and spending. One more advantage is it’s customisable.

Using Cloud Pub/Sub for Slack Alerts

Cloud Pub/Sub event is a way to receive updates efficiently. Think of it as a secret messenger silently carrying important messages about your budget and spending.

If we opt for Cloud Pub/Sub, an event will be published on a topic every 50 minutes approximately. This event will contain important information like the billing account ID, the current threshold limit that has been reached, the budget limit, and the total cost incurred so far for the month.

High-level architecture of the GCP budget alert service should be like this:

In the diagram above we can see how the solution works. Billing Alerts trigger events, which are sent to a Cloud Pub/Sub topic, and then a Cloud Function listens to this topic and handles the billing alert event. The cloud function sends a notification to the Slack Channel.

Setup for the micro-service

Before we start writing the code for our notifications, we need to do a few things first:

1. Create a new Cloud Pub/Sub topic called “billing-alerts” using Cloud Pub/Sub. This topic will be used to send billing alerts. The Topic will be a Pub (Publisher) and an upcoming cloud function acting as a Sub (Subscriber).
2. Go to the Billing section in the Google Cloud Console and set up your budget.
3. In the “Manage notifications” section of the budget setup, connect it to the “billing-alerts” channel we created earlier in step 1.
   Note: In step 1, we created a Cloud Pub/Sub topic named “billing-alerts”.

Set up Slack access: Create a new Slack application by following the instructions at https://api.slack.com/slack-apps. Make sure to keep the Slack Access Token handy, as we will need it later to configure access from our Cloud Function to Slack.

Steps to set up the budget for your project at Google Cloud console

1. Go to the billing section at the GCP console.

2. Click on the Budget & Alert tab on the left side of the page.

3. Click on the Create Budget button on the top.

4. Add a name for your budget, select the time range for the budget alerts, and then select your organization, Project, and Services for which you need the Alerts.

5. Then moving to the next step add the amount of budget as per your requirements or preferences.

6. Now, moving to the next step set the alert threshold rules, and select the PubSub topic we created earlier for “billing-alerts”.

Now the setup for the budget is completed.

We will use a cloud function to subscribe to the Billing Alerts Cloud Pub/Sub topic, we will write the event data to BigQuery, and send a notification to the Slack channel.

Navigate to Cloud Function in the Google Cloud Console and create a new Cloud function with the following values (We can change the values as per our expectations):

• Name: gcp-budget-alerts
• Trigger: Cloud Pub/Sub
• Topic: billing-alerts
• Source code: inline editor
• Runtime: Python 3.8

Function to execute: handle (Endpoint of the cloud function)

The codebase setup for the microservice

The open-source GitHub repository for the codebase setup of GCP Alert Service: gcp-budget-alerts-service. When code setup for the service is done. We just need to deploy the code to the cloud function and monitor the Pub/Sub event trigger.

Our alert message will look like this on Slack:

To ensure the alert messages are sent to the designated channel, it’s necessary to establish and set up a Slack bot as outlined in the repository’s prerequisites.

The alert message contains the percentage of the threshold used, the Total budget amount we fixed, the budget period, the budget name, and The billing account ID.

Note: We can customize the design and data on the alert as needed in the codebase for the GCP alert service.

Impact

Our client wanted to keep an eye on the investments and the spending on the whole project hosted on the GCP so that an informed decision can be taken in case of high spending. 
Implementing this solution helped in the monitoring of the amount of spending on the different services of GCP. As a result, our client now receives regular updates on the current spending amount every month. Furthermore, if the budget is approaching exhaustion, we can promptly take appropriate action.

Outcome:

Now, the client can make informed decisions about budget allocation and optimize resource usage on the GCP platform. With real-time alerts and monthly spending reports, the client has gained better financial visibility into the project. This proactive approach has not only prevented budget overruns but also enabled the client to allocate resources more efficiently, ensuring that the project stays within financial constraints.

Summary/Conclusion

Efficiently monitor your Google Cloud Platform (GCP) budget through Slack alerts. By setting up a micro-service that tracks and monitors your GCP spending, you can receive alerts when your budget surpasses predefined thresholds. 

Configure budget alerts for billing accounts and projects, customize the alerts based on spending limits or previous month’s expenditures and receive notifications via email or Cloud Pub/Sub events. 

The micro-service architecture involves using Cloud Pub/Sub to trigger events, and a Cloud Function to handle the events and send notifications to Slack. Set up the codebase by creating necessary files and dependencies, and deploy the code to the Cloud Function. 

This GCP Alert service improves cost management and financial control, allowing you to take immediate action and ensure efficient budget management for your GCP projects.

The post Efficient GCP Budget Monitoring through Slack Alerts appeared first on ColoredCow.

We reached Bangalore around 6 in the evening where we met Vivek from Thoughtworks at the hotel. He was a volunteer in the conference and shared that he has heard of Glific and wanted to know more about it in the talk next day. It was really exciting for us to talk about something that we were building from the past two years.

I was really excited for the conference the next day.

Food

Sharing this for the organizers that the food was really great throughout the conference and especially Idli in the morning, that was the best Idli I ever had. Usually we don’t get Idli like this in the north (okay enough about food for now).

The Audi dilemma

One thing that really bugged me was that the talks were in two different audis (which means I had to choose which talk interests me most). Sometimes both of the talks were really interesting. Eventually we had to choose one over the other.

There were a wide variety of talks and sharing the ones that I found most interesting:

1. Use of GIS for Planning Rs 80,000 Cr of Rural Road Investments

In this talk Harsh shared about how he used the open street data, QGIS and other open source libraries to help government plan out upgrading roads in rural India. We are also working on with Reap benefit to plot civic data into maps so this captured our attention as to the tools and data they were using.

2. Democratising Open (Geo)Data for Open Innovation and Solutioning

Arjun shared about the importance of open data and how it can help the government in planning better for citizens. They shared the work they did with Kerala government for various routes taken by buses and shared that data on a public repository.

3. Self Hosting with Nomad

Karan shared that data is insecure on the major cloud providers and for simple personal things to store we can easily use self hosting to overcome these issues.

4. The designer-contributor dilemma with FOSS
Krutika shared that it is really difficult for designers to contribute to open source since most of them are not aware of how design can help an open source project.

Apart from this there were project showcases and following are the ones that caught my eye and would love to try these, and see if they can be utilized in other places that I work:

1. Blaze (a peer to peer file sharing across internet)
2. Karate (a test runner)
3. Frappe (low code web framework)
4. ReactPlay (learn react the fun way)
5. Appsmith (low code app builder)

My thoughts on what we can do better as a community

1. We should think about how designers can also contribute to the open source community. As shared by Krutika we can propose a convention for designers on github to look for open source projects using the label contrib-design

2. Social sector organizations should be made more aware about these open source tools and technologies to scale their operations. Tech4dev is already on track for doing this.

3. While wrapping up the conference we touched upon the topic of Women in tech and it led to a good discussion where everyone brainstormed how we can encourage women in the tech sector. This aligns very well with the mission that Tech4dev is working on for Women in Tech with Hyperverge Academy and ColoredCow. Tejas (Reapbenefit) suggested that we should invite these Women In Tech interns to attend such conferences to get more exposure.

4. Collaboration is another part of open source community and conferences like these helps in better collaboration to know and learn from each others work.

Breaks in between talks
Lunch and tea breaks were kept perfectly for people to just talk and get to know each other. The best thing about conferences that I like is that the people are so humble and eager to share no matter if you are the CEO of a big company or an intern at a small startup, they ask and share with genuine curiosity.

At the end of the conference we also came to know that it was all volunteer driven and volunteers from organizations like Zerodha, Frappe, Thoughtworks helped in setting up the whole thing from website to the settings up timelines and stuff.

I also got a chance to share how Glific is helping the social sector organizations and making an impact through a small interview.

Outside conference

After the end of the first day Gautam and Tejas (from ReapBenefit) took us for a Bangalore tour. Since meeting GP in the Tehri Sprint and Tejas in Pune sprint I had always wanted to see the ReapBenefit office and meet the whole team. Because these two people go above and beyond for the work they are doing. They believe in the purpose and work on it to the level that some may find even crazy.

I was really eager to see where and how they work. Gautam took us to the Reap benefit office and showed us where each of them works and how they cannot work if even four of them sits together in the same room because they end up discussing so much. So they have a separate thinking room. 

We went to a fastfood corner next where we had different types of Puri’s and a twister at the end. We went for coffee next and finally to Bob’s  Bar where we met other members of the Reap benefit team. Everyone was really fun and welcoming.

Next day we were thinking of going to places like MTR or Vidyarthi Bhavan but we were suggested a better place than this for south indian breakfast. It was Taaza thindi and it was really really awesome. I have always been a fan of south indian food and didn’t want to leave Bangalore after seeing the possibility of eating the food everyday.

At night we just explored nearby areas and got the famous Mysore Pak back for our homes.

Overall it was and amazing and relaxing journey with lots to learn and lots to share. Will encourage every one to attend one of the conferences. It helps you to meet a lot of like minded people and sharing and collaborating on what you already build.

The post My IndiaFOSS 2.0 experience appeared first on ColoredCow.

When do you think a developer is the happiest?
For me it is when a piece of code written by you is used by someone and they are happy and thankful for it. That is the best feeling ever.

It is so fascinating to be in the technology space in today’s time. You can easily do, whatever you can think of today with all the tools and resources available on the internet. You just have to look for it in the right places and put in some time to understand it.

Early days

From my college days I have always wanted to solve a problem that was faced by many people and tried to look for it in the open source space. I found many problems but was not sure where to think and start. It was really difficult to pick someone else’s problem and think of a solution.

I looked at the various products that came into existence because of a problem the user wanted to solve in his life, and others liked it so they joined.

I decided that whatever problem I will face in my life maybe I can start with that and build a solution for myself. Finally I will make the code available publicly for anyone to use.

Problem

Sharing one of the problems that I faced recently while working on Glific

For this product we have a communication server on discord and we have many channels to communicate with the users and developers of the product. One of them is ngos-support, where our users ask for help, post questions or share issues with us.



Since there is a lot of messaging happening on that channel, sometimes we miss handling or checking on some of the issues as they have not been tracked anywhere.

We try to create issues on Github from there but it feels like a lot to move those messages into issues on a separate platform.

Solution

I saw a problem and worked on the best way to handle this.

I knew that discord provided some automations with the help of bots and tried to look for something that can help manage these missing issues.

First try
I started to look for something and came across a bot called tickets that can create a ticket based on a discord message https://ticketsbot.net/. It looked really cool as now we can create tickets and track them on discord only.



But somehow that did not work in our case as once the ticket is closed all the information and messages in that thread gets lost which was a concern for us.

So we decided to not go through with it.

Second try
I thought if we can directly open a Github issue from here how cool would that be.
So I started reading on how to create a discord bot that can connect with other services.

Discord has a comprehensive and great documentation. I was able to figure out how to interact with messages and capture events and now the next step would be to open an issue on Github through an API

Again thanks to comprehensive Github API and some good articles I was able to figure out how to create an issue through an API call

Final step would be to link all of this and try it out.

It took one full day just to experiment with it and make it somehow work.
Was really happy when it worked for the first time.



I shared it with my colleagues and they tried it out.
Looked good in the start. Let’s see how this will go in the future.

Sharing and open source

Since there was no other solution yet on the internet. I shared it on github for others to use and it seems like there are people like me who want this kind of solution.

Will work on it to improve this in the future for others to use.

Sharing some of the other problems that I faced and the solutions that I published on Github-

Auto close tab: I was really frustrated after opening so many chrome tabs at once and thought of a solution to automatically close them after a certain time of non usage.
This is the chrome extension that does that.

React d3 graphs: d3 is an awesome library for creating customized charts but there is a lot of configuration for it to use. I wanted to create a minimal react wrapper for some of the charts so these can be used easily. This package does exactly that.

Data studio visualization: Data studio has a predefined set of charts that we can use but there are limitations on styling and animations. Created a customized chart with Chart.js that can be imported in Data Studio to visualize data.

Bigquery discord bot: I wanted to display some stats data on a daily basis from bigquery to discord. For that I created this bot.

If these issues are faced by me I am sure that it will be faced by some more people and may this code help them to make some progress on their issues.

“And that is what open source is all about, getting from and giving back to the community.” 

The post Open source, automation and sharing with discord bots appeared first on ColoredCow.

In this blog I will share my journey of Glific (an open source two way communication platform) and how the codebase, tools and processes evolved over time to help the reviewer.

Started with Unit testing

The first thing that we started was to add unit test cases for each component in React. The task was simple: create a component, think of all the possible states for the component and write a test for each of the states.

With Jest (the test runner) we get the code coverage metrics, so it can tell if your component is well covered with all the possible scenarios or not. It even provides the line numbers for which the tests did not evaluate.

Now we have the tool to measure coverage but how do the reviewer get to know if the test cases are added or if the coverage is increased or decreased. For this the reviewer needs to keep track  of coverage percentage and compare it with the new one.

Thankfully we have a service called CodeCov that does this for us, if we integrate it with our Github CI. All one needs to do is run the tests in Github CI and send the coverage report generated by Jest to CodeCov. It will generate the metrics for you.

The coverage details will be added as a comment on the PR from CodeCov.

It even highlights the lines for which the test cases were not added or not reached.

And finally a badge in our readme file for the coverage percentage. We are still at 77 percent and hopefully will reach 90 sometime soon .

Start of integration testing

With unit testing we ensured that individual components worked fine but it was not enough. How do we make sure that the components linked together are working fine?

With this thought we reached out to Cypress. Its an amazing library for automated integration and end to end testing. We can test out a complete feature from one end to another and ensure it works exactly the same way on each test run.

Our initial test cases followed this pattern:
– Open a page
– Create an item,
– Test that the functionality is working as intended
– Delete the item.

With this we ensured our apps functionality remained intact with each new change. Thanks to the open source plan that we got from Cypress we got additional features like the support for the dashboard service. It helps to track all previous tests and to see videos of failing test cases so that it is easier to debug where the test is failing.

With the dashboard service in place, we were able to see screenshots and videos of the errors that helped us to debug the issue really quickly



This ensured that the functionality is not broken on a new change and we covered most of the features in these test cases. 

UI and cross browser testing

We made sure that the functionality remained intact on new changes by adding unit and integration test cases but what about UI and other browsers compatibility. We tried to make our code cross browser compatible but how do we ensure it?

I had a windows system earlier and we got an issue in Safari and I had no way to test that. I took help from my colleagues and resolved that. Fast forward some time later I got a MacBook Pro. And then we got another issue, which was a scroll related issue on chrome in windows .

With these issues coming up related to cross browser and system we thought to get a service for easier testing across multiple devices. Thanks to the team at browserstack to give us a free open source plan. Using browserstack we were able to easily test and debug on multiple devices.




With this we made sure our application is working as expected across multiple devices with a new change, but even this was not enough. Our next challenge was to reduce the reviewer’s time.

Start with a code linter

With the linting process we did something extra. Since we are using TypeScript and there is a compilation step to javascript, we added the linting in the compilation process itself.

Now if we don’t follow any rule provided by the linter, our code will give a warning after compilation. It needs to be fixed to continue working. This way we reduced the overhead for the developer to check warnings at the end which believe me never comes.

Further we added a service called DeepScan in our checks that scans our code for best practices that needs to be followed in javascript.





And last but not the least, checking for security issues in our code. For this we added two services:

Synk– This checks for any vulnerabilities in third party libraries that we have used in our project and tells us the latest version where the vulnerability issue is resolved

GitGuardian Security Checks– This makes sure that the developer does not check in any API key in the code by mistake. Believe me this happens a lot with some devs.

Now with these services and checks a reviewer can be at ease that most of the basic things are working as expected.

The final problem

Testing locally: If a reviewer has to ensure the functionality is working as expected before merging the PR he may have to test the branch in their local system. This takes some time and this time significantly increases if there are some package updates.

Thanks to netlify for giving an open source hosting plan with which we can deploy and preview our branches without the need to check the code on local.



All the services and processes that we added ensures that the quality of the code is good and ensures stability with new releases and most importantly makes the code review process much smoother.

In the end I would like to thank: Cypress, Browser stack, Netlify, CodeCov, Deepscan, Synk and GitGuardian to support us by providing your free open source plans which helped us in our journey to ensure a quality product.

The post Enhance review process with automated tools and services in an open source application appeared first on ColoredCow.