Riccardo Padovani

Kubernetes doesn't care about your uptime

2026-02-07T10:35:54+00:00

Your stateful pod sits in a 'Terminating' state, blocked by the same system that you chose to keep it alive. This is when you realize that Kubernetes doesn't actually care about your uptime; it cares about data integrity, and it will sacrifice your availability to protect it.

We often treat Kubernetes like a magic High Availability agent, and assume that it will always move workloads to healthy hardware. However, Kubernetes prioritizes data integrity over availability. When in doubt, K8s leaves your app down to keep your data safe.

This is a foundational part of how a StatefulSet works: it guarantees “At most one” semantic. No more than one pod to prevent data corruption.

Image generated by Nano Banana Pro

Why the scheduler ignores your pod

The split brain problem

K8s doesn't know, and cannot know if a node is dead for real (for example, due to a hardware failure) or if the node is working but is not able to communicate back. This can lead to a split brain. Split brain happens when the network fails, and the cluster is split in half. Both sides think that they are the surviving healthy side. Why is it dangerous?

The danger of multi-writer

If Kubernetes would blindly reschedule the pod, two pods might attach to the same ReadWriteOnce volume: the existing pod, that Kubernetes thinks dead, and the new one just scheduled.

In this case, your data will be corrupted, due to the concurrent writing. To avoid that, Kubernetes adopts a very conservative default behavior.

Most databases are ACID (Atomicity, Consistency, Isolation, Durability): they would rather be offline than wrong. We tend to consider Kubernetes a BASE system (Basically Available, Soft-state, Eventual consistency). When a node goes dark, Kubernetes chooses the C (Consistency) in the CAP Theorem. It freezes your pod because it cannot guarantee that "Isolation" isn't being violated. The goal is for your data stays durable and uncorrupted.

The default behavior

As soon as the node is seen as unhealthy by Kubernetes, one of two standards taints are applied:

node.kubernetes.io/unreachable: this taint is added when Kubernetes cannot communicate with the node;
node.kubernetes.io/not-ready: this taint is added when the node is actually reachable, but it reports that is not ready;

By default, every pod has a toleration of 300 seconds (5 minutes), to continue existing on a tainted node. You can reduce this time using tolerations, but it doesn't resolve the problem for stateful pods: because the node is unreachable, K8s has no way to confirm that a pod is actually dead. Pod might still be writing data to the disk.

Additionally, reducing the toleration is highly risky also for stateless pods: reducing the toleration too much leads to flapping in unstable network conditions, which is often worse than 5 minutes of downtime.

So, Kubernetes will move the pod to Terminating or Unknown, and it will stay like that forever. And trying to force a new pod in a new node wouldn't help either: the VolumeAttachment would remain in use by the dead node, so your new pod stays in ContainerCreating or Pending with an error in your events saying something like Multiattach error for volume <X>. Volume is already used by pod…

If K8s doesn't help, who will?

Application level replication

Databases, but also event queues, as Postgres, MongoDB, or RabbitMQ, should handle their own failovers. In a robust stateful setup, you don't just run one pod. You run a cluster, with a leader (Read/Write) and one or more followers (Read-only). Kubernetes task is only to run them, but it’s the application itself that manages data synchronization, and especially, the election of a new leader. How does it work?

The leader pod becomes unreachable
The remaining follower pods, on healthy node, fail to observe a heartbeat. They perform a vote, and promote a new leader.
The old “dead” pod is still stuck in Terminating, but it doesn't matter, because it is now ignored by the application

For this to work, your application should share nothing: every pod has its own volume. In this way, K8s is not the data gatekeeper anymore, but the app itself is.

For example, Postgres doesn't know how to handle K8s failover. You can use Patroni, which is a “wrapper” around Postgres, that uses a Distributed Configuration Store (like etcd or Consul) to keep track of who is the leader. If the leader node's network cuts, the leader “lease” in the DCS expires. A standby Patroni instance sees that the lease has expired, and grabs it, promoting itself to primary.

MongoDB, on the other hand, is natively built for this. All MongoDB instances talk to each other constantly (with so-called heartbeats). If the primary goes down, the secondaries hold an election. Each MongoDB pod has its own persistent volume, so they don't need to steal the old disk from the dead node.

Node power off

You can also, manually or via automation, manually power off the “failed” machine. It can be a Virtual Machine or a bare-metal server: if it is off, it cannot write to the volume. At that point, you can force detach the volume, and you are sure that there won't be data corruption.

This is called fencing, or more colloquially, STONITH: shoot the other node in the head. Make sure that the problematic node is isolated and dead, so the integrity of the system is guaranteed.

While you can do this manually, the best course of action would be having automatic systems performing it for you.

CSI Drivers

Modern Container Storage Interfaces are getting smarter. Some of them can handle “fencing” a node by automatically invalidating its access to the disk. However, this is still far from being a universal “out of the box” feature.

Design for distrust

Start thinking about failure scenarios where a pod dies for good: how would your app react? While it is true that, for stateless workloads, scheduling is not a problem anymore, when there is data involved, you want to be prescriptive on how you want to manage data.

Kubernetes is a world-class orchestrator, but it is not a DBA. If your data is important, your database should be smarter than your orchestrator.

And, what are some of your firefighting stories about Kubernetes in the middle of the night?

On the importance of teachability

2026-01-05T09:36:41+00:00

I would argue that “being teachable" is even a more precious skill than being able to give great feedback: we can spend our whole life without giving any feedback, but for sure we can't avoid receiving it.

And given that feedback is one of the most valuable tool for your own growth, career-wise: you definitely want to optimize how you use it, and you want to encourage people to give you more.

Illustration by undraw.co

As "Thanks for the Feedback" by Douglas Stone and Sheila Heen teaches us, there are three types of feedback we could receive: Appreciation, Coaching, and Evaluation.

A first challenge arises when we are expecting one kind of feedback, but we receive a different one. Our own expectations set us up to not listening to what we hear, because we are not ready for it.

Other times, we receive low-quality feedback, so we react in a defensive way. However, feedback is always a gift: maybe a ugly gift, that we will throw away as soon as we are home, but still a gift.

What being “teachable” means

Most people think “being teachable” means “doing what you are told”. This is definitely not it.

Anyone giving feedback has only a partial view of the receiver’s work, so teachability is actually the ability to sustain the discomfort of the conversation, metabolize it quickly, and use the result of that metabolization to grow.

In the same fashion, having low teachability doesn’t mean rejecting feedback. I have seen many people that say they want feedback, they crave feedback, but that still have a very low teachability, because they make the feedback too expensive to give.

Feedback conversation is not a debate: there is no objective truth to find, nor logic to defend. During feedback conversation, we must be in “collection mode”: collecting information and data points on how our work is perceived.

The reaction to feedback

Giving feedback is a lot of work. It requires the giver to think about words, to prepare to a potentially uncomfortable conversation, and to spend time thinking about the impact of the talk before, during, and after.

If we react defensively to the feedback, we are punishing the giver. If we punish the giver, they will stop giving us any more data.

Often we get defensive because we wanted a pat on the back (Appreciation), or we were expecting some guidance (Coaching), but we got only raw data (Evaluation). The mismatch causes us pain, but we shouldn’t let it take over.

The goal should be to lower the “social cost” of telling us the truth: giving us feedback should be a positive, painless, effective effort.

Discerning the valuable parts

Not all feedback is high quality: some of it is driven by the other person’s insecurity, lack of context, bad mood, or simply having a hard time communicating the point they want to make.

But discussing the feedback doesn’t help: as mentioned above, it increases the social cost to get feedback. And word gets out: you don’t want to label yourself as the one that is unable to receive feedback.

Even when the feedback is bad, somewhere there is a 10% of truth. This might just be how actions appear (more on that below in “But … you misunderstood!”), and that in itself is already valuable.

So, the goal is to take the feedback, say “Thank you”, and find that 10% of value, discarding the rest.

Getting more out of feedback

What does all of this mean in practice? How can you get more out of the feedback you get?

I have personally seen the best results when, after having metabolized the feedback, and acted upon it, I went back to the giver explicitly telling them how I used their last piece of feedback.

This has a huge value for different reasons:

I show that I did my homework: I took the time to understand the feedback, integrating it in my daily activities, and seeing results;
The giver can comment on the results, explaining if I metabolized correctly what they shared, or if more adjustments are needed;
It transforms what it could have been a transactional comment in the beginning of deeper relation: with the excuse of discussing the feedback, you can start a relationship with someone that has already shown you the willingness to provide feedback. They could even become mentors!

The feedback budget

Mentors have a limited budget of energy: every minute they spend arguing with you about the feedback they shared, is a minute they are not spending in helping you grow.

If you waste their energy in senseless discussions because you cannot accept feedback, they will stop giving you any feedback at all. It won’t change a thing in their career trajectory, it will be a huge loss for you.

If you are difficult to work with, people will simply stop coming to you. They will route around you.

Being teachable means optimizing how you receive feedback for low friction. You want to make it effortless for people to dump information into your brain.

But… you misunderstood!

This is the most common fallacy I have seen in smart people: they love to say “Ah, but you have misunderstood what I have done!”. They have a reasoning on why they acted in a certain way, and they really want to explain that to you.

However, especially growing into more senior positions, perception is reality. If stakeholders perceive you as difficult, you are difficult. Arguing that you "aren't difficult, you're just precise" is ironic and proves the point.

When receiving feedback, your only job is to understand the perspective of the giver. You don’t necessarily have to agree, or act upon it. But you must understand it!

Explaining away why what you have done was the right thing to do, is wasted time. If I misunderstood you, go and fix how you communicate, don’t fix me.

Wait 24 hours

A very simple rule of thumb to help you be teachable is simply waiting at least 24 hours before discussing the feedback.

When you receive the feedback, you can ask clarifying questions if needed, but you can't argue, you can't say that it is wrong, you can't discuss it.

Simply say “Thank you, I will need some time to reflect on this.". Then, you really reflect on it. You try and find the valuable part of the message. You empathize with the giver. You spend some idle time not thinking about it.

Only after all that, and 24 hours, if you still feel the urge to discuss the feedback, you go back and discuss it.

Talk with a different mentor

Sometimes, also after having let some time pass by, we still find the feedback we received quite cryptic. We see there is value in it, but we are not really sure how to decode it, or how to get something actionable out of it.

If there is someone we trust, it is a great opportunity to share the feedback with them as well, and ask them if they have noticed the same pattern in us, or if they can rephrase the feedback in a way that is maybe more comprehensible for us.

Conclusion

Feedback is always a precious gift, and as that, we should treat it carefully: thanks for having received it, grateful, and making a good use of it.

This can make all the difference in the world in our career: having a tight, quick feedback loop allows to grow exponentially, and integrate in ourselves small changes that make us better.

How Kubernetes picks which pods to delete during scale-in

2025-01-26T21:31:43+00:00

It doesn’t matter if the number of pods is changed manually in a Deployment or through a horizontal pod autoscaler (HPA): Kubernetes (K8s) must pick which pods to delete, and it doesn’t choose randomly.

Users can help K8s choose, setting a value for the annotation controller.kubernetes.io/pod-deletion-cost, which we will cover in the article, but it is just a part of the algorithm. Given the behavior is not properly documented, this article will explain it, based on the source code.

The article will dive deep into the details of the implementation: if you are not interested in the low-level working, at the end of the page there is a summary. If you want to customize the scale-in behavior, please jump to the section about Pod deletion cost.

Scaling-in

Illustration by unDraw

Scaling-in means reducing the number of pods available in a deployment. There could have been a spike in traffic in the application, so more pods were necessary to serve all the clients, and after the spike is gone, we can save resources by reducing the number of pods we request K8s to run.

This can be done manually, patching a Deployment with kubectl, or could be managed automatically by an autoscaler which will monitor some metrics to choose how many pods are necessary.

ReplicaSet

The logic for the scale-in of pods, behind the curtain, is managed by the ReplicaSet controller. When the number of expected pods decreases (but it doesn’t become zero), it does two things:

It creates a rank among the current list of pods managed by itself: this ranking will be one of the metrics used for sorting the pods, but not the only one nor the most important one;
It sorts the pods using 8 different rules (including the ranking above);

Ranking calculation

When the ReplicaSet sees that it must decrement the number of pods it manages, it invokes the function getPodsToDelete(filteredPods, relatedPods, diff)

The three arguments are:

filteredPods: the active pods managed by this ReplicaSet: inactive pods are filtered out because they won’t be deleted;
relatedPods: all the pods owned by any ReplicaSet which has the same owner of this same ReplicaSet, including its own.
diff: the number of pods to delete;

RelatedPods

The relatedPods informs the ranking, so let’s deep dive to understand better what they represent. They are calculated in a dedicated function that returns all pods that are owned by any ReplicaSet that is owned by the given ReplicaSet's owner.

This means that relatedPods is a superset of filteredPods: it will contain all the pods that are also in filteredPods, plus other if there is any other ReplicaSet with the same owner.

In this way, we have a list of all the pods that are somehow related to the ReplicaSet which is scaling-in. As an example, if a Helm Chart is managing two deployments, an application and a database, and the application is being scaled-in, the database pods are part of the relatedPods.

The getPodsToDelete function first invokes getPodsRankedByRelatedPodsOnSameNode, and then it sorts the pods.

The function getPodsRankedByRelatedPodsOnSameNode calculates a rank for each pod, based on how many relatedPodsare running on the same node. The rank is the number of active pods.

Two pods on the same node always have the same ranking.

If there is only one ReplicaSet with a given owner, then the ranking is simply the number of pods on each node: this means that if multiple pods are colocated on the same node, they will have a higher ranking of a pod running alone on a node.

Things become muddier when you have multiple ReplicaSet with the same owner. The ranking will be based on all the pods across the multiple ReplicaSet. Sticking with the example above, let’s say we have two nodes, and two ReplicaSet: a app and db, and we are scaling in the app.

If the pods are deployed in this way:

Node 1: app, db, db
Node 2: app, app

The app pod on the first node will have a ranking of 3, while the two app pods on the second node will have a ranking of 2.

Sorting

Illustration by unDraw

Now that the ReplicaSet has assigned a rank to each pod, it delegates the sorting to the ActivePodsWithRanks structure, that implements the sort.Sort() interface.

The logic of the sorting, that is what we are interested in, is all contained in the Less() implementation. The pods that will be sorted in front of the list will be the first ones to be deleted.

There are 8 different rules: when comparing two pods, each of them is applied in turn until one matches.

The first thing that is compared is if a pod is assigned to a node: the ones that are not assigned are deleted first;
Then, the phase of the pods is the next criteria. A pod in Pending state will be deleted before a pod in Unknown state, and the ones in Ready phase will be deleted last;
Then, the Ready status is compared: pods not Ready will be deleted before pods marked as Ready;
If the feature pod-deletion-cost is enabled, (we will speak about it later, as it is the only way to shape the choice of which pod to delete), the pod with a lower controller.kubernetes.io/pod-deletion-cost (if any), will be deleted first;
Then, Kubernetes uses the rank of the pod: we explained above, is the number of related pods running on the same node. The one with a higher rank will be deleted first;
Then, if both pods are Ready, the pod that has been ready for a shorter amount of time will be deleted before the pod that has been ready for longer;
Then, everything else equal, the pods that have restarted the most will be deleted first;
If nothing else matches, the pod that has been created most recently, according to the CreationTimestamp field, will be deleted first.

If all these 8 criteria are the same, so there is no clear indication of which pod should be deleted first, they are sorted by UUID to provide a pseudorandom order. The one that comes before in alphabetical order will be deleted first.

Pod deletion cost

The pod deletion cost is a feature introduced in Kubernetes v1.22, currently in the Beta state and enabled by default.

It allows users to set an annotation on a pod, controller.kubernetes.io/pod-deletion-cost, which represents the cost of deleting a pod. The cost can be any value between -2147483648 and 2147483647, and the pods with a lower value will be deleted first.

As we have seen, this is on a best-effort basis: it is not the first criteria that Kubernetes will use to pick a pod to delete.

You shouldn’t update this value too often, to not put too much pressure on the api-server, and you shouldn’t update this value manually: if you want to use it, I suggest writing some controller that implements the logic you’d like to see applied.

Summary

Long story short, the algorithm compares all the pods and orders them following these criteria:

Unassigned < assigned;
PodPending < PodUnknown < PodRunning;
Not ready < ready;
Lower pod-deletion-cost < higher pod-deletion cost;
Doubled up < not doubled up;
Been ready for empty time < less time < more time;
Pods with containers with higher restart counts < lower restart counts;
Empty creation time pods < newer pods < older pods;

The first pod in the list will be the first to be deleted;

I hope you found this article somehow useful, please let me know in the comments if you have any feedback or suggestions, or any other questions!

Ciao,

Moving to HyvorBlogs

2025-01-26T21:31:31+00:00

It’s almost 10 years since I wrote my first blog post on this domain. Back at the time, it was a WordPress blog.

After WordPress, I moved to Jekyll, and I hosted the website in many different places: GitHub Pages, my own VPS, and lately on GitLab Pages.

However, I wasn’t satisfied with the setup: Jekyll is a powerful tool, and quite a simple one, but it still requires maintenance. Moreover, it doesn’t have some interesting features, such as a search feature, and I am not good at customizing themes.

Image courtesy of unDraw.co

At the same time, while managing a blog through Git sounds like fun, at the end of the day, it has some limitations, and it proves itself as a barrier to creating more content.

Meaning, given I was spending too much time tweaking Jekyll, and I had some friction in writing content, I basically stopped writing anything.

And this is a shame, at least for me: maybe the Internet doesn’t need yet another blog, but I like writing: it helps me to understand better the topic I am writing about, it forces me doing proper research, and all in all, I like communicating.

So, when I had the opportunity to change the platform to something I liked more, I took it. I spent the last month preparing the migration, and now this blog is hosted on Hyvor Blogs!

But why Hyvor Blogs among all the platforms?

Hyvor Talk

Two years ago, almost to the date, I introduced comments on this blog.

I am thrilled with it: for a small fee, I have a nice product, without any tracked or shady business going on. So, when they launched Hyvor Blogs I was truly interested in them, given how happy I am with the comments system

No maintenance

Once again, for just a small fee I have a platform that just works: I don’t have to worry about software versions, or infrastructure, or anything else. Growing old, I learned to focus on the core of stuff, and delegate everything else. My blog is here for me to write and communicate, not for maintaining software. So, it makes sense to have somebody expert taking care of it.

Pretty theme

I like having a pretty theme for my blog. I customized one of the default ones to have a style similar to the previous one. Moreover, I also gained a dark theme, and a search function. Neither things I would have ever implemented on my own.

Amazing support

Hyvor is still a small company, not a faceless behemoth: they have outstanding support channels, and they helped me so much in setting everything up. It is nice to have somebody to talk to when I encounter bugs, or I am not confident how to do something. The product has room for improvement (which product doesn’t?), but any bug I reported has been fixed within a few days, if not hours.

Writing more

Now I can focus on just writing, and I can also do from my smartphone (I never installed git on it, so it was almost impossible with Jekyll). Maybe I will write more, or perhaps I will continue writing just a couple of posts every year. I definitely aim for the former, let’s see.

Feedback

I hope the look and feel of the blog remembers the old one. If you notice anything broken, if the change introduced any bug, or just if you want to let me know what you think of this change, please leave a comment below :-)

So, thanks Hyvor for this pretty blog and the wonderful commenting system, and see you all in the next blog post!

Ciao,
R.

No heroes needed

2025-01-26T21:31:29+00:00

In Google, we have an internal document, quite often quoted, that says to let things fail if the workload to keep them alive is too high. From Google, many good practices have been exported and adopted in the IT world, and I believe this could be very useful to many teams.

Illustration by unDraw.

The story of how “heroism is bad” is not new. The internal Google document has actually been a presentation at SRE TechCon 2015. And, other companies have a similar take. But let’s start from the principle.

What is the significance of heroism?

When we talk about a hero for the team, we mean a single person taking on a systemic problem to compensate for it. No matter the cost, how many hours are needed, or the consequences. Of course, thanks to the hero, the crisis due to the systemic issue is momentarily avoided.

And, heroes get immediate reward! They have completed a gigantic task, they get praised from teammates, and they feel crucial: after all, without them, everything would have failed.

Why being a hero is bad

Heroic behavior, although quite encouraged in many realities, at the end of the day, brings more damage than benefits. It is bad for the individual: takes a toll on them, brings to burnout, creates unsatisfiable expectations (you did once, why don’t you do it again?), and puts a lot of pressure on them.

It is also bad for the team: it creates incorrect expectations from the team members, and it leads to inaction because the heroes will pick up anything left behind. Not only that, but it damages the morals because it makes it feel like everything has been solved by the hero, and everybody else has no purpose.

And it is bad for systems and processes: they don’t improve because teams don’t realize they are broken, given that at the end of the day, the heroes make everything work. No systemic fixes are carried on, ‘cause such problems are hidden due to the heroes efforts.

We work in teams for a reason: the idea that one person has the solution to everything is ridiculous.

What to do instead?

On YouTube, there is an astonishing TED talk by Lorna Davis, “A guide to collaborative leadership”. If you have 14 minutes of spare time today, you should really use them to watch it.

There is a key concept in the video: radical interdependence.

What’s radical interdependence?

Radical interdependence is just a fancy way to say that we need each other. In a complex, ever evolving, interconnected world, a single individual cannot change the world on its own.

And a team can delivery things that could be impossible to deliver by an individual alone.

However, quoting from the video:

Interdependence is harder than being a hero. It requires us to be open, transparent, and vulnerable. And that’s not what traditional leaders have been trained to do.

Let things fail

It’s hard, but nowadays, every time I feel the urge to step in to save the day, I stop a minute, and ask myself: “Can’t I sit down with the team and work this out?”. And every so often, this requires things to fail: but that’s okay.

When things fail, we sit down all together, and we find out what went wrong, and what we can do better the next time. This allows improvements to the system, to the process, and at the end things are remarkably better than what they would have been if they hadn’t failed.

It’s a complex, difficult approach, but it provides significant improvements. Have you ever tried it, or plan to do so? Can you think of an occasion where it would have been useful? Let me know in the comments!

If you have any question, other than here in the comments, you can always reach me by email at [email protected]" rel="noopener noreferrer">[email protected].

Ciao,
R.

How to prepare for the Certified Kubernetes Administrator (CKA) exam

2025-01-26T21:31:29+00:00

Illustration by unDraw.

Preparation

Before taking the exam, I already had extensive experience using K8s, however always through managed services. I had to learn everything about how a K8s cluster is actually run.

I used the course hosted over A Cloud Guru. It is up-to-date, and covers everything needed for the CKA. The course is very hands-on, so if you sign up for it, I recommend doing the laboratories (they provide the environment) because the exam is a long series of tasks similar to what you would be asked to do during the lab sections of this course.

Furthermore, you should learn how to navigate the K8s doc. You can use it during the exam, so there is no point in learning every single field of every single resource. Do you need to create a pod? Go to the documentation for it, copy the basis example, and customize it.

Simulation

When you register for the CKA certification exam on the Linux Foundation site, you will receive access to a simulated exam hosted by killer.sh. The simulation is significantly more difficult and longer than the actual exam: I attempted a simulation two days before the exam, and I achieved 55%. On the actual exam, 91%.

I highly recommend taking the simulation, not only to see how prepared you are, but to familiarize yourself with the exam environment.

The exam

The exam is 2 hours long, with a variable number of tasks. Points are also assigned based on how many tasks of a question you answered. You need a score of 66% to clear the exam.

During the exam you can access K8s documentation, use this power to your advantage. The environment in which the exam will run is based on XFCE. You will have a browser, a note pad, and a terminal you can use.

Being familiar with vi can help you work more quickly, especially if you spend a lot of time in there, and you just use the browser to read the documentation.

You can use the note pad to keep track of which questions you skipped, or you want to revise later.

During your exam, you will work on multiple K8s clusters. Each question is about one cluster, and at the beginning of the question, it says which context you should use. Don’t jump to answering the question without having switched context!

Requirements

To attend the exam, you will have to download a dedicated browser, called PSI Safe Browser. You cannot download this in advance, it will be unlocked for you only 30 minutes before the beginning of the exam.

Theoretically, it is compatible with Windows, macOS, and Linux: however, many users have problems with Linux. Be aware that there could be issues, and have a macOS or Windows machine available, if possible, as a backup plan.

This browser will make sure that you are not running any background application, and that you don’t have multiple displays (only one monitor, sorry!).

After you installed the browser, you will have to identify yourself with some official document. You will have to show that you are alone in the room, that your desk is clean, that there is nothing under your desk, and so on. You will have to show your smartphone, and then show that you placed it in some unreachable place. The whole check-in procedure takes time, and it is error-prone (I had to show the whole room twice, then the browser crashed, and I had to start from scratch again).

The onboarding procedure is the most stressful part of the exam: read all the requirements on the Linux Foundation Website, try to clear your desk from any unrelated things, and start the procedure as soon as possible. In my case, it took almost 50 minutes.

Their browser is not a perfect piece of software: in my case, the virtual environment inside this browser had a resolution of 800x600, making it impossible to have two windows on a side-by-side. However, you spend the huge majority of your time in the terminal, and sometimes on the browser to copy-paste snippets from the browser.

Tips

Keep a Windows or macOS machine nearby, if Linux doesn’t work for you;
Answering a question partially is better than not answering at all;
Always double-check the K8s context! The first thing you must do for each question, it is switching the context of your kubectl according to the instructions;
Create files for each resource you create, so you can go back and adjust stuff. Moreover, if you have time at the end of the exam, makes way easier to check what you have done;
Remember to have a valid ID document with you for the exam check-in;

Conclusion

Time constraints and the unfamiliar environment will be your greatest challenges: however, if you have used K8s in production before, you should be able to clear the exam without any major difficulty. Spending sometime training before is strongly suggested, no matter your level of expertise, just to understand the format of the exam.

Good luck, and reach out if you have any question, here in the comments or by email at [email protected]" rel="noopener noreferrer">[email protected].

Ciao,
R.

Why K8s deployments need `matchLabels` keyword

2025-01-26T21:31:29+00:00

Illustration by unDraw+.

A Kubernetes (K8s) Deployment provides a way to define how many replicas of a Pod K8s should aim to keep alive. I’m especially bothered by the Deployment spec’s requirement that we must specify a label selector for pods, and that label selector must match the same labels we have defined in the template. Why can’t we just define them once? Why can’t K8s infer them on its own? As I will explain, there is actually a good reason. However, to understand it, you would have to go down a rabbit hole to figure it out.

A deployment specification

Firstly, let’s take a look at a simple deployment specification for K8s:

 1apiVersion: apps/v1
 2kind: Deployment
 3metadata:
 4  name: nginx-deployment
 5spec:
 6  replicas: 3
 7  selector:
 8    matchLabels:
 9      app: nginx # Why can't K8s figure it out on its own?
10  template:
11    metadata:
12      labels:
13        app: nginx
14    spec:
15      containers:
16      - name: nginx
17        image: nginx:latest

This is a basic deployment, taken from the official documentation, and here we can already see that we need to fill the matchLabels field.

What happens if we drop the “selector” field completely?

1➜ kubectl apply -f nginx-deployment.yaml
2The Deployment "nginx-deployment" is invalid:
3* spec.selector: Required value

Okay, so we need to specify a selector. Can it be different from the “label” field in the “template”? I will try with:

1matchLabels:
2      app: nginx-different

1➜ kubectl apply -f nginx-deployment.yaml
2The Deployment "nginx-deployment" is invalid: spec.template.metadata.labels: Invalid value: map[string]string{"app":"nginx"}: `selector` does not match template `labels`

As expected, K8s doesn’t like it, it must match the template. So, we must fill a field with a well-defined value. It really seems that a computer could do that for us, why do we have to specify it manually? It drives me crazy having to do something a computer could do without any problem. Or could it?

Behind a deployment

How does a deployment work? Behind the curtains, when you create a new deployment, K8s creates two different objects: a Pod definition, using as its specification what is available in the “template” field of the Deployment, and a ReplicaSet. You can easily verify this using kubectl to retrieve pods and replica sets after you have created a deployment.

A ReplicaSet’s purpose is to maintain a stable set of replica Pods running at any given time. As such, it is often used to guarantee the availability of a specified number of identical Pods.

A ReplicaSet needs a selector that specifies how to identify Pods it can acquire and manage: however, this doesn’t explain why we must specify it, and why K8s cannot do it on its own. In the end, a Deployment is a high-level construct that should hide ReplicaSet quirks: such details shouldn’t concern us, and the Deployment should take care of them on its own.

Digging deeper

Understanding how a Deployment works doesn’t help us find the reason for this particular behavior. Given that Googling doesn’t seem to bring any interesting results on this particular topic, it’s time to go to the source (literally): luckily, K8s is open source, so we can check its history on GitHub.

Going back in time, we find out that, actually, K8s used to infer it the matchLabels field! The behavior was removed with apps/v1beta2 (released with Kubernetes 1.8), through Pull Request #50164. Such pull request links to issue #50339, that, however, has a very brief description, and lacks the reasoning behind such a choice.

Luckily, other issues provide way more context, as #26202: it turns out, the main problem with defaulting is when in subsequent updates to the resource, labels are mutated: the patch operation is somehow fickle, and apply breaks when you update a label that was used as default.

Many other concerns have been described by Brian Grant in deep in the issue #15894.

Basically, assuming a default value, creates many questions and concerns: what’s the difference between explicitly setting a label as null, and leaving it empty? How to manage all the cases where users left the default, and now they intend to update the resource to manage themselves the label (or the other way around)?

Conclusion

Given that in K8s everything is intended to be declarative, developers have chosen that explicit is better than implicit, especially for corner cases: specifying things explicitly allows a more robust validation on creation and update time, and removes some possible bugs that existed due to uncertainties caused by lack of clarity.

Shortly after dropping the defaulting behavior, developers also made the labels immutable, to make sure behaviors were well-defined. Maybe in the future labels will be mutable again, but to have that working, somebody needs to design a well-though document explaining how to manage all the possible edge cases that could happen when a controller is updated.

I hope you found this deep dive into the question interesting. I spent some time on it, since I was very curious, and I hope that the next person with the same question can find this article and get an answer quicker than what it took me.

If you have any question, or feedback, please leave a comment below, or write me an email at [email protected]" rel="noopener noreferrer">[email protected].

Ciao,
R.

Managing Helm CRDs with Terraform

2025-01-26T21:31:29+00:00

Helm is a remarkable piece of technology to manage your Kubernetes deployments, and used with Terraform is perfect for deploying following the GitOps strategy.

Illustration by unDraw+.

However, Helm has a limitation: it doesn’t manage the lifecycle of Custom Resource Definitions (CRDs), meaning it will only install the CRD during the first installation of a chart. Subsequent chart upgrades will not add or remove CRDs, even if the CRDs have changed.

This can be a huge problem for a GitOps approach: having to update CRDs manually isn’t a great strategy, and makes it very hard to keep in sync with deployments and rollbacks.

For this very reason, I created a small Terraform module that will read from some online manifests of CRDs, and apply them. When parametrizing the version of the chart, it is simple to keep Helm Charts and CRDs in sync, without having to do anything manually.

Example

Let’s use Karpenter as an example of how to use the module. We want to deploy the chart with the Helm provider, and we use this new Terraform module to manage its CRDs as well:

 1resource "helm_release" "karpenter" {
 2  name            = "karpenter"
 3  namespace       = "karpenter"
 4  repository      = "https://charts.karpenter.sh"
 5  chart           = "karpenter"
 6  version         = var.chart_version
 7
 8  // ... All the other parameters necessary, skipping them here ...
 9}
10
11module "karpenter-crds" {
12  source  = "rpadovani/helm-crds/kubectl"
13  version = "0.1.0"
14  
15  crds_urls = [
16    "https://raw.githubusercontent.com/aws/karpenter/v${var.chart_version}/charts/karpenter/crds/karpenter.sh_provisioners.yaml",
17    "https://raw.githubusercontent.com/aws/karpenter/v${var.chart_version}/charts/karpenter/crds/karpenter.k8s.aws_awsnodetemplates.yaml"
18  ]
19}

As you can see, we parametrize the version of the chart, so we can be sure to have the same version for CRDs as the Helm chart. Behind the curtains, Terraform will download the raw file, and apply it with kubectl. Of course, the operator running Terraform needs to have enough permissions to launch such scripts, so you need to configure the kubectl provider.

The URLs must point to just the Kubernetes manifests, and this is why we use the raw version of the GitHub URL.

The source code of the module is available on GitHub, so you are welcome to chime in and open any issue: I will do my best to address problems and implement suggestions.

Conclusion

I use this module in production, and I am very satisfied with it: it brings under GitOps the last part I missed: the CRDs. Now, my only task when I install a new chart is finding all the CRDs, and build a URL that contains the chart version. Terraform will take care of the rest.

I hope this module can be useful to you as it is to me. If you have any question, or feedback, or if you would like some help, please leave a comment below, or write me an email at [email protected]" rel="noopener noreferrer">[email protected].

Ciao,
R.

Why you should contribute to GitLab

2025-01-26T21:31:29+00:00

The new GitLab logo, just announced on the 27th April 2022.

Nowadays, my contributions focus mostly on GitLab, so you will see many references to it in this blog post, but the content is quite generalizable; I would like to share my experience to highlight why you should consider contributing to an open-source project.

And contributing doesn’t mean only coding: there are countless ways to help an open-source project: translating it to different languages, reporting issues, designing new features, writing the documentation, offering support on forums and StackOverflow, and so on.

Before deep diving into this wall of text, be aware there are mainly three parts to this blog post, after this introduction: a context section, where I describe my personal experience with open source, and what does it mean to me. Then, a nice list of reasons to contribute to any project. In closing, there are some tips on how to start contributing, both in general, and something specific to GitLab.

Context

Ten years ago, I was fresh out of high school, without (almost) any knowledge about IT: however, I found out that I had a massive passion for it, so I enrolled in a Computer Engineering university (boring!), and I started contributing to Ubuntu (cool!). I began with the Italian Local Community, and I soon moved to Ubuntu Touch.

We all know how it ended, but it still has been a fantastic ride, with plenty of great moments: just take a look at the archive of this blog, and you can see the passion and the enthusiasm I had. I was so enthusiast, that I wrote a similar blog post to this one! I think it highlights really well some considerable differences 10 years make.

Back then, I wasn’t working, just studying, so I had a lot of spare time. My English was way worse. I was at the beginning of my journey in the computer world, and Ubuntu has ultimately shaped a big part of it. My knowledge was very limited, and I never worked before. Contributing to Ubuntu gave me a glimpse of the real world, I met outstanding engineers who taught me a lot, and boosted my CV, helping me to land my first job.

Since then, I completed a master’s degree in C.S., worked in different companies in three different countries, and became a professional. Nowadays, my contributions to open source are more sporadic (adulthood, yay), but given how much it meant to me, I am still a big fan, and I try to contribute when I can, and how I can.

Why contributing

Friends

During my years contributing to open-source software, I’ve met countless incredible people, with some of whom I’ve become friends. In the old blog post I mentioned David: in the last 9 years we stayed in touch, met on different occasions in different cities: the last time was as recent as last summer. Back at the time, he was a Manager in the Ubuntu Community Team at Canonical, and then, he became Director of Community Relations at GitLab. Small world!

The Ubuntu Touch Community Team in Malta, in 2014. It has been an incredible week, sponsored by Canonical!

One interesting thing is people contribute to open-source projects from their homes, all around the world: when I travel, I usually know somebody living in my destination city, so I’ve always at least one night booked for a beer with somebody I’ve met only online; it’s a pleasure to speak with people from different backgrounds, and having a glimpse in their life, all united by one common passion.

Fun

Having fun is important! You cannot spend your leisure time getting bored or annoyed: contributing to open source is fun ‘cause you pick the problems you would like to work on, and you don’t need all that bureaucracy and meetings that is often needed in your daily job. You can be challenged, and feel useful, and improving a product, without any manager on your shoulder, and at your pace.

Being up-to-date on how things evolve

Contributing to a project typically gives you an idea of how the teams behind it work, which technologies they use, and which methodologies. Many open-source projects use bleeding-edge technologies, or draw a path. Being in contact with new ideas is a great way to know where the industry is headed, and what is the latest news: it is especially true if you hang out in the channels where the community meets, being them Discord, forums, or IRC (well, IRC is not really bleeding-edge, but it is fun).

Learning

When contributing in an area that doesn’t match your expertise, you always learn something new: reviews are usually precise and on point, and projects of a remarkable size commonly have a coaching team that help you to start contributing, and guide you on how to land your first patches.

In GitLab, if you need a help with merging your code, there are the Merge Request Coaches! And for any type of help, you can always join Gitter, or ask on the forum, or write to the [email protected]" rel="noopener noreferrer">dedicated email address.

Feel also free to [email protected]" rel="noopener noreferrer">ping me directly if you want some general guidance!

Giving back

I work as a Platform Engineer. My job is built on an incredible amount of open-source libraries, amazing FOSS services, and I basically have just to glue together different pieces. When I find some rough edge that could be improved, I try to do so.

Nowadays, I find crucial to having well-maintained documentation, so after I have achieved something complex, I usually go back and try to improve the documentation, where lacking. It is my tiny way to say thanks, and giving back to a world that really has shaped my career.

This is also what mostly of my blogs posts are about: after having completed something on which I spent fatigue on, I find it nice to be able to share such information. Every so often, I find myself years later following my guide, and I really also appreciate when other people find the content useful.

Swag

Who doesn’t like swag? :-) Numerous projects have delightful swags, starting from stickers, that they like to share with the whole community. Of course, it shouldn’t be your main driver, ‘cause you will soon notice that it is ultimately not worth the amount of time you spend contributing, but it is charming to have GitLab socks!

A GitLab branded mechanical keyboard, courtesy of the GitLab's security team! This very article has been typed with it!

Tips

I hope I inspired you to contribute to some open-source project (maybe GitLab!). Now, let’s discuss some small tricks on how to begin easily.

Find something you are passionate about

You must find a project you are passionate about, and that you use frequently. Looking forward to a release, knowing that your contributions will be included, it is a wonderful satisfaction, and can really push you to do more.

Moreover, if you already know the project you want to contribute to, you probably know already the biggest pain points, and where the project needs some contributions.

Start small and easy

You don’t need to do gigantic contributions to begin. Find something tiny, so you can get familiar with the project workflows, and how contributions are received.

My journey with Ubuntu started correcting a typo in a README, and here I am, years later, having contributed to dozens of projects, and having a career in the C.S. field. Back then, I really had no idea of what my future would hold.

For GitLab, you can take a look at the issues marked as “good for new contributors”. They are designed to be addressed quickly, and onboard new people in the community. In this way, you don’t have to focus on the difficulties of the task at hand, but you can easily explore how the community works.

Writing issues is a good start

Writing high-quality issues is a great way to start contributing: maintainers of a project are not always aware of how the software is used, and cannot be aware of all the issues. If you know that something could be improved, write it down: spend some time explicating what happens, what you expect, how to reproduce the problem, and maybe suggest some solutions as well! Perhaps, the first issue you write down could be the very first issue you resolve.

Not much time required!

Contributing to a project doesn’t require necessarily a lot of time. When I was younger, I definitely dedicated way more time to open-source projects, implementing gigantic features. Nowadays, I don’t do that anymore (life is much more than computers), but I like to think that my contributions are still useful. Still, I don’t spend more than a couple of hours a month, based on my schedule, and how much is raining (yep, in winter I definitely contribute more than in summer).

GitLab is super easy

Do you use GitLab? Then you should undoubtedly try to contribute to it. It is easy, it is fun, and there are many ways. Take a look at this guide, hang out on Gitter, and see you around. ;-)

Next week (9th-13th May 2022) there is also a GitLab Hackathon! It is a real fun and easy way to start contributing: many people are available to help you, there are video sessions talking about contributing, and just doing a small contribution you will receive a pretty prize.

And in time, if you are consistent in your contributions, you can become a GitLab Hero! How cool is that?

I really hope this wall of text made you consider contributing to an open-source project. If you have any question, or feedback, or if you would like some help, please leave a comment below, or write me an email at [email protected]" rel="noopener noreferrer">[email protected].

Ciao,
R.

Managing Rust crates in private Git repositories

2025-01-26T21:31:29+00:00

Ferris the crab, unofficial mascot for Rust.

A Rust crate can be hosted in different places: on a public registry on crates.io, but also in a private Git repo hosted somewhere. In this latter case, there are some challenges on how to make the crate easily accessible to both engineers and CI/CD systems.

Developers usually authenticate through SSH keys: given humans are terrible at remembering long passwords, using SSH keys allows us to do not memorize credentials, and having authentication methods that just work. The security is quite high as well: each device has a unique key, and if a device gets compromised, deactivating the related key solves the problem.

On the other hand, it is better to avoid using SSH keys for CI/CD systems: such systems are highly volatile, with dozens, if not hundreds, instances that get created and destroyed hourly. For them, a short-living token is a way better choice. Creating and revoking SSH keys on the fly can be tedious and error-prone.

This arises a challenge with Rust crates: if hosted on a private repository, the dependency can be reachable through SSH, such as

1[dependencies]
2my-secret-crate = { git = "ssh://[email protected]/rpadovani/my-secret-crate.git", branch = "main" }

or through HTTPS, such as

1[dependencies]
2my-secret-crate = { git = "https://gitlab.com/rpadovani/my-secret-crate", branch = "main" }

The former is really useful and simple for engineers: authentication is the same as always, so no need to worry about it. However, it is awful for CI/CD: now there is a need to manage the lifecycle of SSH keys for automatic systems.

The latter is awful for engineers: they need a new additional authentication method, slowing them down, and of course, there will be authentication problems. On the other hand, it is great for automatic systems.

How to conciliate the two worlds?

Well, let’s use them both! In the Cargo.toml file, use the SSH protocol, so developers can simply clone the main repo, and they will be able to clone the dependencies without further hassle.

Then, configure the CI/CD system to clone every dependency through HTTPS, thanks to a neat feature of Git itself: insteadOf.

From the Git-SCM website:

url.<base>.insteadOf:
Any URL that starts with this value will be rewritten to start, instead, with . In cases where some site serves a large number of repositories, and serves them with multiple access methods, and some users need to use different access methods, this feature allows people to specify any of the equivalent URLs and have Git automatically rewrite the URL to the best alternative for the particular user, even for a never-before-seen repository on the site. When more than one insteadOf strings match a given URL, the longest match is used.

Basically, it allows rewriting part of the URL automatically. In this way, it is easy to change the protocol used: developers will be happy, and the security team won’t have to scream about long-lived credentials on CI/CD systems.

An implementation example using GitLab, but it can be done on any CI/CD system:

1my-job:
2  before_script:
3    - git config --global credential.helper store
4    - echo "https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.com" > ~/.git-credentials
5    - git config --global url."https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.com".insteadOf ssh://[email protected]
6  script:
7    - cargo build

The CI_JOB_TOKEN is a unique token valid only for the duration of the GitLab pipeline. In this way, also if a machine got compromised, or logs leaked, the code is still sound and safe.

What do you think about Rust? If you use it, have you integrated it with your CI/CD systems? Share your thoughts in the comments below, or drop me an email at [email protected]" rel="noopener noreferrer">[email protected].

Ciao,
R.

The inconsistencies of AWS EKS IAM permissions

2025-01-26T21:31:29+00:00

Image courtesy of unDraw.

The shared responsibility model

First, what’s the shared responsibility model? Well, to design a well-architected application, AWS suggests following six pillars. Among these six pillars, one is security. Security includes sharing responsibility between AWS and the customer. In particular, and I quote,

Customers are responsible for managing their data (including encryption options), classifying their assets, and using IAM tools to apply the appropriate permissions.

Beautiful, isn’t it? AWS gives us a powerful tool, IAM, to manage permissions; we have to configure things in the best way, and AWS gives us the way to do so. Or does it? Let’s take a look together.

Our goal

Our goal is quite straightforward: setting up a Kubernetes cluster for our developers. Given that AWS offers AWS EKS, a managed Kubernetes service, we only need to configure it properly, and we are done. Of course, we will follow best practices to do so.

A proper setup

Of course, we don’t use the AWS console to manually configure stuff, but Infrastructure as Code: basically, we will write some code that will call the AWS APIs on our behalf to set up AWS EKS and everything correlated. In this way, we can have a reproducible setup that we could deploy in multiple environments, and countless other advantages.

Moreover, we want to avoid launching scripts that interact with our infrastructure from our PC: we prefer not to have permissions to destroy important stuff! Separation of concerns is a fundamental, and we wish to write code without worrying about having consequences on the real world. All our code should be vetted from somebody else through a merge request, and after being approved and merged to our main branch, a runner will pick it up and apply the changes.

We are at the core of the problem: our runner should follow the principle of least privilege: it should be able to do only what it needs to do, and nothing more. This is why we will create a IAM role only for it, with only the permissions to manage our EKS cluster and everything in it, but nothing more.

I would have a massive rant about how bad is the AWS documentation for IAM in general, not only for EKS, but I will leave it to some other day.

The first part of creating a role with minimum privileges is, well, understanding what minimum means in our case. A starting point is the AWS documentation: unfortunately, it is always a bad starting point concerning IAM permissions because it is always too generous in allowing permissions.

The “minimum” permission accordingly to AWS

Okay, hardening this will be fun, but hey, do not let bad documentation get in the way of a proper security posture.

You know what will get in the way? Bugs! A ton of bugs, with absolutely useless error messages.

I started limiting access to only the namespace of the EKS cluster I wanted to create. I ingenuously thought that we could simply limit access to the resources belonging to the cluster. But, oh boy, I was mistaken!

Looking at the documentation for IAM resources and actions, I created this policy:

 1{
 2    "Version": "2012-10-17",
 3    "Statement": [
 4        {
 5            "Effect": "Allow",
 6            "Action": [
 7                "eks:ListClusters",
 8                "eks:DescribeAddonVersions",
 9                "eks:CreateCluster"
10            ],
11            "Resource": "*"
12        },
13        {
14            "Effect": "Allow",
15            "Action": "eks:*",
16            "Resource": [
17                "arn:aws:eks:eu-central-1:123412341234:addon/my-cluster/*/*",
18                "arn:aws:eks:eu-central-1:123412341234:fargateprofile/my-cluster/*/*",
19                "arn:aws:eks:eu-central-1:123412341234:identityproviderconfig/my-cluster/*/*/*",
20                "arn:aws:eks:eu-central-1:123412341234:nodegroup/my-cluster/*/*",
21                "arn:aws:eks:eu-central-1:123412341234:cluster/my-cluster"
22            ]
23        }
24    ]
25}

Unfortunately, if a role with these permissions try to create a cluster, this error message appears:

1Error: error creating EKS Add-On (my-cluster:kube-proxy): AccessDeniedException: User: arn:aws:sts::123412341234:assumed-role/<role>/<iam-user> is not authorized to perform: eks:TagResource on resource: arn:aws:eks:eu-central-1:123412341234:/createaddon

I have to say that at least the error message gives you a hint: the /createddon action is not scoped to the cluster.

After fighting with different polices for a while, I asked DuckDuckGo for a help, and indeed somebody reported this problem to AWS before, in this GitHub issue.

What the issue basically says is that if we want to give an IAM role permission to manage an add-on inside a cluster, we must give it permissions over all the EKS add-ons in our AWS account.

This of course breaks the AWS shared responsibility principle, ‘cause they don’t give us the tools to upheld our part of the deal. This is why it is a real and urgent issue, as they also mention in the ticket:

Can’t share a timeline in this forum, but it’s a high priority item.

And indeed it is so high priority, that it has been reported the 3rd December 2020, and today, more than one year later, the issue is still there.

To add insult to the injury, you have to write the right policy manually because if you use the IAM interface to select “Any resource” for the add-ons as in the screenshot below, it will generate the wrong policy! If you check carefully, the generated resource name is arn:aws:eks:eu-central-1:123412341234:addon/*/*/*, which of course doesn’t match the ARN expected by AWS EKS. Basically, also if you are far too permissive, and you use the tools that AWS provides you, you still will have some broken policy.

Do you have some horror story about IAM yourself? I have a lot of them, and I am thinking about a more general post. What do you think? Share your thoughts in the comments below, or drop me an email at [email protected]" rel="noopener noreferrer">[email protected].

Ciao,
R.

How to make Terraform waiting for cloud-init to finish on EC2 without SSH

2025-01-26T21:31:29+00:00

Terraform logo, courtesy of HashiCorp.

I find using SSH in Terraform quite problematic: you need to distribute a private SSH key to anybody that will launch the Terraform script, including your CI/CD system. This is a no-go for me: it adds the complexity to manage SSH keys, including their rotation. There is a huge issue on the Terraform repo on GitHub about this functionality, and the most voted solution is indeed connecting via SSH to run a check:

1provisioner "remote-exec" {
2  inline = [
3    "cloud-init status --wait"
4  ]
5}

AWS Systems Manager Run Command

The idea of using cloud-init status --wait is indeed quite good. The only problem is how do we ask Terraform to run such command. Luckily for us, AWS has a service, AWS SSM Run Command that allow us to run commands on an EC2 instance through AWS APIs! In this way, our CI/CD system needs only an appropriate IAM role, and a way to invoke AWS APIs. I use the AWS CLI in the examples below, but you can adapt them to any language you prefer.

Prerequisites

There are some prerequisites to use AWS SSM Run Command: we need to have AWS SSM Agent installed on our instance. It is preinstalled on Amazon Linux 2 and Ubuntu 16.04, 18.04, and 20.04. For any other OS, we need to install it manually: it is supported on Linux, macOS, and Windows.

The user or the role that executes the Terraform code needs to be able to create, update, and read AWS SSM Documents, and run SSM commands. A possible policy could be look like this:

 1{
 2  "Version": "2012-10-17",
 3  "Statement": [
 4    {
 5      "Sid": "Stmt1629387563127",
 6      "Action": [
 7        "ssm:CreateDocument",
 8        "ssm:DeleteDocument",
 9        "ssm:DescribeDocument",
10        "ssm:DescribeDocumentParameters",
11        "ssm:DescribeDocumentPermission",
12        "ssm:GetDocument",
13        "ssm:ListDocuments",
14        "ssm:SendCommand",
15        "ssm:UpdateDocument",
16        "ssm:UpdateDocumentDefaultVersion",
17        "ssm:UpdateDocumentMetadata"
18      ],
19      "Effect": "Allow",
20      "Resource": "*"
21    }
22  ]
23}

If we already know the name of the documents, or the instances where we want to run the commands, it is better to lock down the policy specifying the resources, accordingly to the principle of least privilege.

Last but not least, we need to have the AWS CLI installed on the system that will execute Terraform.

The Terraform code

After having set up the prerequisites as above, we need two different Terraform resources. The first will create the AWS SSM Document with the command we want to execute on the instance. The second one will execute such command while provisioning the EC2 instance.

The AWS SSM Document code will look like this:

 1resource "aws_ssm_document" "cloud_init_wait" {
 2  name = "cloud-init-wait"
 3  document_type = "Command"
 4  document_format = "YAML"
 5  content = <<-DOC
 6    schemaVersion: '2.2'
 7    description: Wait for cloud init to finish
 8    mainSteps:
 9    - action: aws:runShellScript
10      name: StopOnLinux
11      precondition:
12        StringEquals:
13        - platformType
14        - Linux
15      inputs:
16        runCommand:
17        - cloud-init status --wait
18    DOC
19}

We can refer such document from within our EC2 instance resource, with a local provisioner:

 1resource "aws_instance" "example" {
 2  ami           = "my-ami"
 3  instance_type = "t3.micro"
 4
 5  provisioner "local-exec" {
 6    interpreter = ["/bin/bash", "-c"]
 7
 8    command = <<-EOF
 9    set -Ee -o pipefail
10    export AWS_DEFAULT_REGION=${data.aws_region.current.name}
11
12    command_id=$(aws ssm send-command --document-name ${aws_ssm_document.cloud_init_wait.arn} --instance-ids ${self.id} --output text --query "Command.CommandId")
13    if ! aws ssm wait command-executed --command-id $command_id --instance-id ${self.id}; then
14      echo "Failed to start services on instance ${self.id}!";
15      echo "stdout:";
16      aws ssm get-command-invocation --command-id $command_id --instance-id ${self.id} --query StandardOutputContent;
17      echo "stderr:";
18      aws ssm get-command-invocation --command-id $command_id --instance-id ${self.id} --query StandardErrorContent;
19      exit 1;
20    fi;
21    echo "Services started successfully on the new instance with id ${self.id}!"
22
23    EOF
24  }
25}

From now on, Terraform will wait for cloud-init to complete before marking the instance ready.

Conclusion

AWS Session Manager, AWS Run Commands, and the others tools in the AWS Systems Manager family are quite powerful, and in my experience they are not widely use. I find them extremely useful: for example, they also allows connecting via SSH to the instances without having any port open, included the 22! Basically, they allow managing and running commands inside instances only through AWS APIs, with a lot of benefits, as they explain:

Session Manager provides secure and auditable instance management without the need to open inbound ports, maintain bastion hosts, or manage SSH keys. Session Manager also allows you to comply with corporate policies that require controlled access to instances, strict security practices, and fully auditable logs with instance access details, while still providing end users with simple one-click cross-platform access to your managed instances.

Do you have any questions, feedback, critics, request for support? Leave a comment below, or drop me an email at [email protected]" rel="noopener noreferrer">[email protected].

Ciao,
R.

Adding comments to the blog

2025-01-26T21:31:29+00:00

A picture I took at the CCC 2015 - and the moderation policy for comments!

I’ve done so to make it easier engaging with the four readers of my blabbering: of course, it took some time to choose the right comment provider, but finally, here we are!

I’ve chosen, long ago, to do not put any client-side analytics on this website: while the nerd in me loves graph and numbers, I don’t think they are worthy of the loss of privacy and performance for the readers. However, I am curious to have feedback on the content I write, if some of them are useful, and how can I improve. In all these years, I’ve received different emails about some posts, and they are heart-warming. With comments, I hope to reduce the friction in communicating with me and having some meaningful interaction with you.

Enter hyvor.com

Looking for a comment system, I had three requirements in mind: being privacy-friendly, being performant, and being managed by somebody else.

A lot of comments system are “free”, because they live on advertisements, or tracking user activities, and more often a combination of both. However, since I want comments on my website, I find dishonest that you, the user, has to pay the price for it. So, I was looking for something that didn’t track users, and bases its business on dear old money. My website, my wish, my wallet.

I find performances important, and unfortunately, quite undervalued on the bloated Internet. I like having just some HTML, some CSS, and the minimum necessary of JavaScript, so whoever stumbles on these pages don’t waste CPU and time waiting for some rendering. Being a static website, there isn’t a server side, so I cannot put comments there. I had to find a really light JavaScript based comment system.

Given these two prerequisites, you would say the answer is obvious: find a comment system that you can self-host! And you would be perfectly right - however, since I spend already 8 hours a day keeping stuff online, I really don’t want to have to care about performance and uptime in my free time - I definitely prefer going drinking a beer with a friend.

After some shopping around, I’ve chosen to go with Hyvor Talk, since it checks all three requirements above. I’ve read nice things about it, so let’s see how it goes! And if you don’t see comments at the end of the page, probably a privacy plugin for your browser is blocking it - up to you if you want to whitelist my website, or communicate with me in other ways ;-)

A nice plus of Hyvor is they also support reactions, so if you are in an hurry but want still to leave a quick feedback on the post, you can simply click a button. Fancy, isn’t it?

Moderation

Internet can be ugly sometime, and this is why I will keep a strict eye on the comments, and I will probably adjust moderation settings in the future, based on how things evolve - maybe no-one will comment, then no need for any strict moderation! The only rule I ask you to abide, and I’ve put as moderation policy, is: “Be excellent to each other”. I’ve read it at the CCC Camp 2015, and it sticks with me: as every short sentence, cannot capture all the nuances of human interaction, but I think it is a very solid starting point. If you have any concern or feedback you prefer to do not express in public, feel free to reach me through email. Otherwise, I hope to see you all in the comment section ;-)

Questions, comments, feedback, critics, suggestions on how to improve my English? Leave a comment below, or drop me an email at [email protected]" rel="noopener noreferrer">[email protected]. Or, from today, leave a comment below!

Ciao,
R.

Reading env variables from a Tauri App

2025-01-26T21:31:29+00:00

But being in its first days (the beta has just been released!) a bit of documentation is still missing, and on the internet there aren’t many examples on how to write code.

Tauri is very light, as highlighted by these benchmarks.

One thing that took me a bit of time to figure out, is how to read environment variables from the frontend of the application (JavaScript / Typescript or whichever framework you are using).

The Command

As usual, when you know the solution the problem seems easy. In this case, we just need an old trick: delegating!

In particular, we will use the Command API to spawn a sub-process (printenv) and reading the returned value. The code is quite straightforward, and it is synchronous:

 1import { Command } from "@tauri-apps/api/shell";
 2
 3async readEnvVariable(variableName: string): Promise<string> {
 4    const commandResult = await new Command(
 5        "printenv",
 6        variableName
 7    ).execute();
 8
 9    if (commandResult.code !== 0) {
10      throw new Error(commandResult.stderr);
11    }
12    
13    return commandResult.stdout;
14}

The example is in Typescript, but can be easily transformed in standard JavaScript.

The Command API is quite powerful, so you can read its documentation to adapt the code to your needs.

Requirements

There are another couple of things you should consider: first, you need to have installed in your frontend environment the @tauri-apps/api package. You also need to enable your app to execute commands. Since Tauri puts a strong emphasis on the security (and rightly so!), your app is by default sandboxed. To being able to use the Command API, you need to enable the shell.execute API.

It should be as easy as setting tauri.allowlist.shell.execute to true in your tauri.json config file.

Tauri is very nice, and I really hope it will conquer the desktops and replace Electron, since it is lighter, faster, and safer!

Questions, comments, feedback, critics, suggestions on how to improve my English? Leave a comment below, or drop me an email at [email protected]" rel="noopener noreferrer">[email protected].

Ciao,
R.

Integrating JetBrains Qodana with GitLab pipelines

2025-01-26T21:31:29+00:00

Qodana on Gitlab Pages

In this blog post we will see how to integrate this new tool by JetBrains in our GitLab pipeline, including having a dedicated website to see the cool report it produces.

If you don’t have a proper GitLab pipeline to lint your code, run your test, and manage all that other annoying small tasks, you should definitely create one! I’ve written an introductory guide to GitLab CI, and many more are available on the documentation website.

JetBrains Qodana

Qodana comprises two main parts: a nicely packaged GUI-less IntelliJ IDEA engine tailored for use in a CI pipeline as a typical “linter” tool, and an interactive web-based reporting UI. It makes it easy to set up workflows to get an overview of the project quality, set quality targets, and track progress on them. You can quickly adjust the list of checks applied for the project and include or remove directories from the analysis.

Qodana launching post.

I’m a huge fun of JetBrains products, and I happily pay their license every year: my productivity using their IDEs is through the roof. This very blog post has been written using WebStorm :-)

Therefore, when they announced a new product to bring the smartness of their IDE on CI pipelines, I was super enthusiastic! In their documentation there is also a small paragraph about GitLab, and we’ll improve the example to have a nice way to browse the output.

At the moment, Qodana has support for Java, Kotlin, and PHP. With time, Qodana will support all languages and technologies covered by JetBrains IDEs.

Remember: Qodana is in an early access version. Using it, you expressly acknowledge that the product may not be reliable, work as not intended, and may contain errors. Any use of the EAP product is at your own risk.

I suggest you to also taking a look to the official GitHub page of the project, to see licenses and issues.

Integrating Qodana in GitLab

The basic example provided by JetBrains is the following:

1qodana:
2  image: 
3    name: jetbrains/qodana
4    entrypoint: [sh, -c]
5  script:
6    - /opt/idea/bin/entrypoint --results-dir=$CI_PROJECT_DIR/qodana --save-report --report-dir=$CI_PROJECT_DIR/qodana/report
7  artifacts:
8    paths:
9      - qodana

While this works, it doesn’t provide a way to explore the report without firstly downloading it on your PC. If we have GitLab Pages enabled, we can publish the report and explore it online, thanks to the artifacts:expose_as keyword.

We also need GitLab to upload the right directory to Pages, so we change the artifact path as well:

 1qodana:
 2  image: 
 3    name: jetbrains/qodana
 4    entrypoint: [sh, -c]
 5  script:
 6    - /opt/idea/bin/entrypoint --results-dir=$CI_PROJECT_DIR/qodana --save-report --report-dir=$CI_PROJECT_DIR/qodana/report
 7  artifacts:
 8    paths:
 9      - qodana/report/
10    expose_as: 'Qodana report'

Now in our merge requests page we have a new button, as soon as the Qodana job finishes, to explore the report! You can see such a merge request here, with the button “View exposed artifact”, while here you can find an interactive online report, published on GitLab Pages!

Configuring Qodana

Full reference for the Qodana config file can be found on GitHub. Qodana can be easily customized: we only need to create a file called qodana.yaml, and enter our preferences!

There are two options I find extremely useful: one is exclude, that we can use to skip some checks, or to skip some directories, so we can focus on what is important and save some time. The other is failThreshold. When this number of problems is reached, the container would exit with error 255. In this way, we can fail the pipeline, and enforce a high quality of the code!

Qodana shows already a lot of potential, also if it only at the first version! I am really looking forward to support for other languages, and to improvements that JetBrains will do in the upcoming releases!

Questions, comments, feedback, critics, suggestions on how to improve my English? Leave a comment below, or drop me an email at [email protected]" rel="noopener noreferrer">[email protected].

Ciao,
R.

Fail a Gitlab pipeline when code coverage decreases

2025-01-26T21:31:29+00:00

Photo by Pankaj Patel on Unsplash

If you don’t have a proper Gitlab pipeline to lint your code, run your test, and manage all that other annoying small tasks, you should definitely create one! I’ve written an introductory guide to Gitlab CI, and many more are available on the documentation website.

While there isn’t (unfortunately!) a magic wand to highlight if the code is covered by enough tests, and, in particular, if these tests are of a sufficient good quality, we can nonetheless find some valuable KPI we can act on. Today, we will check code coverage, what indicates, what does not, and how it can be helpful.

Code coverage

In computer science, test coverage is a measure used to describe the degree to which the source code of a program is executed when a particular test suite runs. A program with high test coverage, measured as a percentage, has had more of its source code executed during testing, which suggests it has a lower chance of containing undetected software bugs compared to a program with low test coverage.

Wikipedia

Basically, code coverage indicates how much of your code has been executed while your tests were running. Personally, I don’t find a high code coverage a significant measure: if tests are fallacious, or they run only on the happy path, the code coverage percentage will be high, but the tests will not actually guarantee a high quality of the code.

On the other hand, a low code coverage is definitely worrisome, because it means some parts of the code aren’t tested at all. Thus, code coverage has to be taken, as every other KPI based only exclusively on lines of code, with a grain of salt.

Code coverage and Gitlab

Gitlab allows collecting code coverage from test suites directly from pipelines. Major information on the setup can be found in the pipelines guide and in the Gitlab CI reference guide. Since there are lots of different test suites out there, I cannot include how to configure them here. However, if you need any help, feel free to reach out to me at the contacts reported below.

Gitlab will also report code coverage statistic for pipelines over time in nice graphs under Project Analytics > Repository. Data can also be exported as csv! We will use such data to check if, in the commit, the code coverage decreased comparing to the main branch.

This means that every new code written has to be tested at least as much as the rest of the code is tested. Of course, this strategy can be easily changed. The check is only one line of bash, and can be easily be replaced with a fixed number, or any other logic.

The Gitlab Pipeline Job

The job that checks the coverage runs in a stage after the testing stage. It uses alpine as base, and curl and jq to query the APIs and read the code coverage.

On self hosted instances, or on Gitlab.com Bronze or above, you should use a project access token to give access to the APIs. On Gitlab.com Free, use a personal access token. If the project is public, the API are accessible without any token. It needs three variables: the name of the job which generates the code coverage percentage (JOB_NAME), the target branch to compare the coverage with (TARGET_BRANCH), and a private token to read the APIs (PRIVATE_TOKEN). The job will not run when the pipeline is running on the target branch, since it would be comparing the code coverage with itself, wasting minutes of runners for nothing.

The last line is the one providing the logic to compare the coverages.

 1checkCoverage:
 2    image: alpine:latest
 3    stage: postTest
 4    variables:
 5        JOB_NAME: testCoverage
 6        TARGET_BRANCH: main
 7    before_script:
 8        - apk add --update --no-cache curl jq
 9    rules:
10      - if: '$CI_COMMIT_BRANCH != $TARGET_BRANCH' 
11    script:
12        - TARGET_PIPELINE_ID=`curl -s "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/pipelines?ref=${TARGET_BRANCH}&status=success&private_token=${PRIVATE_TOKEN}" | jq ".[0].id"`
13        - TARGET_COVERAGE=`curl -s "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/pipelines/${TARGET_PIPELINE_ID}/jobs?private_token=${PRIVATE_TOKEN}" | jq --arg JOB_NAME "$JOB_NAME" '.[] | select(.name==$JOB_NAME) | .coverage'`
14        - CURRENT_COVERAGE=`curl -s "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/pipelines/${CI_PIPELINE_ID}/jobs?private_token=${PRIVATE_TOKEN}" | jq --arg JOB_NAME "$JOB_NAME" '.[] | select(.name==$JOB_NAME) | .coverage'`
15        - if  [ "$CURRENT_COVERAGE" -lt "$TARGET_COVERAGE" ]; then echo "Coverage decreased from ${TARGET_COVERAGE} to ${CURRENT_COVERAGE}" && exit 1; fi;

This simple job works both on Gitlab.com and on private Gitlab instances, for it doesn’t hard-code any URL.

Gitlab will now block merging merge requests without enough tests! Again, code coverage is not the magic bullet, and you shouldn’t strive to have 100% of code coverage: better fewer tests, but with high quality, than more just for increasing the code coverage. In the end, a human is always the best reviewer. However, a small memo to write just one more test is, in my opinion, quite useful ;-)

Questions, comments, feedback, critics, suggestions on how to improve my English? Leave a comment below, or drop me an email at [email protected]" rel="noopener noreferrer">[email protected].

Ciao,
R.

Create a PyTorch Docker image ready for production

2025-01-26T21:31:29+00:00

Photo by Michael Dziedzic on Unsplash

You know the drill: your Data Science team has created an amazing PyTorch model, and now they want you to put it in production. They give you a .pt file and some preprocessing script. What now?

Luckily, AWS and Facebook have created a project, called Torch Serve, to put PyTorch images in production, similarly to Tensorflow Serving. It is a well-crafted Docker image, where you can upload your models. In this tutorial, we will see how to customize the Docker image to include your model, how to install other dependencies inside it, and which configuration options are available.

We include the PyTorch model directly inside the Docker image, instead of loading it at runtime; while loading it at runtime as some advantages and makes sense in some scenario (as in testing labs where you want to try a lot of different models), I don’t think it is suitable for production. Including the model directly in the Docker image has different advantages:

if you use CI/CD you can achieve reproducible builds;
to spawn a new instance serving your model, you need to have available only your Docker registry, and not also a storage solutions to store the model;
you need to authenticate only to your Docker registry, and not to the storage solution;
it makes easier keeping track of what has been deployed, ‘cause you have to check only the Docker image version, and not the model version. This is especially important if you have a cluster of instances serving your model;

Let’s now get our hands dirty and dive in what is necessary to have the Docker image running!

Building the model archive

The Torch Serve Docker image needs a model archive to work: it’s a file with inside a model, and some configurations file. To create it, first install Torch Serve, and have a PyTorch model available somewhere on the PC.

To create this model archive, we need only one command:

1torch-model-archiver --model-name <MODEL_NAME> --version <MODEL_VERSION>  --serialized-file <MODEL> --export-path <WHERE_TO_SAVE_THE_MODEL_ARCHIVE>

There are four options we need to specify in this command:

MODEL_NAME is an identifier to recognize the model, we can use whatever we want here: it’s useful when we include multiple models inside the same Docker image, a nice feature of Torch Serve that we won’t cover for now;
MODEL_VERSION is used to identify, as the name implies, the version of the model;
MODEL is the path, on the local PC, with the .pt file acting as model;
WHERE_TO_SAVE_THE_MODEL_ARCHIVE is a local directory where Torch Serve will put the model archive it generates;

Putting all together, the command should be something similar to:

1torch-model-archiver --model-name predict_the_future --version 1.0 --serialized-file ~/models/predict_the_future --export-path model-store/

After having run it, we now have a file with .mar extension, the first step to put in production our PyTorch model!

Probably some pre-processing before invoking the model is necessary. If this is the case, we can create a file where we can put all the necessary instructions. This file can have external dependencies, so we can code an entire application in front of our model.

To include the handler file in the model archive, we need only to add the --handler flag to the command above, like this:

1torch-model-archiver --model-name predict_the_future --version 1.0 --serialized-file ~/models/predict_the_future --export-path model-store/ --handler handler.py

Create the Docker image

Now we have the model archive, and we include it in the PyTorch Docker Image. Other than the model archive, we need to create a configuration file as well, to say to PyTorch which model to automatically load at the startup.

We need a config.properties file similar to the following. Later in this tutorial we will see what these lines mean, and what other options are available.

1inference_address=http://0.0.0.0:8080
2management_address=http://0.0.0.0:8081
3number_of_netty_threads=32
4job_queue_size=1000
5model_store=/home/model-server/model-store

Docker image with just the model

If we need to include just the model archive, and the config file, the Dockerfile is quite straightforward, since we just have to copy the files, all the other things will be managed by TorchServe itself. Our Dockerfile will thus be:

1FROM pytorch/torchserve as production
2
3COPY config.properties /home/model-server/config.properties
4COPY predict_the_future.mar /home/model-server/model-store

TorchServe already includes torch, torchvision, torchtext, and torchaudio, so there is no need to add them. To see the current version of these libraries, please go see the requirements file of TorchServe on GitHub.

Docker image with the model and external dependencies

What if we need different Python Dependencies for our Python handler?

In this case, we want to use a two-step Docker image: in the first step we build our dependencies, and then we copy them over to the final image. We list our dependencies in a file called requirements.txt, and we use pip to install them. Pip is the package installer for Python. Their documentation about the format of the requirements file is very complete.

The Dockerfile is now something like this:

 1ARG BASE_IMAGE=ubuntu:18.04
 2
 3# Compile image loosely based on pytorch compile image
 4FROM ${BASE_IMAGE} AS compile-image
 5ENV PYTHONUNBUFFERED TRUE
 6
 7# Install Python and pip, and build-essentials if some requirements need to be compiled
 8RUN apt-get update && \
 9    DEBIAN_FRONTEND=noninteractive apt-get install --no-install-recommends -y \
10    python3-dev \
11    python3-distutils \
12    python3-venv \
13    curl \
14    build-essential \
15    && rm -rf /var/lib/apt/lists/* \
16    && cd /tmp \
17    && curl -O https://bootstrap.pypa.io/get-pip.py \
18    && python3 get-pip.py
19
20RUN python3 -m venv /home/venv
21
22ENV PATH="/home/venv/bin:$PATH"
23
24RUN update-alternatives --install /usr/bin/python python /usr/bin/python3 1
25RUN update-alternatives --install /usr/local/bin/pip pip /usr/local/bin/pip3 1
26
27# The part above is cached by Docker for future builds
28# We can now copy the requirements file from the local system
29# and install the dependencies
30COPY requirements.txt .
31
32RUN pip install --no-cache-dir -r requirements.txt
33
34FROM pytorch/torchserve as production
35
36# Copy dependencies after having built them
37COPY --from=compile-image /home/venv /home/venv
38
39# We use curl for health checks on AWS Fargate
40USER root
41RUN apt-get update && \
42    DEBIAN_FRONTEND=noninteractive apt-get install --no-install-recommends -y \
43    curl \
44    && rm -rf /var/lib/apt/lists/*
45
46USER model-server
47
48COPY config.properties /home/model-server/config.properties
49COPY predict_the_future.mar /home/model-server/model-store

If PyTorch is among the dependencies, we should change the line to install the requirements from

1RUN pip install --no-cache-dir -r requirements.txt

1RUN pip install --no-cache-dir -r requirements.txt -f https://download.pytorch.org/whl/torch_stable.html

In this way, we will use the pre-build Python packages for PyTorch instead of installing them from scratch: it will be faster, and it requires less resources, making it suitable also for small CI/CD systems.

Configuring the Docker image

We created a configuration file above, but what does it? Of course, going through all the possible configurations would be impossible, so I leave here the link to the documentation. Among the other things explained there, there is a way to configure Cross-Origin Resource Sharing (necessary to use the model as APIs over the web), a guide on how to enable SSL, and much more.

There is a set of configurations parameters in particular I’d like to focus on: the ones related to logging. First, for production environment, I suggest setting async_logging to true: it could delay a bit the output, but allows a higher throughput. Then, it’s important to notice that by default Torch Serve captures every message, including the ones with severity DEBUG. In production, we probably don’t want this, especially because it can become quite verbose.

To override the default behavior, we need to create a new file, called log4j.properties. For more information on every possible options I suggest familiarizing with the official guide. To start, copy the default Torch Serve configuration, and increase the severity of the printed messages. In particular, change

1log4j.logger.org.pytorch.serve = DEBUG, ts_log
2log4j.logger.ACCESS_LOG = INFO, access_log

1log4j.logger.org.pytorch.serve = WARNING, ts_log
2log4j.logger.ACCESS_LOG = WARNING, access_log

We need also to copy this new file to the Docker Image, so copy the logging config just after the config file:

1COPY config.properties /home/model-server/config.properties
2COPY config.properties /home/model-server/log4j.properties

We need to inform Torch Serve about this new config file, and we do so adding a line to config.properties:

1vmargs=-Dlog4j.configuration=file:///home/model-server/log4j.properties

We now have a full functional Torch Serve Docker image, with our custom model, ready to be deployed!

For any question, comment, feedback, critic, suggestion on how to improve my English, leave a comment below, or drop an email at [email protected]" rel="noopener noreferrer">[email protected].

Ciao,
R.

Introducing Daintree.app: an opensource alternative implementation of the AWS console.

2025-01-26T21:31:29+00:00

The AWS Console is an amazing piece of software: it has hundreds of thousands of features, it is reliable, and it is the front end of an incredible world. However, as any software, it is not perfect: sometimes is a bit slow, so many features can be confusing, and it is clear it has evolved over time, so there are a lot of different styles, and if it would be made from scratch today, some choices would probably be different.

Daintree has born wanting to fix one particular problem of the AWS Console: the impossibility to see resources from multiple regions in the same view.

I’ve starting working on it last month, and now I’m ready to publish a first version: it’s still quite young and immature, but I’m starting using it to check some resources on AWS accounts I have access to. A lot of features are still missing, of course, and if you like, you can contribute to its development.

Multiple region support

The main reason Daintree exists is to display resources from multiple regions in the same screen: why limiting to one, when you can have 25?

Also, changing enabled regions doesn’t require a full page reload, but just a click on a flag: Daintree will smartly require resources from the freshly enabled regions.

Fast role switching

If you belong to an AWS organization, and you have multiple accounts, you probably switch often role: such operation on the original AWS console requires a full page reload, and it always brings you to the homepage.

On Daintree, changing roles will only reload the resources in the page you are currently in, without having to wait for a full page reload!

Coherent interface

Beauty is in the eye of the beholder, so claiming Daintree is more beautiful than the original console would be silly: however, Daintree has been built to be coherent: all the styling is made thanks to the Gitlab UI project, and the illustrations are made by Katerina Limpitsouni from unDraw.

This guarantees a coherent and polished experience all over the application.

Free software

Daintree is licensed under AGPL-3.0, meaning is truly free software: this way, everyone can contribute improving the experience of using it. The full source code is available over Gitlab.

The project doesn’t have any commercial goal, and as explained in the page about security, no trackers are present on the website. To help Daintree, you can contribute and spread the word!

Daintree heavily uses Javascript (with Vue.js) to do not have to reload the page: also, it tries to perform as few operations as possible to keep always updated on your resources. In this way, you don’t waste your precious time waiting for the page to load.

And much more!

While implementing Daintree, new features have been introduced to make life easier to whoever uses it. As an example, you can create Internet Gateway and attach them to a VPC in the same screen, without having to wait for the gateway to be created. Or, while visualizing a security groups, you can click on the linked security groups, without having to look for them or remember complicated IDs. If you have any idea on how to improve workflows, please share it with developers!

Supported components

Daintree is still at early stages of development, so the number of supported resources is quite limited. You can report which feature you’d like to see to the developers, or you can implement them!

Daintree allows to view VPCs, Subnets, Internet Gateways, Nat Gateways, Route Tables, Elastic IPs, Security Groups, Instances, SNS, and SQS. You can also create, delete, and edit some of these resources. Development is ongoing, so remember to check the changelog from time to time.

What are you waiting? Go to https://daintree.app and enjoy your AWS resources in a way you haven’t before!

Needless to say, Daintree website is not affiliated to Amazon, or AWS, or any of their subsidiaries. ;-)

For any comment, feedback, critic, suggestion on how to improve my English, leave me a comment below, or drop an email at [email protected]" rel="noopener noreferrer">[email protected].

Ciao,
R.

Leveraging AWS Lambda to notify users about their old access keys

2025-01-26T21:31:29+00:00

The AWS IAM console helps to highlight which keys are old, but if you have dozens of users, or multiple AWS accounts, it is still boring doing it manually. So, I wrote some code to doing it automatically leveraging AWS Lambda - since it has a generous free-tier, this check is free (however, your mileage may vary).

Enter a caption...

Image by Randall Munroe, xkcd.com

Setting up the permissions

Of course, we want to follow the principle of least privilege: the Lambda function will have access only to the minimum data necessary to perform its task. Thus, we need to create a dedicated role over the IAM Console. AWS Guide to create roles for AWS services

Our custom role needs to have the managed policy AWSLambdaBasicExecutionRole, needed to execute a Lambda function. Other than this, we create a custom inline policy with this permissions:

iam:ListUsers, to know which users have access to the account. If you want to check only a subset of users, like filtering by department, you can use the Resource field to limit the access.
iam:ListAccessKeys, to read access keys of the users. Of course, you can limit here as well which users the Lambda has access to.
ses:SendEmail, to send the notification emails. Once again, you can (and should!) restrict the ARN to which it has access to.

And that are all the permissions we need!

The generated policy should look like this, more or less:

 1{
 2    "Version": "2012-10-17",
 3    "Statement": [
 4        {
 5            "Sid": "VisualEditor0",
 6            "Effect": "Allow",
 7            "Action": [
 8                "ses:SendEmail",
 9                "iam:ListAccessKeys"
10            ],
11            "Resource": [
12                "arn:aws:iam::<ACCOUNT_ID>:user/*",
13                "arn:aws:ses:eu-central-1:<ACCOUNT_ID>:identity/*"
14            ]
15        },
16        {
17            "Sid": "VisualEditor1",
18            "Effect": "Allow",
19            "Action": "iam:ListUsers",
20            "Resource": "*"
21        }
22    ]
23}

Setting up SES

To send the notification email, we use AWS Simple Email Service.

Before using it, you need to move out of the sandbox mode, or to verify domains you want to send emails to.

After that, you don’t have to do anything else, SES will be used by the Lambda code.

Setting up Lambda

You can now create an AWS Lambda function. I’ve written the code that you find below in Python, since I find it is the fastest way to put in production such a simple script. However, you can use any of the supported languages.

You need to assign the role we created before as an execution role. As memory, 128 MB is more than enough. About the timeout, it’s up to how big your company is. More or less, it can check 5/10 users every second. You should test it and see if it goes in timeout.

Lambda Code

Following there is the code to perform the task. To read it better, you can find it also on this Gitlab’s snippet.

  1from collections import defaultdict
  2from datetime import datetime, timezone
  3import logging
  4
  5import boto3
  6from botocore.exceptions import ClientError
  7
  8
  9# How many days before sending alerts about the key age?
 10ALERT_AFTER_N_DAYS = 100
 11# How ofter we have set the cron to run the Lambda?
 12SEND_EVERY_N_DAYS = 3
 13# Who send the email?
 14SES_SENDER_EMAIL_ADDRESS = '[email protected]'
 15# Where did we setup SES?
 16SES_REGION_NAME = 'eu-west-1'
 17
 18iam_client = boto3.client('iam')
 19ses_client = boto3.client('ses', region_name=SES_REGION_NAME)
 20
 21# Helper function to choose if a key owner should be notified today
 22def is_key_interesting(key):
 23    # If the key is inactive, it is not interesting
 24    if key['Status'] != 'Active':
 25        return False
 26    
 27    elapsed_days = (datetime.now(timezone.utc) - key['CreateDate']).days
 28    
 29    # If the key is newer than ALERT_AFTER_N_DAYS, we don't need to notify the
 30    # owner
 31    if elapsed_days < ALERT_AFTER_N_DAYS:
 32        return False
 33    
 34    return True
 35    
 36# Helper to send the notification to the user. We need the receiver email, 
 37# the keys we want to notify the user about, and on which account we are
 38def send_notification(email, keys, account_id):
 39    email_text = f'''Dear {keys[0]['UserName']},
 40this is an automatic reminder to rotate your AWS Access Keys at least every {ALERT_AFTER_N_DAYS} days.
 41
 42At the moment, you have {len(keys)} key(s) on the account {account_id} that have been created more than {ALERT_AFTER_N_DAYS} days ago:
 43'''
 44    for key in keys:
 45        email_text += f"- {key['AccessKeyId']} was created on {key['CreateDate']} ({(datetime.now(timezone.utc) - key['CreateDate']).days} days ago)\n"
 46    
 47    email_text += f"""
 48To learn how to rotate your AWS Access Key, please read the official guide at https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html#Using_RotateAccessKey
 49If you have any question, please don't hesitate to contact the Support Team at [email protected].
 50
 51This automatic reminder will be sent again in {SEND_EVERY_N_DAYS} days, if the key(s) will not be rotated.
 52
 53Regards,
 54Your lovely Support Team
 55"""
 56    
 57    try:
 58        ses_response = ses_client.send_email(
 59            Destination={'ToAddresses': [email]},
 60            Message={
 61                'Body': {'Html': {'Charset': 'UTF-8', 'Data': email_text}},
 62                'Subject': {'Charset': 'UTF-8',
 63                            'Data': f'Remember to rotate your AWS Keys on account {account_id}!'}
 64            },
 65            Source=SES_SENDER_EMAIL_ADDRESS
 66        )
 67    except ClientError as e:
 68        logging.error(e.response['Error']['Message'])
 69    else:
 70        logging.info(f'Notification email sent successfully to {email}! Message ID: {ses_response["MessageId"]}')
 71
 72def lambda_handler(event, context):
 73    users = []
 74    is_truncated = True
 75    marker = None
 76    
 77    # We retrieve all users associated to the AWS Account.  
 78    # Results are paginated, so we go on until we have them all
 79    while is_truncated:
 80        # This strange syntax is here because `list_users` doesn't accept an 
 81        # invalid Marker argument, so we specify it only if it is not None
 82        response = iam_client.list_users(**{k: v for k, v in (dict(Marker=marker)).items() if v is not None})
 83        users.extend(response['Users'])
 84        is_truncated = response['IsTruncated']
 85        marker = response.get('Marker', None)
 86    
 87    # Probably in this list you have bots, or users you want to filter out
 88    # You can filter them by associated tags, or as I do here, just filter out 
 89    # all the accounts that haven't logged in the web console at least once
 90    # (probably they aren't users)
 91    filtered_users = list(filter(lambda u: u.get('PasswordLastUsed'), users))
 92    
 93    interesting_keys = []
 94    
 95    # For every user, we want to retrieve the related access keys
 96    for user in filtered_users:
 97        response = iam_client.list_access_keys(UserName=user['UserName'])
 98        access_keys = response['AccessKeyMetadata']
 99        
100        # We are interested only in Active keys, older than
101        # ALERT_AFTER_N_DAYS days
102        interesting_keys.extend(list(filter(lambda k: is_key_interesting(k), access_keys)))
103    
104    # We group the keys by owner, so we send no more than one notification for every user
105    interesting_keys_grouped_by_user = defaultdict(list)
106    for key in interesting_keys:
107        interesting_keys_grouped_by_user[key['UserName']].append(key)
108
109    for user in interesting_keys_grouped_by_user.values():
110        # In our AWS account the username is always a valid email. 
111        # However, you can recover the email from IAM tags, if you have them
112        # or from other lookups
113        # We also get the account id from the Lambda context, but you can 
114        # also specify any id you want here, it's only used in the email 
115        # sent to the users to let them know on which account they should
116        # check
117        send_notification(user[0]['UserName'], user, context.invoked_function_arn.split(":")[4])

Schedule your Lambda

You can schedule your Lambda to run thanks to CloudWatch Events. You can use a schedule expression such rate(3 days) to run the email every 3 days. Lambda will add necessary permissions to the role we created before to invoke the Lambda. If you need any help, AWS covers you with a dedicated tutorial!

Conclusions

This is just an idea on how to create a little script, leveraging AWS Lambda and AWS SES, to keep your AWS account safe. There are, of course, plenty of possible improvements! And remember to check the logs, sometimes ;-)

If you have hundreds or thousands of users, the function will go in timeout: there are different solutions you can implement, as using tags on users to know when you have lasted checked them, or checking a different group of users every hour, leveraging the PathPrefix argument of list_users.

Also, in my example it’s simple to know to whom send the notification email - but what if your users don’t have their email as username? You can use tags, and set their contact email there. Or, you maybe have to implement a lookup somewhere else.

We could also send a daily report to admins: since users usually ignore automatic emails, admins can intervene if too many reports have been ignored. Or, we can forcibly delete keys after some time - although this could break production code, so I wouldn’t really do it - or maybe yes, it’s time developers learn to have a good secrets’ hygiene.

And you? How do you check your users rotate their access keys?

For any comment, feedback, critic, suggestion on how to improve my English, leave a comment below, or drop an email at [email protected]" rel="noopener noreferrer">[email protected].

Ciao,
R.

My year on HackerOne

2025-01-26T21:31:29+00:00

Last year, totally by chance, I found a security issue over Facebook - I reported it, and it was fixed quite fast. In 2018, I also found a security issue over Gitlab, so I signed up to HackerOne, and reported it as well. That first experience with Gitlab was far from ideal, but after that first report I’ve started reporting more, and Gitlab has improved its program a lot.

2019

Since June 2019, when I opened my first report of the year, I reported 27 security vulnerabilities: 4 has been marked as duplicated, 3 as informative, 2 as not applicable, 9 have been resolved, and 9 are currently confirmed and the fix is ongoing. All these 27 vulnerabilities were reported to Gitlab.

Especially in October and November I had a lot of fun testing the implementation of ElasticSearch over Gitlab. Two of the issues I have found on this topic have already been disclosed:

Why just Gitlab?

I have an amazing daily job as Solutions Architect at Nextbit that I love. I am not interested in becoming a full-time security researcher, but I am having fun dedicating some hours every month in looking for securities vulnerabilities.

However, since I don’t want it to be a job, I focus on a product I know very well, also because sometimes I contribute to it and I use it daily.

I also tried to target some program I didn’t know anything about, but I get bored quite fast: to find some interesting vulnerability you need to spend quite some time to learn how the system works, and how to exploit it.

Last but not least, Gitlab nowadays manages its HackerOne program in a very cool way: they are very responsive, kind, and I like they are very transparent! You can read a lot about how their security team works in their handbook.

Can you teach me?

Since I have shared a lot of the disclosed reports on Twitter, some people came and asked me to teach them how to start in the bug bounties world. Unfortunately, I don’t have any useful suggestion: I haven’t studied on any specific resource, and all the issues I reported this year come from a deep knowledge of Gitlab, and from what I know thanks to my daily job. There are definitely more interesting people to follow on Twitter, just check over some common hashtags, such as TogetherWeHitHarder.

Gitlab’s Contest

I am writing this blog post from my new keyboard: a custom-made WASD VP3, generously donated by Gitlab after I won a contest for their first year of public program on HackerOne. I won the best written report category, and it was a complete surprise; I am not a native English speaker, 5 years ago my English was a monstrosity (if you want to have some fun, just go reading my old blog posts), and still to this day I think is quite poor, as you can read here.

Indeed, if you have any suggestion on how to improve this text, please write me!

Congratulations to Gitlab for their first year on HackerOne, and keep up the good work! Your program rocks, and in the last months you improved a lot!

HackeOne Clear

HackerOne started a new program, called HackerOne Clear, only on invitation, where they vet all researchers. I was invited and I thought about accepting the invitation. However, the scope of the data that has to be shared to be vetted is definitely too wide, and to be honest I am surprised so many people accepted the invitation. HackerOne doesn’t perform the check, but delegates to a 3rd party. This 3rd party company asks a lot of things.

I totally understand the need of background checks, and I’d be more than happy to provide my criminal record. It wouldn’t be the first time I am vetted, and I am quite sure it wouldn’t be the last.

More than the criminal record, I am a puzzled about these requirements:

Financial history, including credit history, bankruptcy and financial judgments;
Employment or volunteering history, including fiduciary or directorship responsibilities;
Gap activities, including travel;
Health information, including drug tests;
Identity, including identifying numbers and identity documents;

Not only the scope is definitely too wide, but also all these data will be stored and processed outside EU! Personal information will be stored in the United States, Canada and Ireland. Personal information will be processed in the United States, Canada, the United Kingdom, India and the Philippines.

As European citizen who wants to protect his privacy, I cannot accept such conditions. I’ve written to HackerOne asking why such a wide scope of data, and they replied that since it’s their partner that actually collects the information, there is nothing they can do. I really hope HackerOne will require fewer data in the future, preserving privacy of their researchers.

2020

In these days I’ve though a lot about what I want to do in my future about bug bounties, and for the 2020 I will continue as I’ve done in the last months: assessing Gitlab, dedicating not more than a few hours a month. I don’t feel ready to step up my game at the moment. I have a lot of other interests I want to pursue in 2020 (travelling, learning German, improve my cooking skills), so I will not prioritize bug bounties for the time being.

That’s all for today, and also for the 2019! It has been a lot of fun, and I wish to you all a great 2020! For any comment, feedback, critic, leave a comment below, or drop an email at [email protected]" rel="noopener noreferrer">[email protected].

Ciao,
R.

Updates

29th December 2019: added paragraph about having asked to HackerOne more information on why they need such wide scope of personal data.

Exploring Gitlab Visual Reviews

2025-01-26T21:31:29+00:00

If you already have Continuous Integration and Continuous Delivery enabled for your websites, adding this feature is blazing fast, and will make life of your reviewers easier! If you want to start with CI/CD in Gitlab, I’ve written about it in the past.

The feature

While the official documentation has a good overview of the feature, we can take a deeper look with some screenshots:

We can comment directly from the staging environment! And additional metadata will be collected and published as well, making easier to reproduce a bug.

Our comment (plus the metadata) appears in the merge request, becoming actionable.

Implementing the system

Adding the snippet isn’t complicate, you only need some information about the MR. Basically, this is what you should add to the head of your website for every merge request:

1<script
2  data-project-id="CI_PROJECT_ID"
3  data-merge-request-id="CI_MERGE_REQUEST_IID"
4  data-mr-url='https://gitlab.example.com'
5  data-project-path="CI_PROJECT_PATH"
6  id='review-app-toolbar-script'
7  src='https://gitlab.example.com/assets/webpack/visual_review_toolbar.js'>
8</script>

Of course, asking your team to add the HTML snippet, and filling it with the right information isn’t feasible. We will instead take advantage of Gitlab CI/CD to inject the snippet and autocomplete it with the right information for every merge request.

First we need the definition of a Gitlab CI job to build our client:

 1buildClient:
 2  image: node:12
 3  stage: build
 4  script:
 5    - ./scripts/inject-review-app-index.sh
 6    - npm ci
 7    - npm run build
 8  artifacts:
 9    paths:
10      - build
11  only:
12    - merge_requests
13  cache:
14    paths:
15      - .npm

The important bit of information here is only: merge_requests. When used, Gitlab injects in the job a environment variable, called CI_MERGE_REQUEST_IID, with the unique ID of the merge request: we will fill it in the HTML snippet. The official documentation of Gitlab CI explains in detail all the other keywords of the YAML.

The script

The other important bit is the script that actually injects the code: it’s a simple bash script, which looks for the </title> tag in the HTML, and append the needed snippet:

 1#!/bin/sh
 2
 3repl() {
 4  PATTERN=$1 REPL=$2 awk '
 5    {gsub(ENVIRON["PATTERN"], ENVIRON["REPL"]); print}'
 6}
 7
 8TEXT_TO_INJECT=$(cat <<-HTML
 9</title>
10<script
11  data-project-id="${CI_PROJECT_ID}"
12  data-merge-request-id="${CI_MERGE_REQUEST_IID}"
13  data-mr-url='${CI_SERVER_URL}'
14  data-project-path="${CI_PROJECT_PATH}"
15  id='review-app-toolbar-script'
16  src='${CI_SERVER_URL}/assets/webpack/visual_review_toolbar.js'>
17</script>
18HTML
19)
20
21repl "</title>" "${TEXT_TO_INJECT}" < public/index.html > tmpfile && mv tmpfile public/index.html

Thanks to the Gitlab CI environment variables, the snippet has already all the information it needs to work. Of course you should customize the script with the right path for your index.html (or any other page you have).

Now everything is ready! Your team needs only to generate personal access tokens to login, and they are ready to go! You should store your personal access token in your password manager, so you don’t need to generate it each time.

Future features

One of the coolest things in Gitlab is that everything is always a work in progress, and each feature has some new goodies in every release. This is true for the Visual Reviews App as well. There is an epic that collects all the improvements they want to do, including removing the need for an access token, and adding ability to take screenshots that will be inserted in the MR comments as well.

That’s all for today, I hope you found this article useful! For any comment, feedback, critic, leave a comment below, or drop an email at [email protected]" rel="noopener noreferrer">[email protected].

I have also changed the blog theme to a custom version of Rapido.css. I think it increases the readability, but let me know what you think!

Ciao,
R.

Using AWS Textract in an automatic fashion with AWS Lambda

2025-01-26T21:31:29+00:00

Updated on January 4th, 2023: removed references to SQS, that is not actually used in the proposed implementation.

In that case, we need to invoke some asynchronous APIs, poll an endpoint to check when it has finished working, and then read the result, which is paginated, so we need multiple APIs call. Wouldn’t be super cool to just drop files in an S3 bucket, and after some minutes, having their content in another S3 bucket?

Let’s see how to use AWS Lambda and SNS to automatize all the process!

Overview of the process

This is the process we are aiming to build:

Drop files to an S3 bucket;
A trigger will invoke an AWS Lambda function, which will inform AWS Textract of the presence of a new document to analyze;
AWS Textract will do its magic, and push the status of the job to an SNS topic;
The SNS topic will invoke another Lambda function, which will read the status of the job, and if the analysis was successful, it downloads the extracted text and save to another S3 bucket (but we could replace this with a write over DynamoDB or others database systems);
The Lambda function will also publish the state over Cloudwatch, so we can trigger alarms when a read was unsuccessful.

Since a picture is worth a thousand words, let me show a graph of this process.

While I am writing this, Textract is available only in 4 regions: US East (Northern Virginia), US East (Ohio), US West (Oregon), and EU (Ireland). I strongly suggest therefore to create all the resources in just one region, for the sake of simplicity. In this tutorial, I will use eu-west-1.

S3 buckets

First of all, we need to create two buckets: one for our raw file, and one for the JSON file with the extracted test. We could also use the same bucket, theoretically, but with two buckets we can have better access control.

Since I love boring solutions, for this tutorial I will call the two buckets textract_raw_files and textract_json_files. Official documentation explains how to create S3 buckets.

Invoke Textract

The first part of the architecture is informing Textract of every new file we upload to S3. We can leverage the S3 integration with Lambda: each time a new file is uploaded, our Lambda function is triggered, and it will invoke Textract.

The body of the function is quite straightforward:

 1from urllib.parse import unquote_plus
 2
 3import boto3
 4
 5s3_client = boto3.client('s3')
 6textract_client = boto3.client('textract')
 7
 8SNS_TOPIC_ARN = 'arn:aws:sns:eu-west-1:123456789012:AmazonTextract'    # We need to create this
 9ROLE_ARN = 'arn:aws:iam::123456789012:role/TextractRole'   # This role is managed by AWS
10
11def handler(event, _):
12    for record in event['Records']:
13        bucket = record['s3']['bucket']['name']
14        key = unquote_plus(record['s3']['object']['key'])
15
16        print(f'Document detection for {bucket}/{key}')
17
18        textract_client.start_document_text_detection(
19            DocumentLocation={'S3Object': {'Bucket': bucket, 'Name': key}},
20            NotificationChannel={'RoleArn': ROLE_ARN, 'SNSTopicArn': SNS_TOPIC_ARN})

You can find a copy of this code hosted over Gitlab.

As you can see, we receive a list of freshly uploaded files, and for each one of them, we ask Textract to do its magic. We also ask it to notify us, when it has finished its work, sending a message over SNS. We need therefore to create an SNS topic. It is well explained how to do so in the official documentation.

When we have finished, we should have something like this:

We copy the ARN of our freshly created topic and insert it in the script above in the variable SNS_TOPIC_ARN.

Now we need to actually create our Lambda function: once again the official documentation is our friend if we have never worked with AWS Lambda before.

Since the only requirement of the script is boto3, and it is included by default in Lambda, we don’t need to create a custom package.

At least, this is usually the case :-) Unfortunately, while I am writing this post, boto3 on Lambda is at version boto3-1.9.42, while support for Textract landed only in boto3-1.9.138. We can check which version is currently on Lambda from this page, under Python Runtimes: if boto3 has been updated to a version >= 1.9.138, we don’t have to do anything more than simply create the Lambda function. Otherwise, we have to include a newer version of boto3 in our Lambda function. But fear not! The official documentation explains how to create a deployment package. Update Oct ‘19: this is no longer the case, AWS has updated the boto3 package included in the Lambda runtime.

We need also to link an IAM role to our Lambda function, which requires some additional permission:

AmazonTextractFullAccess: this gives access also to SNS, other than Textract
AmazonS3ReadOnlyAccess: if we want to be a bit more conservative, we can give access to just the textract_raw_files bucket.

Of course, other than that, the function requires the standard permissions to be executed and to write on Cloudwatch: AWS manages that for us.

We are almost there, we need only to create the trigger: we can do that from the Lambda designer! From the designer we select S3 as the trigger, we set our textract_raw_files bucket, and we select All object create events as Event type.

If we implemented everything correctly, we can now upload a PDF file to the textract_raw_files, and over Cloudwatch we should be able to see the log of the Lambda function, which should say something similar to Document detection for textract_raw_files/my_first_file.pdf.

Now we only need to read the extracted text, all the hard work has been done by AWS :-)

Read data from Textract

AWS Textract is so kind to notify us when it has finished extracting data from PDFs we provided: we create a Lambda function to intercept such notification, invoke AWS Textract and save the result in S3.

The Lambda function needs also to support pagination in the results, so the code is a bit longer:

 1import json
 2import boto3
 3
 4textract_client = boto3.client('textract')
 5s3_bucket = boto3.resource('s3').Bucket('textract_json_files')
 6
 7
 8def get_detected_text(job_id: str, keep_newlines: bool = False) -> str:
 9    """
10    Giving job_id, return plain text extracted from input document.
11    :param job_id: Textract DetectDocumentText job Id
12    :param keep_newlines: if True, output will have same lines structure as the input document
13    :return: plain text as extracted by Textract
14    """
15    max_results = 1000
16    pagination_token = None
17    finished = False
18    text = ''
19
20    while not finished:
21        if pagination_token is None:
22            response = textract_client.get_document_text_detection(JobId=job_id,
23                                                                   MaxResults=max_results)
24        else:
25            response = textract_client.get_document_text_detection(JobId=job_id,
26                                                                   MaxResults=max_results,
27                                                                   NextToken=pagination_token)
28
29        sep = ' ' if not keep_newlines else '\n'
30        text += sep.join([x['Text'] for x in response['Blocks'] if x['BlockType'] == 'LINE'])
31
32        if 'NextToken' in response:
33            pagination_token = response['NextToken']
34        else:
35            finished = True
36
37    return text
38
39
40def handler(event, _):
41    for record in event['Records']:
42        message = json.loads(record['Sns']['Message'])
43        job_id = message['JobId']
44        status = message['Status']
45        filename = message['DocumentLocation']['S3ObjectName']
46
47        print(f'JobId {job_id} has finished with status {status} for file {filename}')
48
49        if status != 'SUCCEEDED':
50            return
51
52        text = get_detected_text(job_id)
53        to_json = {'Document': filename, 'ExtractedText': text, 'TextractJobId': job_id}
54        json_content = json.dumps(to_json).encode('UTF-8')
55        output_file_name = filename.split('/')[-1].rsplit('.', 1)[0] + '.json'
56        s3_bucket.Object(f'{output_file_name}').put(Body=bytes(json_content))
57
58        return message

You can find a copy of this code hosted over Gitlab.

Again, this code has to be published as a Lambda function. As before, it shouldn’t need any special configuration, but since it requires boto3 >= 1.9.138 we have to create a deployment package, as long as AWS doesn’t update the Lambda runtime.

After we have uploaded the Lambda function, from the control panel we set as trigger SNS, specifying as ARN the ARN of the SNS topic we created before - in our case, arn:aws:sns:eu-west-1:123456789012:AmazonTextract.

We need also to give the IAM role which executes the Lambda function new permissions, in addition to the ones it already has. In particular, we need:

AmazonTextractFullAccess
AmazonS3FullAccess: in production, we should give access to just the textract_json_files bucket.

This should be the final result:

And that’s all! Now we can simply drop any document in a supported format to the textract_raw_files bucket, and after some minutes we will find its content in the textract_json_files bucket! And the quality of the extraction is quite good.

Known limitations

Other than being available in just 4 locations, at least for the moment, AWS Textract has other known hard limitations:

The maximum document image (JPEG/PNG) size is 5 MB.
The maximum PDF file size is 500 MB.
The maximum number of pages in a PDF file is 3000.
The maximum PDF media size for the height and width dimensions is 40 inches or 2880 points.
The minimum height for text to be detected is 15 pixels. At 150 DPI, this would be equivalent to 8-pt font.
Documents can be rotated a maximum of +/- 10% from the vertical axis. Text can be text aligned horizontally within the document.
Amazon Textract doesn’t support the detection of handwriting.

It has also some soft limitations that make it unsuitable for mass ingestion:

Transactions per second per account for all Start (asynchronous) operations: 0.25
Maximum number of asynchronous jobs per account that can simultaneously exist: 2

So, if you need it for anything but testing, you should open a ticket to ask for higher limits, and maybe poking your point of contact in AWS to speed up the process.

That’s all for today, I hope you found this article useful! For any comment, feedback, critic, leave a comment below, or drop an email at [email protected].

Regards,
R.

Responsible disclosure: improper access control in Gitlab private project.

2025-01-26T21:31:29+00:00

This issue was firstly reported on HackerOne and was managed on the Gitlab issues’ tracker. Both links are now publicly accessible.

Summary of the issue

Rogue user is added to a private group with dozens of projects
The user’s role in some projects changes
Rogue is fired, and removed from the group: they still have access to projects where their role was changed

The second step could happen for a lot of different reasons:

rogue is added as master - knowing this vulnerability, they decrease their privileges to stay in some projects (this is the only malicious one)
rogue is added as developer, but they become responsible for some projects, and are promoted to master role
rogue is added as reporter, and then they are promoted for a project, and so on.

When an admin removes a user from a private group, there is no indication that the user still has access to private projects, if their role was changed.

Impact

User can still see all resources of a project of a secret group after they have been removed from the parent’s group.

Timeline

29 January 2018: First disclosure to Gitlab
9 February 2018: Gitlab confirmed the issue and triaged it, assigning a medium priority
25 February 2018: I ask for a timeline
27 February 2018: They inform me they will update me with a timeline
16 March 2018: Almost two months are passed, I ask again for a timeline or suggest to go public since administrators of groups can easily check and avoid this vulnerability
17 March 2018: They inform me they will update me with a timeline, and ask to do not go public
Somewhere around December 2018: the team think the issue has been fixed, and close the internal issue - without communicating with me
17 January 2019: I ask for an update - they will never reply to this message
25 January 2019: the security team sees this is still an issue
31 January 2019: the fix is deployed in production and publicly disclosed, without informing me
5 March 2019: I ask again for another update
12 March 2019: Gitlab says the issue has been fixed and awards me a bounty

Bounty

Gitlab awarded me a $2000 bounty award for the disclosure.

If you follow my blog, you know I deeply love Gitlab: I contribute to it, I write blog posts, and I advocate for it any time I can. Still, I think this experience was awful, to say the least. There was a total lack of communication by their side, they thought they fixed the issue the first time, but actually, it wasn’t fixed. If they had communicated with me, I would have double checked their work. After that, they deployed the fix and went public, without telling me. I was not interested in the bounty (for which I am grateful), I reported the issue because I care about Gitlab. Nonetheless, my love for Gitlab is still the same! I just hope they will improve this part of communication / contributing to Gitlab: in the last couple of years the community around the project grew a lot, and they are doing amazing with it, maybe the Community team should step in and help also the security community?

For any comment, feedback, critic, leave a comment below, or drop an email at [email protected].

Regards,
R.

Glasnost: yet another Gitlab's client.

2025-01-26T21:31:29+00:00

I travel often, and be able to work on issues and pipelines on the go is something essential for me. Unfortunately, Gitlab’s UX on small screens is far from ideal (while it has improved over the years).

Enter Glasnost

My good friend Giovanni has developed a new opensource mobile client for Gitlab, with a lot of cool features: Glasnost!

In his words:

Glasnost is a free, currently maintained, platform independent and opensource mobile application that is aiming to visualize and edit the most important entities handled by Gitlab.

Among the others features, I’d like to highlight support for multiple Gitlab hosts (so you can work both on your company’s Gitlab and on Gitlab.com at the same time), two different themes (a light one and a dark one), a lite version for when your data connection is stuck on edge, and support for fingerprint authentication.

The application is still in an early phase of development, but it already has enough features to be used daily. I am sure Giovanni would love some feedback and suggestions, so please go on the Glasnost’s issues tracker or leave a feedback on the PlayStore.

If you feel a bit more adventurous, you can contribute to the application itself: it is written in React+Redux with Expo: the code is hosted on Gitlab (of course).

Enjoy yet another client for Gitlab, and let Giovanni knows what you think!

For any comment, feedback, critic, leave a comment below or drop an email at [email protected].

Regards,
R.

Responsible disclosure: retrieving a user's private Facebook friends.

2025-01-26T21:31:29+00:00

On the pratical side, how we will see, disclose of private data is often a unwanted side effect of an useful feature.

Facebook and Instagram

Facebook bought Instagram back in 2012. Since then, a lot of integrations have been implemented between them: among the others, when you suscribe to Instagram, it will suggest you who to follow based on your Facebook friends.

Your Instagram and Facebook accounts are then somehow linked: it happens both if you sign up to Instagram using your Facebook account (doh!), but also if you sign up to Instagram creating a new account but using the same email you use in your Facebook account (there are also other way Instagram links your new account with an existing Facebook account, but they are not of our interest here).

So if you want to create a secret Instagram account, create a new mail for it ;-)

Back in topic: Instagram used to enable all its feature to new users, before they have confirmed their email address. This was to do not “interrupt” usage of the website / app, they would have been time to confirm the email later in their usage.

Email address confirmation is useful to confirm you are signing up using your own email address, and not one of someone else.

Data leak

One of the features available before confirming the email address, was the suggestion of who to follow based on the Facebook friends of the account Instagram automatically linked.

This made super easy to retrieve the Facebook’s friend list of anyone who doesn’t have an Instagram account, and since there are more than 2 billions Facebook accounts but just 800 millions Instagram accounts, it means that at least 1 billion and half accounts were vulnerable.

The method was simple: knowing the email address of the target (and an email address is all but secret), the attacker had just to sign up to Instagram with that email, and then go to the suggestions of people to follow to see victim’s friends.

Conclusion

The combination of two useful features (suggestion of people to follow based on a linked Facebook account, being able to use the new Instagram account immediately) made this data leak possible.

It wasn’t important if the attacker was a Facebook’s friend with the victim, or the privacy settings of the victim’s account on Facebook. Heck, the attacker didn’t need a Facebook account at all!

Timeline

20 August 2018: first disclosure to Facebook
20 August 2018: request of other information from Facebook
20 August 2018: more information provided to Facebook
21 August 2018: Facebook closed the issue saying wasn’t a security issue
21 August 2018: I submitted a new demo with more information
23 August 2018: Facebook confirmed the issue
30 August 2018: Facebook deployed a fix and asked for a test
12 September 2018: Facebook awarded me a bounty

Bounty

Facebook awarded me a $3000 bounty award for the disclosure. This was the first time I was awarded for a security disclosure for Facebook, I am quite happy with the result and I applaude Facebook for making all the process really straightforward.

For any comment, feedback, critic, leave me a comment below or drop an email at [email protected].

Regards, R.