Table Bloat

This week I’ve been dealing with PostgreSQL table bloat issues with some production data. When tables accumulate dead tuples from frequent updates and deletes, they can become severely bloated, impacting query performance.

I explored pg_repack, a powerful extension that reorganizes tables online without holding exclusive locks.

Why pg_repack?

Unlike VACUUM FULL, pg_repack allows concurrent reads and writes during the reorganization process. This makes it perfect for production environments where downtime isn’t an option.

The tool works by creating a new table copy, applying all changes via triggers, and then swapping the tables atomically.

pg_repack removes bloat from tables and indexes, and optionally restores the physical order of clustered indexes.

How to run it?

Note: This was tested on macOS with a local PostgreSQL instance.

First we installed pg_repack using the PostgreSQL Extension Network client:

$ brew install pgxnclient
$ cd /path/to/development-kit
$ pgxn install pg_repack

Then we enabled the extension in our target database:

CREATE EXTENSION IF NOT EXISTS pg_repack;

Finally we ran pg_repack on a specific partitioned table that was experiencing heavy bloat:

$ pg_repack \
  --dbname="postgresql://user:[email protected]:5432/development_db" \
  --table='gitlab_partitions_dynamic.ci_builds' \
  --jobs=4 \
  --no-order \
  --wait-timeout=36000 \
  -w \
  --echo \
  2>&1 | ts '[%Y-%m-%d %H:%M:%S]' | tee -a "pg_repack_ci_builds_$(date +%Y%m%d_%H%M%S).log"

The --echo flag shows all SQL commands being executed, which is helpful for understanding what pg_repack is doing behind the scenes.

The entire repack of our 500GB partition completed in 3 hours, achieving a 30% reduction in table size.

Links

• pg_repack documentation
• PostgreSQL bloat explained
• PGXN - PostgreSQL Extension Network
• Understanding MVCC and bloat

🚙 Transport

• Avion: 2 841€

💰 Total: €

Manger

🏠 Dormir

💰 Total: 1 225€

A voir

• Tegallalang Rice Terrace
• Pura Ulun Danu Bratan
• Pura Agung Besakih

Sortir

Planning

💰 Total: €

💰 Budget

Liens

• Blog

🚙 Transport

• Parking: 148€
• Avion: 485€
• Location voiture: 357€

💰 Total: 990€

Manger

🏠 Dormir

💰 Total: 735€

A voir

Sortir

Planning

💰 Total: €

💰 Budget

Liens

• Blog

Hackathon

This week I have been playing around with AI models from Anthropic and Vertex AI at GitLab to familiarize myself with this new technology.

I have learned a lot of new skills and discovered how powerful those recents models are.

Detecting duplicate content

We have been working on a small project in order to identify when an issue contains the same information.

The idea was to convert the content of an issue an generating an embedding for it.

This technique is being called Semantic Similarity.

An embedding is a compressed numerical representation of a piece of text that captures its meaning and is generated by an ML model.

How we did it?

Note: The code is available here.

First we installed pgvector, a new postgres extension in order to store those records in our database:

$ brew install pgvector

CREATE EXTENSION vector;

Then we added a new column to persist those records in the issue table:

ALTER TABLE issues ADD COLUMN embedding vector(768);

And a new index to speed up the queries:

CREATE INDEX ON issues USING hnsw (embedding vector_l2_ops);

Finally we ran a script to import all existing issues for a given team and generating embedding for it.

To determine how similar an issue is with another one we leveraged the neighboor gem to measure the euclidean distance between those vectors:

nearest_issue = issue.nearest_neighbors(:embedding, distance: "euclidean").first
nearest_issue.neighbor_distance
=> 0.6025755637811935

This gives us a great approximation of how similar an issue with the rest of them and we need to adjust the distance to determine good threshold with our entire dataset.

For our use case, a distance of 0.3 gave us a good approximation of similar issues.

Links

• Semantic Similarity
• pgvector
• Detect duplicate issues
• upgrade gdk
• neighboor

Padi

• 18m avec un instructeur
• Certifat medical
• A 20m la pression est de 3bar
• Palier de securité à 5m
• Equilibré par palier
• Gérer sa flottabilité
• Connaître les signes
• Avoir au minimum 50 bars/500 psi restants
• Sortir de l’eau: flotabilite positive -> enlever palme -> garder masque et respirer sur tuba
• Toujours coucher la bouteille
• L’eau est environ 800 fois plus dense que l’air
• Le son se déplace environ quatre fois plus vite dans l’eau que dans l’air
• L’hydrodynamisme est simplement le processus consistant à porter son matériel près de soi
• L’eau absorbe davantage de chaleur que l’air à la même température et 20 fois plus vite.
• Contrôle de sécurité pré-plongée BALLO = Bouée; Air; Lestage; Largage; OK.
• Gérer sa réserve d’air
• Descente en 5 etapes
• Un palier de sécurité est un arrêt de trois à cinq minutes fait entre 6 mètres/20 pieds et 6 mètres/20 pieds, 3 mètres/10 pieds.
• Toujours porter un object tranchant sur soi pour des raisons de sécurité
• Le ressac ou la zone de ressac est l’endroit où les vagues se brisent.
• Les courants d’arrachement se produisent quand les vagues font passer l’eau au-dessus d’une longue obstruction (comme un banc de sable ou un récif).
• Planification préalable > Planification préparatoire > Planification de pré-plongée > Préparatifs de dernière minute
• Enregistrer ses plongées (elogs)
• Air = 21% Oxygène + 79% Azote

Deroulement

Log Book

Liens

• Frejus
• Flashcard

Lausanne

Manger

Resto

• Leonardo, Pizza
• Man’ouchy, Libanais
• Amici, Italien
• BeefOr, Viande
• Assador, Brésilien
• La Bavaria, Brasserie
• Un po di piu, Italien
• Petit boeuf, Viande
• Inglewood, Burger
• Aux Trois Tonneaux, Suisse

Café/Snack

• Snack du marché, Kebab
• Ajo, Italien
• Saveurs do Brazil, Brésilien

Vevey

• Les Bains Payes, Brasserie

Bern

• Pier 17

Valais

• Accademia della Pizza - Saillon

Rando

• Bisse du Ro
• Glacier Aletsch

🚙 Transport

💰 Total: €

Manger

🏠 Dormir

💰 Total: €

A voir / A faire

Planning

💰 Total: €

💰 Budget

Liens

• Choses a faire

✈️ Transport

💰 Total: $2769

• Max: $0
• Mil: $1050

• Marco: $119

• Car (16 days): $1764

🏠 Dormir

💰 Total: $2541

A voir / A faire

• Alcatraz

Planning

💰 Total: $

💰 Budget

Liens

• Choses a faire
• Spot a faire
• Louer RV
• Itinéraire
• Blog
• America the beautiful - Pass

🚙 Transport

• Avion: 275€
• Voiture: 190€

💰 Total: 370€

Manger

• Le 20123
• Boulangerie - Croissant
• Restaurant Chez Alain
• L’Atelier Pizza
• Café De La Poste
• Pozzo di Mastri - Vendredi soir cochon grillé

🏠 Dormir

💰 Total: 220€

A voir / A faire

• Plage de Santa Giulia
• Piscines Naturelles de Cavu

Planning

💰 Total: 600€

💰 Budget

Liens

• Choses a faire Corse

🚙 Transport

💰 Total: 100€

Manger

• El Brellin
• Osteria del Gnocco Fritto
• Panzerotti Luini
• La Prosciutteria Milano Brera
• Gino Sorbillo
• Antica Pizza Fritta da Zia Esterina
• Osteria Conchetta

🏠 Dormir

• Airbnb

💰 Total: 110€/nuit

A voir / A faire

• Duomo
• Galleria Vittorio Emanuele II
• Quadrilatero d’Oro
• Via Monte Napoleone
• Via della Spiga
• Santa Maria delle Grazie
• Castello Sforzesco
• Parco Sempione
• Musée des sciences et des techniques Léonard de Vinci
• Brocante le dimanche à Navigli

Planning

💰 Total: 600€

💰 Budget

Liens

• Choses a faire Milan