Jonathas Ribeiro

How I replaced Dropbox Camera Uploads with an automated photo sync pipeline

2026-03-08T16:39:19+00:00

I run a weekly photo cleanup routine because I want two things at the same time:

Only curated photos in iCloud
Real files on disk for archive and backup

I like having my photos as normal files. I can inspect them, back them up to external drives, and know exactly where they live instead of relying entirely on a cloud service.

For years, Dropbox Camera Uploads handled an important part of this workflow: photos arriving on my MacBook as normal files.

I use Apple Photos for curation, not as my archive, so that behavior mattered a lot.

When I started considering alternatives to Dropbox (for example Proton Drive), the gap became obvious. Sync exists, but getting camera photos automatically as files on disk is not guaranteed.

Once Dropbox Camera Uploads stopped being the center of my workflow, I had to rebuild that behavior myself.

Before

My weekly flow looked like this:

Let Dropbox Camera Uploads sync iPhone photos and videos to my MacBook.
Check duplicates and curate what I wanted to keep in iCloud.
Run the pbc-organizer script I implemented to rename and move files by EXIF date.
Move files into backup folders.
Choose photos for my digital frame.
Resize and convert them.
Send them to the digital frame/tablet using LocalSend.
Run backup scripts to external drives.

This workflow worked, but it was repetitive and fragile. The curation decisions were valuable. The glue work was not.

After I replaced my old Frameo digital frame with an Android tablet running Fotoo, the display and system improved, but the workflow was still mostly manual.

What I wanted instead

The idea was simple:

Apple Photos can handle syncing and curation, while automation handles exporting and filesystem organization.

So I built a small pipeline to restore the capability I cared about from Dropbox Camera Uploads:

Photos arriving through sync
Photos available as normal files on disk
Deterministic backups

Core tools:

osxphotos for exporting from Apple Photos
sips for image conversion and resizing
ADB over Wi-Fi for tablet transfer
rsync for backups

The automated pipeline

You can find the scripts in the photo-sync-pipeline repository.

I still manually decide what to delete, keep, and send to the digital frame. After that, scripts handle the mechanical steps.

The pipeline now does the following:

Export photos and videos since a given date
Place them into predictable directories
Export the curated Digital frame album
Normalize file extensions
Convert HEIC to JPG
Resize and compress frame photos
Push them to Android via ADB
Trigger a media scan
Restart Fotoo
Clear the Digital frame album
Run backup sync jobs

Typical commands:

export-photos-since 2026-01-25
export-digital-frame

export-photos-since effectively replaces the “files on disk” behavior of Dropbox Camera Uploads.

Small pipeline diagram

This is the operational flow I usually follow each week:

flowchart TD A[Open Apple Photos] --> B[Delete duplicates] B --> C[Wait for iCloud sync] C --> D[Run export-photos-since] D --> E[Add selected photos to Digital frame album] E --> F[Run export-digital-frame] F --> G[Tablet updated] F --> H[Backups synced]

It is still a curated workflow, not a fully automatic one. That is intentional, since I want the decisions to stay manual and the repetitive steps to be scripted.

Example exported filesystem structure

One of the main goals was to keep the exported files easy to inspect and back up. A simplified structure looks like this:

2026/
├── 01 - January                           
│   ├── 2026-01-01                                                                          
│   │   ├── 2026-01-01 09.29.20.heic       
│   │   ├── 2026-01-01 09.29.23.heic                                                        
│   │   ├── 2026-01-01 09.29.24.heic       
│   │   ├── 2026-01-01 09.35.04.heic                                                        
│   │   ├── 2026-01-01 09.35.07.heic      
│   │   ├── 2026-01-01 09.35.38.heic      
│   │   └── 2026-01-01 18.32.27.heic      
│   ├── 2026-01-02                        
│   │   ├── 2026-01-02 18.26.47.mov 
│   │   └── 2026-01-02 18.26.51.jpg        
│   ├── 2026-01-03                  
│   │   ├── 2026-01-03 09.32.47.heic       
│   │   ├── 2026-01-03 09.32.48.heic       
│   │   ├── 2026-01-03 09.32.52.heic       
│   │   ├── 2026-01-03 09.32.54.heic       
│   │   ├── 2026-01-03 09.32.55.heic       
│   │   ├── 2026-01-03 09.32.59.heic       
│   │   ├── 2026-01-03 10.46.40.heic       
│   │   ├── 2026-01-03 10.46.42.heic
...

The exact layout can vary, but the important parts are:

Deterministic paths
Files visible in normal directories
A separate place for digital frame exports
Backup destinations that can be synced repeatedly

Why this is better

The main improvement is reduced mental overhead.

Before, I had to remember a long sequence of manual steps every week.

LocalSend was a good intermediate step for transferring photos to the tablet, but ADB made the process fully repeatable.

Now the Digital frame album acts as a signal, not storage:

Add photos to the album
Run the pipeline
The batch is exported, transferred, and cleared

Final setup

The system now looks like this:

Apple Photos + iCloud for sync and curation
Filesystem directories as the archive
ADB for transferring images to the digital frame
rsync for backups

No single vendor controls the entire workflow.

That is exactly what I wanted.

How I replaced Dropbox Camera Uploads with an automated photo sync pipeline was originally published by Jonathas Ribeiro at Jonathas Ribeiro on March 08, 2026.

Running two Zigbee networks side by side in Home Assistant

2026-02-28T09:30:00+00:00

For a long time, my Zigbee setup in Home Assistant was fully based on ZHA running on Home Assistant Yellow.

It was stable, reliable, and already managing more than 40 devices. So when I needed better support for a few specific devices (especially a Tuya button), I did not want to migrate everything and risk breaking a working network.

Instead, I built a hybrid architecture:

Keep ZHA exactly as it is for the existing mesh
Add a second coordinator for Zigbee2MQTT
Pair only selected devices to the new Zigbee2MQTT network

This turned out to be the safest way to extend compatibility without downtime.

Why I did not migrate everything

A full migration from ZHA to Zigbee2MQTT would require re-pairing every device.

In a production home setup, that means unnecessary risk:

Temporary automation outages
Broken device references
Time-consuming reconfiguration

My ZHA network was already healthy, so replacing it would have solved no real problem.

What I actually needed was targeted compatibility for some devices where Zigbee2MQTT has better converters and cleaner action modeling.

Final architecture

I now run two independent Zigbee networks in parallel:

Network 1: ZHA

Coordinator: Home Assistant Yellow internal Zigbee radio
Zigbee channel: 25
Purpose: Existing core mesh and legacy devices

Network 2: Zigbee2MQTT

Coordinator: SONOFF Zigbee 3.0 USB Dongle Plus V2 (EFR32MG21)
Zigbee channel: 15
Adapter type: ember
Purpose: Devices with better support in Zigbee2MQTT

Important: these networks are independent. A device belongs to one network only.

Hybrid architecture diagram

This is the mental model I use. ZHA integrates directly with Home Assistant, while Zigbee2MQTT goes through MQTT (Mosquitto) and then shows up in Home Assistant via MQTT discovery.

flowchart LR subgraph Zigbee_Network_1[Zigbee network 1: ZHA] ZHA_COORD[Coordinator: HA Yellow internal radio] ZHA_DEV[Zigbee devices paired to ZHA] ZHA_DEV --> ZHA_COORD end subgraph Zigbee_Network_2[Zigbee network 2: Zigbee2MQTT] Z2M_COORD[Coordinator: Sonoff USB Dongle Plus E] Z2M_DEV[Zigbee devices paired to Zigbee2MQTT] Z2M_DEV --> Z2M_COORD end ZHA_COORD --> HA[Home Assistant] Z2M_COORD --> Z2M[Zigbee2MQTT] Z2M --> MQTT[Mosquitto MQTT broker] MQTT --> HA

Prerequisites

Before pairing anything in Zigbee2MQTT, I installed:

Mosquitto Broker add-on
Zigbee2MQTT add-on
MQTT integration in Home Assistant

This gives Zigbee2MQTT a stable transport layer and allows Home Assistant to auto-discover exposed entities and triggers.

How Zigbee2MQTT integrates with Home Assistant via MQTT discovery

Unlike ZHA, which integrates directly with Home Assistant’s device registry, Zigbee2MQTT communicates entirely through MQTT.

The flow looks like this:

A Zigbee device talks to the Zigbee2MQTT coordinator (the USB dongle)
Zigbee2MQTT publishes messages to the MQTT broker (Mosquitto)
Home Assistant’s MQTT integration subscribes to discovery topics and creates devices/entities/triggers automatically

In practice, that means Zigbee2MQTT devices only show up inside Home Assistant if:

Mosquitto is running
The MQTT integration is enabled
Zigbee2MQTT can reach the broker

Without that last step (MQTT discovery), Zigbee2MQTT can still pair devices and show them in its own UI, but Home Assistant will not see them.

Zigbee2MQTT baseline config

This is the core configuration I used:

mqtt:
  server: mqtt://core-mosquitto:1883

serial:
  port: /dev/ttyUSB0
  adapter: ember

advanced:
  channel: 15
  pan_id: GENERATE
  network_key: GENERATE

The key point is coordinator isolation:

Internal radio stays dedicated to ZHA
USB dongle stays dedicated to Zigbee2MQTT

Do not try to share the same coordinator between both integrations.

Adapter type and firmware: why `ember` matters

My Sonoff Zigbee dongle uses a Silicon Labs EFR32MG21 chip, so Zigbee2MQTT must use the ember adapter type.

Zigbee2MQTT supports multiple adapters depending on the coordinator chipset:

ember: Silicon Labs EFR32 (my setup)
zstack: Texas Instruments CC2652 based coordinators
deconz: ConBee coordinators

If you pick the wrong adapter type, Zigbee2MQTT typically fails to start, cannot open the serial connection correctly, or behaves unreliably.

Channel planning and RF stability

To reduce interference, I separated channels deliberately:

ZHA on channel 25
Zigbee2MQTT on channel 15
Wi-Fi on channel 1 or 6 where possible

I also used a USB extension cable for the Zigbee2MQTT dongle and kept it away from metal surfaces and other noisy electronics.

That small physical change improves link quality more than many software tweaks.

Pairing workflow for the second network

The pairing flow is simple:

Enable Permit Join in Zigbee2MQTT.
Put the target device in pairing mode.
Wait for Zigbee2MQTT interview completion.
Confirm the device appears in Home Assistant via MQTT discovery.

I only pair new or problematic devices to Zigbee2MQTT.

Everything already stable in ZHA stays untouched.

Real-world example: Tuya TS0041 button

This button was the main reason I introduced Zigbee2MQTT.

In my setup:

ZHA exposed it as a switch instead of a button, with an On-Off toggle instead of action events.
Zigbee2MQTT exposed clear action events like single, double, and hold

That made automations significantly cleaner and easier to maintain.

Instead of parsing raw event payloads, I can build automations directly with device triggers in the Home Assistant UI.

What this hybrid model gives me

Running both integrations side by side delivered exactly what I wanted:

No migration downtime
No mass re-pairing effort
Better support for selected edge-case devices
Incremental rollout with low risk

In practice, ZHA remains the stable baseline while Zigbee2MQTT acts as a compatibility layer for specific devices.

Lessons learned

If you already have a stable ZHA mesh, a full migration is often unnecessary.

A second Zigbee network with Zigbee2MQTT can be a cleaner strategy when you need better support for specific devices.

The two rules that mattered most in my setup were:

Keep coordinators strictly isolated.
Plan channels and physical dongle placement carefully.

With those in place, both networks can coexist reliably in Home Assistant.

Running two Zigbee networks side by side in Home Assistant was originally published by Jonathas Ribeiro at Jonathas Ribeiro on February 28, 2026.

Automating winter humidity control with Home Assistant and dehumidifiers

2026-02-21T14:30:00+00:00

Every winter, the same problem returned in my flat:

Wet windows in the morning
Condensation around glass doors
Bedrooms sitting at 60-70% relative humidity overnight

One morning I noticed water pooling on the floor near the glass door. That was the moment I stopped treating this as a minor annoyance.

Outside humidity here during winter is often 70–90%, while temperatures frequently sit around 0 °C and regularly drop below freezing at night. Even with heating and regular ventilation, indoor humidity stayed too high for comfort.

The result was obvious: moisture everywhere.

I wanted a fully automated and resilient setup in Home Assistant that could keep humidity under control without constantly toggling devices manually.

Why humidity control gets tricky in winter

A common mistake is aiming for one perfect number (for example 45% RH) and forcing the system to stay there all the time.

In practice, that can overwork dehumidifiers, increase noise, and still produce unstable cycling.

What worked better for me was:

Room-level control instead of one global target
Hysteresis (different ON and OFF thresholds)
Stability timers to avoid oscillation
Independent automations per room

This made the system quieter, more efficient, and much more reliable.

Final hardware setup

I ended up with distributed moisture control:

1x Comfee dehumidifier in the bathroom
2x Xiaomi Smart Dehumidifier Lite (main bedroom + children’s bedroom)
Wall-mounted Zigbee humidity sensors in each room
Home Assistant Yellow with stable Zigbee routing

To connect everything cleanly in Home Assistant, I had to install two integrations:

Xiaomi Miot from HACS (for Xiaomi dehumidifiers)

Midea Air Appliances (for the Comfee dehumidifier)

The key design decision:

Wall sensors are the source of truth, not dehumidifier internal sensors.

Internal sensors measure intake air and are heavily affected by device placement. Wall sensors reflect what people actually feel in the room.

Main bedroom automation logic

The main bedroom follows a conservative hysteresis model:

Turn ON above 50% RH (after 2 minutes of stability)
Turn OFF below 48% RH (after 10 minutes of stability)

I also use periodic checks and Home Assistant startup triggers so the automation self-heals after restarts.

ON automation

alias: Main Bedroom - Turn dehumidifier ON
trigger:
  - platform: numeric_state
    entity_id: sensor.main_bedroom_temperature_and_humidity_humidity
    above: 50
    for: "00:02:00"
  - platform: time_pattern
    minutes: "/5"
  - platform: homeassistant
    event: start
condition:
  - condition: state
    entity_id: humidifier.xiaomi_lite_814e_dehumidifier
    state: "off"
  - condition: numeric_state
    entity_id: sensor.main_bedroom_temperature_and_humidity_humidity
    above: 50
action:
  - service: humidifier.set_humidity
    target:
      entity_id: humidifier.xiaomi_lite_814e_dehumidifier
    data:
      humidity: 45
  - service: humidifier.turn_on
    target:
      entity_id: humidifier.xiaomi_lite_814e_dehumidifier
mode: single

OFF automation

alias: Main Bedroom - Turn dehumidifier OFF
trigger:
  - platform: numeric_state
    entity_id: sensor.main_bedroom_temperature_and_humidity_humidity
    below: 48
    for: "00:10:00"
  - platform: time_pattern
    minutes: "/5"
  - platform: homeassistant
    event: start
condition:
  - condition: state
    entity_id: humidifier.xiaomi_lite_814e_dehumidifier
    state: "on"
  - condition: numeric_state
    entity_id: sensor.main_bedroom_temperature_and_humidity_humidity
    below: 48
action:
  - service: humidifier.turn_off
    target:
      entity_id: humidifier.xiaomi_lite_814e_dehumidifier
mode: single

That 2% gap plus different hold times was enough to eliminate constant ON/OFF flapping.

Tank-full / fault detection for Xiaomi units

The Xiaomi integration exposes fault codes through:

sensor.xiaomi_lite_814e_device_fault
Attribute: dehumidifier.fault

0 means normal state, and values above 0 usually indicate a fault (most commonly a full tank).

Instead of silently failing, I trigger push notifications when faults persist for a few seconds.

alias: Main Bedroom Dehumidifier Tank Full
trigger:
  - platform: numeric_state
    entity_id: sensor.xiaomi_lite_814e_device_fault
    attribute: dehumidifier.fault
    above: 0
    for:
      seconds: 10
action:
  - parallel:
      - action: notify.mobile_app_jon_iphone
        data:
          title: Main Bedroom Dehumidifier Tank Full
          message: >-
            The main bedroom dehumidifier reported a fault (code: {{ state_attr('sensor.xiaomi_lite_814e_device_fault', 'dehumidifier.fault') }}).
            This is usually a full tank. Please check and empty it.
mode: single

This simple alert removed one of the most annoying failure modes: the dehumidifier is technically “on”, but nothing is being dehumidified.

Children’s bedroom and bathroom strategy

I do not use exactly the same thresholds everywhere.

Children’s bedroom

I keep slightly softer thresholds for comfort:

ON around 52-55%
OFF around 48-50%

Bathroom (Comfee)

The bathroom uses more aggressive source control:

ON around 58-60%
OFF around 52%

This room creates short humidity spikes (showers), so local response works much better than trying to dry the entire flat globally.

The 45% experiment that failed

I initially pushed all rooms toward 45% RH because it sounded like the “ideal” number.

In my real setup, it was not ideal:

Devices ran too often
Noise increased
Comfort did not improve proportionally
Practical maintenance (tank emptying) got worse

After tuning, the 48-52% band became the sweet spot:

Condensation mostly gone
Better cycling behavior
Lower operational stress on devices

This was the biggest lesson: optimize for a stable system, not a theoretically perfect number.

Reliability patterns that mattered most

A few implementation details made a huge difference:

Multi-trigger automations (numeric_state + periodic checks + HA startup)
Explicit hysteresis thresholds per room
Longer OFF stability windows than ON windows
Fault-code-based alerts instead of brittle string checks

These patterns made the setup resilient to restarts, temporary sensor noise, and common real-world device issues.

Final result

By the end of winter, humidity control became mostly invisible:

Windows stayed dry most mornings
Bedrooms remained in a controlled range overnight
Manual intervention dropped to occasional tank emptying

And because everything is room-level and event-driven, I can keep iterating each zone independently without breaking the whole system.

If you are battling winter condensation, start with three things:

Use external room sensors as your source of truth.
Implement hysteresis with stability timers.
Tune each room based on real behavior, not one universal target.

That combination made all the difference in my flat.

Automating winter humidity control with Home Assistant and dehumidifiers was originally published by Jonathas Ribeiro at Jonathas Ribeiro on February 21, 2026.

How I made my physical mailbox send me real-time push notifications

2026-02-15T12:06:30+00:00

I live in Prague, and like many apartment buildings here, the mailboxes are grouped in the building entrance instead of inside the flat. Letters arrive through a slot and fall directly inside the box.

There is something deeply satisfying about turning physical events into software events. Not just monitoring systems, not dashboards, not metrics. Real-world events, like receiving a letter.

This project started with a simple goal:

Receive a push notification the exact moment a letter arrives in my mailbox.

Not when I open it, not when I check it, but when it physically arrives.

This required bridging the physical and digital worlds using Zigbee, MQTT, and Home Assistant.

What sounded like a tiny automation turned into a deeper project involving Zigbee range problems, metal interference, firmware flashing, and creative power solutions.

The physical constraints

My mailbox is located outside my apartment, in the building entrance hallway.

This introduces multiple problems:

It is physically separated from my apartment
There are several walls between the mailbox and my Zigbee coordinator
There are no power outlets near the mailbox

This immediately ruled out:

Wi‑Fi devices
Mains‑powered Zigbee routers
Any solution requiring permanent power

This forced a battery‑powered Zigbee routing architecture.

High‑level architecture

To understand how everything connects, here is the full architecture:

flowchart TB Mailbox["📬 Mailbox"] Sensor["👁️ Aqara P1 Motion Sensor"] Router["🔋 Sonoff Zigbee Router Dongle
+ Powerbank"] subgraph HAHost["🖥️ Home Assistant Host"] Z2M["Zigbee2MQTT"] HA["Home Assistant"] Z2M --> HA end Phone["📱 iPhone Notification"] Mailbox --> Sensor Sensor --> Router Router --> Z2M HA --> Phone

This converts a physical event into a software event:

Physical motion → Zigbee message → MQTT event → Automation → Push notification

What I tried first

Direct pairing

My first attempt was simply pairing the motion sensor and placing it inside the mailbox. It worked occasionally, but the signal was unreliable. Metal enclosures and distance are brutal for 2.4 GHz signals.

Normal Zigbee repeater

The standard fix would be placing a mains-powered Zigbee repeater near the entrance. That would likely solve coverage, but there are no power sockets near the door.

Alternative technologies

I also considered 433 MHz sensors and a bridge. Technically viable, but it would add a second wireless stack only for this use case. I wanted to keep everything in the existing Zigbee ecosystem.

The key idea

A Zigbee USB dongle can run in router mode and be powered entirely from a battery. That unlocked the final design.

Hardware components

Motion sensor

Aqara Motion Sensor P1 was selected because of:

Extremely low power consumption
Reliable Zigbee performance
Long battery life
Excellent Zigbee2MQTT compatibility

It detects motion when letters fall inside the mailbox.

Zigbee router

I used a Sonoff Zigbee Dongle‑E flashed with router firmware.

Normally, this device acts as a coordinator, but when flashed with router firmware, it becomes a dedicated Zigbee router.

This extends the mesh network range. I powered the router with a power bank because there are no power outlets near my apartment’s entrance.

Full hardware list

Aqara Motion Sensor P1
Sonoff Zigbee 3.0 USB Dongle Plus (ZBDongle-E)
Xiaomi 20,000 mAh power bank
USB keep-alive module (adjustable dummy load)
Existing Zigbee2MQTT setup

Critical problem: Power bank shutdown

Initially, everything worked perfectly, then it stopped working and the router disappeared randomly.

Cause: power bank auto shutdown.

Power banks automatically shut down when the current draw is too low. Because the Zigbee router consumes very little power, the power bank assumed nothing was connected and turned itself off.

This silently killed the Zigbee network extension and broke mailbox notifications.

Final fix: USB keep‑alive module

The solution was adding a configurable USB dummy load. This device draws a small constant current, tricking the power bank into believing an active device is connected, preventing automatic shutdown.

You can find the exact model I bought on Amazon

Architecture update

flowchart LR PowerBank --> Router PowerBank --> DummyLoad["USB Keep‑Alive Module"] DummyLoad --> Load["Constant current draw"] Router --> Zigbee["Zigbee Mesh Network"]

This keeps the router powered continuously and stabilizes the Zigbee mesh.

Step-by-step implementation

Flash the Sonoff dongle with router firmware. I’ve done it via the Sonoff website
Pair it into Zigbee2MQTT and name it zigbee_router_hallway.
Power it from the power bank and place it high in the hallway.
Pair the Aqara motion sensor and mount it inside the mailbox.
Verify routing in the Zigbee2MQTT network map.
Configure occupancy timeout to 90 seconds.

Software architecture

The software flow:

sequenceDiagram participant Sensor participant Router participant Coordinator participant Zigbee2MQTT participant HomeAssistant participant Phone Sensor->>Router: Motion detected Router->>Coordinator: Forward Zigbee message Coordinator->>Zigbee2MQTT: Publish MQTT message Zigbee2MQTT->>HomeAssistant: Forward event HomeAssistant->>Phone: Push notification

Motion automation

Home Assistant automation:

alias: Mailbox - Notify on motion
trigger:
  - platform: state
    entity_id: binary_sensor.aqara_p1_mailbox_occupancy
    from: "off"
    to: "on"
action:
  - service: notify.mobile_app_jon_iphone
    data:
      title: "📬 Mailbox"
      message: "You've got a new letter!"

Router reliability monitoring

Because the hallway router runs on a power bank, monitoring is as important as the mailbox notification. If the battery dies, the automation silently stops working.

Why availability sensors were not enough

Zigbee2MQTT availability works for many devices, but a battery-powered router can stop sending updates without reliably publishing an offline transition. For this case, last_seen is more reliable than plain availability state.

Stale sensor in `configuration.yaml`

I used a stale detector based on last_seen in configuration.yaml:

template:
  - binary_sensor:
      - name: "Zigbee Router Hallway Stale"
        unique_id: zigbee_router_hallway_stale
        device_class: connectivity
        state: >
          
          {% set s = states('sensor.zigbee_router_hallway_last_seen') %}
          {% if s in ['unknown', 'unavailable', 'none', ''] %}
            true
          {% else %}
            {{ (now() - states.sensor.zigbee_router_hallway_last_seen.last_updated).total_seconds() > 900 }}
          {% endif %}
          

This sensor turns on when the router has not reported anything for more than 15 minutes.

Offline notification automation

alias: Notify Zigbee Router Hallway Offline (Last seen watchdog)
trigger:
  - platform: state
    entity_id: binary_sensor.zigbee_router_hallway_stale
    to: "on"
    for: "00:02:00"
action:
  - service: notify.persistent_notification
    data:
      title: "⚠ Zigbee Router Offline"
      message: "No Zigbee router heartbeat for ~10+ minutes. Power bank likely empty."
  - service: notify.mobile_app_jon_iphone
    data:
      title: "⚠ Zigbee Router Offline"
      message: "No Zigbee router heartbeat for ~10+ minutes. Power bank likely empty."
  - service: input_boolean.turn_on
    target:
      entity_id: input_boolean.zigbee_router_hallway_was_offline
mode: single

Back online automation

alias: Notify Zigbee Router Hallway Back Online (Last seen watchdog)
trigger:
  - platform: state
    entity_id: binary_sensor.zigbee_router_hallway_stale
    to: "off"
action:
  - service: notify.persistent_notification
    data:
      title: "✅ Zigbee Router Online"
      message: "Hallway Zigbee router is reporting again."
  - service: notify.mobile_app_jon_iphone
    data:
      title: "✅ Zigbee Router Online"
      message: "Hallway Zigbee router is reporting again."
  - service: input_boolean.turn_off
    target:
      entity_id: input_boolean.zigbee_router_hallway_was_offline
mode: single

This approach proved more reliable than relying on MQTT availability alone.

Power consumption considerations

Router consumption: ~0.5W
Dummy load consumption: configurable

Power bank runtime depends on configured load, which creates a trade‑off:

Lower load → longer runtime
Higher load → more reliability

Final result

Now the system works reliably.

When a letter arrives:

Phone receives notification instantly.
No manual checking required.

This small project turned an ordinary mailbox into an event-driven system, proving that software architecture does not have to stop at the boundaries of a server.

Why this matters

Software architecture is fundamentally about converting events into actions.

Most engineers apply this thinking inside software systems:

HTTP requests
Database changes
Queue messages

But the same principles apply to the physical world.

A mailbox opening becomes an event. A motion sensor becomes an event producer. Zigbee becomes a transport layer. MQTT becomes an event bus. Home Assistant becomes an event processor.

This is event-driven architecture applied to reality itself.

Once you start thinking this way, the boundary between software and physical systems begins to disappear.

Everything becomes part of your architecture.

Lessons learned

Zigbee routers are flexible when used intentionally.
Power banks can replace mains power in constrained locations.
last_seen-based monitoring is more reliable than availability flags in this case.
Conservative timeouts reduce false alerts.

Future improvements

I’m planning to make Home Assistant play an audio announcement on Siri/HomePod when new mail arrives. It would be awesome to have that in Matt Berry’s voice!

How I made my physical mailbox send me real-time push notifications was originally published by Jonathas Ribeiro at Jonathas Ribeiro on February 15, 2026.

Optimizing File Upload Architecture with AWS S3 Presigned POST

2025-07-12T14:00:10+00:00

In many systems, file uploads follow a familiar pattern:

A client sends the file to your application server.
The backend processes it (e.g., for validation or virus scanning).
Then it forwards the file to a cloud storage service like Amazon S3.

While this works, it’s highly inefficient. Your app becomes a middleman consuming bandwidth, CPU, and memory, just to pass files along.

Why This Is a Problem

This upload pattern:

Consumes resources on your app server.
Increases latency for users.
Stresses your infrastructure (especially under load).
Breaks down for large files, especially when you hit limits in intermediaries like API Gateway.

Let’s look at two common architectures where these problems show up.

Two Upload Architectures Compared

🔸 Architecture 1: Upload with Nginx

This is typical in smaller-scale deployments with a single server behind a reverse proxy.

flowchart LR Client["Client (Browser/App)"] Nginx["Nginx (Reverse Proxy)"] App["App Server"] S3["S3 Bucket"] Client --> Nginx Nginx --> App App --> S3

Problems:

Nginx must buffer the file in memory or disk (an exception can be thrown here).
App still receives and processes the full payload.
Memory and CPU are wasted just to forward the upload.

🔸 Architecture 2: Upload via API Gateway + Load Balancer

This setup is common in cloud-native apps using AWS infrastructure.

flowchart LR Client["Client (Browser/App)"] APIGW["AWS API Gateway"] ALB["Application Load Balancer"] App["App Server"] S3["S3 Bucket"] Client --> APIGW APIGW --> ALB ALB --> App App --> S3

Problems:

API Gateway limits uploads to 10 MB max per request.
Both ALB and your app must handle the file payload.
This creates bottlenecks and risks timeouts.

✅ The Solution: Presigned POST to S3

Instead of routing uploads through your backend, you can let clients upload directly to S3, securely, using a presigned POST policy.

How It Works

The client asks your backend for a signed upload policy.
Your backend returns a temporary URL and form fields.
The client uploads the file directly to S3 using multipart/form-data.

sequenceDiagram autonumber participant Client as Client (Browser/App) participant API as App Server participant S3 as S3 Bucket Client->>API: GET /attachments/presigned-post-policy API-->>Client: 200 OK (URL + fields) Client->>S3: POST file upload (multipart/form-data) S3-->>Client: 204 No Content

Benefits of Presigned Uploads

🚀 Bypass Infrastructure: No more file handling by Nginx, ALB, or API Gateway.
📉 Reduce Server Load: No memory or CPU usage from streaming file data.
⏱️ Improve Latency: Fewer hops and no buffering.
💡 Avoid Limits: Works around API Gateway’s 10MB payload cap.
🔐 Secure: You control what, where, and for how long uploads are allowed.

Example Server-Side implementation

import { Injectable } from '@nestjs/common';
import { S3Client } from '@aws-sdk/client-s3';
import { createPresignedPost } from '@aws-sdk/s3-presigned-post';

@Injectable()
export class S3Service {

  public getPresignedPostPolicy(
    filename: string,
    bucketConfig: S3BucketResourceConfig,
    contentType?: string
  ): Promise<{ url: string; fields: Record<string, string> }> {
    const expiresInSeconds = 600; // Default 10 minutes
    const maxSizeBytes = 150 * 1024 * 1024; // Default 150 MB

    const s3ClientConfig: S3ClientConfig = {
      region: bucketConfig.region
    };

    return createPresignedPost(new S3Client(s3ClientConfig), {
      Bucket: bucketConfig.name,
      Key: filename,
      Conditions: [
        ['content-length-range', 0, maxSizeBytes],
        ['starts-with', '$Content-Type', contentType || '']
      ],
      Fields: {
        ...(contentType && { 'Content-Type': contentType })
      },
      Expires: expiresInSeconds
    });
  }

}

Example Client Implementation

Here’s how a client might perform the upload:

import axios from 'axios';

async function uploadFile(file: File) {
  // Step 1: Get the presigned policy
  const res = await axios.get('/api/get-presigned-policy', {
    params: {
      filename: file.name,
      contentType: file.type,
    },
  });

  const { url, fields } = res.data;

  // Step 2: Prepare the upload form
  const formData = new FormData();
  Object.entries(fields).forEach(([key, val]) => formData.append(key, val as string));
  formData.append('file', file); // must be last

  // Step 3: Upload to S3
  const uploadRes = await axios.post(url, formData, {
    headers: { 'Content-Type': 'multipart/form-data' },
  });

  if (uploadRes.status === 204) {
    console.log('Upload successful');
  } else {
    console.error('Upload failed');
  }
}

When Should You Use Presigned POST?

When handling large file uploads.
When deploying behind API Gateway.
When you want to scale uploads without scaling your backend.
When you want better performance and lower infrastructure costs.

Summary

If you’re still uploading files through your backend, you’re likely wasting resources, risking timeouts, and introducing unnecessary complexity.

Presigned POST policies in AWS S3 offer a better approach:

Simple to implement.
Secure and time-limited.
Scalable and resilient.

Offload uploads to your storage layer, which is where they belong, and keep your app fast, simple, and maintainable.

More info about the AWS feature

Optimizing File Upload Architecture with AWS S3 Presigned POST was originally published by Jonathas Ribeiro at Jonathas Ribeiro on July 12, 2025.

Creating a CV in LaTeX: Clean, Structured, and Versioned

2025-07-05T10:02:00+00:00

For years, I maintained my CV using various online builders. They got the job done, but there was always a subscription fee. This last time I tried to update my CV through one of them, their payment process got stuck. They withdrew the subscription fee from my card but didn’t enable any premium features. That was the last nail in the coffin for me.

This year, I decided to rebuild my CV from scratch using LaTeX. The result is a clean, professional-looking CV that’s easy to maintain and version-controlled with Git.

Why LaTeX?

LaTeX is a typesetting system widely used in academia and technical fields. It lets you separate content from presentation, gives you full control over layout and styling, and produces high-quality PDFs.

For me, LaTeX offered a few major benefits:

Precise control over formatting
Clean separation of content and style
Easy versioning using Git
Great print and PDF output
No reliance on visual editors or WYSIWYG templates

I also liked the idea of making the whole project open source, so others could see how it was done and adapt it to their own needs.

Tools I Used

Editor: Overleaf + local editing in VS Code with LaTeX Workshop
Template: A custom minimalist layout using the article class, with optional support for the AltaCV class
Version control: GitHub repository with full LaTeX source and compiled PDF
Licensing: Reusable for personal use under a non-commercial license

Sample LaTeX Code

Here’s a basic example using the article class:

\documentclass[11pt,a4paper]{article}
\usepackage[utf8]{inputenc}
\usepackage{geometry}
\geometry{margin=1in}

\begin{document}

\section*{Profile}
Senior Software Engineer with 15+ years of experience across diverse technologies and teams.

\section*{Skills}
\begin{itemize}
  \item Backend: Node.js, TypeScript, Java, PHP
  \item DevOps: AWS, Terraform, Docker, GitHub Actions
\end{itemize}

\section*{Experience}
\textbf{Senior Software Engineer} \hfill Jan 2024 -- Present \\
Vacasa – Prague, Czech Republic (Hybrid) \\
Worked on cloud architecture and workflow automation projects supporting large-scale field operations.

\end{document}

If you want a more modern layout, I highly recommend AltaCV, a great open-source LaTeX CV class. You can get started with:

git clone https://github.com/liantze/AltaCV.git

Or download the class file directly and use it when building your CV.

Structure and Philosophy

My CV is structured to be clean, readable, and compliant with applicant tracking systems (ATS). It includes:

A brief summary
A key skills section
Chronological work experience
A compact education section
Language skills and legal residency info (since I live in the EU)

Everything fits into two pages, and updates are now fast and consistent. I no longer have to worry about formatting bugs or version mismatches.

Final Thoughts

LaTeX isn’t the quickest way to build a CV, but it’s one of the most reliable and rewarding. It’s a great fit for developers, researchers, or anyone who wants full control over how their work is presented.

If you’re curious, I encourage you to clone my repo or use it as inspiration for your own LaTeX CV.

Creating a CV in LaTeX: Clean, Structured, and Versioned was originally published by Jonathas Ribeiro at Jonathas Ribeiro on July 05, 2025.

How I Transformed My Flat into a Smart Home with Home Assistant

2025-06-29T14:30:00+00:00

When I first ventured into home automation, like many, I started with Alexa. Voice commands and plug-and-play devices felt futuristic—until I began hitting limitations. Many devices required separate hubs, cloud access, or their own apps just to function. Even something as simple as turning on a socket often involved sending a request to a remote server before the action happened locally. The delay, fragmentation, and privacy concerns all started piling up. That’s when I turned to Home Assistant.

Today, my entire flat runs on Home Assistant Yellow, which comes with a built-in Zigbee radio and is built specifically for local-first automation. It was a game-changer. I went from juggling multiple ecosystems to a cohesive, responsive, and private smart home that works exactly the way I want.

Why Home Assistant?

The biggest draw was flexibility and local control. Unlike Alexa or Google Home, Home Assistant doesn’t depend on the cloud to work. That means no delays, no vendor lock-in, and no worries if the internet goes down. Nearly all my devices now run on Zigbee, connecting directly to the Home Assistant Yellow hub. This drastically reduced latency and increased reliability.

My choice of Zigbee over Wi-Fi wasn’t just about responsiveness—it was also practical. In a modern flat with multiple smart devices, phones, laptops, TVs, and work equipment all sharing the same Wi-Fi network, congestion becomes inevitable. Zigbee devices operate on a separate mesh network that’s purpose-built for low-power, low-bandwidth communication. This means they don’t compete for bandwidth with streaming video, Zoom calls, or backups. It also makes the entire system more scalable and robust.

Take my early experience with TP-Link TAPO smart sockets. While they were affordable and easy to set up, they required Wi-Fi and a round trip to the cloud to toggle a switch—even when everything was physically on the same network. It felt absurd to send an internet request just to turn on some devices. I’ve since replaced them all with Zigbee switches, which respond instantly and don’t require an internet connection at all.

Built via the UI (No YAML Required)

Although Home Assistant gives you full access to YAML configuration, I opted for a UI-first approach. The modern Home Assistant dashboard and automation editor are incredibly powerful—I was able to build nearly everything through the web interface without manually editing YAML files. This made the process smoother, safer, and more approachable. You can still find my configuration and structure documented on GitHub, but most of the logic was built visually.

Everyday Automations

Here’s where the system really shines: automation. I’ve built dozens of small automations that make daily life more convenient, comfortable, and safe. Some of my favorites include:

You’ve Got Mail: When the mailbox sensor is triggered, Home Assistant sends a notification to my phone so I know mail has arrived—even before I open the door.
Smoke Alert: I have three Zigbee-powered smoke sensors—one in the hallway, one in the living room, and another in the office. If any of them detect smoke, Home Assistant instantly notifies me.

Low Battery Warnings: All sensors are monitored for battery status, and when one falls below a certain threshold, I get notified—no more dead devices without warning.
Shutter Control: My shutters automatically close after sunset and open at sunrise, using sun elevation as the trigger.
Gym Prep Mode: When my Withings sleep tracking detects I’ve woken up, a webhook triggers Home Assistant to turn on the lights and get me ready for the gym—before I even touch anything.
Kitchen LED Strip: A battery-powered Zigbee switch controls a sleek LED strip mounted under the kitchen counter.

Smart Sockets and Switches: Other battery-powered switches control lights through Zigbee-connected smart sockets.
Bathroom Fan Logic: If the bathroom door is closed and the light is on, the fan activates automatically.
Humidity Control: A dehumidifier switches on when the indoor humidity passes a predefined level.
Alarm System via Alarmo: I’ve integrated a smart siren using the Alarmo custom integration, which is connected to door and window sensors. If the system is armed and a breach is detected, the siren triggers and notifications are sent immediately.

Seamless Integration with Apple Ecosystem

One feature that elevates the experience is the HomeKit Bridge integration. Through this, I’ve exposed all my Home Assistant devices to Apple’s Home app. My HomePods can now see and control every switch, light, sensor, and more. It feels native—“Hey Siri, turn off all the lights in the flat” just works—even for Zigbee devices that have no official Apple support. No subscriptions, no iCloud Home Hub required.

Smarter Lighting, Simpler Bulbs

Instead of relying on smart bulbs—which can be expensive and often require constant power—I opted for Zigbee-enabled wall switches. This approach allows me to use standard light bulbs throughout the flat while maintaining complete control via Home Assistant. The lights behave like any normal wall-controlled circuit, but are also fully accessible for automation, scheduling, and remote control. It keeps things simple for guests and family members while preserving smart functionality in the background.

NFC Tags and Touch-Based Automation

Another layer of interaction I’ve embraced is NFC tags. These small, inexpensive stickers are placed throughout the flat and programmed to trigger specific automations when tapped with my iPhone.

For instance, a tag near the bed activates a “Good Night” routine—dimming lights, turning off devices, and arming sensors. A tag by the front door toggles “Leaving Home” mode, shutting everything down and confirming the flat is secure. It’s a subtle, tactile way to control your environment, and it works offline.

Outdoor Sensing

Initially, I used a standalone weather station—similar to this model—to track outdoor temperature. While it served its purpose, it had a major limitation: it wasn’t smart. It couldn’t connect to Home Assistant or any other system. To fully integrate weather data into my automations, I replaced it with an Aqara temperature and humidity sensor placed just outside. Since it’s Zigbee-powered, it communicates seamlessly with Home Assistant, just like my indoor sensors. Now, outdoor conditions can influence automations too, such as adjusting ventilation or sending alerts when it’s too cold or humid outside.

Hallway Control Panel

To make the smart home even more accessible to everyone in the flat, I’ve mounted a Huawei tablet in the hallway that runs the Home Assistant dashboard full-time. It acts as a dedicated control panel where we can view sensor data, toggle lights, check the status of the alarm, and access custom dashboards. It’s a great fallback for guests or anyone who prefers touch over voice or automation.

Xiaomi Devices and Wi-Fi Integration

Not everything in my smart home runs on Zigbee. I also use Xiaomi smart fans and a Roborock vacuum, both connected to Home Assistant via Wi-Fi using the Xiaomi Miio integration. While I generally prefer local control, these devices are reliable enough and offer valuable capabilities—like scheduling fan usage during the night or triggering vacuum runs while we’re away.

Even though these devices connect via Wi-Fi, the Miio integration communicates with them locally over the network, not via the cloud. That means Home Assistant sends commands directly to the device’s IP address on my local network. This keeps the experience fast, reliable, and private—without requiring any round trips to external servers.

Shutter Integration with Somfy TaHoma Switch

My shutters weren’t Zigbee-compatible out of the box—they required a TaHoma Switch, which acts as a bridge between the proprietary shutter protocol and Home Assistant. Once configured, the TaHoma Switch exposed each shutter as an entity in Home Assistant, allowing me to control them through automations or directly from the dashboard. And because I’ve integrated Home Assistant with the HomeKit Bridge, these shutter controls are also available via Siri on my HomePods. Voice commands like “Hey Siri, set the office shutter to 50%” are now fully supported.

If you’re curious about how it all fits together, I’ve open-sourced the entire setup at github.com/jonathas/homeassistant-config. It’s modular, UI-driven, and designed to scale—from simple automations to complex routines.

How I Transformed My Flat into a Smart Home with Home Assistant was originally published by Jonathas Ribeiro at Jonathas Ribeiro on June 29, 2025.

How I Replaced My Frameo Digital Frame with an Android Tablet

2025-06-08T08:10:08+00:00

Having a digital photo frame at home has always been a nice way to keep our memories alive. Over the years, I’ve collected thousands of photos from family trips, special events, and everyday moments. This turned our digital frame into a slideshow of our best experiences. However, recently, I had issues with my Frameo-powered digital frame, leading me to look for a better solution. Here’s how I moved to a Lenovo Yoga Tab 11 running Fotoo and why it’s been a much better choice.

The Issues with Frameo

Initially, Frameo seemed like a good idea—sending photos directly from an iPhone app to the frame. But in reality, it wasn’t very good. First, the Frameo iOS app took up a lot of space, sometimes more than 4GB on my phone, which is too much for just sending photos.

Even worse, after uploading a few thousand photos, my digital frame became slow and unresponsive. Navigating photos caused it to crash frequently, needing regular restarts. The touch screen became slow, and the overall experience was frustrating. This made me realize the Frameo digital frame wasn’t reliable at its basic job—showing photos smoothly.

The Difficult Frameo Workflow

My workflow with Frameo was annoying:

Selecting and sending photos in small groups (only 10 photos at a time unless paying extra).
Manually resizing and positioning each photo on the frame to fit the display.
Regularly inserting an SD card to manually back up the photos.

Not exactly easy!

Moving to Fotoo on Lenovo Yoga Tab 11

Looking for something better, I decided to use a used Lenovo Yoga Tab 11 as my new digital frame. The tablet might be a bit overkill for this task, but it already has a built-in stand, perfect for displaying photos.

My new setup includes:

Installing Fotoo, a simple digital frame app for Android.
Using LocalSend to easily transfer photos directly from my MacBook to the tablet.
Automatic backup through Google Photos using the tablet’s built-in syncing feature, so all photos are safely stored in Google Drive for free.

Optimizing Images with a Custom Script

The Frameo app on iOS automatically reduced photo quality to save storage space. LocalSend doesn’t have this feature, so I made a simple script to resize and convert my images before sending them:

#!/bin/bash

for f in *.heic; do
  sips -Z 2000 -s format jpeg -s formatOptions 75 "$f" --out "${f%.*}.jpg"
done

This script resizes images to save space without losing much quality, so after selecting the photos, I run the script to resize them and then can just transfer all at once to the tablet.

Keeping the Same Screen Schedule

In Frameo, I had my frame set to turn off at 11 pm and turn on again at 7 am. Fotoo allows me to easily set up the tablet to do exactly the same, matching my previous settings perfectly.

Free Cloud Backup Compared to Frameo

One great benefit of using an Android tablet is automatic free backup with Google Photos. In Frameo, cloud backup is only available with a paid subscription, but with my tablet setup, backups happen automatically for free using Google Drive.

Conclusion

Switching from a Frameo digital frame to an Android tablet running Fotoo has greatly improved my experience. Using Fotoo, LocalSend, and Google Photos is an efficient and simple way to display our family’s favorite moments. If you’re frustrated with slow or limited digital frames, this setup could save you headaches and storage space!

How I Replaced My Frameo Digital Frame with an Android Tablet was originally published by Jonathas Ribeiro at Jonathas Ribeiro on June 08, 2025.

Composition versus Inheritance in Software Engineering

2023-11-25T20:10:08+00:00

Programming paradigms such as functional programming (FP) and object-oriented programming (OOP) are used in conjunction with some important principles and concepts in order to create modular and maintainable code. Today we’ll take a look at the concepts of composition and inheritance, demonstrating their use with some examples.

What is Composition?

When organizing your codebase using composition, you need to think of a “has-a” relationship.

In functional programming, functions are first-class citizens, and composition is a powerful tool for building complex behavior by combining simpler functions.

const square = (x: number): number => x * x;
const double = (x: number): number => x * 2;

const squareAndDouble = (x: number): number => double(square(x));

const result: number = squareAndDouble(3);
console.log(result); // Result is 18

squareAndDouble is composed of the square and double functions. This allows us to create a pipeline of operations, making the code more modular and readable.

Not only limited to functional programming, composition can also be used in object-oriented projects.

class Engine {
  public start(): void {
    console.log("Engine started");
  }
}

class Car {
  private engine: Engine;

  public constructor() {
    this.engine = new Engine();
  }

  public start(): void {
    this.engine.start();
  }
}

const myCar = new Car();
myCar.start(); // This will start the car's engine

In this example, a Car has an Engine. Through composition, a Car is composed of an Engine. This allows for a flexible and modular design where the Car class can leverage the functionality of the Engine class without directly inheriting from it.

Pros of using Composition

Flexibility: Composition allows for a more flexible and dynamic relationship between classes. Objects can be composed of other objects at runtime, providing more flexibility in creating complex behaviors.
Code Reusability: By encapsulating objects within other objects, you can reuse components in different contexts. This promotes code reusability and modular design.
Avoidance of tight coupling: Changes to one class do not directly impact other classes, reducing the risk of unintended side effects. This makes the codebase more maintainable.
No fragile base class problem: Unlike inheritance, changes to the base class do not affect the functionality of the derived classes. This avoids breaking existing code in derived classes, which is an issue that happens with inheritance.

Cons of using Composition

Indirect access: Accessing the functionality of a composed object may require additional methods in the containing class. This can lead to more verbose code when compared with inheritance.
More boilerplate code: In some cases, composition can lead to more boilerplate code, especially when dealing with multiple levels of nesting.

What is Inheritance?

When organizing your codebase using inheritance, you need to think of a “is-a” relationship.

Inheritance is a core concept in object-oriented programming, allowing for the creation of specialized classes that inherit behavior and attributes from more general classes.

class Animal {
  protected speak(): void {
    console.log("Animal speaks");
  }
}

class Dog extends Animal {
  public bark(): void {
    console.log("Dog barks");
  }
}

const myDog = new Dog();
myDog.speak(); // This will call the speak method inherited from Animal
myDog.bark();  // This will call the bark method specific to Dog

In this example, the Animal class has a speak method and a Dog class extends Animal, as a dog is an animal. Through inheritance, the Dog class inherits the speak method and adds its own bark method. This demonstrates a “is-a” relationship.

Pros of using Inheritance

Code Reusability: Inheritance allows for the reuse of code from a base class in derived classes. This can lead to a more concise and DRY (Don’t Repeat Yourself) codebase.
Polymorphism: Inheritance supports polymorphism, allowing objects of derived classes to be treated as objects of the base class. This promotes a more generalized and abstract design.
Less boilerplate code: In some scenarios, inheritance can result in less boilerplate code compared to composition, especially for simple hierarchies.

Cons of using Inheritance

Tight Coupling: Changes in the base class can affect all derived classes, leading to tight coupling. This can make the code more fragile and less adaptable to changes.
Inflexibility: The “is-a” relationship enforced by inheritance may lead to a less flexible design. If the relationship between classes changes, it can be challenging to adapt the code.
Limited to single inheritance: Many languages support single inheritance only, which restricts the ability to inherit from multiple classes. This limitation can be addressed with interfaces or mixins, but it adds complexity.

Composition over inheritance

Composition over inheritance is a design principle that suggests what the name says: favoring composition over inheritance. This approach promotes flexibility, code reuse, and maintainability.

For example, if you have a Logger class, it can be part of other classes via composition:

// Base class representing a Logger
class Logger {
  log(message: string): void {
    console.log(`[Log]: ${message}`);
  }
}

// Class representing a User with logging capability using composition
class User {
  private logger: Logger;

  constructor(name: string) {
    this.logger = new Logger();
    this.name = name;
  }

  private name: string;

  greet(): void {
    this.logger.log(`Hello, my name is ${this.name}`);
  }
}

// Class representing an Admin with logging capability using composition
class Admin {
  private logger: Logger;

  constructor(name: string) {
    this.logger = new Logger();
    this.name = name;
  }

  private name: string;

  announce(): void {
    this.logger.log(`Admin announcement: ${this.name} is here!`);
  }
}

// Example usage
const regularUser = new User('John');
regularUser.greet(); // Outputs: [Log]: Hello, my name is John

const systemAdmin = new Admin('Admin1');
systemAdmin.announce(); // Outputs: [Log]: Admin announcement: Admin1 is here!

This approach provides:

Flexibility, as you can easily change or extend the behavior of User and Admin without being constrained by the limitations of a fixed inheritance hierarchy.
Code reusability, as the Logger class can be reused in other contexts as well, making it modular and reusable
Maintainability: Changes to the Logger class do not affect the functionality of other classes, reducing side effects.

If instead we used inheritance for this:

// Using inheritance
class UserWithLogger extends Logger {
  private name: string;

  constructor(name: string) {
    super();
    this.name = name;
  }

  greet(): void {
    this.log(`Hello, my name is ${this.name}`);
  }
}

class AdminWithLogger extends Logger {
  private name: string;

  constructor(name: string) {
    super();
    this.name = name;
  }

  announce(): void {
    this.log(`Admin announcement: ${this.name} is here!`);
  }
}

We’d achieve the same logging functionality, but now there’s a tight coupling between the Logger and User or Admin classes. Changes to the Logger class might affect the behavior of User and Admin. Using composition over inheritance helps avoid this issue.

Conclusion

Choosing between composition and inheritance often involves a trade-off between flexibility, code reusability, and maintainability. In practice, a combination of both approaches is frequently used to leverage the benefits of each while mitigating their weaknesses. This allows for a more adaptable and modular design, where composition and inheritance are used where they make the most sense in the context of the application’s requirements.

I’d be more inclined to use inheritance when thinking about a “is-a” relationship in the codebase and to use composition when thinking about a “has-a” relationship, so it depends on the specific requirements of the project. In a way, composition should always be there if you want to use Dependency Injection in your project, since it would be a prerequisite for it, as DI uses composition.

Composition versus Inheritance in Software Engineering was originally published by Jonathas Ribeiro at Jonathas Ribeiro on November 25, 2023.

Optimizing Data Access: A Deep Dive into Database Indexes

2023-11-23T16:30:08+00:00

When considering database optimization, the efficiency of retrieving information is crucial. One key player in this quest for efficiency is the database index. Let’s dive into the world of database indexes and uncover what happens under the hood.

What are Database Indexes?

At its core, a database index is a data structure that provides a faster way to locate rows in a table based on the values in one or more columns. Imagine a well-organized library where books are arranged by genres or authors. This way of organizing them significantly accelerates the process of finding a specific book. In a similar way, a database index works as a roadmap, guiding the database engine to the relevant rows in a table.

Most database indexes utilize a B-tree (balanced tree) structure. This structure allows for efficient insertion, deletion, and search operations. Each entry in the index is a key-value pair, with the key representing the indexed column(s) and the value pointing to the corresponding row in the table.

Types of Indexes

Database indexes can be created manually by administrators or automatically by the database management system (DBMS). Indexes can be applied to one or more columns, and there are various types to cater to different needs:

Unique Indexes: Ensure the uniqueness of values in the indexed columns, preventing duplicate entries.
Clustered Indexes: Determine the physical order of data rows in the table based on the index, influencing data storage layout. In a clustered index, the rows in the table are stored in the same order as the index. Each table can have only one clustered index.
Non-clustered Indexes: Create a separate structure for the index, leaving the data rows unchanged. Unlike a clustered index, a non-clustered index does not affect the physical order of the rows in the table and allows for the creation of multiple non-clustered indexes on a single table.

When a query involving the indexed column(s) is executed, the database engine utilizes the index to quickly locate the relevant rows. This process reduces the need to scan the entire table, resulting in a significant reduction in disk I/O operations. B-tree indexes, often sorted, enable efficient range queries, making them particularly powerful in enhancing read operation performance.

Trade-offs and Considerations

While database indexes offer immense benefits in terms of query performance, they come with trade-offs. Indexes consume additional disk space, and their maintenance can impact the performance of write operations such as inserts, updates, and deletes. Careful consideration is needed when deciding which columns to index, taking into account the selectivity of the index and the specific needs of the database.

Updating or deleting data in indexed columns requires corresponding updates to the index to maintain accuracy. Periodic rebuilding or reorganizing of indexes may be necessary to optimize their performance over time.

Indexes with high selectivity (a large number of unique values) are more effective in improving query performance.

Composite indexes involve multiple columns, optimizing queries that filter on multiple criteria.

Practical examples

Theory yada-yada.. let’s see some examples.

Continuing on the library idea that was mentioned before, we have the following tables represented in SQL below:

Authors

CREATE TABLE authors (
  id INT PRIMARY KEY,
  first_name VARCHAR(50),
  last_name VARCHAR(50)
);

Publishers

CREATE TABLE publishers (
  id INT PRIMARY KEY,
  name VARCHAR(100)
);

Books

The books table includes information about each book, such as title, publication year, ISBN, and the publisher.

CREATE TABLE books (
  id INT PRIMARY KEY,
  title VARCHAR(255),
  publication_year INT,
  isbn VARCHAR(13),
  publisher_id INT,
  FOREIGN KEY (publisher_id) REFERENCES publishers(id)
);

AuthorsBooks (Association Table)

This table represents the many-to-many relationship between authors and books. It allows multiple authors to be associated with multiple books.

CREATE TABLE authors_books (
  author_id INT,
  book_id INT,
  PRIMARY KEY (author_id, book_id),
  FOREIGN KEY (author_id) REFERENCES authors(id),
  FOREIGN KEY (book_id) REFERENCES books(id)
);

Each book can have multiple authors (many-to-many relationship).
Each author can write multiple books (many-to-many relationship).
Each book is associated with one publisher (many-to-one relationship).

Inserting some data

-- Inserting authors
INSERT INTO authors VALUES (1, 'J.K.', 'Rowling');
INSERT INTO authors VALUES (2, 'George R.R.', 'Martin');
INSERT INTO authors VALUES (3, 'Jane', 'Austen');

-- Inserting publishers
INSERT INTO publishers VALUES (1, 'Penguin Books');
INSERT INTO publishers VALUES (2, 'HarperCollins');

-- Inserting books
INSERT INTO books VALUES (1, 'Harry Potter and the Philosopher''s Stone', 1997, '978-0747532743', 1);
INSERT INTO books VALUES (2, 'A Game of Thrones', 1996, '978-0553103540', 2);
INSERT INTO books VALUES (3, 'Pride and Prejudice', 1813, '978-1503290563', 1);

-- Inserting author-book associations
INSERT INTO authors_books VALUES (1, 1);  -- J.K. Rowling wrote Harry Potter
INSERT INTO authors_books VALUES (2, 2);  -- George R.R. Martin wrote A Game of Thrones
INSERT INTO authors_books VALUES (3, 3);  -- Jane Austen wrote Pride and Prejudice
INSERT INTO authors_books VALUES (1, 2);  -- J.K. Rowling also wrote A Game of Thrones

Retrieving all books written by J.K. Rowling

SELECT books.title
FROM authors
JOIN authors_books ON authors.id = authors_books.author_id
JOIN books ON authors_books.book_id = books.id
WHERE authors.first_name = 'J.K.' AND authors.last_name = 'Rowling';

Retrieving the details of the authors of “A Game of Thrones”

SELECT authors.first_name, authors.last_name
FROM books
JOIN authors_books ON books.id = authors_books.book_id
JOIN authors ON authors_books.author_id = authors.id
WHERE books.title = 'A Game of Thrones';

Examples of index usage

Some good examples of index usage are:

The primary key index in the authors table, on the “id” attribute, as it is crucial for efficiently locating specific authors based on their unique id.
The foreign key index in the books table, on the “publisher_id” attribute, as it facilitates quick retrieval of books associated with a particular publisher, optimizing join operations.
The composite index on the authors_books association table, which is made of author_id and book_id and supports the many-to-many relationship between authors and books. It enhances the performance of queries involving specific author-book associations.
Index on frequently queried columns, like for example on the “title” attribute of the books table. If queries frenquently search for books based on their titles, creating an index on the “title” column can significantly improve query performance.

Some bad examples of index usage are:

Over-indexing: Creating an index on both first_name and last_name on the authors table might be unnecessary if queries rarely involve searching for authors based on both criteria. It could lead to index bloat and increased maintenance overhead without substantial performance gains.
Indexing on columns with low selectivity: In the books table, if most books have the same publication year, indexing on this column might not significantly enhance query performance. Indexing columns with high selectivity is generally more beneficial.
Unused indexes: If there are no queries that involve searching or joining based on the publishers table, creating an index on this table might be unnecessary and add unnecessary overhead.
Overreliance on clustered index: In the books table, while clustering on the primary key (id) is common, overreliance on the clustered index for all types of queries might lead to suboptimal performance for certain query patterns.

What should be considered when improving index usage

Selective indexing: Choose indexes based on the selectivity of columns. Indexing highly selective columns enhances the efficiency of the index.
Query patterns: Consider the types of queries that are commonly executed. Indexes should align with the patterns of data retrieval in your application.
Balance between read and write operations: While indexes can significantly improve read performance, they may impact write operations. Evaluate the trade-offs based on your application’s requirements.
Regular monitoring and maintenance: Periodically review and optimize indexes based on the evolving usage patterns of your application. Unused or redundant indexes can be identified and removed.
Use tools for analysis: Utilize database management tools to analyze query execution plans and identify areas where indexes can be beneficial. This helps in making informed decisions about index creation.

Conclusion

When considering to optimize your database, understanding how indexes work is essential. They are very important for efficient data retrieval, but their effectiveness depends on factors such as selectivity, indexing strategies and maintenance challenges.

Optimizing Data Access: A Deep Dive into Database Indexes was originally published by Jonathas Ribeiro at Jonathas Ribeiro on November 23, 2023.

Bringing Back the Flags! Keyboard Layout Indicator as Country Flags on Gnome

2023-01-21T12:10:08+00:00

Gnome is a free and open-source desktop environment for Linux and Unix-like operating systems. It is one of the most popular desktop environments for Linux, and is the default desktop environment for many Linux distributions. Gnome is known for its simplicity, ease of use and a user-friendly interface. It is designed to be easy to use for people of all ages and skill levels and it’s also fully themeable and customizable.

One of the features that Gnome used to offer was the ability to display country flags as the keyboard layout indicator. This feature was present in earlier versions of Gnome, but was removed in later versions. The thing is, users don’t always agree with the decisions a product takes, so in this case as the product is open-source and highly configurable, it’s much easier to customize it to our liking.

A few months ago I implemented a Python script which updates the config files related to X11, where Gnome fetches the information for the keyboard layout indicator, and changes the country codes to their flags instead.

Current state

On Gnome, the keyboard layout is presented as the code of the language which is currently selected. In the example below, the English (US) keyboard layout is selected:

After the script

A country flag there looks nicer to me instead of the language. After this script runs, this is what that would look like for the English (US) keyboard layout:

Ps: Other keyboard layouts/languages are also available

The switcher now looks like this

In context

Installing

If you use Arch Linux or any distribution based on it, you can get it from AUR using yay:

yay -S x11-keyboard-flags

On other distributions which have nothing to do with Arch Linux, you can get the source code and install it manually.

Using

Run it with sudo:

sudo x11-keyboard-flags

Then reset gnome-shell by pressing Alt+F2 and entering the “r” command.

The script makes a backup of the original evdev.xml file from X11 before running and then sets the flags in the original evdev.xml file

And that’s it! After these steps you’ll see the flags in your keyboard layout selector.

Conclusion

This customization can enhance the visual representation of the keyboard layout and make the language switching process more intuitive. As a further step, you can include this script in the startup process of your OS so that even if the package related to the file that was changed gets updated, the country flags will still be there.

Bringing Back the Flags! Keyboard Layout Indicator as Country Flags on Gnome was originally published by Jonathas Ribeiro at Jonathas Ribeiro on January 21, 2023.

Efficient Software Release Management with Automated Changelog Generation

2023-01-15T21:10:08+00:00

The software release process is a critical aspect of any software development project. A manual software release process can be time-consuming, error-prone, and often leaves room for human error. This is where an automated release process comes in. The tools mentioned in this article (used in TypeScript/Javascript projects) can help streamline and optimize the software release process, making it more efficient and less prone to errors, by reducing human involvement.

The importance of the items and the order in which they are presented in this article, is that each of them is part of the final result which we wish to achieve. In the end you’ll see they all match together to make the development and release process work.

Notes

When I mention the “master/main” branch, it can be either master or main, depending on what you agreed with your team.
When I mention Github Actions it can also be done in any other CI/CD service or software

Versioning

When thinking about a software release, it all starts with deciding how the versioning will be handled. I’ve been following Semantic Versioning (semver) in my projects for years now and recommend it.

This means that:

Given a version number MAJOR.MINOR.PATCH, increment the:

MAJOR version when you make incompatible API changes
MINOR version when you add functionality in a backwards compatible manner
PATCH version when you make backwards compatible bug fixes Additional labels for pre-release and build metadata are available as extensions to the MAJOR.MINOR.PATCH format.

Ps: There’s also a variation of it for purely frontend repositories.

A convention to use in all commits

In most projects, if the team wants to keep some kind of changelog, it’s necessary to write the latest changes somewhere manually (Confluence, Notion, a CHANGELOG.md file, Slack, etc). We want to automate that and remove the manual steps required in order to create a release, but we’ll get to this part on how to automate the changelog generation in a bit. There’s one step which is a requirement for us to get there:

The Conventional Commits specification provides a group of rules on how to format every commit message so that they all follow the same pattern across the project. It is used by the Angular team and many other big projects.

Example of a commit message: <type>[optional scope]: <description>

A commit contains the following structural elements, to communicate intent:

fix: a commit of the type fix patches a bug in your codebase (this correlates with PATCH in Semantic Versioning).
feat: a commit of the type feat introduces a new feature to the codebase (this correlates with MINOR in Semantic Versioning).
BREAKING CHANGE: a commit that has a footer BREAKING CHANGE:, or appends a ! after the type/scope, introduces a breaking API change (correlating with MAJOR in Semantic Versioning). A BREAKING CHANGE can be part of commits of any type.
types other than fix: and feat: are allowed, for example @commitlint/config-conventional (based on the Angular convention) recommends build:, chore:, ci:, docs:, style:, refactor:, perf:, test:, and others.
footers other than BREAKING CHANGE: <description> may be provided and follow a convention similar to git trailer format.

Additional types are not mandated by the Conventional Commits specification, and have no implicit effect in Semantic Versioning (unless they include a BREAKING CHANGE). A scope may be provided to a commit’s type, to provide additional contextual information and is contained within parenthesis, e.g., feat(parser): add ability to parse arrays.

Making sure that the Conventional Commit rules are followed

In order to enforce that the correct format is followed, the commitlint library can be automatically called via git hooks locally when the developer tries to create a new commit. It works as a linter, but for commit messages!

Install the following packages:

Then add a file called commitlint.config.js to the root of your project, with the following content:

module.exports = { extends: ['@commitlint/config-conventional'] };

Local Git hooks

Git hooks are scripts that run automatically before or after executing Git commands like Commit and Push. With Git hook scripts, users can customize Git’s internal behavior by automating specific actions at the level of programs and deployment, like for example validating that the commit is following the standards and not breaking the tests.

They can be managed with the use of a library called husky.

The hooks I usually configure are: pre-commit, commit-msg and pre-push

pre-commit: When the developer tries to commit what was implemented, the pre-commit hook will be automatically called and run eslint with the predefined rules. If the code is invalidated by these rules, the commit doesn’t happen until the issues are fixed. It’s better to have the linter at commit time to allow for smaller and timely iterations instead of only checking these during the push to the repository
commit-msg: When the developer writes the commit message, it is automatically checked against commitlint, which validates the commit message against the Conventional Commits specification. If the commit message is not valid, the commit doesn’t happen until the message is fixed.
pre-push: When the developer tries to push the commit(s) to the repository, the pre-push hook is called automatically and runs “npm test”. The push to the repository only happens if all tests are passing.

Configuring husky

Install the husky library as a dev dependency:

npm i husky --save-dev

Run the following to add the following to the package.json file:

npm pkg set scripts.prepare="husky install"
npm run prepare

Add the pre-commit hook that runs the linter:

npx husky add .husky/pre-commit "npm run lint"

Ps: My eslintrc example can be found here.

Add the commit-msg hook that runs commitlint:

npx husky add .husky/commit-msg "npx commitlint --edit $1"

Then finally the pre-push hook that runs the tests:

npx husky add .husky/pre-push "npm test"

After these steps, commit the .husky directory to the git repository.

Way of working

For the automated changelog generation to work well, it’s important to have a well defined development process.

When a developer starts to work on a ticket, a new branch needs to be created in the git repository. This branch must be created out of the develop branch.

Branch naming conventions

The name of this new branch that the developer will create can be anything. It doesn’t really matter in this case, as it must be automatically deleted after the PR is merged so that the repository doens’t become a mess of hundreds or thousands of branches. What I recommend, though, is to follow a similar format to what is done in the commits, so that it’s easy to identify what that branch is about.

For example, if it’s a new feature:

feat/name-of-the-feature

if it’s a fix:

fix/name-of-the-fix

Pull Request title

When the developer is ready to submit changes to the repository, a new Pull Request needs to be created. The Pull Request title matters more than the name of the branch, since that’s what will end up in our git history and, consequently in our automatically generated changelog! Hence, when creating a Pull Request, its name must follow the Conventional Commits specification.

An example of how the PRs following Conventional Commits would look like in the repository history:

If you’d like to also link the PRs to Jira tickets, it’s recommended to add the Jira ticket to the PR title as well, following this format: <type>(scope): description [jiraticket-number]

For example: fix(calendar): default pagination limit [AG-1103]

AG-1103 in this case being the Jira ticket.

Ps: You’ll need to, of course, configure this sync between Github and Jira separately. This part is not covered by this article.

Making sure the Pull Request title format is followed

Using Github Actions, it’s easy to implement a workflow that checks the Pull Request title and validates it against the Conventional Commits specification.

You can find the gist with the code here.

Save this code to the following path in your project: .github/workflows/lint-pr-title.yml

Whenever a PR is created, this workflow runs and doesn’t allow the PR to be merged if the title is incorrect.

Imagine dependabot creates 20 PRs and all of them run your Github Actions workflows. That would spend a lot of precious Github Actions minutes, which cost money! So as a bonus, this workflow doesn’t run on PRs created automatically by dependabot, which should already be following the Conventional Commits specification.

Pull Request content

A good pull request should allow developers to review it quickly, so it needs to be small and well explained. The best way of thinking about it to keep it smaller is that it should cover one thing only, as in the Single Responsibility Principle (SRP).

Here’s an interesting and worth reading article I found the other day that gets deeper into this topic: The anatomy of a perfect pull request

Merging strategy

In order for the changelog generation process to work correctly, the git history needs to be clean, so there must be no merge commits in the develop nor in the master/main branches. The repository needs to be configured to only allow squash merging, so that only PRs end up in the git history. In Github that can be done in the following area:

And then the developers can configure git locally to always use rebase so that they won’t need to create merge commits when fixing conflicts in PRs. This can be done globally for all projects by running the following command:

git config --global pull.rebase true

Then when synching the changes from the develop branch to the branch they’re working on, they can just pull the latest changes from develop into their branch and the rebase will happen automatically:

git checkout feat/my-feature-branch
git pull origin develop

Status checks on Pull Requests

As with the Github Action workflow to validate the PR title, it’s recommended to run other status checks on PRs to validate that the PR can only be merged to the develop branch if all the status checks are passing. These are recommended to be added to the repository:

Run integration tests
Run unit tests
SonarCloud Code Analysis

Bear in mind that if you have dependabot enabled in your repository, it’s usually a good idea to change the validations above so that they don’t run on PRs created by dependabot. Otherwise you’ll see a big increase in Github Actions minutes. I won’t go deeper in details about them as it’s not the focus of this article.

Deployment to Staging

From “What is a Staging Environment?”

A staging environment or staging site is a copy of your live website and is the last step in the deployment process before changes are deployed to your live website. By having a staging environment that is a copy of your live environment you are able to test new changes made by your developers before they are released to your live website. Using multiple environments is not necessary, but it comes with a long list of advantages, which are especially important if you work on big or complex projects. Testing new changes on a staging environment before deploying them to your live website also reduces the risk of any errors or issues that will affect your users. This effectively means happier users and more uptime for your website.

The deployment to Staging should happen every time a PR or commit is merged to the develop branch. This can be done by configuring Github Actions, for example, to start and handle the deployment once that happens.

Deployment to Production

If there’s no well defined process yet for deployments to production, it’s always good to follow the golden rule of “no deployments on Fridays!”

Whenever we want to deploy to production, a release needs to be created.

A release branch can be created out of the develop branch, then a PR can be created so that after it’s approved it’s merged to the master/main branch.

Then after that, the release process can be started manually with the help of the release-it wizard or the Github Action can pick it up, run the release-it lib inside of it in automated mode and do what else needs to be done for deployment. We’ll talk more about this part below, so keep reading.

The release process should always be started on the master/main branch and then synched back to develop.

Rolling back

In case the release breaks production even after all the status checks and QA approval, we need to have a way to rollback what is deployed to production to a stable version. This can be done via a Github action workflow, which exists for this purpose and can be started manually.

From the repository, the developer should be able to select which branch this workflow should run from, so that this branch with the fix replaces what is in production. Or this workflow can instead always revert the master/main branch to the previous version. It’s a good idea to decide with your team how this process will be handled in your specific case.

Creating a Release

Finally after all is configured in the repository and the team is following a well defined development process, it’s time to talk about creating the release!

This process is handled by the release-it library, which can be used in two ways:

Manually: One of the developers runs npm run release locally, which starts the release-it wizard, which in turn takes care of what’s needed.
Automatically via CI/CD: After the PR is merged, the Github Action with the release-it library starts from there and takes care of what’s needed.

When the release-it library starts, it:

Identifies which commits happened after the last version
Bumps the version in package.json following the Semver convention
Adds the changelog related to the new version to the CHANGELOG.md file (using the auto-changelog library)
Creates a commit with the new version. For example: chore(release): 0.3.1
Creates a Git tag with the version
Pushes these changes to the master/main branch of the repository
Creates a Github release
Merges the master/main branch back to develop and pushes develop, to keep the branches in sync

Configuring release-it

Install the release-it library:

npm i release-it --save-dev

Add the following to the “scripts” part of the package.json file to enable the npm run release command:

"release": "release-it",

Create a .release-it.json file on the root of your project:

{
  "hooks": {
    "before:init": ["npm test"],
    "after:bump": ["npx auto-changelog -p"],
    "after:git:release": ["git checkout develop", "git merge master", "git push origin develop"]
  },
  "git": {
    "requireBranch": "master",
    "commit": true,
    "commitMessage": "chore(release): ${version}",
    "commitArgs": "",
    "tag": true,
    "tagName": "${version}",
    "tagAnnotation": "${version}",
    "push": true,
    "requireCommits": true,
    "changelog": "npx auto-changelog --stdout --commit-limit false -u --template https://raw.githubusercontent.com/release-it/release-it/master/templates/changelog-compact.hbs"
  },
  "github": {
    "release": true,
    "releaseName": "${version}",
    "tokenRef": "GITHUB_TOKEN"
  },
  "npm": {
    "publish": false
  }
}

Ps: If your project is an npm library, you can also make release-it push it to npmjs.com automatically by changing npm.publish to true in the config above.

Automatic Changelog generation

With the use of the release-it library, the changelog is generated automatically during the release process. Every item in git history from the last version until now is added to the beginning of the CHANGELOG.md file in the root of the project. It’s important that the previous steps mentioned in this article are followed so that the resulting changelog is well formatted.

The automatically generated changelog looks like the following:

CHANGELOG.md

As the changelog file is in Markdown format, it also contains links to the specific PRs.

Configuring auto-changelog

Install the auto-changelog library:

npm i auto-changelog --save-dev

And that’s it, as the integration is already configured with release-it.

Further improvements

As an idea for further improvement, in case your company has a Slack channel where people need to post changelogs to, you can extend the idea presented in this article to post the automatically generated changelog to Slack.

The release-it library allows us to add a post release hook that could be integrated with that or it can be done via a job in the same Github Actions workflow.

Conclusion

By using the tools and processes presented in this article, development teams can improve the software release process and deliver high-quality software faster with efficiency, better accuracy and quality. Do you have any idea for improvement or have you been working differently? Let me know in the comments and let’s discuss so I can learn more from your experience as well!

Efficient Software Release Management with Automated Changelog Generation was originally published by Jonathas Ribeiro at Jonathas Ribeiro on January 15, 2023.

REST API Best Practices

2019-01-27T19:22:08+00:00

I’ve been working a lot with APIs during the last few years, either developing them or integrating with third party APIs. So I’ve decided to gather some of the knowledge I gained along the way and write about it here. Many of the APIs I’ve integrated with didn’t follow any kind of pattern, so their documentation had to be consulted more often than if they were following some of the points below. Because of that, some of them couldn’t even be considered RESTful APIs.

What is REST?

REST means Representational State Transfer, and it is a software architectural style that defines a set of rules to be used when creating APIs. The APIs that conform to the REST style are called RESTful APIs. First, you need to create one or more endpoints and expose them to your clients. Then, your clients are able to access them via HTTP using the correct HTTP methods.

HTTP methods

The GET method happens when you open a website and it retrieves an image from the server, for example. The POST method is often used on forms in websites, so your data is not sent in the URL (query string), but in the body of the request in this case. A RESTful API will use these two HTTP methods in their specific cases, but also some others like PUT and DELETE.

Let’s see some points to take into consideration when designing an API.

Endpoints should be nouns, not verbs

Think of a resource you have in the system. For example, you have a collection of employees. If your client wants to retrieve a list of employees, he doesn’t need to remember he has to access the URL /getEmployees. Or even worse, what if this method was called /getAllEmployees, or something else? Then your client would have to remember if the method for deleting an employee was /removeEmployee or /deleteEmployee, for example, and use the HTTP Method POST for that?

What if you could follow a pattern that would make much more sense for everybody? The HTTP protocol already comes with methods you can implement on your API for your client to interact with, so that makes it more intuitive.

Let’s suppose you have a collection of employees in your database, and you want to return all of these employees.

Then your client will make an HTTP GET request to the /employees endpoint for that.

The most used form when naming an endpoint is the plural, so employees instead of employee in this example, as we’re returning a list with all the employees.

It makes much more sense this way, but if you insist in using the singular form (I wouldn’t recommend), it’s important that you keep the same pattern also in the other endpoints you create.

Use the correct HTTP methods

Following our GET /employees example, which returns a list of employees, what happens if we need to create, update, delete an employee, or retrieve just a single employee? There are HTTP methods for each of these cases:

POST

The HTTP POST method is the one for when you want to create a resource, so the client will POST to /employees, sending the data in the request body instead of in the query string. If the request body contains any array, the best practice is to stringify that array. I’ve seen people sending something like this for the server to parse:

1
2
3
4
5
6
{
  "termsOptions[0].key": "optionSMS",
  "termsOptions[0].value": "true",
  "termsOptions[1].key": "optionPhone",
  "termsOptions[1].value": "true"
}

instead of just simply sending a stringified array. This would be better sent like that:

1
{ "termsOptions": [{"optionSMS": "true"}, {"optionPhone": "true"}] }

or even better, use an object instead:

1
{ "termsOptions": {"optionSMS": "true", "optionPhone": "true"} }

PUT

The HTTP PUT method is the one you use when you want to update a resource. Before accessing this endpoint, the client will need to retrieve the employees, so it is able to know the id of the employee it wants to update.

An endpoint with a PUT request is accessed by sending HTTP PUT to /employees/:id, being :id the id of the chosen employee. In the body of the request must be the object that represents the employee, with the changes the client needs to make.

Ps: I’ve seen some (very few) APIs implementing the HTTP PATCH method to allow the partial update of a resource, so from experience, that’s not much used. If you need to update your resource only partially, though, then use PATCH instead.

DELETE

The HTTP DELETE method is used when you want to delete a resource. Same way as with the PUT request, the client will need to have retrieved the employees before, so it is able to know the id of the employee it wants to delete.

An endpoint with a DELETE request is accessed by sending HTTP DELETE to /employees/:id, being :id the id of the chosen employee. In this request, nothing goes in the body.

GET

Following this same pattern then, what if we want to get a single employee? That’s right, we can call it the same way we did when using the PUT request, but changing the HTTP method from PUT to GET. So it would be GET /employees/:id, being :id the id of the chosen employee.

Safe or idempotent

A GET request is safe and idempotent. An HTTP method is considered safe when invoking that method doesn’t change the state of the resource. It doesn’t matter how many times you run GET /employees/1, for example, that will not change anything in that employee (it won’t be updated or deleted). When you get the same response no matter how many times you call the same endpoint, this means it is idempotent.

A POST request, for example, is not safe nor idempotent, as it always creates something and the response is not the same every time you call it.

PUT and DELETE requests are not safe but are idempotent.

When any of these methods are consumed in your API, a response should be returned to the client, right? Let’s see more about the response then in the next topic.

Use the correct HTTP codes

The HTTP protocol has a series of codes that can be used for specific cases. Many people know already (even some who don’t work directly with IT), that 404 means NOT FOUND, so that’s a valid HTTP code that should be returned by our API when a resource is not found.

The code 200 means OK and should be returned when the request happened successfully. It is mostly used for GET requests.

When you have an endpoint that creates a resource, it should return the status 201, which means CREATED. Makes sense, right?

If when trying to create a resource, the client sent malformed data to the endpoint, it should return 400, which means BAD REQUEST.

401 is for when the client is unauthorized, 500 for when there’s an internal error in the server, and so on.

Unfortunately, I’ve seen many APIs always returning 200 OK for everything. Please be nice and don’t do that, as there are codes widely used to explain each case.

I’ve also seen some returning always 200 as the status code, but some different status code in the body of the response. That is redundant, as there’s already a place for the status code in the response, outside the body. That would be less worse than returning always 200 as the status code but no other status anywhere at all, though.

Another thing I have seen but wouldn’t recommend would be to create some random status code, like 398 or 521, for example and return it in the body. There are several status codes that exist and can be used already, so there’s no need to invent new ones. Let’s follow the correct HTTP status codes?

If you wanna know more about other HTTP status codes, here are some links:

If you like cats
If you prefer dogs
If you are a very serious person / would like a deeper explanation

Document your endpoints

If you are building an API, then it means you’ll be integrating one or more clients (frontend, mobile, desktop) with it soon. It is important to write a good documentation on how to use the endpoints you’re exposing with your API, so other developers that will implement these clients, or even you in the future, are able to know how to proceed.

If you document well your API endpoints and how to work with them (and keep the documentation updated), developers will ask you less questions and your company will save time/money.

I’ve written a post about APIDoc for that purpose, but lately I’ve been using Swagger instead (I’ll write a post about it soon).

Version your API

Software is like a living organism, constantly changing and evolving. Your business and processes will change over time, requirements will change. If you work with Software Development, you need to learn to embrace change, so do yourself a favor and start versioning your API from the start.

The best way to version an API is to include the version in the URL. For example, if your API base for the employees endpoint is /api/employees, change it to /api/v1/employees

If the new requirements will introduce a breaking change, then you can create an endpoint under v2, as /api/v2/employees and keep it running side by side with the /api/v1/employees endpoint, so the clients that are using the v1 are not broken with the update.

Send the authorization token in the header instead of in the URL

There’s something called “Authorization” header which you can use to send your token, then there’s no reason to expect that to be sent in the query string, even less in the request body.

Use pagination

Imagine you have thousands of employees in your collection, and the client consumes the GET /employees endpoint. If you have no pagination, then the response can take a really long time or even make the service unavailable if there are thousands of requests for the same endpoint without pagination.

So always return paged results when there’s a list in your response. The API can accept parameters in the query string, such as page and limit, to inform which page and how many rows to be returned, for example.

Check response size and never include images

Watch out for the size of your responses. Imagine a response of some hundreds of Kb multiplied by thousands, when your API is live (think about the future). It’s better to keep it as slim as possible and not include unecessary data in it.

Also, please use an object storage service like AWS S3 for images, for example, and return only the link for those images instead of the code of the image as base64. Yes, I’ve seen that happening as well and the response gets huge!

You can easily check the size of the response on Postman.

Use SSL, but don’t configure that directly in your API

SSL is very imporant in order for your API to be more secure. Your API doesn’t need to implement it, though. It’s better to leave this to your load balancer, reverse proxy or API Gateway, so your API can be started in cluster mode behind it and not be exposed to the outside world, receiving data via local network or unix sockets.

Always validate input

When web development used to be all in one layer, with the frontend code built by the server and the HTML outputted as static content, it seems that there used to be more focus on input validation. Now that we have frontend and backend separated in two layers, many times we see the validation happening only in the frontend. That can obviously lead to security risks, so it’s very important to always validate user input in your API as well.

Do not rely on the clients to implement validation, but implement it in the API as well, returning the correct HTTP status codes according to the case.

Test your API with automated tests

Improve the quality of your code and be sure your API is responding with the correct HTTP status codes for each case, by writing automated tests. When you have tests, it’s easier to add new features because you know better if your changes are breaking what is already working, for example. This can take longer in the beginning to implement, but will save you time in the future.

KISS (Keep it simple, stupid)

I love this software principle, because it avoids me doing unecessary stuff and makes the code easier to understand and evolve in the future.

For example:

Always return the same format of data. Stay consistent. Don’t have many different kinds of error objects, but stick to only one, so your clients know what to expect.
Do not include unecessary data in the response. Don’t expose more than you think needs exposing. If something is not needed for any client, then there’s no need to expose it.

Conclusion

Building an API sometimes can be a complicated task, but if we follow best practices and if we are consistent and keep it simple, a lot of time can be saved and future headache avoided.

What are some of the best practices you’ve been using? If you have any tips or know of something I forgot to mention, leave a comment below :)

REST API Best Practices was originally published by Jonathas Ribeiro at Jonathas Ribeiro on January 27, 2019.

Caching API responses with Redis for faster endpoints

2017-11-04T08:14:08+00:00

Have you ever implemented an API endpoint that takes a while to respond? Maybe some seconds?

Imagine when you have thousands of clients accessing your API at the same time and this response time increases because of that.

An I/O call such as reading content from a file or a very complicated query in the database may take a while to complete. If this content doesn’t change at all, it’s much better then to leave it in the RAM, as accessing something there will be much faster than accessing something that is on disk. If it changes from time to time, a script to update the cache can be implemented to be called via a cron job when needed and the endpoint code can be refactored to fetch data only from the cache.

In this post I expect you to have Node.js and Redis already configured.

Redis

Taken from their website: Redis is an open source (BSD licensed), in-memory data structure store, used as a database, cache and message broker. It supports data structures such as strings, hashes, lists, sets, sorted sets with range queries, bitmaps, hyperloglogs and geospatial indexes with radius queries.

As Redis is a database that stores everything in memory, the access to it is lightning fast! Much faster than accessing something from disk. This is perfect for what we need.

Use case

This code was developed for Node.js in TypeScript and uses Redis, Bluebird, async/await and a database file from Maxmind for the lookup of IP related data (not included in the example). This is an example of blocking the content or part of it according to the IP address that was detected, comparing it with the regions that are allowed to receive that response (geo-targeting).

If the IP address is not from one of the configured regions, the code would filter the response and respond with only the parts of it that were configured for its region. In order to do that, we need to know all the regions in the country, so they can be compared with the region that was detected for the IP address of the request.

Inside the config directory I have the configuration for Redis inside the cache.ts file:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
const log = require("pino")();
import * as bluebird from "bluebird";
const redis = bluebird.Promise.promisifyAll(require("redis"));

const cacheAddress = process.env.CACHE_HOST || "127.0.0.1";
const cachePort = process.env.CACHE_PORT || 6379;

let cache = redis.createClient(cachePort, cacheAddress);

if (process.env.CACHE_AUTH === "true") {
    cache.auth(process.env.CACHE_PASS);
}

cache.on("error", (err) => {
    if (err.message.indexOf("ECONNREFUSED") !== -1) {
        log.error("Error: The server was not able to reach Redis. Maybe it's not running?");
        process.exit(1);
    } else {
        throw err;
    }
});

export default cache;

Below is an example of a class with a method to load the regions for a country.

The regions need to be loaded in order to be compared with the region of the IP address. If it is not in the cache, the method loads the file from disk into cache, so for the next request it will be serving from cache already (The return statement makes sure the rest of the code is not executed if the content is cached).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
import * as fs from "fs";
import * as util from "util";
import cache from "../config/cache";
const readFile = util.promisify(fs.readFile);

class GeoIP {

    public filterAllowedInstances = async (instances: Array<IConfigResponse>, ipAddress: string): Promise<any> => {
        try {
            if (instances.length === 0) return [];

            let allowedInstances = [];

            let lookup = this.ipLookup(ipAddress);
            let ipGeonameId = lookup.subdivisions[0].geoname_id;

            for (let instance of instances) {
                let response = await this.isIPAllowed(ipGeonameId, instance.regions);
                if (response) allowedInstances.push(instance);
            }

            return allowedInstances;
        } catch (err) {
            if (err.message === "Local ip" || err.message === "Wrong database") {
                return instances;
            }
            Validation.logErrorInternal("GeoIP", err);
            return [];
        }
    }

    private isIPAllowed = async (ipGeonameId: number, instanceRegions: Array<string>): Promise<Boolean> => {
        if (!instanceRegions || instanceRegions.length === 0) return true; // No region is configured, so allow all

        if (this.countryRegions.length === 0) { // Load all the regions for the country
            this.countryRegions = await this.loadRegionsForCountry();
        }

        for (let region of instanceRegions) {
            let filteredRegion = this.countryRegions.filter(regionAll => regionAll.name === region);
            if (filteredRegion.length === 0) continue;
            if (filteredRegion[0].geoname_id === ipGeonameId) return true;
        }

        return false;
    }

    private loadRegionsForCountry = async (): Promise<any> => {
        let cachedRegions = await cache.getAsync("regions");
        if (cachedRegions) return JSON.parse(cachedRegions);

        let countries = await readFile(`${__dirname}/../config/country_region_data.json`, "utf8");
        const parsedCountries = JSON.parse(countries);
        const countryCode = process.env.COUNTRY_CODE;
        const regions = parsedCountries.filter(code => code.countryShortCode === countryCode.toUpperCase())[0].regions;

        await cache.setAsync("regions", JSON.stringify(regions));
        return regions;
    }

}

If thousands of requests would need to have their IP addresses filtered against regions that were in a file on disk or in the database, then it would take a lot of file reading or calls to the database for that, so when the information comes from RAM it’s much faster.

Redis methods usually don’t have the “Async” suffix and use callbacks, but because of bluebird’s promisifyAll method, these methods with this suffix were created with a Promise implementation.

Conclusion

This was an example of loading a file into Redis, so it can be served from cache instead of from the disk. You can use this idea for your project with some other cases, for example when viewing a statistics page, where the backend code needs to calculate many things before responding so the frontend can show it. In this case, the results could be cached on Redis and updated from time to time via a cron job. This way when the user opens the statistics page, the page would load instantly instead of making him wait every time for the calculation.

Do you have another interesting example, any doubt or suggestion? Leave a comment :)

Caching API responses with Redis for faster endpoints was originally published by Jonathas Ribeiro at Jonathas Ribeiro on November 04, 2017.

Token based authentication in Node.js with Passport, JWT and bcrypt

2017-10-21T11:14:08+00:00

When you develop an API, most of the times you’ll need part or all of its endpoints to require authentication. How to do that using Node.js?

A combination of passport.js with JWT and bcrypt is one of the best ways to implement it. Time to go stateless!

This post is based on the authentication implemented in the TODO List API project that you can find here. In this post, all the code is in TypeScript and I expect you to have Node.js and MongoDB already configured on your OS.

What is Passport

As its website states: “Passport is an authentication middleware for Node.js. Extremely flexible and modular, Passport can be unobtrusively dropped in to any Express-based web application. A comprehensive set of strategies support authentication using a username and password, Facebook, Twitter, and more”.

So Passport allows us to integrate login strategies for many kinds of services and they have currently more than 300 strategies that can be just plugged in, ready to be used.

What is JWT

JWT (JSON Web Token) is an open, industry standard RFC 7519 method for representing claims securely between two parties. If we use Passport with a strategy for JWT, then it generates tokens that look for example like this:

eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiYWRtaW4iOnRydWV9.TJVA95OrM7E2cBab30RMHrHDcEfxjoYZgeFONFh7HgQ

A token contains its expiration date and can also contain data we need for checking the user.

The token goes in the Authorization header of the HTTP method call, so the Passport middleware extracts and validates it.

What is bcrypt

In order to save the user in the database and later compare the password he/she enters for generating the authentication token, we need to encrypt his/her password, as it’s not safe to keep passwords with no encryption in the database. Also, it would be a joke to use md5 for that and sha1 recently became unsafe. Enter bcrypt.

From Wikipedia: “bcrypt is a password hashing function designed by Niels Provos and David Mazières, based on the Blowfish cipher, and presented at USENIX in 1999”. Here is more on why you should use bcrypt to hash passwords.

Configuring the project

In order to implement authentication in our API, we need to install the following packages:

npm i bcryptjs dotenv moment express bluebird body-parser mongoose express-validator@2.20.8 jwt-simple passport passport-jwt@2.2.1 --save

And the dev dependencies (As I’ve been using TypeScript, I also install some types):

npm i @types/bcryptjs @types/jwt-simple @types/passport @types/passport-jwt @types/node @types/mongoose @types/superagent chai gulp gulp-apidoc gulp-sourcemaps gulp-typescript gulp-util istanbul@1.0.0-alpha.2 mocha@3.2.0 supertest tslint typescript vinyl-fs --save-dev

The passport-jwt package is Passport’s strategy for JWT.

Inside the package.json file, in “scripts”, let’s leave it like this so istanbul runs integrated with mocha for generating the code coverage report:

1
2
3
4
5
6
"scripts": {
    "pretest": "gulp",
    "test": "istanbul cover node_modules/mocha/bin/_mocha",
    "prestart": "gulp",
    "start": "nodemon bin/server.js"
}

Inside the test directory, create a file called mocha.opts with the following content:

bin/test/*_test.js
--reporter spec

Let’s copy these files and save them to the root of our project:

In the root of our project, let’s create a file called .env if it isn’t there yet. This file needs to have the JWT_SECRET environment variable, with a random hash we need to generate. It’s very important not to repeat this hash among your projects, as it has to be something unique and secure, because this is what is used for generating and comparing the token.

You can generate a hash using the LastPass password generator, for example. I recommend you to use all kinds of characters and to have the length of at least 25. An example of an .env file then would be:

API_BASE='/api/v1/'
DEFAULT_TIMEZONE='Europe/Prague'

JWT_SECRET='ogA9ppB$S!dy!hu3Rauvg!L96'

DB_HOST='127.0.0.1'
DB_PORT='27017'
DB_AUTH='false'
DB_USER=''
DB_PASS=''

In the config directory, let’s create a db.ts file for the database connection:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
import * as mongoose from "mongoose";
(<any>mongoose).Promise = require("bluebird");
let dbName;

switch (process.env.NODE_ENV) {
    case "test":
        dbName = "todo_test";
        break;
    case "production":
        dbName = "todo";
        break;
    default:
        dbName = "todo_dev";
}

const dbAddress = process.env.DB_HOST || "127.0.0.1";
const dbPort = process.env.DB_PORT || 27017;

let options = {
    useMongoClient: true
};

if (process.env.DB_AUTH === "true") {
    options["user"] = process.env.DB_USER;
    options["pass"] = process.env.DB_PASS;
}

mongoose.connect(`mongodb://${dbAddress}:${dbPort}/${dbName}`, options).catch(err => {
    if (err.message.indexOf("ECONNREFUSED") !== -1) {
        console.error("Error: The server was not able to reach MongoDB. Maybe it's not running?");
        process.exit(1);
    } else {
        throw err;
    }
});

Auth tests

Now let’s write some tests for what we need our authentication to do.

Create a file named common.ts inside the test directory:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
process.env.NODE_ENV = "test";

import "mocha";
import { IUser, model as User } from "../models/user";

const express = require("../config/express")();

export const request = require("supertest")(express);

export const chai = require("chai");
export const should = chai.should();

const testUser = { "username": "testuser", "password": "mytestpass" };

const createUser = async (): Promise<void> => {
    const UserModel = new User(testUser);
    await UserModel.save();
};

const getUser = async (): Promise<IUser> => {
    let users = await User.find({});
    if (users.length === 0) {
        await createUser();
        return getUser();
    } else {
        return users[0];
    }
};

export const login = async (): Promise<any> => {
    let user = await getUser();
    return request.post(process.env.API_BASE + "login")
        .send({ "username": user.username, "password": testUser.password })
        .expect(200);
};

And then some tests for authentication in a file called auth_test.ts inside the test directory:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
import { request, login } from "./common";
import { cleanCollection } from "../models/user";

describe("# Auth", () => {
    const endpoint = process.env.API_BASE + "login";

    it("should retrieve the token", () => {
        return cleanCollection().then(res => {
            return login().then(res => {
                res.status.should.equal(200);
                res.body.token.should.not.be.empty;
            });
        });
    });

    it("should not login with the right user but wrong password", () => {
        return request.post(endpoint)
            .send({ "username": "testuser", "password": "anythingGoesHere" })
            .expect(401);
    });

    it("should return invalid credentials error", () => {
        return request.post(endpoint)
            .send({ "username": "testuser", "password": "" })
            .expect(401)
            .then(res => {
                return request.post(endpoint)
                    .send({ "username": "anotherusername", "password": "mypass" })
                    .expect(401);
            });
    });

    it("should return token expired message", () => {
        return request.post(process.env.API_BASE + "tasks")
            .set("Authorization", "JWT eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJleHAiOjE0OTg5Mzk1MTksInVzZXJuYW1lIjoidGVzdHVzZXIifQ.FUJcVCzZTkjDr62MCJj5gvCFvmxewmz2jotiknuVbOg")
            .send({
                name: "Do the dishes"
            })
            .expect(res => res.body.message.should.equal("Your token has expired. Please generate a new one"))
            .expect(401);
    });
});

Now we have tests for some things related to our JWT implementation. We need then to configure Express to use Passport as a middleware.

Auth middleware

Inside the config directory, let’s create a file called express.ts to configure everything related to Express.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
const dotenv = require("dotenv").config();
const express = require("express");
const auth = require("../controllers/auth").default;
const bodyParser = require("body-parser");
const expressValidator = require("express-validator");
const db = require("./db");

export = () => {
    let app = express();

    app.use(bodyParser.json());
    app.use(expressValidator());

    app.use(auth.initialize());

    app.all(process.env.API_BASE + "*", (req, res, next) => {
        if (req.path.includes(process.env.API_BASE + "login")) return next();

        return auth.authenticate((err, user, info) => {
            if (err) { return next(err); }
            if (!user) {
                if (info.name === "TokenExpiredError") {
                    return res.status(401).json({ message: "Your token has expired. Please generate a new one" });
                } else {
                    return res.status(401).json({ message: info.message });
                }
            }
            app.set("user", user);
            return next();
        })(req, res, next);
    });

    const routes = require("../routes")(app);

    return app;
};

In the above code, if our endpoint is “login”, it will not check for authentication (return next()), as that is the endpoint the clients will use for sending the username and password for generating the token they need for the other endpoints. Any other endpoint will go through the authenticate method inside our Auth controller, which is using Passport.

As you see in the code, the Auth middleware must come before the routes are required, as the authentication process needs to happen before them.

Auth controller and route

Inside the controllers directory, let’s create a file called auth.ts with the following content:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
import * as jwt from "jwt-simple";
import * as passport from "passport";
import * as moment from "moment";
import { Strategy, ExtractJwt } from "passport-jwt";
import { model as User, IUser } from "../models/user";

class Auth {

    public initialize = () => {
        passport.use("jwt", this.getStrategy());
        return passport.initialize();
    }

    public authenticate = (callback) => passport.authenticate("jwt", { session: false, failWithError: true }, callback);

    private genToken = (user: IUser): Object => {
        let expires = moment().utc().add({ days: 7 }).unix();
        let token = jwt.encode({
            exp: expires,
            username: user.username
        }, process.env.JWT_SECRET);

        return {
            token: "JWT " + token,
            expires: moment.unix(expires).format(),
            user: user._id
        };
    }

    public login = async (req, res) => {
        try {
            req.checkBody("username", "Invalid username").notEmpty();
            req.checkBody("password", "Invalid password").notEmpty();

            let errors = req.validationErrors();
            if (errors) throw errors;

            let user = await User.findOne({ "username": req.body.username }).exec();

            if (user === null) throw "User not found";

            let success = await user.comparePassword(req.body.password);
            if (success === false) throw "";

            res.status(200).json(this.genToken(user));
        } catch (err) {
            res.status(401).json({ "message": "Invalid credentials", "errors": err });
        }
    }

    private getStrategy = (): Strategy => {
        const params = {
            secretOrKey: process.env.JWT_SECRET,
            jwtFromRequest: ExtractJwt.fromAuthHeader(),
            passReqToCallback: true
        };

        return new Strategy(params, (req, payload: any, done) => {
            User.findOne({ "username": payload.username }, (err, user) => {
                /* istanbul ignore next: passport response */
                if (err) {
                    return done(err);
                }
                /* istanbul ignore next: passport response */
                if (user === null) {
                    return done(null, false, { message: "The user in the token was not found" });
                }

                return done(null, { _id: user._id, username: user.username });
            });
        });
    }

}

export default new Auth();

As you can see, here we set the Passport initialize method, configured the token generation with a validity of 7 days, implemented the login method to be used in the login endpoint and the strategy using JWT, extracting the token from the Authorization header.

We need now a route for the login endpoint, for this to work. Let’s create a file called auth.ts inside the routes directory:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
import Auth from "../controllers/auth";

export = (app) => {
    /**
    * @api {post} /api/v1/login Generate a token
    * @apiVersion 1.0.0
    * @apiName Login
    * @apiGroup Auth
    * @apiPermission public
    * @apiDescription In order to generate a token, you will need to already have a user in the database.
    *
    * @apiParam (Request body) {String} username The username
    * @apiParam (Request body) {String} password The password
    *
    * @apiExample {js} Example usage:
    * const data = {
    *   "username": "test@email.com",
    *   "password": "yourpassword"
    * };
    *
    * $http.post(url, data)
    *   .success((res) => doSomethingHere())
    *   .error((err) => doSomethingHere());
    *
    * @apiSuccess {String} token The token that must be used to access the other endpoints
    * @apiSuccess {String} expires The expiration datetime (YYYY-MM-DDTHH:mm:ssZ)
    * @apiSuccess {String} user The user id
    *
    * @apiSuccessExample {json} Success response:
     *     HTTPS 200 OK
     *     {
     *      "token": "JWT eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9 ... and the rest of the token here",
     *      "expires": "2017-10-28T14:50:17+00:00",
     *      "user": "57e12cab65c0c892381b8b44"
     *    }
    */
    app.post(process.env.API_BASE + "login", Auth.login);
};

This example is ready to be used with apiDoc for generating the API documentation, explained in my previous post.

We also need an index.ts file inside the routes directory, in case you don’t have it:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
export = (app) => {

    // Require the routes files in the routes directory
    require("./auth")(app);

    app.get("/", (req, res) => res.status(200).json({ message: "Welcome to the TODO API. Check the documentation for the list of available endpoints" }));

    // If no route is matched by now, it must be a 404
    app.use((req, res, next) => {
        res.status(404).json({ "error": "Endpoint not found" });
        next();
    });

    app.use((error, req, res, next) => {
        if (process.env.NODE_ENV === "production") {
            return res.status(500).json({ "error": "Unexpected error: " + error });
        }
        next(error);
    });

};

Now that we have the project configured with the packages, the auth tests, auth middleware, auth controller and routes, we need to implement the bcrypt functions for checking the user password before generating the token.

User model

Let’s create the file called user.ts inside the models directory:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
import * as mongoose from "mongoose";
import * as bcrypt from "bcryptjs";

export interface IUser extends mongoose.Document {
    name: string;
    username: string;
    password: string;
    comparePassword(candidatePassword: string): Promise<boolean>;
}

export const schema = new mongoose.Schema({
    name: String,
    username: {
        type: String,
        required: true,
        unique: true
    },
    password: {
        type: String,
        required: true
    }
}, { timestamps: { createdAt: "created_at", updatedAt: "updated_at" } });

schema.pre("save", function (next) {
    bcrypt.hash(this.password, 10, (err, hash) => {
        this.password = hash;
        next();
    });
});

schema.pre("update", function (next) {
    bcrypt.hash(this.password, 10, (err, hash) => {
        this.password = hash;
        next();
    });
});

schema.methods.comparePassword = function (candidatePassword: string): Promise<boolean> {
    let password = this.password;
    return new Promise((resolve, reject) => {
        bcrypt.compare(candidatePassword, password, (err, success) => {
            if (err) return reject(err);
            return resolve(success);
        });
    });
};

export const model = mongoose.model<IUser>("User", schema);

export const cleanCollection = () => model.remove({}).exec();

export default model;

Here we added bcrypt methods to hash a new password when saving or updating a user and implemented a method to compare the password the user will post to the login endpoint when he wants to generate a token. This comparePassword method is used in the Auth controller.

How to create a user for the first login?

The common.ts file inside the test directory does that using the User model we just created, so when our tests run and try to login, if no user exists, it creates a user before accessing the login endpoint for posting the credentials. For a production case you can create a users controller inside the controllers directory or copy the one I have here, then create user routes inside the routes directory or copy the one I have here to make the endpoints available for a web panel to create new users, for example. Or you could create a script that runs in the CLI and uses bcrypt and the User model to create new users in the database. I will not cover user management in this post, as the focus here is the authentication part.

The server file

We need to run our Express config and create a server with it. Let’s create a server.ts file in the root of our project:

1
2
3
4
5
6
7
const app = require("./config/express")();

const port = process.env.PORT || 3000;

const server = app.listen(port, () => {
    console.log(`Express server listening on port ${port}.\nEnvironment: ${process.env.NODE_ENV}`);
});

Running the tests

Now that we configured everything that is used for requiring authentication for our endpoints, we can run the tests and check if the code is working (be sure to have MongoDB running before):

npm test

Calling an endpoint from a client

After you integrate the authentication on your API, you can use your client (javascript on the browser, mobile, desktop, postman, etc) to call the login endpoint and get the token for your user.

Run the project with node, nodemon or pm2 (edit that in the package.json file):

npm start

then:

The response comes with “JWT” before the actual token. This is the token bearer and it is required when sending the token in the Authorization header.

If you are using the TODO List API example, you can use the tasks endpoint sending the Authorization header using your client (Postman in the example), with the token generated in the previous step:

Conclusion

In this post we saw how to implement authentication with Passport, JWT and bcrypt, integrating it to our existing API, then calling it from a client. As our authentication is stateless, when we need to deploy the API behind a load balancer, we don’t need to have sticky sessions and the authentication process is much simpler and easily scalable.

If you have any suggestion, doubt or use something else, leave a comment below :)

Token based authentication in Node.js with Passport, JWT and bcrypt was originally published by Jonathas Ribeiro at Jonathas Ribeiro on October 21, 2017.

Documenting your Node.js API with apiDoc

2017-10-14T09:10:08+00:00

When you are developing an API and one or more developers (frontend, desktop, mobile, etc) will have to integrate their code with it, it’s very important to have it well documented so they know what and how they can use.

For that, in Node.js projects I’ve been using apiDoc, as it is able to generate documentation in HTML from annotations in the source code.

For this post, I’ll use the TODO List API I developed as an example, once more. So you can clone or download it from here.

Routes and annotations

On my post about tests with mocha and code coverage with istanbul, I showed the example of the Task endpoints in our TODO List API:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
import Task from "../controllers/tasks";

export = (app) => {

    const endpoint = process.env.API_BASE + "tasks";

    app.post(endpoint, Task.create);

    app.delete(endpoint + "/:id", Task.delete);

    app.get(endpoint + "/:id", Task.getOne);

    app.get(endpoint, Task.getAll);

    app.put(endpoint + "/:id", Task.update);

};

This represents all our endpoints related to tasks in the system. How can we use them? What should the developers that will consume the API send to each of these endpoints?

So far there is no way for them to know other than looking at the code, which shouldn’t be needed.

Because of apiDoc, we can achieve that with annotations. The way I configure it, is by writing them before each endpoint configured in the files inside the routes directory. If you are not sure of what I’m talking about, check here when I mentioned how I configure and organize my Node.js projects.

With annotations, our Task endpoints (inside routes/tasks.ts) would look like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
import Task from "../controllers/tasks";

export = (app) => {

    const endpoint = process.env.API_BASE + "tasks";

    /**
    * @api {post} /api/v1/tasks Create a task
    * @apiVersion 1.0.0
    * @apiName Create
    * @apiGroup Task
    * @apiPermission authenticated user
    *
    * @apiParam (Request body) {String} name The task name
    *
    * @apiExample {js} Example usage:
    * const data = {
    *   "name": "Do the dishes"
    * }
    *
    * $http.defaults.headers.common["Authorization"] = token;
    * $http.post(url, data)
    *   .success((res, status) => doSomethingHere())
    *   .error((err, status) => doSomethingHere());
    *
    * @apiSuccess (Success 201) {String} message Task saved successfully!
    * @apiSuccess (Success 201) {String} id The campaign id
    *
    * @apiSuccessExample {json} Success response:
    *     HTTPS 201 OK
    *     {
    *      "message": "Task saved successfully!",
    *      "id": "57e903941ca43a5f0805ba5a"
    *    }
    *
    * @apiUse UnauthorizedError
    */
    app.post(endpoint, Task.create);

    /**
    * @api {delete} /api/v1/tasks/:id Delete a task
    * @apiVersion 1.0.0
    * @apiName Delete
    * @apiGroup Task
    * @apiPermission authenticated user
    *
    * @apiParam {String} id The task id
    *
    * @apiExample {js} Example usage:
    * $http.defaults.headers.common["Authorization"] = token;
    * $http.delete(url)
    *   .success((res, status) => doSomethingHere())
    *   .error((err, status) => doSomethingHere());
    *
    * @apiSuccess {String} message Task deleted successfully!
    *
    * @apiSuccessExample {json} Success response:
     *     HTTPS 200 OK
     *     {
     *      "message": "Task deleted successfully!"
     *    }
     *
     * @apiUse UnauthorizedError
    */
    app.delete(endpoint + "/:id", Task.delete);

    /**
    * @api {get} /api/v1/tasks/:id Retrieve a task
    * @apiVersion 1.0.0
    * @apiName GetOne
    * @apiGroup Task
    * @apiPermission authenticated user
    *
    * @apiParam {String} id The task id
    *
    * @apiExample {js} Example usage:
    * $http.defaults.headers.common["Authorization"] = token;
    * $http.get(url)
    *   .success((res, status) => doSomethingHere())
    *   .error((err, status) => doSomethingHere());
    *
    * @apiSuccess {String} _id The task id
    * @apiSuccess {String} name The task name
    *
    * @apiSuccessExample {json} Success response:
     *     HTTPS 200 OK
     *     {
     *        "_id": "57e8e94ea06a0c473bac50cc",
     *        "name": "Do the disehs",
     *        "__v": 0
     *      }
     *
     * @apiUse UnauthorizedError
    */
    app.get(endpoint + "/:id", Task.getOne);

    /**
    * @api {get} /api/v1/tasks Retrieve all tasks
    * @apiVersion 1.0.0
    * @apiName GetAll
    * @apiGroup Task
    * @apiPermission authenticated user
    *
    * @apiExample {js} Example usage:
    * $http.defaults.headers.common["Authorization"] = token;
    * $http.get(url)
    *   .success((res, status) => doSomethingHere())
    *   .error((err, status) => doSomethingHere());
    *
    * @apiSuccess {String} _id The task id
    * @apiSuccess {String} name The task name
    *
    * @apiSuccessExample {json} Success response:
    *     HTTPS 200 OK
    *     [{
    *       "_id": "57e8e94ea06a0c473bac50cc",
    *       "name": "Do the disehs"
    *      },
    *      {
    *       "_id": "57e903941ca43a5f0805ba5a",
    *       "name": "Take out the trash"
    *     }]
    *
    * @apiUse UnauthorizedError
    */
    app.get(endpoint, Task.getAll);

    /**
    * @api {put} /api/v1/tasks/:id Update a task
    * @apiVersion 1.0.0
    * @apiName Update
    * @apiGroup Task
    * @apiPermission authenticated user
    *
    * @apiParam {String} id The task id
    *
    * @apiParam (Request body) {String} name The task name
    *
    * @apiExample {js} Example usage:
    * const data = {
    *   "name": "Run in the park"
    * }
    *
    * $http.defaults.headers.common["Authorization"] = token;
    * $http.put(url, data)
    *   .success((res, status) => doSomethingHere())
    *   .error((err, status) => doSomethingHere());
    *
    * @apiSuccess {String} message Task updated successfully!
    *
    * @apiSuccessExample {json} Success response:
     *     HTTPS 200 OK
     *     {
     *      "message": "Task updated successfully!"
     *    }
     *
     * @apiUse UnauthorizedError
    */
    app.put(endpoint + "/:id", Task.update);

};

As you can see, we have the HTTP method type (post, put, get, delete), the endpoint address, the api version, the kind of permission it needs, the params it requires, the response and the error if the user is not authorized.

In the official website you can check the documentation and available params for the annotations.

Where is this UnauthorizedError coming from?

apiDoc settings

There are some settings that can be done for apiDoc and this UnauthorizedError is the one I usually implement.

Create a file called _apidoc.js inside the routes directory with the following content:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
// ------------------------------------------------------------------------------------------
// General apiDoc documentation blocks and old history blocks.
// ------------------------------------------------------------------------------------------

// ------------------------------------------------------------------------------------------
// Current Success.
// ------------------------------------------------------------------------------------------


// ------------------------------------------------------------------------------------------
// Current Errors.
// ------------------------------------------------------------------------------------------


// ------------------------------------------------------------------------------------------
// Current Permissions.
// ------------------------------------------------------------------------------------------
/**
 * @apiDefine UnauthorizedError
 * @apiVersion 1.0.0
 *
 * @apiError Unauthorized Only authenticated users can access the endpoint.
 *
 * @apiErrorExample  Unauthorized response:
 *     HTTP 401 Unauthorized
 *     {
 *       "message": "Invalid credentials"
 *     }
 */

// ------------------------------------------------------------------------------------------
// History.
// ------------------------------------------------------------------------------------------

I also create another file, also inside the routes directory, called apidoc.json

This file contains the following content (example):

1
2
3
4
5
6
7
{
    "name": "Test API - This is my API",
    "version": "1.0.0",
    "description": "This is my very powerful API",
    "title": "Test API - This is my API",
    "url": "https://testapi.com"
}

These two files will be used by apiDoc when generating the documentation.

Generating the documentation

After writing annotations for every endpoint and configuring our project, we need to configure a task to generate the documentation.

The way I do that is using gulp. Install the required dependencies:

npm i gulp gulp-apidoc --save-dev

Then create a file called gulpfile.js in the project’s root directory. If it’s there already, just add the parts related to apiDoc to it:

1
2
3
4
5
6
7
8
9
10
11
12
13
const gulp = require("gulp");
const apidoc = require("gulp-apidoc");

gulp.task("apidoc", (done) => {
    apidoc({
        src: "./routes",
        dest: "../docs/apidoc"
    }, done);
});

gulp.task("watch", () => {
    gulp.watch(["./routes/**"], ["apidoc"]);
});

You can change the “dest” there to another directory that suits you better. That’s where you want the output to be generated to.

Now all you need to do is to run:

gulp apidoc

After that, you just need to open the index.html file inside the directory you configured in the “dest” above and you’ll see something like this (Click on the image below to enlarge):

So the other developers can generate the same with gulp as well or you can even serve this generated directory via Nginx, for example.

Conclusion

In this post we saw how to document our APIs with annotations and generate the HTML for them using apiDoc.

Do you use another software for documenting your APIs or do you use apiDoc in a different way? Leave a comment below!

Documenting your Node.js API with apiDoc was originally published by Jonathas Ribeiro at Jonathas Ribeiro on October 14, 2017.

Backup your MongoDB databases to Amazon S3

2017-10-07T09:55:08+00:00

I’ve developed a backup script for MongoDB using TypeScript. It’s available as GPL3 here.

If you download or clone the repository, you can choose to run it with Docker (so cron is already configured) or run it without Docker (If you want to configure cron on your host OS by yourself).

The cron example there is the same I wrote about on my previous post about cron with Docker.

In any case, you’ll need to have Node.js and yarn installed.

This script generates a dump from your MongoDB databases, compresses this dump as tar.bz2, generates a hash file for integrity check and then uploads both files to your Amazon S3 bucket.

Configuring

I expect you to have your Amazon Web Services (AWS) account ready and know your access key and secret access key before, also to have a bucket created on S3 in order to store the backup files.

In order to configure the backup script, you need to open the .env file and enter your data there, such as your timezone and AWS credentials.

Cron is configured to run the backup script every day at midnight. If you want to change the time there, change the file called root inside /infra/cron. If you need to change the timezone of the cron container, then open /infra/cron/Dockerfile

Building

In order to build the script and start the cron process with Docker, run the following command:

make

This will run the following steps:

Install the required packages
Run the build script
Build the cron Docker image
Start the cron Docker container
Run all the tests with npm

If you chose not to use Docker, but configure cron yourself on your host OS, then run this instead:

make no-docker

This will run the following steps:

Install the required packages
Run all the tests with npm

Enabling daemon on system startup

In order to start this container automatically if your server reboots:

Change the path inside the file infra/host/etc/systemd/system/mongo-backup.service to be the path where you put the mongo-s3-backup source code.

As root, copy this file to /etc/systemd/system in your server

Reload the daemons:

sudo systemctl daemon-reload

Enable and start it:

sudo systemctl enable mongo-backup
sudo systemctl start mongo-backup

Conclusion

Today we saw how to configure and run the MongoDB backup script.

If you have any idea or suggestion, leave a comment or send a pull request.

Backup your MongoDB databases to Amazon S3 was originally published by Jonathas Ribeiro at Jonathas Ribeiro on October 07, 2017.

Scheduling tasks with cron on Docker

2017-09-30T07:30:10+00:00

When you need to setup a cron job, you can do it using Docker. The best way to do it with Docker is using an Alpine Linux image, as it has only 5MB initial size.

Don’t know what is Docker? Check more about it here.

In this post I expect you to already have Docker and docker-compose installed on your OS.

Creating the image

In the case below I needed an Alpine Linux container with Node.js inside, as I wanted cron to run a Node.js script.

First we create a file called Dockerfile with the following content (yes, with no file extension):

1
2
3
4
5
6
7
8
9
FROM node:8-alpine
MAINTAINER Jonathas Ribeiro <contact@jonathas.com>

RUN apk update && apk add tzdata &&\ 
    cp /usr/share/zoneinfo/Europe/Prague /etc/localtime &&\ 
    echo "Europe/Prague" > /etc/timezone &&\ 
    apk del tzdata && rm -rf /var/cache/apk/*

CMD chown root:root /etc/crontabs/root && /usr/sbin/crond -f

Let’s save this file inside a directory called cron.

The FROM command tells Docker which image we want to use for building our image.

The MAINTAINER command tells Docker who is maintaining this Dockerfile.

The RUN command tells Docker which commands to run on image creation.

It’s very important for your Dockerfile to have the least number of commands possible, as another RUN command for example, would create another layer in the resulting image. That’s why we concatenate the commands with && instead of writing a RUN command for each of them. Simplicity should be the goal.

In the example, I set the timezone to Prague (my current timezone), but you can change it to yours instead, of course.

The CMD command tells Docker what will run inside the container that will be created from this image, which is cron in this case. Exactly what we need.

Configuring cron

Now we need to configure what this image will run using cron.

I created this silly example below just to illustrate what could be done.

Save the content below to a file called root in the same directory where you saved the Dockerfile (the cron directory you created before).

1
2
3
4
5
6
7
8
9
10
11
# ┌───────────── minute (0 - 59)
# │ ┌───────────── hour (0 - 23)
# │ │ ┌───────────── day of month (1 - 31)
# │ │ │ ┌───────────── month (1 - 12)
# │ │ │ │ ┌───────────── day of week (0 - 6) (Sunday to Saturday;
# │ │ │ │ │                                       7 is also Sunday on some systems)
# │ │ │ │ │
# │ │ │ │ │
# * * * * *  command to execute

0 7,19 * * * /usr/local/bin/node /home/node/hello.js >> /var/log/crontest/hello.log 2>&1

Now create a directory called scripts on the same level of the cron directory, not inside of it.

Inside this scripts directory, create our hello.js script with the following content:

console.log("Hello world");

In this example I told cron to execute a JavaScript file using the Node.js executable that comes inside the container and output the response to a hello.log file. This cron command, as you can see in the explanation in the comments, will run every day at 7:00 and 19:00.

Ok, so how to add this to our container?

Docker-compose

I like using docker-compose as it enables me to orchestrate many containers in a simple way, using only one docker-compose.yml file.

With docker-compose you can for example create a file with all the containers you need and then run docker-compose up for it to start all of them at once, instead of dealing with docker run and passing all parameters every time, etc.

Let’s create a file called docker-compose.yml with the following content in same level as our scripts and cron directories:

1
2
3
4
5
6
7
8
9
version: "2"
services:
    cron:
        build: cron
        container_name: crontest
        volumes:
            - /var/log/crontest:/var/log/crontest
            - cron/root:/etc/crontabs/root
            - sripts:/home/node

Spacing is very important here. Visual Studio Code can help you with that, as it detects yml files.

The cron directory is informed in the build atribute there, so docker-compose will build the image using the Dockerfile we created inside the cron directory.

In “volumes”, the crontest directory inside your host OS is being mounted inside the container using the same path there, so when our script outputs to hello.log as configured above, the log will be easily accessible from the host OS instead of having to enter the cron container to read it. If this directory doesn’t exist on your host OS, docker-compose will create it.

For the next lines configured inside “volumes”, the root file we created inside the cront directory is being mounted inside the container in the /etc/crontabs/root path. On the 3rd line, everything from the scripts directory is being mounted inside /home/node inside the container, as this is the home directory this Node.js container comes with.

Instead of mounting these files inside the container using docker-compose, you could have added them on image creation using the ADD command, but then you’d have to rebuild the image if you needed to change any of these files and the resulting image would also have more layers. Because of that, I prefer to mount them from the host OS instead, so if I need to modify any of these files, they are not builtin with the image in any way.

After saving the docker-compose.yml file, you can run the command to build our docker image and start our container from it:

docker-compose up

This way all the commands will show up on your terminal and you won’t be able to use this terminal for anything else. If you add -d in the end of the command, then it runs as a daemon and your terminal will be free, but you wont see any output about the image build process or when it’s running.

If you run docker-compose witout the daemon option, you just need to use Ctrl + c when you want to stop the containers specified in your docker-compose.yml file. If you run as a daemon, you can run docker-compose stop to stop the containers and then if you want to remove them you can run docker-compose rm

In order to list the current containers:

docker ps -a

And the images:

docker images

Running on startup

After creating the image and checking that everything works as expected, we need to create a startup script so if our OS gets restarted the cron container runs automatically.

The servers I configure are usually running Debian Linux, my personal notebook runs Arch Linux and at work I run Manjaro. All these Linux distributions come with systemd, so I’ll show an example of how the startup could be configured for that.

Let’s save a file called docker-infra.service inside /etc/systemd/system/ with the following content:

1
2
3
4
5
6
7
8
9
10
11
12
[Unit]
Description=Docker infra
Requires=docker.service
After=docker.service

[Service]
Restart=always
ExecStart=/usr/local/bin/docker-compose -f /home/jon/crontest/docker-compose.yml up
ExecStop=/usr/local/bin/docker-compose -f /home/jon/crontest/docker-compose.yml stop

[Install]
WantedBy=default.target

You need to change the path to your docker-compose.yml file to the one where you saved yours inside your server, of course.

Then let’s reload the system daemons:

sudo systemctl daemon-reload

and enable our newly created daemon to run on system startup:

sudo systemctl enable docker-infra

If you need to stop or check the status of the daemon that controls our docker container, just change the enable in the systemctl command above to stop or status and that’s it.

Conclusion

In this post we saw how to create a Docker image to use as cron to run Node.js scripts. Then we saw how to use docker-compose and create a daemon for it to run on system startup.

If you have any suggestion or doubt, let me know in the comments below :)

Scheduling tasks with cron on Docker was originally published by Jonathas Ribeiro at Jonathas Ribeiro on September 30, 2017.

My favorite fitness apps for Android

2017-09-24T08:20:08+00:00

Some apps are very useful when you decide to keep track of your fitness activities and habits.

Here I list some apps I have used and others I still use and recommend.

Runkeeper

This is one of the most well known apps out there. If you like taking long walks, running or biking, it’s perfect for that. It tracks your speed, pace and distance (using your phone’s GPS). In the end of your activity it shows you a map with where you’ve been through and it also allows you to add a photo to your activity so you can create good memories for yourself to look at in the future or for your friends. Talking about friends, it’s even better when you have many friends added there, so you can compete with each other and set goals in the app and website.

Get it on Google Play

Jefit

In my opinion this is the best workout app ever. I always use it when working out at the gym, as I’ve saved all my trainings and routines there. It allows you to create new workout plans by days and body parts, it has several registered exercises you can add to your routines and when you start a workout, you can log the weights you’re using so you keep track of your progress. After each exercise it starts a counter for your rest time (60 seconds or 30 seconds, for example) before it shows you the next exercise on your list. The free version comes with ads but the Pro version is worth it.

Get it on Google Play

HIIT Interval Training Timer

If you need to burn fat, HIIT (High-intensity interval training) is the best way there is. Many articles and studies say it’s better than steady state cardio (running having distance as a goal, for example). On a HIIT workout you run as fast as you can for some seconds, then go slowly, then repeat that for some series. This app helps with that by counting the intervals and sounding a whistle when they begin.

Get it on Google Play

Fitocracy

As their description states: “Track your workouts, earn points, unlock achievements, beat quests, and slay the laziness dragon”. This app and website represent a community of people who are working hard every day to level up. Fitocracy works with a lot of gamification, so you can earn badges and level up according to your points. It’s great for when you need some extra motivation for your achievements. It also has many challenges you can complete in order to earn more badges and points. It integrates with Runkeeper but, unfortunately, not with Jefit. I’ve always used Jefit, so I had to log the exercises to Fitocracy manually, back when I was using it more often.

Get it on Google Play

Stretch

Stretching is always important and it brings many benefits to your health. This app has many stretching poses arranged in categories for training various body parts. You can pick a few stretching poses as a daily routine.

Get it on Google Play

Muay Thai

I don’t believe in this “learn Muay Thai by yourself” crap, but this is a good app if you need some suggestions for a trainig with a punching bag or with a friend, also for some shadow boxing.

Get it on Google Play

Hydro

It is very important to drink a certain amount of water every day and it even helps you to lose weight. This app allows you to set hourly reminders so you are able to achieve your goal for the day. If you do not drink the correct amount of water, this is the perfect application, which will restore your healthy habits and will help you care for proper hydration.

Get it on Google Play

MyFitnessPal

This is a great app for tracking what you eat and counting calories intake. It comes with a barcode scanner and it has a huge database of products, so it will almost certainly identify what you ate if you scan its barcode. You just need to put there the product and the amount and it will count how many calories, protein, carbs, etc. I tested it with products in Brazil and products in Czech Republic and the barcode scan worked very well, identifying many products. It also has a log for how much water you drink during your day and you can integrate the app with Google Fit or with Fitbit so it sends the calories you log to these apps.

Get it on Google Play

Google Fit

This app allows you to track your walking, running or biking activities using your phone or Android wear. It shows real time stats so you can see your speed, pace, distance, etc. It can be integrated with Runkeeper, MyFitnessPal and others. You can set goals based on steps, time, distance, calories burned and reach your fitness goals. Last time I used it was long ago, so it seems to have evolved pretty well. Nowadays I don’t use it anymore as I use Fitbit instead.

Get it on Google Play

Fitbit

I just bought a Fitbit Charge 2 band and I’m really enjoying it. This app works integrated with your fitness band by Fitbit and it tracks your steps, distance (using your phone’s GPS), calories burned, allows you to set daily goals and has some gamification as well, with badges. It also tracks how well you sleep so you can set goals to sleep better. You can also tack your weight changes, body fat, heart rate, water and food intake. On the fitness band you can check the current time, your current heart rate, start an activity, start the stop watch, have a 2 minutes relaxation exercise and set silent alarms. A silent alarm is an alarm that vibrates the fitness band instead of sounding any alarm, so you can set reminders or even set it to wake you up with no sound (doesn’t work for me, as I keep sleeping after that). It also sets a reminder to get you ready to go to bed some minutes before the time you configured that you plan to start sleeping and it also shows on the fitness band screen notifications from your favorite messenger (Facebook messenger, whatsapp, whatever you use more often) and phone calls.

Get it on Google Play

Conclusion

Do you know another app that could be included on this list? Please let me know in the comments below :)

My favorite fitness apps for Android was originally published by Jonathas Ribeiro at Jonathas Ribeiro on September 24, 2017.

One year working with Node.js after PHP and .NET - Challenges, comparisons and improvements

2017-09-16T08:20:08+00:00

In my software development career I’ve worked so far with both frontend and backend development, also with Desktop apps development (.NET) and Hybrid mobile development (Ionic Framework), but always enjoyed and focused more on the backend. I started my career in 2008 with PHP, so around this time JavaScript was mostly for creating alert messages, form validation and a bit of user interaction. Who could tell it would become what it is today?

In 2010 I’ve worked with Java web for a while, then PHP again, then around 2013 I changed to .NET for desktop and web, then AngularJS (Ionic Framework), then PHP again and then Node.js (since 2016). So this year (2017) I complete 1 year working with Node.js and related technologies. It has been a great journey so far, full of ups and downs.

Asynchronous vs Synchronous

When you write your code in PHP, .NET or Python, the instructions you write are followed line by line when the code is executed. That doesn’t happen with JavaScript when you write a function that reads a file or fetches something from the database, for example. That function is called and the code continues its execution on the next line before that function is finished, as it works asynchronously. This can be mind blowing sometimes for someone who is coming from other languages, so it was the biggest change for me when starting to work with Node.js. Even though Node.js has many methods that have synchonous versions, it’s not recommended to use them, as the Node process would be waiting for the method execution to finish before doing anything else. So it’s always recommended to use async functions in order to allow the event loop to work in an optimized way.

You get used to it after a while, but the callback hell was probably what made all look more confusing.

Callbacks vs Promises vs async/await

As I started learning more about Node.js, all the courses I took online and books I read were showing how to do everything with callbacks. Sometimes there were callbacks inside callbacks, creating what is called the “callback hell”, code with many levels of indentation.

1
2
3
4
5
6
7
8
9
const fs = require("fs");

fs.readFile("a.txt", "utf8", (err, a) => {
     fs.readFile("b.txt", "utf8", (err, b) => {
           fs.writeFile("ab.txt", a + b, () => {
               console.log("we are done");
           });
     });
});

As a result, the first system I wrote in Node.js was full of callbacks. When I had to use a for loop with callbacks inside, then it was an even bigger problem. So I found out I could use the async module for that, to orchestrate these callbacks and loop through them and that way the code would wait instead of going to the next line and ignoring the loop.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
const async = require("async");
const fs = require("fs");

const concatFiles = (callback) => {
    let directories = ["dir01", "dir02", "dir03", "dir04"];

    async.each(directories, (dir, loopCallback) => {
        fs.readFile(`${dir}/a.txt`, "utf8", (err, a) => {
            fs.readFile(`${dir}/b.txt`, "utf8", (err, b) => {
                fs.writeFile(`${dir}/ab.txt`, a + b, () => {
                    console.log(`We are done for directory ${dir}`);
                    loopCallback();
                });
            });
        });
    }, (err) => {
        if (err) {
            callback(err);
        } else {
            callback(null, true);
        }
    });
}

concatFiles((err, res) => {
    if (err) {
        // Do something with the error
    } else {
        console.log("All done!");
        // Do something else here that depends on the ab.txt files
    }
});

Promises looked like a cleaner alternative for the callbacks, so I started refactoring parts of the code to use them and they looked indeed much cleaner.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
const fs = require("fs");

const readFileContent = (filePath) => {
    return new Promise((resolve, reject) => {
        fs.readFile(filePath, "utf8", (err, data) => {
            if (err) return reject(err);
            return resolve(data);
        });
    });
}

const writeFileContent = (filePath, content) => {
    return new Promise((resolve, reject) => {
        fs.writeFile(filePath, content, (err) => {
            if (err) return reject(err);
            return resolve(true);
        });
    });
}

const concatFiles = () => {
    let concatContent = "";
    return readFileContent(`a.txt`)
        .then(a => {
            concatContent = a;
            return readFileContent(`b.txt`);
        })
        .then(b => {
            concatContent += b;
            return writeFileContent(`ab.txt`, concatContent);
        });
}

concatFiles().then(res => /* etc etc */ );

Then on ES7 we finally were allowed to make our code look normal like in some other languages, with instruction below instruction, instead of instruction inside instruction. When you have less levels of indentation, your code looks clearer and simpler. So I started refactoring all code to async/await and using util.promisify. Now it is much shorter and looks much simpler and easier to understand. (Code refactored by alsiola on reddit)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
const fs = require("fs");
const util = require("util");
const readFile = util.promisify(fs.readFile);
const writeFile = util.promisify(fs.writeFile);

const concatDir = async (dir) => {
    const [a, b] = await Promise.all([
        readFile(`${dir}/a.txt`, "utf-8"),
        readFile(`${dir}/b.txt`, "utf-8")
    ]);

    await writeFile(`${dir}/ab.txt`, a + b);
}

const concatFiles = async () => {
    try {
        const dirs = ["dir01", "dir02", "dir03", "dir04"];
        await Promise.all(
                dirs.map(concatDir)
        );
        return "All Done!";
    } catch (err) {
        return err;
    }
}

const doSomethingElse = async () => {
    await concatFiles();
    // Do something else here that depends on the ab.txt files inside every directory
    // Return a promise
}

It is easier to work with that on Node.js than on the frontend, as you’ll only need to update Node version to 8 or later, while on the frontend you’d have to use Babel to transpile to older ECMAScript versions in order to support older browsers that don’t have async/await implemented in their old versions.

MongoDB and the NoSQL paradigm

This database has nothing an everything to do with Node.js. It can be used with many other languages as well, but it’s the most used one by Node.js developers, forming the MEAN (MongoDB, Express, AngularJS, Node.js) or MERN (MongoDB, Express, React, Node.js) stack. Before starting to work with Node.js, I had used MySQL, SQL Server, Oracle and studied about PostgreSQL, which are all relational databases. So this was also a change of paradigm for me, as MongoDB is a NoSQL, non relational database. It doesn’t have queries, tables nor relationships. In the beggining I ended up modelling the database with the relational paradigm in mind, which didn’t work well for MongoDB (got really slow), so then I learned better and had to model it again to use the NoSQL way. Instead of queries with joins, I had to learn about the Aggregation framework, which works with pipelines filtering the results from the database. In my opinion that looks more confusing than any SQL query.

Vagrant and Docker

Some years ago while working with PHP, I started using Vagrant with Puppet for provisioning development environments, but it would use a full virtual machine with Virtualbox for that with everything in one place and most of times it would become huge. Since starting to work with Node.js I also started using Docker, which I wanted to start using long before already. Docker is much lighter and not a full virtual machine, but a container, so it can be used for development and I started using it even for production in order to make the deployment process more automatic and much simpler. The same environment you have on development you have on production, so this takes away the “works on my machine” excuse.

Deployment challenges

This part is where I had the biggest challenge. Some years ago when I was studying about Ruby on Rails, I wanted to learn how to deploy it to production, as what I had done the most before was LAMP (Linux, Apache, MySQL, PHP). I found out about Phusion Passenger at that time, and that many people were using it to deploy their Ruby on Rails systems.

When I started thinking about how I would deploy Node.js, my company wanted to deploy to a Debian Linux server (IaaS) instead of to a managed platform like Heroku or Openshift (PaaS). I saw no problem with that, as I had been using Linux since 2006 and had already configured many LAMP servers before. So I rememberd about Phusion Passanger, went to look for it and saw that they also supported Node.js. I found out they had a Docker image as well, with everything installed and ready to be used on production, so I went with that. I read through their whole documentation for Node.js and the best way to deploy Passenger integrated with Nginx for Node.js.

Their documentation stated that in order to deploy in an optimized way for thousands of simultaneous requests, several Passenger instances should be started, according to the amount of RAM available on the server. So following a calculation they had on their documentation, I configured the number of Passenger instances according to the amount of RAM to handle the load. Then it’s when the problems started to happen. When we started testing it with thousands of simultaneous clients, the Passenger instances would start handling the load but would also start bloating the RAM. After a while all the RAM was full and the server started swapping, until it couldn’t take it anymore and would shutdown. We tried increasing the amount of available RAM several times, but it would use as much RAM as it had available.

Looking into the problem and discussing it on some IRC channels, I realised the issue was that Passenger’s deployment documentation was written for several languages and technologies (As Passenger can be used for Rails and Python as well), not specifically for a Node.js system, so that’s why it had that kind of calculation there to discover the number of instances that should be spinned in cluster mode. Because of that configuration, the CPU was becoming crazy with the context switching because of so many requests and the RAM was getting bloated. For Node.js, a non multi-thread technology, it should be maximum 1 process per CPU core.

After researching more and more about Node.js deployment, I decided to change from Passenger integrated with Nginx (all inside one Docker container) to a pm2 Docker container with a separate container for Nginx in front of it as a reverse proxy. Passenger is written in C++ and its Docker image is based on Ubuntu with Nginx and even Redis or Memcached inside, which is a wrong way of using Docker, putting everything inside the same container as if it was a virtual machine with everything you need. The pm2 Docker image comes only with pm2 and is based on Alpine Linux, which has only 5mb size, so it’s much more optimized.

After this change of process manager for Node.js, I also decided to split the monolith API into some microservices and run a pm2 docker container for each of these microservices, with Nginx proxying to the right address inside the network according to the accessed endpoint. The amount of pm2 instances to handle the load was configured not to be bigger than the amount of CPU cores the server has, so this time the server handled the load beautifully and consumed not even 10% of the available RAM, even under heavy load.

Resources I used for learning Node.js

Some of the resources I’ve used for learning Node.js are listed below:

Courses on alura.com.br (Brazilian Portuguese). If you understand Portuguese, I always recommend these guys.
Lynda’s course Node.js Essential Training
Udemy’s course Master NodeJS: The Complete Front-End Developer Course
Pluralsight - Building Web Apps With Node.js
Chalkstreet - Node.js: Learn Node.js from scratch
Coursera - Server-side Development with NodeJS: The Hong Kong University of Science and Technology
Book from Casa do Codigo: Aplicações web real-time com Node.js - Caio Ribeiro Pereira (Brazilian Portuguese)
Node.js documentation
Google

I can recommend Alura (if you know Portuguese) or Pluralsight for anything you need to learn, actually. They are absolutely amazing. I also strongly recommend using TypeScript, as it helps you organize your code much better than pure Javascript.

Conclusion

These were some of my experiences so far with Node.js and some comparisons with other technologies I’ve worked with.

In my opinion Node.js courses could cover more about deployment, but perhaps that’s more related to insfrastructure/DevOps.

Do you work with Node.js or are planning to? Leave your comment below :)

One year working with Node.js after PHP and .NET - Challenges, comparisons and improvements was originally published by Jonathas Ribeiro at Jonathas Ribeiro on September 16, 2017.

Jonathas Ribeiro

How I replaced Dropbox Camera Uploads with an automated photo sync pipeline

Before

What I wanted instead

The automated pipeline

Small pipeline diagram

Example exported filesystem structure

Why this is better

Final setup

Running two Zigbee networks side by side in Home Assistant

Why I did not migrate everything

Final architecture

Network 1: ZHA

Network 2: Zigbee2MQTT

Hybrid architecture diagram

Prerequisites

How Zigbee2MQTT integrates with Home Assistant via MQTT discovery

Zigbee2MQTT baseline config

Adapter type and firmware: why ember matters

Channel planning and RF stability

Pairing workflow for the second network

Real-world example: Tuya TS0041 button

What this hybrid model gives me

Lessons learned

Automating winter humidity control with Home Assistant and dehumidifiers

Why humidity control gets tricky in winter

Final hardware setup

Main bedroom automation logic

ON automation

OFF automation

Tank-full / fault detection for Xiaomi units

Children’s bedroom and bathroom strategy

Children’s bedroom

Bathroom (Comfee)

The 45% experiment that failed

Reliability patterns that mattered most

Final result

How I made my physical mailbox send me real-time push notifications

The physical constraints

High‑level architecture

What I tried first

Direct pairing

Normal Zigbee repeater

Alternative technologies

The key idea

Hardware components

Motion sensor

Zigbee router

Full hardware list

Critical problem: Power bank shutdown

Final fix: USB keep‑alive module

Architecture update

Step-by-step implementation

Software architecture

Motion automation

Router reliability monitoring

Why availability sensors were not enough

Stale sensor in configuration.yaml

Offline notification automation

Back online automation

Power consumption considerations

Final result

Why this matters

Lessons learned

Future improvements

Optimizing File Upload Architecture with AWS S3 Presigned POST

Why This Is a Problem

Two Upload Architectures Compared

🔸 Architecture 1: Upload with Nginx

🔸 Architecture 2: Upload via API Gateway + Load Balancer

✅ The Solution: Presigned POST to S3

How It Works

Benefits of Presigned Uploads

Example Server-Side implementation

Example Client Implementation

When Should You Use Presigned POST?

Summary

More info about the AWS feature

Creating a CV in LaTeX: Clean, Structured, and Versioned

Why LaTeX?

Adapter type and firmware: why `ember` matters

Stale sensor in `configuration.yaml`