Nothing motivates a migration quite like a sunset notice. A few weeks after we moved our client's CI/CD from CodeShip to GitHub Actions, this appeared in the console:

Turns out we migrated just in time.

Why We Moved

Moving away from CodeShip was a matter of improving developer experience and providing a more consistent release experience for our client. While CodeShip Basic offered simplicity, GitHub Actions has better documentation and a larger ecosystem. In addition, CodeShip was outdated and lacked modern CI/CD features.

1. Inconsistent failures due to resource restraints. Limited resources in the CI made for a sluggish headless browser, inability to run tests in parallel, and unpredictable build times.
2. Poor debugging experience. CodeShip's SSH debugging feature sounded useful until we discovered the SSH session wasn't the container that ran our build. Build artifacts, test screenshots, and application logs aren’t there for inspection. You're debugging a clean environment that may or may not reproduce your issue.
3. IaC locked behind Pro tier. CodeShip Basic's configuration lives entirely in the UI. If you wanted to version control your build triggers, environment variables, or deployment steps, that would require a Pro license. This meant no audit trail for configuration changes, no ability to review CI/CD changes in pull requests and tribal knowledge about build settings that disappeared when team members left.
4. Limited cache control. CodeShip's caching is remarkably primitive. You dump files into $HOME/cache, and that's it. No cache keys. No expiration control. No branch isolation. The only way to invalidate the cache is through the UI.
5. Limited Conditional Logic. GitHub provides rich conditional expressions that can use paths, commit messages, PR labels and other context. CodeShip provides only a few options for choosing branches or PR events.

The Writing Was On the Wall

A few weeks after our migration, CodeShip's sunset notice appeared in the console. While we didn't know the exact timeline, CodeShip's stagnation was evident: outdated documentation, missing modern CI/CD features, and a developer experience that hadn't evolved in years. The sunset notice validated what we already knew—it was time to move on.

Starting Point

CodeShip Basic operated on a straightforward premise: write shell commands in a text input, and they run on build triggers. Since configuration lived in the UI, we tracked our actual build logic in versioned shell scripts.

The environment came pre-configured with common tooling: nvm for Node.js, rvm for Ruby, JDK version switchers, and a running PostgreSQL instance on the default port. This "batteries included" approach reduced initial setup but limited flexibility.

Build steps and deployment pipelines were both configured through the UI:

Our typical CodeShip workflow looked like this:

1. Setup commands - Set language versions, install dependencies
2. Test commands - Run test suites (sequentially, due to resource constraints)
3. Deployment - Push artifacts to AWS Elastic Beanstalk via UI-configured deployment step

This simplicity was CodeShip's strength and its limitation. When our needs outgrew what the UI could express, we hit a wall.

Mapping CodeShip → GitHub Actions

Migrating our shell files to GitHub action’s workflow syntax was a significant task. Theoretically, we could have just copied our scripts into a single multiline step. But we would be missing out on core GitHub Actions features. Instead of a simple shell script, GitHub Actions uses a YAML configuration file. Nearly everything about your workflow is declared in this file, from test commands to deployment commands, even container image configuration. Here are the major migrations that were made:

Dependencies

In CodeShip, we set the Ruby version using the included rvm version manager and install gems with the bundler gem

rvm use 3.2.2
gem install bundler
bundle package --all

In Github Actions, we use an official action called ruby/setup-ruby@v1

- uses: ruby/setup-ruby@v1
  with:
    ruby-version: 3.2.2
    bundler-cache: true

A similar mapping exists for NodeJS and it’s dependencies.

PostgreSQL

In CodeShip, a PostgreSQL server is available on the default port. In GitHub Actions, we setup a service with a postgres image:

services:
  postgres:
    image: postgres:14
    ports:
      # Maps tcp port 5432 on service container to the host
      - 5432:5432
    env:
      POSTGRES_HOST_AUTH_METHOD: trust

Elasticsearch

Elasticsearch required special handling due to our parallel test setup. Our test suite spawns multiple Elasticsearch processes—one per parallel worker—which means we need the Elasticsearch binary available on the PATH rather than a single service container.

We download the Elasticsearch tarball and cache it between builds to avoid repeated downloads:

- name: Set Path
  run: |
    ES_HOME="$HOME/.cache/elasticsearch-${ES_VERSION}"
    echo "ES_HOME=$ES_HOME" >> "$GITHUB_ENV"
    echo "$ES_HOME/bin" >> "$GITHUB_PATH"
    
- name: Cache Elasticsearch
  uses: actions/cache@v4
  with:
    path: ${{ env.ES_HOME }}
    key: ${{ runner.os }}-es-${{ env.ES_VERSION }}
    
- name: Install Elasticsearch
  run: |
    if [ -e $ES_HOME/bin/elasticsearch ]; then
      echo "Elasticsearch found in cache"
    else
      echo "Elasticsearch not found in cache"
      mkdir -p "$ES_HOME"
      curl -sSLO <https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-${ES_VERSION}-linux-x86_64.tar.gz>
      curl -sSLO <https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-${ES_VERSION}-linux-x86_64.tar.gz.sha512>
      shasum -a 512 -c elasticsearch-${ES_VERSION}-linux-x86_64.tar.gz.sha512
      tar -xzf elasticsearch-${ES_VERSION}-linux-x86_64.tar.gz
      mv elasticsearch-${ES_VERSION}/* "$ES_HOME/"
    fi

Deployment

CodeShip provided a UI for configuring deployment whereas GitHub Actions expects this to be declared in a workflow. Our deployment involves uploading a zip file to AWS S3 and triggering a deployment in Elastic Beanstalk. Thankfully, a community action exists for this case.

- name: Deploy to Elastic Beanstalk
  uses: einaregilsson/beanstalk-deploy@v21      
  env:
    EB_ENV_NAME: ${{ github.ref == 'refs/heads/master' && vars.EB_ENV_NAME_PRODUCTION || vars.EB_ENV_NAME_STAGING }}
    EB_APP_NAME: ${{ vars.EB_APP_NAME }}
  with:
    aws_access_key: ${{ secrets.AWS_ACCESS_KEY_ID }}
    aws_secret_key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
    application_name: ${{ env.EB_APP_NAME }}
    environment_name: ${{ env.EB_ENV_NAME }}
    region: us-east-1
    version_label: "github-deployment-${{ github.sha }}"
    version_description: ${{ env.VERSION_DESC }}
    deployment_package: ${{ env.ZIP_FILE }}
    existing_bucket_name: elasticbeanstalk-us-east-1-xxxxxxxxxx
    use_existing_version_if_available: true

Debugging

CodeShip supported little in the realm of debugging builds. There was an option to SSH into a container but it’s not the instance that the tests ran on. This means that any build assets, logs, or screenshots were not available in the SSH session. In Github Actions, a community member offers an action called mxschmitt/action-tmate. This action creates a tmate session and allows you to SSH into it. You can place this action at any step in a job to inspect and debug. This proved extremely useful during the migration.

- name: Step that requires inspection
	run: ...
	
- name: Setup tmate session
  uses: mxschmitt/action-tmate@v3
  
# The workflow continues after the session ends.  
- name: Next Step
	run: ...

Workflow Highlights

Some things to note about the new workflow:

1. More Infrastructure as Code. This centralizes our process in our codebase and allows us to track with version control. Things that moved from the UI console to IaC include build triggers, environment variables, container images, and artifact deployments.
2. Different runners for different jobs. For running tests, we use a Linux instance with 8 cores to run our tests in parallel. For other things like building assets, deploying artifacts or invalidating caches, we use the default runner. GitHub offers 3,000 minutes/month of free actions using the default runners in private repositories and we want to take advantage of that.
3. Avoid compiling twice. We compile assets in staging/production mode at the beginning of the workflow and use that for testing. This saves time and also makes our testing environment more similar to production. Later jobs access the build artifact using GitHub Action’s official actions/upload-artifact and actions/download-artifact.

# At the end of the build job
- name: Create deploy artifact
  run: zip -r "$ZIP_FILE" . -x "*.git*" "log/*" "tmp/*" "node_modules/*" "vendor/bundle/*"
- name: Upload artifact
	uses: actions/upload-artifact@v4
  with:
    name: ${{ env.ZIP_FILE }}
    path: ${{ env.ZIP_FILE }} 
    
# ....

# At the beginning of both test and deploy jobs
- name: Download and unpack artifact to workspace
  uses: actions/download-artifact@v4
  with:
    name: ${{ env.ZIP_FILE }}
    path: .

4. Concurrency control. This basically cancels any in-progress build when there is a new trigger on the same branch.

concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

Outcome

This migration was about more than just replacing one tool with another. It was about rethinking our delivery pipeline to achieve faster feedback, reproducible builds, and first-class debugging.

The results validated the effort:

1. Faster builds. On CodeShip, our full pipeline duration ranged from 24 to 48 minutes. Now, a full pipeline takes about 12-14 minutes. This is mainly due to increased parallelization with larger runners.
2. Smaller Bill. A similarly sized image in CodeShip Pro would have cost $299/mo at the time. With GitHub Actions, we pay based on minutes and our bill last month was around $47.
3. Zero resource-related failures since the migration. No more flaky builds!
4. Actually debugging builds instead of making guesses because the build is not reproducible.

And we’re just getting started! There is still more to explore in GitHub Actions like matrix builds, reusable workflows and automated dependency updates, but for now, we're thrilled with the results.

TailwindCSS does a fantastic job of organizing malleable configuration for development teams and allows for building out reusable design systems, allowing new developers to get up and running quickly. But because TailwindCSS entirely circumvents a large part of the complexity of CSS (the whole "cascade" part), it can't necessarily always match the power of CSS pound-for-pound. But there's a cool upcoming CSS feature that will soon close one of the more problematic capability gaps that I've encountered.

The problem

If you visit the TailwindCSS documentation for how to represent dynamic theming (e.g. dark & light mode), you'll see an arguably misleading example.

This shows the TailwindCSS documentation site, which by default renders in either light-mode or dark-mode depending on the viewer's system theme as reported by their browser. But inside of the docs site, there is an example with associated code for how to handle a light-theme as opposed to a dark theme with code to match. This might lead you to assume that it is safely possible to nest content of one theme inside of content of another theme. As that's what must be happening on the TailwindCSS documentation site, as you can see the dark-mode website rendering a light-mode card.

The problem here is that the TailwindCSS site is actually lying to the reader. The code below the example is not actually what's running on the example itself at all. If you inspect each card, you'll see that they are actually entirely different component instances with entirely separate TailwindCSS classes, not using a dark: or light: variant system at all.

As it turns out, nesting themes in TailwindCSS hasn't actually been possible historically. TailwindCSS's recommended solution to multiple theme variants is all-or-nothing, only allowing one active theme at a time. The recommended configurations look like one the following two options.

Side note: The configurations seen in the documentation exclusively uses a shorthand syntax equivalent to what's seen below, but I find the expanded syntax to be more explanatory to what a variant selector is representing. This expanded syntax is only necessary if your variant uses multiple selectors with each variant needing multiple instances of @slot.

@custom-variant dark {
  @media (prefers-color-scheme: dark) {
    @slot;
  }
}
@custom-variant light {
  @media (prefers-color-scheme: light) {
    @slot;
  }
}

This first option prescribes the proper theme to the user based on their system settings. This is effectively how the TailwindCSS documentation site works.

@custom-variant dark {
  &:where(.dark, .dark *) {
    @slot;
  }
}
@custom-variant light {
  &:where(.light, .light *) {
    @slot;
  }
}

This option would allow the theme choice to be made and dynamically updated by users, by assigning a .dark or .light class to a root document element, e.g. <body>.

And of course, you can combine both options by prescribing the default applied class based on the result of calling window.matchMedia("(prefers-color-scheme: dark)") from JavaScript.

However, all of these solutions still have the same problem of not supporting nesting. Of course, the @media queries all represent a document-wide setting for your whole page. But the class solution also has the same limitations. You might think that you could nest use of the theme classes, e.g.

<body class="light">
  <CardComponent />
  <CardComponent class="dark" />
</body>

However, the problem here is that the <CardComponent> with the .dark class is also underneath the .light class. So the styles of both variants will apply. This creates undefined behavior in TailwindCSS where the styles which actually get applied is whichever classes are resolved last in the generated output CSS, and TailwindCSS cannot guarantee a generation order. Which means that in this example you'll likely get a combination of the styles of multiple themes. The problem is that CSS has historically lacked a selector to identify proximity, e.g. "prefer this selector because this class is closer to the target element."

The solution

A somewhat recent CSS update has added a new feature called Container Queries. This new tool gives developers a bunch of cool new powers for selectors in CSS. However, the specific features which I've found to be the most revolutionary is Container Style Queries. This API gives us the ability to create a selector that applies styles based upon the value of a particular property. And because the values of properties are inherited by default, this means that we have the ability to bypass the former limitation and do proximity-based-selectors!

So in TailwindCSS, we can create this new configuration!

.dark {
  --theme: dark;
}

@custom-variant dark {
  @container style(--theme: dark) {
    @slot;
  }
}

.light {
  --theme: light;
}

@custom-variant light {
  @container style(--theme: light) {
    @slot;
  }
}

This new configuration allows for the CardComponent example above to run as expected, because the new theme selectors are just selecting "all elements whose --theme property is either light or dark." So for any element, the value of --theme will match the nearest parent in the tree (or themselves) which assigns the --theme property.

Here's a running example of the actual code from the TailwindCSS documentation working as it should!

The problems with the solution

So based on the existence of the example linked above, you might wonder why we can't use this solution right now. Unfortunately this API is very new and is still being implemented in its entirety in browsers.

If you check the caniuse.com support matrix, it's not looking great. But it's also not nearly as bleak it it looks upon first glance. Notably, no browsers identify as fully supporting the feature, but thankfully it doesn't matter all that much. What most browsers decidedly do not support is selecting based on the value of native CSS properties. But what they do support is selectors based upon custom-properties (also known as CSS variables). So that is to say, if you wanted to get tricky and define your dark mode selector like the following, it wouldn't work in any browser currently.

.dark {
  background: black;
  color: white;
}

@custom-variant dark {
  @container style(color: white) {
    @slot;
  }
}

It seems that browser developers have deemed the need for this as very low and have neglected to implement it. However, as things work today, you could still just do something like the next example (assuming you're utilizing a property whose default value is inherit).

.dark {
  background: black;
  --text-color: white;
  color: var(--text-color);
}

@custom-variant dark {
  @container style(--text-color: white) {
    @slot;
  }
}

So generally that isn't really an issue worth stopping anyone from using this solution. The unfortunate leftover problem is that Firefox has thus far failed to support the feature at any level. But there's good news! We know they're close because if you go to about:config in Firefox, there is a flag for this feature!

If you flip the layout.css.style-queries.enabled flag to true, for all my testing, it seems to work perfectly well. This could be blocked from official default support by any number of reasons. So as it stands, it remains disabled and thus using Container Style Queries would exclude Firefox users (or rather, make your site look janky to them). That being said, this will be a powerful tool for use with TailwindCSS, and tons of other contexts, once it gets more wide-reaching support.

Whether you're a developer seeking a quick API fix, a marketer requiring platform-specific steps, or a customer simply trying to get something done, generic, static documentation feels like flipping through a dusty manual.

Modern users expect documentation that feels like it was written just for them, tailored to their role, device, and preferences. Think personalized, dynamic, and interactive content that adapts to your taste, just like Netflix tailors shows to yours.

Adobe Experience Manager (AEM) Guides help you deliver highly personalized documents, whether for PDFs or dynamic web outputs, using conditional attributes, profiling, and smart publishing strategies.

In this blog, we’ll explore:

• How to personalize content in AEM Guides using attributes like audience and platform
• How to set up publishing filters
• How to manage dynamic multi-version outputs from a single DITA source
• Best practices for clean and scalable personalized documentation

How does dynamic content personalization work in AEM guides?

Dynamic personalization helps you create multiple versions of documentation from a single source by applying filters based on audience, platform, region, product, or any other business-specific need.

Instead of maintaining multiple documents, you use DITA attributes like audience, platform, product, region, etc., and AEM Guides dynamically generate filtered outputs during publishing.

Examples:

• Admin Manual vs End User Manual Instructions (Software)
• Mobile App User Guide vs Desktop Web User Guide (Platform)
• Country-specific product instructions vs Global version (Product)
• Healthcare Practitioner vs Patient User Guides (Healthcare)
• Different vehicle models and their feature availability based on regions (Automotive)
• Internal vs External policy documents (Finance, Human Resources)
• Free Users Vs. Premium Users (Subscriptions)

Methods of personalization in AEM guides

You can personalize content in AEM Guides using:

• Conditional Attributes in DITA Topics
• Profiling Rules
• Filtered Output Presets

These allow fine-grained control over what content appears in each output without duplicating your DITA source.

Example 1: Persona-based personalization 

— Service technician manual vs. end user manual

Business Scenario: A company that manufactures electric scooters needs two different manuals:

• One for Service Technicians who repair and maintain the scooters.
• Another for End Users who simply operate the scooters.

While much of the content overlaps (like the braking system), the level of detail differs significantly:

• Service Manual: Technical, procedural, and diagnostic content.
• End User Manual: Simple, usage-focused instructions.

End User Manual ➔ Filter: audience=end-user
Service Manual ➔ Filter: audience=service-tech

Output Presets Setup:
audience="end-user" → Visible only in the End User Manual
audience="service-tech" → Visible only in the Service Manual

Example 2: Platform-based personalization 

— Mobile vs desktop

Business Scenario: A mobile app and desktop platform have different navigation instructions.

platform="mobile" → Visible in Mobile user guide
platform="desktop" → Visible in Desktop user guide

Output Presets Setup:

Mobile Documentation ➔ Filter: platform=mobile
Desktop Documentation ➔ Filter: platform=desktop

Example 3: Combining multiple attributes 

— Audience + Platform

Now, let’s make it even more dynamic!

Business Scenario: A banking app has different features for Admins and Users on Mobile and Desktop platforms.

Different content based on both audience and platform.

Output Presets Setup:

Mobile Admin Guide ➔ Filters: audience=admin, platform=mobile
Desktop Admin Guide ➔ Filters: audience=admin, platform=desktop
Mobile End User Guide ➔ Filters: audience=end-user, platform=mobile
Desktop End User Guide ➔ Filters: audience=end-user, platform=desktop

Step-by-step personalization process in AEM Guides

Step 1: Apply conditional attributes in DITA topics

Goal: Tag specific content pieces that should only appear for certain audiences, platforms, or products.

How to do it:

• In AEM Guides Editor, select the paragraph, table, or image you want to condition.
• In the Properties Panel, assign attributes like audience, platform, or product.

Example:

Step 2: Define Profiling Rules 

Goal: Set up rules that define how filtered outputs should behave during publishing.

How to do it:

• Navigate to Profiles and Conditions in AEM Guides.
• Define a new Profiling Rule Set.

• Specify the attributes, values, and labels you need. You can also assign colors to differentiate content visually.
• Once done, save and publish the profiling rule.

This enables AEM Guides to manage content inclusion or exclusion during output generation automatically.

Step 3: Create and Configure Filtered Output Presets

Goal: Associate profiling rules with specific publishing outputs (such as PDFs or Sites).

How to do it:

• Open the Map Dashboard.

• Navigate to Condition Presets and click Create.
• Click Add All on the preset creation screen to import the attributes defined in the profiling rules. Rename the conditions based on the scenario. (For example: Include "admin" and exclude "end-user" for admin-specific outputs.)

• Save the conditional preset.

• Go to Output Presets and create a new preset (e.g., PDF Output for Admin). In the General section, select the previously saved conditional preset.

• Click "Generate" to produce the final output.

• Now, when you publish, only content that matches the conditions will be included in the final output. You can also preview filtered content before publishing.

Best practices for dynamic personalization

• Plan attributes carefully during documentation design (audience, platform, region, etc.).
• Use consistent attribute naming (case-sensitive in DITA!).
• Keep profiling manageable — avoid too many overlapping conditions.
• Test outputs separately to ensure content renders correctly.
• Document your profiling strategy for the whole team (new writers need to understand the rules easily).

Integrating user profile attributes for advanced personalization

While AEM Guides supports rule-based filtering using conditional attributes like audience, platform, and product, personalization can be extended further by integrating with external user repositories. You can configure AEM’s publish instances to connect with systems such as LDAP, SAML providers, Adobe IMS, or other user authentication and profile management APIs and databases to fetch user-level profile attributes, including roles, job titles, or departments. These attributes can then be used to dynamically apply filters and personalize documentation experiences based on the user's identity. This setup enables AEM Guides to deliver truly personalized content tailored to individual user profiles.

Dynamic personalization in AEM Guides is a powerful strategy that simplifies content management and delivers highly targeted, user-specific documentation — all from a single source of truth.

By applying simple conditional attributes and smart publishing filters, you can significantly reduce maintenance effort, enhance accuracy, and provide a better experience for every user type.

It’s not just about documentation — it's about delivering the right information to the right user at the right time. 

Imagine opening a technical document packed with dense paragraphs and endless tables of numbers. Overwhelming, right?

Now imagine the same content — but this time, with vibrant graphs and dynamic charts that instantly tell the story behind the data.

Visuals don’t just beautify documents. They make understanding effortless.

As AEM Guides users, we have the power to transform complexity into clarity. And the best part? Adobe Experience Manager (AEM) Guides gives us the flexibility to integrate stunning visual representations into both PDF outputs and Sites experiences.

Industry use cases for graphs within product documentation

Integrating charts in technical documentation (techdocs, functional/technical manuals, brochures, etc.) adds a strong visual impact. Here are some of the practical industry examples:

Manufacturing: Show production rates and machine uptime/downtime analytics.

Healthcare and pharmaceuticals: Visualize clinical trial results, medication efficacy, or treatment comparisons.

Automotive: Represent vehicle performance stats and maintenance intervals.

Software and Technology: Showcase system performance metrics, application error rates, or API response times.

Finance: Graphs for loan trends, risk assessments, or insurance claims analysis.

In this post, we will explore how to integrate graphs inside Adobe Experience Manager (AEM) Guides for both PDF and Sites outputs.

We will walk through the complete process — from selecting the proper chart library, importing it into AEM Guides, implementing custom JavaScript, and ensuring adequate rendering for different outputs.

Whether you are working on structured content for PDFs or dynamic content for web experiences, this blog will help you achieve seamless graph integration.

Overview: Graph Integration in AEM Guides

Adobe Guides supports graph/chart integrations by including JavaScript libraries in templates and authored sections. However, there are some important considerations to keep in mind:

• Supported Libraries: Only vanilla JavaScript (ES5 compatible) libraries are currently supported. Libraries that use ES6+ features (like arrow functions, classes, let, const) may not work properly, especially when generating PDFs.
• Use Case Demonstrated: For this tutorial, we will integrate a simple Polar Area Chart and Line Chart using a vanilla JavaScript chart library to demonstrate the steps. The JavaScript example provided while demonstrating PDF generation can also be applied similarly for Sites output. Therefore, in the Sites section, only the integration steps are detailed, as the core JavaScript implementation remains the same.

Part 1: Graph Integration for PDF Output

Let's first see how to integrate a graph that should be rendered when generating PDFs from AEM Guides.

Step 1: Select the Chart Library

• Choose a JavaScript chart library compatible with ES5.
• In this demonstration, we use a simple Polar Area Chart library written in vanilla JavaScript.

Note: Ensure the library does not rely on modern JavaScript features beyond ES5.

Step 2: Import the Library into Templates

In AEM Guides:

• Go to the Template Editor section.
• Open the specific Template where you want to integrate the graph.
• Use the Import Section to upload and include the JavaScript chart library.

Step 3: Create an Output Class

Next, you need to create an Output Class:

• Navigate to the Output Class section.
• Create a new class (for example: .chart-container).
• Assign this class name where you are authoring your graph/chart in the AEM Guides content (in the Author mode).

Step 4: Implement Custom JavaScript

• Create a separate JavaScript file dedicated to handling the chart functionality.
• Target the container where the chart should appear — specifically using the .chart-container class that was set in the previous steps.
• Initialize and render the chart dynamically inside that container by creating a <canvas> element, preparing the chart configuration, and rendering it using a chart library (like Chart.js).
• Support both static and dynamic data sources:
  • Static data: You can directly define datasets within the JavaScript file (for example, hardcoding the labels and values).
  • Dynamic data: You can fetch the chart data from various endpoints such as:
    • AEM Content Fragments via GraphQL APIs
    • AEM Assets or DAM JSON files
    • External REST APIs from third-party services
    • Custom servlets or backend endpoints developed within AEM
• Depending on the use case, the fetch URLs and data processing logic can be customized to feed the chart dynamically at runtime.

The example below is the JS snippet for the polar chart and Line Chart respectively:

JavaScript Example: Static Chart Initialization
window.addEventListener('DOMContentLoaded', function () {
window.pdfLayout.onBeforePagination(function () {

// Create a canvas element
var parent = document.querySelector(".chart-container");
var canvas = document.createElement("canvas");
canvas.classList.add("eligible-categories-bar");

// Avoid duplicate canvas creation
if (document.querySelector(".chart-container canvas.eligible-categories-bar")) {
  return;
}

parent.appendChild(canvas);

var ctx = canvas.getContext('2d');

var mixedChart = new Chart(ctx, {
  type: 'bar', // Define the type of the chart
  data: {
    labels: ["2020", "2021", "2022", "2024"], // X-axis labels
    datasets: [
      {
        label: 'Assets Evaluation', // Dataset label
        data: [12, 19, 3, 5], // Dataset values
        backgroundColor: '#FFD700', // Bar color
        borderColor: '#FFD700', // Border color
        borderWidth: 1 // Border width
      },
      {
        label: 'Selection Process', // Second dataset label
        data: [15, 9, 7, 8], // Second dataset values
        backgroundColor: '#01EA57',
        borderColor: '#01EA57',
        borderWidth: 1
      }
    ]
  },
  options: {
    scales: {
      yAxes: [{
        ticks: {
          beginAtZero: true // Y-axis starts from 0
        }
      }]
    },
    barPercentage: 0.4, // Adjusts width of bars
    categoryPercentage: 0.5 // Adjusts space between groups
  }
});

});
});

JavaScript Example: Static Integration for polar chart

JavaScript Example: Dynamic API Integration for Line chart

window.addEventListener('DOMContentLoaded', function () {

window.pdfLayout.onBeforePagination(function () {
    // Prevent multiple chart creations
    if (document.querySelector(".chart-container-dynamic .canvas-container")) {
        return;
    }
    // Create a <canvas> for Chart.js 
    var canvas = document.createElement("canvas");
    canvas.className = "canvas-container";
    var chartContainer = document.querySelector(".chart-container-dynamic");
    if (!chartContainer) {
        console.error("Chart container not found");
        return;
    }
    chartContainer.appendChild(canvas);
    var ctx = canvas.getContext("2d");
    // Fetch and render Chart.js chart
    fetch("https://canvasjs.com/services/data/datapoints.php?xstart=1&ystart=10&length=100&type=json")
        .then(function (response) {
            if (!response.ok) {
                throw new Error("Network response was not ok");
            }
            return response.json();
        })
        .then(function (data) {
            var dataPoints = [];
            // Convert to Chart.js format
            for (var i = 0; i < data.length; i++) {
                dataPoints.push({
                    x: data[i][0],
                    y: parseInt(data[i][1], 10)
                });
            }
            // Render Chart.js chart
            new Chart(ctx, {
                type: 'line',
                data: {
                    datasets: [{
                        label: 'External Data',
                        data: dataPoints,
                        borderColor: '#04C2C7',
                        backgroundColor: '#04C2C7',
                        fill: false,
                        tension: 0.3
                    }]
                },
                options: {
                    responsive: true,
                    parsing: false,
                    scales: {
                        x: {
                            type: 'linear',
                            position: 'bottom',
                            title: {
                                display: true,
                                text: 'X'
                            }
                        },
                        y: {
                            title: {
                                display: true,
                                text: 'Y'
                            }
                        }
                    },
                    plugins: {
                        legend: {
                            display: true,
                            labels: {
                                font: {
                                    size: 14,
                                    family: 'Manrope, sans-serif'
                                },
                                color: '#2C343B'
                            }
                        }
                    },
                    layout: {
                        padding: {
                            top: 40,
                            bottom: 20,
                            left: 10,
                            right: 10
                        }
                    }
                }
            });
        })
        .catch(function (error) {
            console.error("Fetching data failed:", error);
        });
});

});

Step 5: Include JavaScript in Page Layout

Finally:

• Include both JavaScript files (the library and your implementation script) inside the Page Layout of the PDF output configuration.

This ensures that when the PDF is generated, both the library and your custom logic are loaded correctly.

Step 6: Enable JavaScript in PDF Generation Settings

When generating the PDF:

• Go to the Output Preset settings.
• Make sure the "Enable JavaScript" option is checked.

Without enabling this option, dynamic scripts won't execute while generating the PDF, and charts will not appear.

That’s it!

Now, when you generate the PDF, your chart will render properly inside the document. The polar chart data can be authored as needed, and upon PDF regeneration, the chart will reflect the updated information accordingly.

Part 2: Graph Integration for Sites Output

Now, let's see how you can integrate the same graph into AEM Sites output (for web).

Fortunately, the process is even simpler.

Step 1: Import Library in Clientlibs

In AEM Sites:

• Navigate to Client Libraries (clientlibs).
• Create or update a clientlib specific to your project.
• Add the charting library's JavaScript file inside the clientlib's JS folder.

Example structure:

/apps/your-site/clientlibs/yourclientlib/js/chart-library.js
/apps/your-site/clientlibs/yourclientlib/js/custom-graph-implementation.js

Ensure the clientlibs are properly included via categories in your page templates.

Step 2: Write Custom Implementation (for Sites Output)

Inside your custom JavaScript file (e.g., custom-graph-implementation.js):

• Target the same container class (e.g., .chart-container).
• Write the logic to render the graph when the page loads or when the DOM is ready (DOMContentLoaded or similar event).
• Support both static and dynamic data sources:
  • Static Data: Hardcoded inside the JS itself.
  • Dynamic Data: Fetched from external APIs, AEM Content Fragments (via GraphQL), AEM DAM JSON, or custom AEM servlets.

Important for Sites Output:

• By default, if you fetch data dynamically, it will be retrieved in real-time every time the page is rendered.
• However, if real-time fetching is not desired (for performance or consistency reasons), you can configure AEM to fetch the data once during publish time and cache it for all subsequent page loads.
• This way, the chart will behave like it’s loading static data even though it originally came from a dynamic source.

Example:

document.addEventListener('DOMContentLoaded', function() {
var chartContainer = document.querySelector('.chart-container');
if(chartContainer) {
drawPolarChart(chartContainer);
}
});

Step 3: Generate and Test

Once the clientlib is properly included:

• Publish or preview your page.
• You should see the chart render dynamically on your AEM Site page.

No special settings like "Enable JavaScript" are required here because Sites outputs naturally support dynamic JavaScript rendering.

Important Tips

• Testing: Always test your integration both in Preview Mode and after Publishing.
• Performance: Keep the chart library lightweight to avoid bloating the PDF or Site loading times.
• Fallback Handling: Consider adding fallback content if JavaScript is disabled or the chart fails to render.

Conclusion

Integrating graphs into AEM Guides is a powerful way to make your technical documents and Sites more interactive and visually appealing.

While PDF generation requires careful handling (due to JavaScript execution limitations), the site's output is relatively straightforward.

By following the structured approach discussed above — choosing the right library, correctly importing it, writing clean custom scripts, and configuring output settings — you can successfully bring dynamic, informative charts into your AEM-based documentation.

Happy graphing!

As engineering leaders, many of us find ourselves further from the codebase than we'd like.

The days of all-night coding sessions fade into memory, replaced by strategic meetings, team management, and architectural oversight. But the itch to build often remains. Recently, I scratched that itch using AI coding tools, leading to some crucial reflections for all of us leading technical teams.

The experiment: Vibe coding on a long haul

Inspired by "vibe coding" videos showcasing rapid development, I experimented on a long flight. Armed with Cursor (an AI-first code editor) and a rough app idea, I dove into tackling frameworks I wasn't familiar with.

The results were pretty striking. AI handled much of the heavy lifting – setting up dependencies, generating boilerplate code, and navigating unfamiliar territory. Within hours, a non-trivial web app was functional. This demonstrated AI co-pilots' raw speed and potential to accelerate development, particularly in bootstrapping projects or exploring new technologies.

The reality check: The "overly enthusiastic junior dev"

However, the experience wasn't seamless. It wasn’t far from the experience of collaborating with an "overly enthusiastic junior dev" – fast and full of suggestions, but lacking coherence, consistency, and sometimes making obvious errors (like introducing a redundant CSS framework).

This, I feel, is where experience is critical. Identifying flawed AI suggestions, debugging generated code, and ensuring architectural soundness requires the seasoned judgment that comes from years of building, shipping, and maintaining software. The AI could generate code, but it couldn't consistently generate wisdom.

Challenging the "young person's game" narrative

This experiment prompted reflection on a persistent industry stereotype: is software development still primarily a "young person's game"? 

Statistics support this perception – globally, the largest cohort of developers is 25-34, significantly younger than the average workforce age. The "whizz-kid" archetype endures, often unfairly painting senior engineers as out of touch.

But if AI tools increasingly automate repetitive tasks and lower the barrier to entry for complex frameworks, the differentiating factor shifts. Boilerplate, syntax, and basic implementation details become less critical. What becomes more critical?

• Judgment: Knowing what to build and how it fits into the larger picture.
• Trade-offs: Understanding the long-term implications of technical decisions (scalability, maintainability, security).
• Quality: Spotting subtle flaws, code smells, and architectural weaknesses before they become major problems.
• Direction: Guiding the development process, whether the "developer" is human or AI.

These are the hallmarks of experience. Often, the critical 10% of our skills – the deep understanding and judgment – provides exponential value.

Critical considerations for engineering leaders

The rise of AI co-developers isn't just about individual productivity; it forces us, as leaders, to confront significant challenges:

1. Developing future seniors: How do we nurture junior talent? If AI handles the foundational tasks, how do aspiring developers build the deep understanding needed to become tomorrow's seniors? Relying solely on AI risks creating "prompt experts" who lack fundamental coding principles. We must consciously design training, mentorship programs, and team structures that cultivate this deeper knowledge, even as abstraction layers increase.
2. Maintaining quality & oversight: Rapid, AI-generated code demands rigorous validation. Ensuring that human expertise remains in the loop is crucial. Experienced engineers are vital for reviewing AI output, catching subtle errors, and preventing the propagation of bad practices, lest we grumble and clean up messes later.
3. Adaptability of senior talent: Experience is invaluable, but only if coupled with curiosity. Senior engineers must embrace new tools and adapt their workflows. Those who remain stuck in old ways risk becoming less efficient, even with their deep knowledge. 
4. The power of collaboration: The ideal future isn't about choosing between youthful energy and seasoned wisdom. It's about combining them. Young devs might bring the fire, but seasoned devs bring the fire extinguisher. Our role as leaders is to foster teams where these strengths complement each other, leveraging AI as a tool for everyone.

Engineering experience matters more with AI code generation 

AI is undeniably reshaping software development. But rather than making experienced engineers obsolete, it appears poised to amplify their value. When the grunt work is automated, the focus sharpens on strategic thinking, architectural integrity, and sound judgment – precisely the areas where experience shines.

As leaders, we must encourage our teams to explore these tools, integrate them thoughtfully into our workflows, and critically evaluate their outputs. Most importantly, we need to actively cultivate and value the deep expertise within our teams and challenge the outdated notion that innovation belongs solely to the young.

In the age of AI co-pilots, experience isn't just relevant; it's the rudder steering the ship. Let's ensure we're building teams – and a culture – that recognizes its enduring, and perhaps increasing, importance.

1. Introduction & Overview

In this tutorial, we will be going through how to integrate the email marketing platform Mailchimp with Next.js and TypeScript. When integrating it into an application recently, I ran across numerous issues and thought this tutorial might be useful to others. A GitHub repository is referenced at the end of this tutorial. 

Prerequisites

For your reference, below are the versions we are using in this application. We will be using the app router setup for Next.js.

Versions

• Next: 15.1.4
• React: 19.0.0
• TypeScript: 5.6.2
• Node: 22.12.0
• NPM: 10.5.2

To integrate Mailchimp, you will need to sign up for an account. Install the npm packages @mailchimp/mailchimp_marketing and @types/mailchimp__mailchimp_marketing. For this tutorial, we are using @mailchimp/mailchimp_marketing version 3.0.80 and @types/mailchimp__mailchimp_marketing version 3.0.21. 

npm install @mailchimp/mailchimp_marketing @types/mailchimp__mailchimp_marketing

Note, the node:crypto module is a built-in module included with Node.  It doesn’t require installing, but it needs to be imported into the API endpoint to use the createHash function. 

Environment variables

Next you’ll need to set up MAILCHIMP_API_KEY, MAILCHIMP_API_SERVER, and MAILCHIMP_AUDIENCE_ID environment variables. Add these variables to your .env and then include them in your Next config file. 

1. MAILCHIMP_API_KEY: Mailchimp API key

This resource shows how to find an API key in your Mailchimp account. 

2. MAILCHIMP_API_SERVER: Mailchimp server value

To find the server value for your account, login to Mailchimp. After authentication, the browser URL will show the server value appended before “admin.mailchimp.com.” For example, if the URL was https://us19.admin.mailchimp.com/ the “us19” portion is the server prefix. 

3. MAILCHIMP_AUDIENCE_ID: Mailchimp audience ID

To find the Mailchimp audience ID, go to the Audience section in your account and then go to All contacts, and then to Settings. In the settings, there is an Audience ID field, which you can copy. 

2. Endpoint instructions

Since we are using the Next.js app router, we used this path for the endpoint: /src/app/api/mc/subscribeUser. If you are not using the app router, your setup will look slightly different. At the end of this tutorial, I will provide a GitHub repo that you could spin up to test out the setup with your account. 

For route handling, we are using the helpers NextRequest and NextResponse which are imported from next/server. There are other ways to accomplish route handling, check this resource for more information. 

In this file, we will also be importing Mailchimp, which we installed in the initial steps. We will also be importing createHash from the crypto module (node:crypto), which is included with Node. 

import { NextRequest, NextResponse } from 'next/server';
import mailchimp from '@mailchimp/mailchimp_marketing';
import { createHash } from "node:crypto";

From here we are ready to set the configuration for Mailchimp. The API key and server environment variables will be passed in as the apiKey and server parameters, as shown below. 

mailchimp.setConfig({
  apiKey: process.env.MAILCHIMP_API_KEY,
  server: process.env.MAILCHIMP_API_SERVER,
});

Next, we will be setting up a POST request function. We are only requiring an email to create a list member, but first name and last name will be included if provided. We get the form values passed in using the json method (request.json()) and immediately fail if the email is not provided, since this will be required to create a member. We will be sending a failure response using the NextResponse route handler helper. 

We will also check to make sure the audience ID environment variable is available since this will also be required for the request and send a failure response if it is not available. 

export async function POST (request: NextRequest) {
  const body = await request.json();
  const { email, firstName, lastName } = body;
  if (!email) {
	return NextResponse.json({ error: 'Email is required.' }, { status: 400 });
  }
  const audienceId = process.env.MAILCHIMP_AUDIENCE_ID;
  if (!audienceId) {
	return NextResponse.json({ error: 'Audience required.' }, { status: 400 });
  }
  try {
	
   } catch (error: any) {
	
  }
};

In this implementation, we check if the member exists in Mailchimp for that list so that we can communicate to the user that the failure was due to an account already being created. First, we need to create an MD5 hash of the email using the createHash function provided by the crypto Node module. Then, using the mailchimp.lists.getListMember method, we pass in the audience ID and the email hash.

If there was a member found, we can check if the status was subscribed. The resource for listing member info is available here and shows the other values for the status include: "subscribed", "unsubscribed", "cleaned", "pending", "transactional", or "archived". You can choose to handle each of those situations, but here, we are only handling already subscribed members. 

From there, we catch the error response so that it doesn’t fail if the user doesn’t have an account already. 

NOTE: Mailchimp has a method for adding or updating a list member, which is listed in the resources at the end if you’d prefer to use that method. 

const emailHash = createHash('md5').update(email).digest('hex');
const isEmailExisting = await mailchimp.lists.getListMember(audienceId, emailHash)
  .then((r) => {
    	const isSubscribed = r?.status === 'subscribed';
    	return isSubscribed;
  })
  .catch(() => false);
if (isEmailExisting) {
  	return NextResponse.json({ error: 'Email already subscribed.' }, { status: 400 });
}

Now we can create the list member. The resource for adding a list member is available here. We will be using the mailchimp.lists.addListMember method and will pass in member data as well as the status of “subscribed.” From there we return the data if the request was successful.

const data = await mailchimp.lists.addListMember(audienceId, {
  email_address: email,
  status: 'subscribed',
  merge_fields: {
  	FNAME: firstName ?? "",
  	LNAME: lastName ?? "",
  },
});
return NextResponse.json({ data });

Finally, we will handle the error using the NextResponse json helper function (more information here). Add the following to the catch statement in the case of an unexpected error. 

let errorMessage = "";
if (error instanceof Error) {
  	errorMessage = error?.message;
} else {
  	errorMessage = errorMessage ?? error?.toString();
}
console.error(errorMessage);
return NextResponse.json(
  	{ error: "Something went wrong." },
  	{ status: 500 }
);

Full snippet for API endpoint:

import { NextRequest, NextResponse } from "next/server";
import mailchimp from "@mailchimp/mailchimp_marketing";
import { createHash } from "node:crypto";

mailchimp.setConfig({
  apiKey: process.env.MAILCHIMP_API_KEY,
  server: process.env.MAILCHIMP_API_SERVER,
});

export async function POST(request: NextRequest) {
  const body = await request.json();
  const { firstName, lastName, email } = body;
  if (!email) {
	return NextResponse.json({ error: "Email is required." }, { status: 400 });
  }
  const audienceId = process.env.MAILCHIMP_AUDIENCE_ID;
  if (!audienceId) {
	return NextResponse.json({ error: "Audience required." }, { status: 400 });
  }
  try {
	// Check if the email exists:
	const emailHash = createHash("md5").update(email).digest("hex");
	const isEmailExisting = await mailchimp.lists
    	.getListMember(audienceId, emailHash)
      	.then((r) => {
          	const isSubscribed = r?.status === "subscribed";
          	return isSubscribed;
    	})
    	.catch(() => false);
	if (isEmailExisting) {
    	return NextResponse.json(
        	{ error: "Email already subscribed." },
        	{ status: 400 }
    	);
	}
	// If the email doesn't exist, subscribe:
	const data = await mailchimp.lists.addListMember(audienceId, {
    	email_address: email,
    	status: "subscribed",
    	merge_fields: {
        	FNAME: firstName ?? "",
        	LNAME: lastName ?? "",
    	},
	});
	return NextResponse.json({ data });
  } catch (error: unknown) {
	let errorMessage = "";
	if (error instanceof Error) {
  		errorMessage = error?.message;
	} else {
  		errorMessage = errorMessage ?? error?.toString();
	}
	console.error(errorMessage);
	return NextResponse.json(
    	{ error: "Something went wrong." },
    	{ status: 500 }
	);
  }
}

3. UI instructions

We can now set up the form component, which will display a form that results in a list of members being created in Mailchimp. Create a React component that renders a basic form that has a submit form and input fields for email, first name, and last name. 

The file needs to start with the “use client” directive, which designates a component to be rendered on the client side. This should be used when creating interactive user interfaces (UI) that require client-side JavaScript capabilities; see resource here for more information. 

The CSS module file we’re importing (embeddedForm.module.css) has style specific to the site we were working on, so I’ll be glossing over that. 

  "use client";
import css from "./embeddedForm.module.css";

export function EmbeddedForm() {
  return (
	<form onSubmit={subscribeUser} className={css.form}>
    	<h2 className={css.header}>Subscribe to our newsletter!</h2>
    	<div className={css.inputWrapper}>
        	<label className={css.inputAndLabel}>
            	<span className={css.inputFieldLabel}>First name</span>
            	<input name="firstName" className={css.inputField} />
        	</label>
        	<label className={css.inputAndLabel}>
            	<span className={css.inputFieldLabel}>Last name</span>
            	<input name="lastName" className={css.inputField} />
        	</label>
        	<label className={css.inputAndLabel}>
            	<span className={css.inputFieldLabel}>Email</span>
            	<input name="email" type="email" className={css.inputField} />
        	</label>
    	</div>
    	<button type="submit" value="" name="subscribe" className={css.button}>
        	Submit
    	</button>
	</form>
  );
}

export default EmbeddedForm;

Now, we will set up the function that uses our new subscribeUser endpoint which creates a list member in Mailchimp. Since we are rendering a form element, the event parameter type will be FormEvent with HTMLFormElement passed in. We will need to import FormEvent from react to accomplish this. 

We will also need to import useState from React. From there, we can set up the isLoading state that we will use to display a loader component, as well as a message state that will show the user the result after they submit the form. 

Within the subscribeUser function, we will use the preventDefault method that cancels the event if it is a default action, such as if someone hits Submit without filling out any information. Then, we will set the isLoading state to true so that the loading spinner will display while attempting the request. 

The form contains a first name, last name, and email, which will be sent to the Mailchimp endpoint for creating a list member. To retrieve the form data, create a new FormData object using the FormData constructor and pass in the currentTarget property from the event which will have the first name, last name and email values provided in the form. 

From there, we use the fetch function to hit the endpoint we created (/api/mc/subscribeUser) and provide the form data included in the form. We will also need to set the message state to share the status with the user, which will be either a success or error message,  and set isLoading to false so that the message displays to the user instead of the loader. 

const subscribeUser = async (e: FormEvent<HTMLFormElement>) => {
	e.preventDefault();
	setIsLoading(true);
	const formData = new FormData(e.currentTarget);
	const firstName = formData.get("firstName");
	const lastName = formData.get("lastName");
	const email = formData.get("email");
	const response = await fetch("/api/mc/subscribeUser", {
    	body: JSON.stringify({
        	email,
        	firstName,
        	lastName,
    	}),
    	headers: {
      		"Content-Type": "application/json",
    	},
    	method: "POST",
	});
	const json = await response.json();
	const { data, error } = json;
	if (error) {
    	setIsLoading(false);
    	setMessage(error);
    	return;
	}
	setMessage("You have successfully subscribed.");
	setIsLoading(false);
	return data;
};

After creating the function, add a conditional that displays the CircularLoader component when isLoading is set to true. The CircularLoader is featured in the Github repo, but any loader component will do. 

We also need to add a conditional that displays the message if there is one set, which will happen if the request was successful or not. 

Full snippet for UI component:

"use client";
import { FormEvent, useState } from "react";
import CircularLoader from "@/components/loader/circular-loader";
import css from "./embeddedForm.module.css";

export function EmbeddedForm() {
  const [isLoading, setIsLoading] = useState(false);
  const [message, setMessage] = useState("");

  const subscribeUser = async (e: FormEvent<HTMLFormElement>) => {
	e.preventDefault();
	setIsLoading(true);
	const formData = new FormData(e.currentTarget);
	const firstName = formData.get("firstName");
	const lastName = formData.get("lastName");
	const email = formData.get("email");
	const response = await fetch("/api/mc/subscribeUser", {
    	body: JSON.stringify({
        	email,
        	firstName,
        	lastName,
    	}),
    	headers: {
      		"Content-Type": "application/json",
    	},
    	method: "POST",
	});
	const json = await response.json();
	const { data, error } = json;
	if (error) {
    	setIsLoading(false);
    	setMessage(error);
    	return;
	}
	setMessage("You have successfully subscribed.");
	setIsLoading(false);
	return data;
  };

  if (message) {
	return <p className={css.errorMessage}>{message}</p>;
  }

  if (isLoading) {
	return <CircularLoader />;
  }

  return (
	<form onSubmit={subscribeUser} className={css.form}>
    	<h2 className={css.header}>Subscribe to our newsletter!</h2>
    	<div className={css.inputWrapper}>
        	<label className={css.inputAndLabel}>
            	<span className={css.inputFieldLabel}>First name</span>
            	<input name="firstName" className={css.inputField} />
        	</label>
        	<label className={css.inputAndLabel}>
            	<span className={css.inputFieldLabel}>Last name</span>
            	<input name="lastName" className={css.inputField} />
        	</label>
        	<label className={css.inputAndLabel}>
            	<span className={css.inputFieldLabel}>Email</span>
            	<input name="email" type="email" className={css.inputField} />
        	</label>
      	</div>
    	<button type="submit" value="" name="subscribe" className={css.button}>
      	Submit
    	</button>
	</form>
  );
}

export default EmbeddedForm;

4. Additional resources

To test this functionality from this tutorial, you can clone this Github repository and change the environment variables to reference your own Mailchimp account: https://github.com/dallashuggins/mailchimp-nextjs. 

Below are the resources listed throughout this tutorial for quick reference.

• Find API key for Mailchimp
• @mailchimp/mailchimp_marketing 
• @types/mailchimp__mailchimp_marketing
• node:crypto module
• Add Members to List
• List Members
• Add or Update List Member
• Route handling with Next.js
• Use client directive for Next.js

5. Conclusion

We hope you found this tutorial useful for integrating Mailchimp into your Next.js application. 

We welcome any feedback or questions in the comments here or in the linked Github repo. Thanks for reading! 

Many organizations are pursuing a headless strategy for enterprise content management. A common term to describe this approach is MACH: microservices, API-first, cloud, headless. This architecture consists of many services, deployed in the cloud, communicating and producing data via APIs. In MACH architectures, it's good practice to have an integration layer to synthesize these sources of data before providing it to a front-end.

CMS content is a common integration point, as it's close to the front-end and benefits from a view of all systems whose data ends up exposed to the end user. Recognizing this, many CMS platforms have a marketplace for integrations with third-party services. However, for various reasons you may want to implement your own integration layer.

One reason to build your own integration layer is gaps in the integration marketplace. While these marketplaces are mature and offer integrations with many services, they generally depend on the service provider to build and maintain the integration. As such, an integration may be available in one CMS platform but not another. Additionally, because these integrations must cater to all users of the CMS they are typically designed for the most standard use cases and may not work for your situation. Similarly, if your software architecture includes custom-built services that must be synthesized with managed content, you'll need to build the integration yourself.

You may also want to avoid pre-made integrations to avoid vendor lock-in. Building your own integration layer allows you (but doesn't require you) to separate the CMS platform from the synthesis of content with external data. This kind of separation can be beneficial if you want to replace your current CMS platform, or if you're adding a new one but see some risks that may necessitate a change.

In this post I'll detail how we built a performant and scalable system of custom integrations that empowered editors to conduct A/B tests and other experiments without depending on developers.

What we built

This retail client brought us in to replace a homegrown CMS, feeding their React Native app, with the enterprise platform Optimizely. They wanted to integrate personalization, experimentation, and a product information management system. In addition, to save on platform costs and boost performance, they needed thorough server-side caching.

We delivered on these requirements while also providing front-end access to all integrated data in a single network call. A GraphQL Backend-for-Frontend microservice was built to orchestrate the various systems and serve enriched content.

Configurable Redis and in-memory caching were also added. But we wanted to knock it out of the park with performance and leave their lean engineering team with a way to quickly build new integrations.

To accomplish both of these goals, we designed and implemented a unique content enrichment module.

The content enrichment module

We had three goals in mind when designing the module:

1. To ensure world-class user experience, execute as efficiently as possible by
   1. minimizing time waiting for network calls to external services, and
   2. avoiding redundant iteration of content nodes
2. To support the client's lean engineering team, allow developers to add new integrations without needing to manipulate the content tree
3. To avoid vendor lock-in, don't couple it to any particular CMS platform or tech stack

Content enrichment is a hot path—mobile screens make a content request each time they're loaded—so we needed to go beyond standard caching. Knowing that integration performance is primarily constrained by network requests, we decided to put some limitations on how new integrations are built:

1. To update a content block, developers must first define which external data they need. Then, they can define how the data changes the block.
2. All external data must be retrieved by implementing a special interface and allowing the module to control how and when data is requested.

These rules limit developer flexibility when building new integrations. However, in exchange, we get some really nice benefits.

First, by requiring developers to define enrichment data requirements and entrusting the module to make requests, we can ensure we never request duplicate data and that all data of the same type are requested at once.

Second, development of a new integration is simple and therefore fast—just implementing a couple of interfaces—and carries low risk of defects.

Finally, because application of changes is defined on a block-by-block basis and the module handles the complex (and thoroughly tested) task of mutating the content tree, developers spared from understanding the plumbing and can focus on the important business logic.

Taken together, this means that when developers add new integrations, the most efficient implementation is actually the easiest to build! This module enabled us to quickly and efficiently build out powerful capabilities for content editors, including no-code feature flags, no-code experiments, and real-time native mobile preview.

The best code is no code

When implementing a content architecture it's crucial to consider the editor experience. Editors are responsible for meeting and adapting to business goals and technology is what enables them to do so. However, while technology serves the editorial experience, it can also get in the way. The slow pace of change requests and software releases can severely limit editors' need to adapt to changing business requirements.

Our client wanted to roll out a full suite of experiments with their new system. For the system to keep up, editors couldn't be made to depend on engineering changes for each experiment. To empower editors, we designed special meta-blocks which allowed A/B tests to be fully implemented from within the CMS—no tickets or PRs required! These blocks connect multiple bits of content to an experiment key and variation flag, which allows the orchestration service to ask the experimentation platform which content to serve to the user. In a similar way, meta-blocks also allow editors to feature-flag their content blocks so they can build while front-end components are still being developed. And, because it's all implemented with our enrichment module, editors can add as many experiments as they need without impacting performance.

Setting a foundation

To summarize, we built a content integration microservice that uses flexible caching and a unique enrichment module to serve personalized content. In addition to superior customer experience, we helped content editors run A/B tests and feature-flag content without waiting for an engineer to code it. Finally, we improved developer velocity by freeing them from the nitty-gritty of integration code to focus on the big picture. The client team loved what we built, and are now using the design as a foundation for their enterprise content architecture.

]]>https://engineering.deptagency.com/code-cuisine-a-savory-recipe-app/663d2e98cce62600017f6ae5Thu, 20 Jun 2024 11:55:41 GMT

Welcome to Simple Recipes.

This app simplifies the way you discover and manage your favorite recipes. It provides a user-friendly interface made using Jetpack Compose and ensures effortless navigation and accessibility of data for both an online and offline experience.

This project aims to showcase many of the latest tools and best coding practices recently introduced by Google for Android development. It demonstrates the latest architecture that should now be the standard for all new apps. Implementing this foundation in any new project can instantly address several problems from past development practices. These include boilerplate code, unresponsive UI, tightly coupled components, and a lack of testability and scalability.

---

Setting the stage with flavorful Kotlin and Gradle

At the core of this app is a pure Kotlin codebase. With Gradle as our build system, managing dependencies using version catalogs allows us to add and centralize dependencies and plugins in a scalable way. A fresh configuration or migration are both made easy when using a TOML file for version catalog. This specifies libraries, plugins, and version comprehensively to avoid hardcoding dependency names and scattered versions in individual files. The build process is further accelerated by using the new Kotlin KSP API, which enhances the power of Kotlin. Unlike KAPT, KSP can processes annotations directly in Kotlin code, which enables annotation processors to run up to 2x faster.

For more info, see: Github/KSP

Separation and prep for cooking success

Emphasizing the modular approach to app development, the code is separated into distinct layers. The domain layer contains the business logic, entities, and use cases. The data layer handles data access, manipulation, and interactions with remote and local data sources. Finally, the UI layer focuses on data presentation and event handling.

Koin plays a crucial role in facilitating this approach by providing dependency injection. This promotes modularity and testability by decoupling components and excellent management of component dependencies.

Whipping up connectivity with Retrofit

In a world that vastly relies on networking, Retrofit reigns supreme. For a number of years, Retrofit has stood strong as the leading, most developer-trusted library to simplify the process of making HTTP requests and parsing responses.

By transforming complex networking into straightforward reusable methods, it enhances readability and maintainability. In a well-structured architecture, only the data layer is aware of data sources. This ensures that changes to the data fetching logic do not affect the UI layer, which simply only makes the data requests. Further, Kotlin Coroutines cohesively works alongside Retrofit to ensure a responsive user experience by streamlining background tasks.

Spicing up storage with room

With Room, managing recipe data becomes a piece of cake (Pun maybe intended, maybe not). As a powerful ORM solution, Room has become the industry-leading solution for managing persistent data in modern Android applications. Configuring Room starts with creating a database class that specifies entities. Next, entities are defined as data classes where each field represents a column in the table. Finally, a DAO (Data Access Object) manages all the requied database operations.

Room has introduced robust capabilities to perform database operations asynchronously, and compile time verification of SQL queries never seen before in the world of Android. It seamlessly integrates with other Jetpack components such as ViewModel and StateFlows, and enables a reactive data flow to ensure that the UI remains up to date with the underlying data, and ensures a truly responsive user experience.

Composing the cherry on top

Jetpack Compose has transformed the way UIs are crafted. Gone are the days of boilerplate XML layout, with forced view binding and a non-responsive layout. With Compose, UI becomes an art form. Components are carefully designed and provide excellent scalability to be reused throughout the app. With its declarative approach and intuitive APIs, UI updates are seamlessly ingrained and straightforward animations empower the developers to bring their UI visions to life.

Please explore the complete application by visiting the following link: Simple Recipes

For developers interested in elevating this architecture for their own apps, a skeleton can be found on this link: Android Starter Code (2024)

]]>https://engineering.deptagency.com/adobe-experience-manager-authoring-capabilities-in-2024/664f69d21666a00001531bb1Fri, 24 May 2024 15:58:13 GMT

Sitting in the “leaders” spot of CMS analysts’ reports for the last 15+ years, AEM has long been known as the content author’s paradise.

Although the way content editing works in 2024 is very different from what it was a decade ago, Adobe is a major player in this evolution. With the rise of SPAs and frontend frameworks, an abundance of new channels consuming content as structured data, and everything going cloud, we’re happy to see AEM continue to set standards and stay ahead of the pack. In this article, we’ll take a look at everything AEM has to offer to an enthusiastic content author with a penchant for productivity.

AEM’s classic webpage editor

Funny how time flies — the new state-of-the-art website management "Touch" UI launched with AEM 6.0 in 2014 is what we now (unofficially) call “classic”, while the legacy UI called “Classic” proper is but a fading memory… Anyway, that’s what this editor is — a classic standard for visual component-based webpage assembly with everything edited in context, much copied across the industry.

Authors create pages based on templates. Templates are user-editable and very flexible — they can carry collections of allowed components, apply policies to configure the same components differently for each template, and drive structure with mandatory and optional components and initial content.

Once in the page editor, authors drop components from a sidebar or click to add them where they need them. Later they can just drag the components around the page into a new position. Selecting a component in the page brings up relevant commands and the dialog for component configurations, behaviours and styling. The page’s own metadata like titles, SEO properties, or data and commerce integrations is also quickly accessed from the page editor.

Importantly, this editor is where we also work with Experience Fragments (XFs). This allows authors to extract random parts of webpages with one or more components and extract them into reusable preformatted blocks. This is a great way to author and single-source offers, banners, and such — edited in separation and without a URL of their own, XFs can be placed in many pages and updated across multiple locations with a single edit. XFs are also easily pushed from AEM to Adobe Target to power testing and personalization activities — that way, an AEM author can create XF-based offers, and Target users can apply them as needed.

From the editor, authors have immediate access to AEM Assets, the central DAM holding all of the images, videos and documents. They can navigate DAM locations and filter assets by titles or metadata like tags or media type, then drag directly into the page or into component dialogs. No need to link multiple crops of the same image for responsive websites like with many other CMS — AEM Assets can take care of these automatically, so only one master asset needs to be managed and placed in pages.

This is the proven experience in managing web-channel content specifically. And when some content is common across several channels, that data is managed in a headless fashion but placed in webpage components just as easily as content fragments. With many companies recently learning the hidden costs of going fully headless the hard way, the WYSIWYG webpage editor is as popular as ever, upholding the golden mean of modern content management — manage web-only content in visual web components; keep omnichannel data headless and channel-agnostic.

You might have also heard of the SPA editor. While, indeed, it allows editing single-page applications in AEM, it is not a different editor. You can treat it more like a tech enabler — once configured, it simply allows editing SPAs in the same familiar page editor. Given the complexities and limitations of setting up the SPA editor in AEM, it’s giving way to the new Universal Editor, which we’ll look at later.

Visual site tree management

Alongside the page editors, AEM offers visual management of website structure in intuitive trees (among other useful views). Any node in that tree functions as a folder and a page at the same time, so just go ahead and create pages under pages without worrying about 404s for folders without any content to display. Pages can be copied and moved in the tree, which directly translates into the URL structure of the published website. This being visual and intuitive is a welcome change from simpler CMS that requires tedious — and pun intended, sluggish — slug and route management for URLs. This is exactly the case with most headless CMS that have no love lost with the web channel, treating it as no more special than any others and not even offering website sitemaps or site searches.

This intuitive tree management makes AEM particularly strong at multisite and multi-language management. With AEM’s translation framework facilitating the process, authors maintain copies of translated content as "language masters" that are manually updated after changes accumulate in the base website language. These “language copies” are then dynamically cloned (as “live copies”) for reuse to one or more country sites that need content in that language. For example, France, Switzerland, and Canada sites would clone the content from the French language master branch, and Austria, Germany, and Switzerland would mirror the German language master. Alongside these managed pages, authors can add ad hoc ones at any time on any country site, creating effective combinations of global vs. local content.

In this browsing UI, once one or more pages or sections are selected, all relevant management, publication, and approval commands are immediately available in context.

Content Fragment editor

So what about headless content?

AEM natively supports every scenario on a spectrum between traditional ("coupled" or "headful") website operation and fully headless content with API delivery. Every instance of any headless content type in AEM is a directly editable object in the DAM, called a content fragment (CF). Admins can visually model the structure for content fragments (an exercise similar to crafting content models in other CMS), and authors get a slick, focused editor.

The editor provides access to CF fields and variations (alternate sets of fields), surfacing the relevant controls for each field type and practical metadata. This editor can also be customized to expose custom commands or additional fields.

Content fragments are, in essence, JSON data, which immediately allows serving them via AEM’s APIs and GraphQL interface, as well as linking them to the familiar web authoring components. However, their headless nature makes them impossible to preview — the editor can’t know which fields will every possible channel picks and how they will be visualized. And that’s where the Universal Editor comes to the rescue.

Universal Editor

As one of the latest additions to the author’s toolset in AEM, the Universal Editor (UE) delivers on a seemingly simple promise: no matter how your website is engineered, it can be edited in the UE, and your changes will seamlessly flow back to whatever content sources there are behind it. In time, that can even be a mix of Adobe and non-Adobe sources, but for the time being, any websites managed in AEM are supported — traditional (HTL-based), headless, hybrid, SPAs and the latest ultra-performant addition, sites on Adobe Edge Delivery Services.

It takes minimum developer effort to add UE compatibility to any website front end. Then the magic kicks in: authors visit the webpage’s URL in the editor, click around and see the page come alive with editable components and their configurations. Moreover, any changes they make sync right back to the content source. If that source is a content fragment — it will be modified without the author needing to find that fragment and change it through the non-visual CF editor. And if the source is a traditional page or an EDS website, the changes will persist in AEM's content repository just as well.

Most practical editing features, like the rearrangement of components on the page and adding new ones, are already supported, but as of 2024, the UE is still new and not yet at feature parity with the classic page editor. Yet it’s easy to envision the future where it fully takes over, replacing the Touch UI page editor first for the use cases covered with the SPA editor and gradually for all the other ones. E.g., with Edge Delivery Services, Adobe is betting on the UE rather than the classic editor to provide visual authoring for content natively stored in AEM (as opposed to content served via cloud documents — more on that below).

AEM and Edge Delivery Services

First things first: EDS is not a feature of AEM, it is actually a globally distributed content delivery infrastructure used by AEM as well as other Adobe solutions like Experience Platform and Adobe I/O. However, AEM benefits immensely from the ability to deliver lightweight websites at top speeds with Lighthouse 100 web vitals thanks to this global delivery layer and content loading optimisations.

As far as authors are concerned, this very new addition to AEM also brings a radically new concept of document-based authoring. The underlying idea is simple: every webpage starts as a document somewhere. It undergoes some editing, proofreading, and approvals — but then it has to be copied and pasted into CMS components or content fragments piece by piece. If only we could skip the copypasting and just continue using that document as a page source directly as is! Well, with EDS, you do exactly that: put your docs and sheets in Microsoft Office 365 or Google Workspace in a folder structure that mimics the website, and their content magically turns into your EDS website’s pages. Talk about shortening the learning curve! Was there any resume in the last 20 years where the candidate did not claim to be an expert in Office apps? With EDS in AEM, that’s all you need to author professional websites (although it’s not the only way to for author EDS).

With very few simple formatting conventions, documents feed the pages and their metadata, and sheets power the forms, tables and configurations. Adding a component is just a matter of adding a table with data.

It’s important to remember that cloud documents are not the only way to publish to EDS. AEM can deliver content that it stores and manages natively to EDS as well — authors edit it using the Universal Editor. Being native to AEM, such websites support most AEM features including multi-site management.

It is also possible to use EDS to render super-performant Adobe Commerce web fronts and easily extend the standard PDP pages with custom offers.

EDS is a great solution for smaller projects and campaigns, but it’s not too difficult to start migrating entire website sections. With AEM as a Cloud Service, Adobe provides CDN-level controls to serve parts of the same website with EDS and others from AEM Publishers. The authors would simply edit different parts of the website in different editors offered by AEM.

DITA editor for AEM Guides

Say what? Let’s unpack that a little:)

Editing technical documentation and manuals, product information sheets, legal documents and annual reports, corporate filings and other thorough, well-organised or regulated information is a world in itself. A world where a lot depends on consistent application of terminology and convenient reuse of copy fragments, assembly from modular copy blocks, versioning, and easy repurposing of text for publishing in various formats.

DITA is an XML-based file format purpose-built for exactly those purposes. Modular “topics” are authored with clear structure (think focused content blocks) and can reference other topics. They are then assembled into maps that represent articles or collections of articles. With AEM Guides, DITA data is natively supported with another dedicated editor (as well as with a FrameMaker integration for those who prefer it for DITA work).

With AEM Guides, the base experience can be likened to working with content fragments. Authors pick templates that define topic structure similar to how the CF models define CF structure. Unlike CFs, a DITA topic template is not as restrictive — authors can repeat and rearrange elements. The elements represent different sections like body text, headings, info blocks, process steps, tables, etc. DAM media is immediately available for placement as it sits in the same repository — AEM Guides is built on top of AEM Assets.

AEM Guides authors also get DITA-specific tools to assemble topics into maps and then further link some maps into larger maps, depending on how they organize their body of work. They have explicit control over document versions and metadata to automate the data-driven assembly of documentation for different products or customers. Right from the editor, they can take part in reviews and then publish the final maps as website content, PDF, or other document formats — or serve the content headlessly. The look and feel are defined in templates for each individual output type, so DITA editors don’t need to worry about anything but the logical structure of their content.

The modular nature of DITA content makes it very practical for translation. Rather than send entire large documents for (re)translation, with modular content, it’s easy to only translate new or changed topics, saving time and budgets. And the whole process is managed with AEM’s native translation framework, so the incoming translations are directly integrated into the right context, avoiding error-prone copypasting.

DEPT® is one of very few Adobe solution partners in the world implementing AEM Guides for our clients worldwide. If modular document management sounds like something your business needs, we can explain everything you need to know about the solution and implement it on your existing or new AEM setup.

DEPT® as your Adobe partner

We skipped over a couple of things, like the visual editor for Adaptive Forms in AEM Forms or most things DAM, but we are always there to cover anything and everything AEM or Adobe Experience Cloud if you need help or consultation. Do reach out, and we’ll help you get the most out of your Adobe investment with the help of our 500+ experts worldwide.

As an Adobe Experience Cloud solution partner, we take in the requirements of your business and implement AEM to make sure it is set up to bring the most value for your investment. A big part of that is not tech — it’s the process for efficient management of your content operations and content supply chain. We help you set up and optimize the content practice and enable authors to be productive and self-sufficient in their content entry and management activities.

Together, we assess the most suitable setup to help your editors do their job with the least friction and implement your AEM websites and headless content pipelines using an optimum combination of the options explained in the article. And suppose ever the capacity of your in-house content teams maxes out. In that case, we step in to offer quick and competent authors who take care of your BAU tasks while at the same time contributing improvements for the optimization of the process and toolset. Get in touch!

]]>https://engineering.deptagency.com/how-to-break-away-from-the-standard-chatgpt-interface/664cc7411666a00001531a6bThu, 23 May 2024 11:46:29 GMT

A while back, someone tossed around this cool idea to build an app that would leverage AI to provide users with a goal-oriented product strategy.

By using an LLM, specifically ChatGPT in this case, we could crunch the data of things like market analysis and pricing structures. This all happens with the natural language processing abilities LLMs provide. On top of that, we then get natural language back, which makes things easier to understand and implement.

The basic concept was that a user enters information about a goal or set of goals, and optionally some context about the company. A goal could be something as simple as, “We want to increase conversion rates on our landing page.” The additional context could be anything from what the landing page is for, target audience, amount of traffic, where you advertise this page, or pretty much any other information a user may think applies to achieving that goal.

At first glance, this was dead simple. OpenAI could not have made talking to ChatGPT with their API easier, as long as you want to build an interface similar to a chatbot or assistant. What if you wanted to break out of that design, though? Why would you want to break away from the standard chat interface? Great question, let me explain.

Most use cases with the current batch of AI apps work as a conversation. You ask a question, you get an answer, repeat. But In our use case the responses had more data than just a conversational message. The response we get back after ChatGPT does its magic, contains an initiative a company would undertake to achieve the goal they set out. An initiative will have activities to perform for that initiative, and there are plans to add more items such as details and documents for those activities.

This kind of data isn’t going to work well in a chat bubble. That led to the idea of having a more stylized and easily readable UI, and that led to how we talk to ChatGPT.

Unpacking ChatGPT

As mentioned above, ChatGPT interacts with the user in a chat-like way. You type something in, ChatGPT shows the lovely animated bubble to let us know it is replying, and eventually, you see text typing onto the screen as if someone were typing it in real time. The result is that most ChatGPT apps follow this same pattern and design.

But what happens behind the scenes while ChatGPT is “typing” its response?

At first, I thought the answer was simple: streaming data. Upon further digging, however, I learned that ChatGPT uses a type of stream called Server-Sent Events (SSEs). Not only was this format new to me, but it was also not what I was expecting.

In my experience, I usually see streaming data responses in somewhat usable chunks of data, the equivalent of complete sentences as opposed to snippets or phrases. But, with ChatGPT, users get back tokens in each SSEs. Tokens are generally 1-4 characters long. This could be a single punctuation mark or four characters within a word.

Either way, we’re not receiving a complete sentence, and definitely not a usable JSON object in a single response.

While there is an option not to stream the responses and simply let ChatGPT return the complete response when done, in our use case, that option was too slow.

We needed to find some sort of middle ground to get the streaming speed but have complete sets of data to pass to the UI.

Building a buffer

After asking around and getting some direction from others here at DEPT®, it became clear the best solution for creating that middle ground would be to build some sort of buffer. We need a place to catch the data coming from ChatGPT, handle formatting and validating that data, and then pass that on to the rest of the application.

In our app, we ask the user for at least a single input consisting of a business goal, typically something long or short-term. We send prompt ChatGPT with some information, including:

• The role it is acting as
• A desired format
• Two to three examples, also called Multi-Shot Prompting
• The user's input

ChatGPT tokenizes both requests coming in and responses going out. It essentially takes the text and chops it up into small bits for things like validating requests/responses that aren't too large and calculations for billing. The tokens are not usually a full word. The general rule of thumb is roughly four characters is equal to a token, but that’s definitely not a hard rule.

Example of how ChatGPT turns text into tokens.

Each SSE contains a token from the ChatGPT response, so we need to keep track of what was sent previously and what just came in.

As the store is updated, we constantly check to see if what we have in the store can be used in the UI. Everyone loves RegEx, so we have one here that checks to see if the data returned contains the pieces we need for the UI. Once that condition is satisfied, that entire chunk of the message is passed off to a formatter that converts the text to JSON to be stored in another part of the Svelte store.

When a completed message is found, we also remove it from the main buffer so we don’t have duplicate results.

In our results section of the page we have a UI component that is subscribed to (Svelte magic here again) the results array in the store. As a new complete (i.e. formatted and validated) result gets into the array, the UI updates to display it. The end result from the stream of SSEs ends up looking like this:

The results on the right have been through the buffer and formatter.

Final thoughts and lessons learned

When we first wrote this buffer system, ChatGPT 3.5 was good at getting us the format we wanted but not great.

Recently, however, both ChatGPT 3.5 and 4 have been updated with the ability to return JSON directly. With this update, the reliability of getting the format we coded the UI around has increased significantly, enough that we’re planning on refactoring it to use the updated JSON output.

Going forward, I think we’ll always need a validator in place to ensure the response we receive contains what we need for the UI. As we continue to explore, I’m hoping we can simplify the validation process and maybe remove the formatting helper altogether.

Alongside the model updates, there have also been updates to the OpenAI library we use for types. That library didn’t support streaming easily when we first wrote this system, so I’d like to revisit that to try and improve efficiency and reliability there as well.

]]>http://ashwinsundar.com/blog/diagrams-as-code65e5023b984e4d0001c5fb2fWed, 03 Apr 2024 12:46:11 GMT

A major pain point in the process of maintaining documentation is that, while a product is in development, documentation tends to go stale quickly. This can occur for a number of reasons:

• Engineers don't know how to create useful documentation
• Documentation is kept separately from the work being done
• Only a small subset of engineers are tasked with creating and maintaining documentation

The first problem is a large challenge. Learning how to write good documentation is an entire course. Learning how to create good diagrams is an entire course.

Fortunately, the last two problems can be partially addressed relatively easily - by saving diagrams as code.

Definitions

• Diagram: A visual representation of an engineered system
• Code: Text that makes a machine do things
• Version Control: A system that tracks atomic changes to a file

Put that all together:

• Diagrams as code: A text file that is parsed to generate an image, and which can be committed to version control.

Why save diagrams as code?

• Diagrams should not be an artistic exercise
• Diagrams should be version-controlled with reliable tools
• Diagrams should be useful for new team members

1. Engineering diagrams aren't a form of artistic expression

Picture this scenario - you construct a perfect symmetrical system diagram, arranging subsystem components in rounded boxes at the vertices of an equilateral pentagon. It is beautiful; it is pristine.

And then someone decides to add a subsystem.

The solution is simple:

• Care less about whether boxes in a diagram line up
• Care more about what the boxes actually communicate

2. Diagrams should be committed to version-control

Many WYSIWYG/visual-first tools have poor internal implementations of "version-control." These tools typically allow a user to "checkpoint" an image manually. However, the checkpoints often have cryptic names, such as "v.203". If a mistake is made, there is no way to easily figure out the last "good" state of a diagram.

The solution here is to use a text-based diagramming tool, so one may take advantage of fully-featured version-control systems, such as git. Mistakes can be traced with git bisect.The commit history can easily be searched from the command line. All of the powerful capabilities of git can be used to track changes to a diagram.

3. Diagrams should be useful to new team members

Finally—and most importantly—diagrams must be useful to new team members. Imagine a new member joining the team who needs to understand the architecture of a codebase. Naturally, they will reach for documentation, but they discover that the documentation is out of date.

Stale documentation can be worse than no documentation. New team members cannot distinguish stale from up-to-date documentation, and will develop an incorrect mental model of the system. This can be very difficult to correct once the misunderstanding is complete.

The solution is to keep documentation as close to code as possible. Ideally, it should live in the same repo as the code. Every pull request should involve reviewing relevant documentation, and making updates as needed. Fifteen minutes of extra documentation work in each PR will save significant time trying to re-explain how a system works to a team member who has learned the wrong information.

How does one actually create a "diagram as code" diagram?

There has recently been a renaissance of "diagram as code" tools. With support from GitHub (including native rendering in repositories), Mermaid.js appears to be leading the pack. Other popular options include ZenUML and PlantUML.

But what about tools like LucidChart, diagrams.net, and Microsoft Visio? These tools are popular for remote whiteboarding sessions. Why can't the outputs of those tools simply be committed to version control?

Tool	Can be VC'd in e.g. git	Text -> Image	Addressable in PR
Mermaid.js	Yes	Yes	Yes
ZenUML	Yes	Yes	Yes
PlantUML	Yes	Yes	Yes
draw.io/diagrams.net	Yes	No	No
LucidChart	Yes	No	No
MS Visio	Yes	No	No
Cell phone pictures of whiteboards	Yes	No	No

In the chart above, the following criteria have been selected:

• Artifacts can be version-controlled in e.g. git
• Artifacts can be defined using pure text, which is then parsed to create a diagram
• Artifacts can be atomically addressed in a pull request

In theory, one may commit any file type to version control. In practice, there is limited value to using version-control to track changes to a .svg or .jpeg file type, file types which are used to represent vector graphics and images, respectively. A .svg contains too much non-value-add information, used to describe what a graphic looks like. The signal-to-noise ratio in a diff'd image file is extremely low, in other words.

On the other hand, diff'd text files have a much higher signal-to-noise ratio. Each diff'd character corresponds to a visible change in the generated output of the diagramming tool.

Examples

Enough pedantry, let us take a look at a couple of examples. I have taken a liking to a diagramming tool called Mermaid.js lately, so all of the following examples will use that tool.

Example 1: Sequence Diagrams

...a sequence diagram captures the behavior of a single scenario. The diagram shows a number of example objects and the messages that are passed between these objects within the user case.

-- Fowler, Martin. UML Distilled: A Brief Guide to the Standard Object Modeling Language. 3rd ed., 2003

As the textbook definition alludes to, a sequence diagram can be used to describe any set of systems that share messages. To keep the analogy concrete, let us look at an example of a theoretical message transit service.

Consider a system composed of an API subsystem, Platform subsystem, and IoT Service subsystem. The API is responsible for handling the external interface. The Platform is responsible for handling "business logic." The IoT Service is responsible for hosting the MQTT messaging service.

A minimalist, clean, and informative diagram (such as the one above) is created with the following mermaid.js code:

sequenceDiagram

participant API as API
participant F as Platform
participant IoT as IoT Service

F->>IoT: attempt authenticated connection to MQTT broker
IoT-->>F: confirm connection

loop Every 20s
    F->>API: request messages
    API-->>F: send messages

    F->>IoT: post message to MQTT broker at topic {deviceID}/{msgId}
end

What happens if one wants to add a new database service to the diagram, perhaps in-between the Platform and IoT Service subsystems?

In a traditional WYSIWYG editor, this task could take some time and incur significant frustration because many distinct GUI elements must be manually moved or re-drawn. Not the case in a text-first diagramming tool:

 sequenceDiagram
 
 participant API as API
 participant F as Platform
+participant Pg as Postgres DB
 participant IoT as IoT Service
 
 F->>IoT: attempt authenticated connection to MQTT broker
 IoT-->>F: confirm connection
 
+F->>Pg: attempt authenticated connection to DB
+Pg-->>F: confirm connection
+
 loop Every 20s
+    F->>Pg: request timestamp of last message pull 
+    Pg-->>F: send timestamp
+
+    F->>Pg: update start_timestamp to now
+
     F->>API: request messages
     API-->>F: send messages
 
+    F->>Pg: request device ID 
+    Pg-->>F: send device ID
+
     F->>IoT: post message to MQTT broker at topic {deviceID}/{msgId}
 end

One new participant and a handful of new messages are all that need to be defined, and Mermaid.js takes care of figuring out how the boxes and arrows should be arranged. As mentioned earlier, every highlighted line in the diff corresponds to a visible change in the diagram. Excellent!

Example 2: Activity Diagrams

Activity diagrams are a technique to describe procedural logic, business process, and work flow.

-- Fowler, Martin. UML Distilled.

Activity diagrams are similar to state diagrams, except that they model the activity of system, as opposed to the various states that a system can exist in. UML purists may cringe at the use of state diagram syntax to describe an activity diagram, but the behavior of a system can still be effectively communicated.

Imagine editing this diagram in a WYSIWYG editor. Not fun. In a text-based diagramming tool, the task is a breeze - this entire diagram can be defined in less than 75 lines of code, including comments for clarity:

stateDiagram-v2
  # State Definitions
  ## Main start conditions
  Q_cache_exists : Cache exists?
  Q_checkLastRecovery : lastRecoveryAttempt > 15 mins?

  ## Composite States
  mbRecovRoutine : Mailbox Recovery Routines
  msgRetrievalRoutine : Message Retrieval Routines

  ## Mailbox Recovery Routines
  retrieveInvalidMbs : SELECT * FROM mailbox \n WHERE errorMsg IS NOT NULL
  errCorrect : Attempt error correction
  writeLog : Write to log
  deletePgError : UPDATE mailbox SET errorMsg = NULL

  ## Message Retrieval Routines
  retrieveValidMbs : SELECT * FROM mailbox \n WHERE errorMsg IS NULL \n AND updatedAt > global.lastKnownUpdatedAt
  checkMsgs : Check for new messages 
  Q_maxRetryExceed : Max retry exceeded?

  ### Success States
  retrieveMsgs : Retrieve messages from Api
  sendToMqtt : Post messages to MQTT broker

  ### Failure States
  removeMbFromCache : Remove Mailbox from local cache
  writeErrToPg : UPDATE mailbox SET errorMsg = json(error)

  # State Transitions
  ## Start state
  [*] --> Q_cache_exists

  ## Mailbox Recovery Routines
  Q_cache_exists --> Q_checkLastRecovery: yes
  Q_checkLastRecovery --> retrieveInvalidMbs: yes
  retrieveInvalidMbs --> mbRecovRoutine
  state mbRecovRoutine {
    [*] --> errCorrect
    errCorrect --> writeLog : correction fails
    writeLog --> [*]
    errCorrect --> deletePgError: correction succeeds
    deletePgError --> [*]
  }

  ## Message Retrieval Routines
  Q_cache_exists --> retrieveValidMbs: no
  Q_checkLastRecovery --> retrieveValidMbs : no
  retrieveValidMbs --> msgRetrievalRoutine
  mbRecovRoutine --> retrieveValidMbs
  state msgRetrievalRoutine {
    [*] --> checkMsgs
    checkMsgs --> retrieveMsgs: Mailbox connection succeeds
    checkMsgs --> Q_maxRetryExceed  : Mailbox connection fails
    Q_maxRetryExceed --> checkMsgs : no
    Q_maxRetryExceed --> removeMbFromCache : yes
    removeMbFromCache --> writeErrToPg
    writeErrToPg --> [*]: sleep 15s
    retrieveMsgs --> sendToMqtt
    sendToMqtt --> [*]: sleep 15s
  }

Conclusion

Prefer diagrams as code.

• It makes developers want to work on documentation because it looks like (and is) code.
• It allows one to take advantage of powerful open-source version-control tools, such as git.
• It helps documentation stay up-to-date and remain useful for new team members.

]]>https://engineering.deptagency.com/real-life-software-project-lessons/65f45df55185580001012316Wed, 27 Mar 2024 11:36:58 GMT

I've worked on a substantial software project for one of our clients for the last eight months. It was a great and fun project with lots of lessons learned, excellent teammates, and an incredible client. We hit our milestones and the final deadline despite unexpected challenges. In the spirit of sharing what we learned, this post summarizes what we did, what worked, and what didn't.

About the project

We set out to build a minimum viable product for a new online learning platform with custom native iOS and Android apps and a Node.js backend to support both apps and handle third-party integrations. On the DEPT® side, we added to the team over time, but it consisted of three iOS developers, three Android developers, two backend developers, one designer, and one project manager. The client, an owner of various physical product brands, was pretty new to agile software development, so we helped educate them on the processes involved throughout the project.

When I joined the project, we had already completed a discovery sprint for the client and had set the scope, requirements, and deadline for the MVP launch—all essential things to set us up for success.

Lessons learned

Lesson 1: manage expectations with thought and care

The first step to ensuring we hit the MVP launch deadline was to get a list of the main features (or epics) and assign a rough estimate for their implementation time. I made a Gantt chart to visualize this for the client, help us determine whether we could achieve our goal, and plan resource usage. However, we stayed flexible and adapted throughout the project. We work in an agile way, after all.

Unfortunately, that chart put us a couple of months after the desired deadline. One common practice is to add extra time to consider various parameters, such as:

• Feature freeze date milestone
• Final QA and polish
• Security audits
• App Store/Play Store review delays
• Holidays, PTO, and illness
• Unexpected technical challenges

This list is not exhaustive, but it illustrates that things can take a left turn in many ways. So, accounting for these is essential to keeping you honest about the needed time. By accepting more risk, you can significantly reduce the time. The client wanted this in our case, and we obliged. However, we only did this after negotiating some features as stretch goals and setting the expectation that cutting down the time will carry additional risk and may require us to punt features after launch.

Lesson 2: define roles and responsibilities

Starting a project with new team members, a client not overly familiar with an agile process, and no officially dedicated PM meant we were on our own in managing ceremonies. To at least get us started, we had daily stand-ups and some weekly meetings with the client to ensure we kept moving forward on features. While it was a rough start, we did get things done as most of the team was very experienced, but the solo Android developer needed help to keep up.

Eventually, one additional Android developer joined; a month later, a third joined. The client stepped in to help fill the need for a PM to get more structure, although this was a bandaid at best as this person also had other responsibilities, which meant they couldn't focus on keeping a nice backlog of features. It pressured the team to add tickets and determine their requirements and scope, which took away their work time. I tried to take on this role initially, but it did not scale as I had other responsibilities as a tech lead and one of the two backend developers. In short, my workload increased, and my productivity suffered.

This setup lasted a few months until the temporary PM left the client. We had a good flow going but needed a real PM to step in and take pressure off the team. Finally, the client agreed that we could bring one on. Doing their best to avoid upsetting how we worked, our PM took ownership of the ceremonies and provided a more structured approach to our processes. Once the dust settled, we were full steam ahead.

We had a typical set of agile ceremonies:

• Daily stand-ups lasted up to 15 minutes.
• Backlog refinement sessions were held twice per sprint and lasted 30-60 minutes. They require detailed features, well-defined scopes, and developers preparing estimates.
• Sprint retrospectives once per sprint for about 1 hour. They are essential to help streamline processes and celebrate wins.
• We held stakeholder meetings as often as needed but usually once weekly for 30-60 minutes. We usually had them without the broader team for efficiency and to avoid polluting their calendars.
• Design reviews once per week for about one hour. These worked great for us as the design was still evolving throughout the project. The whole team could discuss and fine-tune features and requirements based on what was technically feasible.

We also had a weekly internal developer-only meeting to discuss technical implementations and a meeting with someone on the client's IT team to help unblock us as needed. That last one was important as we used systems set up by the client, so we had limited permissions to adjust configurations on our own.

Lesson 3: be intentional with communication

Ceremonies gave us plenty of time for scheduled synchronous communication with specific contexts and topics to discuss. But, having options for asynchronous and ad-hoc synchronous communication was also crucial for our success. Err on the side of over-communicating. It's vital when working in a fully remote team.

While we didn't have official communication guidelines, we shared a sense of how we wanted to communicate. In retrospect, formalizing these guidelines would've been helpful, mainly as we added new team members.

Some examples of asynchronous communication are:

• We used direct and group Slack messages mainly to initiate synchronous huddles. We kept most messages in shared channels and tagged the relevant people to ensure the team was aware of ongoing conversations. Figma was our design tool of choice. Within it, the whole squad sent asynchronous messages (comments) to follow up on specifics and implementation details.
• Backlog tickets held not only the scope and specs of the feature but also comments and references to the design files.
• We had code reviews for the developers to discuss implementation details and keep us honest about code quality.
• Finally, we also had the occasional email for things that didn't fit any other mediums or involved people needing access to the above tools.

We had a few unspoken rules of thumb when it made sense to move from async messages to a huddle or scheduled meeting:

• Three or fewer involved: typically an unscheduled huddle.
• Three or more involved: should schedule a meeting, ideally on the same day.
• Does my question have follow-up questions?
• Is there likely some back and forth to figure out the answer?
• Is there an issue affecting a feature, and will we need to devise a compromise?
• Do I need technical feedback for a design decision?

Lesson 4: be flexible on tooling

As you might've guessed from how we communicate, we're big fans of Slack for productivity here. However, as much as possible, we work with the client and their needs, whether it's simply a preference on their end, established systems, or security and compliance reasons. In this case, we used Microsoft Teams. It worked, but we had severe connection issues. These issues caused communication problems and made it hard for us to stay in touch. Thankfully, the client agreed to move most communication to a separate Slack workspace.

Another tool we used based on the client's preference was Azure DevOps. Unfortunately, in our case, no one on our project team had experience with it. Similar to Teams, it does what you need it to:

• Tracking features and bugs in tickets
• Store source code using Git
• Build and deploy pipelines.

Due to our inexperience, it took us some time to get familiar with it. Considering all the customization options, we could have set it up better. However, it was too late to make significant changes when our PM joined.

While it's a bit out of my wheelhouse, we also used Figma for the designs and graphical resources such as icons and colors. I'm glad we did; it is a great tool to collaborate on a design. We all dropped in comments there, asking for clarifications and requesting changes. Cleverly, our designer set up a separate file for the design work ready to be implemented by developers, so we only had to go back and make changes sometimes.

In an ideal world, you want to pick your favorite tools. Sometimes what you have works out, and sometimes it won't. It's essential to recognize the latter and adjust as needed. If the client is willing, this may be easier, but be flexible and work with the client.

Lesson 5: allow time for testing third-party services

The project required some specific technologies. Specifically, streaming videos and storing content in a way that a non-technical person can easily manage. Like tooling, picking these services can make your life as a developer easy or difficult.

Additionally, as we were building mobile apps with a subscription feature, we were also at the mercy of Apple's App Store and Google's Play Store, neither of which is trivial to work with.

I won't give away the names of the services we chose here, but I can share some nuggets on things to consider. First, for any third-party service you need to use, take advantage of a trial and build a small proof-of-concept to test it out. It can inform you and the client which option works best for your scenario.

Second, some services might be optional but will likely help simplify the implementation and speed things up. If so, build up a good case for why the client should budget for that, too. It would likely have helped us out a lot with subscriptions.

Lesson 6: be prepared to compromise

As mentioned earlier, we set the expectation early with the client that we had taken on a significant risk with a tight timeline. It paid off when we hit a snag with subscriptions and notifications. For both, we relied on some members not directly associated with the project on the client's side. We lacked permission to make the necessary changes ourselves.

We were halfway through the project timeline when we finally configured subscriptions. I had hoped it would be one of the first significant features we had finished as it carried the most risk. But it got delayed beyond our control. We worked through all that and got it working because we worked with the client to understand the effort put into this and the cause of the delays.

We had to bump another technically complex feature, which put us at risk of failing to hit our milestones. So we talked it through with the client and compromised a bit. Instead of supporting native notifications on day one, we'd support email notifications. It also meant the frontend developers could avoid that feature almost entirely and instead focus on finishing up other, more important features.

To summarize, have a plan but adapt as the circumstances change.

In the end, we did hit all of our target dates. We've wrapped things up with a big bow and wrote a ton of documentation for future travelers. We impressed the client and made some new friends along the way. And isn't that the most important thing for any journey?

]]>https://engineering.deptagency.com/nix/659825cceb9611000171ad1eTue, 12 Mar 2024 11:12:11 GMT

Scripting is among the most common tasks in the world of DevOps.

But have you ever run into the situation where the moment someone else goes to run one of your scripts, it immediately fails because their environment is slightly different from your own? What about your CI environment, is it different too? Creating consistent environments is a consistent pain point.

So what next? Write up a set of instructions for what utilities need to be installed? Create a Dockerfile with all of the apt-get installs? Do instructions for people, Dockerfile for CI? Containerize everything and map host paths to the host machine's config files and whatnot?

Each of these solutions comes with their own compromises. Additionally, these solutions are very likely to break over time when packages update with breaking changes, packages get removed, or any number of other scenarios. This leads to having to update instructions, Dockerfiles, scripts, and so forth at what is likely a very inconvenient time and after you've already forgotten how those scripts work!

Well, we don't have to deal with that... ok, maybe still a little, but at least our packages won't change underneath us.

Introducing Nix

So what is Nix exactly? Just another package manager? It's a bit more than that, but to fully understand, consider how these problems are solved elsewhere. We are likely all familiar at this point with the package management systems that come with various languages.

• Nuget for C#
• NPM for node.js
• pip/venv for Python
• The list here could get very long...

If you think about a generic package management solution for scripting, you are probably thinking about apt-get or yum. These, however, have several distinct disadvantages over the solutions listed above:

• These utilities are tied to the OS and thus vary significantly between machines (redhat, debian, arch, etc.)
• It is unlikely that the versions of the utilities two people end of using are the same (my version of bash is 5.2.15, what's yours?)
• They don't provide any package locking type solutions
• Each package manager doesn't even have the same library of utilities available

Nix fixes all of these problems. Additionally, it fixes it regardless of the OS the user is using and allows you to fully define everything about the execution environment of the scripts you are running. Nix is powerful enough that it can even be used to replace the package management systems above, though usually the preferred route is to simply tie into those systems.

Too good to be true? Well, no system is perfect. There are significant drawbacks to Nix, including poor documentation and a difficult to understand configuration language. Fortunately, for our purposes here, these drawbacks will be mitigated by setting up a relatively unchanging scaffolded environment.

Bowls of Nix flakes

Nix Flakes is the system we use to provide both package locking and execution environment management. I have created this demo here to show a solid setup for using flakes, so let's break down the demo and see how it works!

From the above demo, you can see that when you run one of the scripts in the demo are run for the first time, all of the dependent utilities for the script are automatically downloaded and made available to the script environment! This means that any utilities that you need in your script, such as jq, kubectl, or just about anything else are automatically pulled without you as a user needing to think about it at all! Your end users do not need to preinstall anything except for nix itself!

Our customized scaffold

The scaffolding is primarily handled through the flake.nix file, but additionally includes an easy-of-use wrapper for running scripts.

The ./run Wrapper

The ./run wrapper is used to make executing individual scripts easier, especially for users of your project who are not familiar with the (quite quirky) nix command line. Additionally, the wrapper includes some basic messaging if the user doesn't have the nix tools installed or forgets to specify which script to execute. As most of this wrapper is self-explanatory, let's move on to the guts of our scaffolding: flake.nix

Diving in to our Customized Nix Flake

One of the biggest concerns I have had with using the Nix tooling in projects with a wider and mixed skillset is that we don't want to require everyone on the team or users of the project to need to learn and understand Yet Another Domain Specific Language™ just to solve package management requirements around our primary scripts. So here's out attempt to gain these advantages without requiring users to fully understand Nix.

The basic structure of using inputs = {... and outputs = {... is defined by the Flake schema. You may notice that we are using nixpkgs-unstable as one of our inputs and that might be a tad alarming to some to see that, however it's worth remembering that our automatically generated flake.lock will ensure that whatever versions of packages we use do not change without us intentionally changing them. If the use of the unstable channel is concerning, however, it can be locked to one of the stable releases such as 23.11 at the time of this writing.

Next up, we define our dependencies:

scriptDeps = with pkgs; [
    nixFormatter
    jq
    git
    gnugrep
    curl
    kubectl
];

Every one of these packages will be installed to the environment that our scripts are executed in. For any utilities you want to add, all you have to do is search for them in the NixOS package search and add them to the list! For example, we added kubectl to the list from this search

Next, we want to make it so that the addition or removal of scripts can be done without needing to touch our Nix code at all. The way we accomplish this is by having our flake scan the scripts/ directory for files that have some predetermined extensions. This means that if someone wants to add a new script to our project, all they have to do is add it to the scripts/ directory with one of the extensions that we specify; no nix code required!

We also want to allow multiple "types" of scripts, automatically determined by extension, so that we can change script headers, footers, or even dependencies based on the extension of the script. While the demo only includes BASH scripts, this could also allow for running any other types of scripts such as python, go, or so on. In our demo, we specify two extensions .std.sh and .tf.sh like so:

# headers here, defined outside the list so they can refer to each other
stdShHeader = ''
    #!${pkgs.stdenv.shell}
    set -Eeou pipefail
    export PATH="$PATH:${scriptEnv}/bin"
'';
tfShHeader = ''
    ${stdShHeader}
    echo 'Running the extra tasks for .tf.sh'
'';

# Define metadata for each file suffix and the headers/exec command to attach to them
scriptSuffixes = [
    {
    suffix = ".std.sh";
    header = stdShHeader;
    command = "exec";
    }
    {
    suffix = ".tf.sh";
    header = tfShHeader;
    command = "exec";
    }
];

In our usage here, we're defining stdShHeader and tfShHeader outside of our list of suffixes to keep it easy to refer to other headers, but these could just as easily be defined inline with our list.

Finally, the real workhorse of our customized flake. This is where we do the actual walking of the scripts/ directory using the metadata defined in the list above:

scriptMappings = builtins.map
        # for each script suffix we...
        (typeAttrs:
        let
            # Find all scripts in the directory with our expected suffix
            scriptsFound = builtins.filter (name: lib.hasSuffix typeAttrs.suffix name) scriptDirScripts;
            # Map those found scripts to command names such that `format.std.sh` becomes `format`
            scriptNames = builtins.map (name: builtins.replaceStrings [ typeAttrs.suffix ] [ "" ] name) scriptsFound;
            # Create a list of maps where the command is set to "name" and the path to the script is set to "value", e.g. [{"name":"format","value":"./scripts/format.std.sh", ...}]
            scriptAttrLists = builtins.map (name: { name = name; value = scriptDir + "/${name}${typeAttrs.suffix}"; }) scriptNames;
            # Convert that list of maps into a single mapping where command name is the key, path is the value. e.g. {"format":"./scripts/format.std.sh", ...}
            scriptAttrs = builtins.listToAttrs scriptAttrLists;
            # Finally, instead of JUST the path, add the full formatting of the wrapper script (including the header and exec command) to the values
            # {"format": "<all contents of wrapper script>", ...}
            scriptContents = builtins.mapAttrs
            (name: value: ''
                ${typeAttrs.header}
                ${typeAttrs.command} ${value} "$@"
            '')
            scriptAttrs;
        in
        scriptContents)
    scriptSuffixes;

We'll let the comments in the above snippets do most of the talking, but basically we are creating a script mapping from each of the scripts found and adding the script headers and exec command. This mapping effectively creates a "wrapper" script for each one of the scripts found that might look something like this for the command ./run format:

#!/usr/bin/env bash
set -Eeou pipefail
export PATH="$PATH:/nix/store/generated-env-dir/bin"
exec ./scripts/format.std.sh "$@"

The Help Command

We also want to ensure that our code is as self-documenting as possible while also keeping documentation simple. While this isn't a replacement for complete documentation by other means, maintaining quick help text can be made semi-automatic. To do this, we can just add another script file to our project called help.std.sh. You can see the full contents of the demo help script here.

In a nutshell, what we do is this:

• Scan the scripts/ directory for files that end in the extensions that we care about (.std.sh and .tf.sh)
• For each script that we find, use grep to find the first line of code that starts with # HELPTEXT:
• Output each script command along with the HELP output in a human readable format.

The result looks like this:

$ ./run help
Usage: ./run <command> [args...]

Standard commands:
  format                        Format flake.nix
  help                          Display this help output
  update-packages               Update the flake.lock with the latest version of all dependencies

Terraform commands:
  test                          Demo for files with a .tf.sh extension instead

note: any arguments passed after <command> are passed directly to the script that handles that command.

Final results

What we're left with after doing this project is a structure that looks like this:

• 📂 scripts/
  • 📄 help.std.sh
  • 📄 format.std.sh
  • 📄 test.tf.sh
  • 📄 update-packages.std.sh
• 📄 flake.nix
• 📄 flake.lock
• 📄 run

The maintenance once the boilerplate is done is simple:

• Any scripts that we want to add to this project can then simply be added to the scripts/ folder with one of our extensions and it will be autodetected as a new run command. Remember that scripts need to be in git git add and need to be executable chmod +x.
• Any new dependencies can be added to scriptDeps inside of flake.nix
• Updating the dependency lock can be done with ./run update-packages
• New types of scripts, such as python scripts, can be supported by amending scriptSuffixes in flake.nix and updating dependencies
• New users of the project do not need to install any dependencies except for Nix itself. Our run script will dump a message telling them to install Nix if it's missing.

So with a little up-front work, we have a project with scripts that will run on anyone's machine, works in any CI environment, self-documents its own commands, is easy to extend and maintain, and best of all it doesn't require anything other than Nix to run anywhere!

Would you use this yourself?

]]>https://engineering.deptagency.com/achieving-agility-and-flexibility-with-aem-hybrid-implementation/65cf74d5a736620001ba9df4Thu, 29 Feb 2024 13:00:56 GMT

Introduction

Implementing AEM with a hybrid approach offers benefits like agility, scalability, and flexibility.

Yet, it requires careful planning, design, and execution, considering technical and organizational factors to ensure robustness and scalability.

AEM solution approaches

AEM (Adobe Experience Manager) is a content management system (CMS) that allows organizations to manage their digital content across various channels. AEM offers three primary approaches to building web applications: fullstack, headless and hybrid.

A Full Stack AEM architecture is a unified system, often referred to as the Traditional CMS. 

This approach reduces the overall platform management costs significantly, as it consolidates all technology stacks into a single architecture. Additionally, it enables the utilization of Out-of-the-Box (OOTB) components, streamlining and expediting the development process.

Once an AEM project is created, the JS and CSS from the ui.frontend module are compressed and stored in the app/project-name/clientlibs directory.

Within the AEM component, Sightly (HTL - HTML Template Language) can be employed within HTML to dynamically render content from Sling models. All aspects of content creation, styling, delivery, and presentation are centralized within AEM, affording complete control over content editing.

However, it's important to note that content cannot be directly exposed and disseminated to multiple external channels outside of the system in this setup.

On the other hand, Headless architecture decouples the frontend and backend, with the term "Headless" indicating the absence of a specific presentation channel being tied to it.

It offers the potential for enhanced performance and facilitates multi-channel delivery. It has an API-driven model, where AEM content is transmitted in JSON format to various channels, enabling these channels to create tailored designs.

In the Headless Architecture, content fragments and content fragment models are accessed using the GraphQL API or Asset Rest API, reducing reliance on the backend. In this setup, content is provided as a service API, enabling interaction between the backend and any frontend, accessible via any device. It necessitates increased involvement from frontend developers.

In this framework, content creation occurs within AEM, while styling, presentation, and delivery occur on a separate platform. It represents a contemporary development approach for implementing experiences across websites.

A Hybrid CMS represents an ideal amalgamation of strengths. Within AEM, we can employ Hybrid Single Page Applications (SPAs) to seamlessly blend the characteristics of both Traditional and Headless architectures.

This approach combines the efficiency and user-friendliness of a traditional CMS with the adaptability and scalability inherent in a headless development framework.

Depending on specific business requirements, we can opt for either the traditional or headless approach. However, once a choice is made regarding the frontend, the hybrid solution may not offer the same degree of flexibility that a purely headless approach can provide.

With a hybrid CMS, we can either create our own templates or utilize pre-existing ones, enabling us to make content readily available and easily reusable across various channels and devices by utilizing APIs. Out-of-the-box components can streamline the delivery of content to the web.

Options for Hybrid solutions within the AEM ecosystem

AEM SPA

The SPA Editor, provided by AEM, offers a solution for integrating React and Angular applications directly with AEM for in-context editing. It is most effective when:

• You intend to develop your entire frontend for a site using React or Angular.
• You aim to minimize the familiarity required with AEM for your frontend developers.
• You require SPA-controlled routing and want to serve SPA web pages from AEM.

Content authoring for SPA components remains stored in the JCR but is presented as a JSON representation through component mapping instead of crafting AEM HTL templates. This JSON can then be exposed to other channels similarly to before.

An advantageous aspect of this approach is that Adobe has re-implemented many Core Components in React and Angular, facilitating direct parent/child SPA component relationships and familiar data flows for SPA developers.

However, this solution has some limitations within AEM due to SPA-controlled rendering instead of HTML templates. For an up-to-date list of limitations, please refer to the Adobe documentation.

Despite these limitations, the SPA Editor is actively maintained and developed by Adobe, raising hopes for the gradual removal of these constraints.

AEM Remote SPA

Remote SPA, another AEM-provided solution, enables externally hosted React applications to be editable within AEM. In function, it parallels the SPA Editor but with the SPA server delivering the pages instead of AEM. This allows the engineering team to predominantly construct site components outside of AEM and independently scale page traffic from AEM.

Drawbacks of this approach include the necessity for separate server infrastructure, distinct deployment cycles, the same limitations as SPA Editor, and it only supports React currently.

Nevertheless, this might represent the optimal choice for integrating authoring into an existing SPA application or meeting project deadlines for a team less familiar with AEM.

Final thoughts

In conclusion, an AEM hybrid approach using React combines the benefits of headless and headful approaches, providing organizations with the flexibility to use the latest front-end technologies while still leveraging AEM's powerful content management and personalization features and even when the trade off can look higher the benefits in the long term are bigger.

Need AEM support? DEPT® is a platinum Adobe solution partner.

]]>https://engineering.deptagency.com/dynamic-components-in-your-mpa/65ca2960dc04860001d5d70aTue, 27 Feb 2024 13:04:05 GMT

You have a multi-page application (MPA) which renders your frontend and it meets 99% of your needs. Your HTML is rendered from a server-side templating language like ASP, Razor, JSP, Django Templates, Pug, Handlebars, etc.

It's lightweight, fast, and ideal in its technical simplicity. Up until now, any dynamic behavior you've needed has been easy enough to add via progressive enhancement in vanilla JS using imperative programming. But then the newest business requirement comes in and it's a real doozy. You know immediately: this would be easy to do as part of an SPA using a frontend framework like React or Vue, but this feels like a huge ask under the current architecture.

So you need to introduce some dynamic components to your MPA code-base.

What should you do? This article aims to discuss different scenarios and how they can be handled in the order of most ideal to least ideal. These scenarios and my recommended solutions are not entirely exhaustive, but recent work forced me to consider these scenarios more directly. Hopefully my experience and research can be an aid to you and your work.

Scenario 1: Your existing stack already accounts for this!

The first step I recommend would be to see if your current toolchain already has a built in solution that fits your needs.

The examples in this category are somewhat artificially limited, as there are many tech stacks which account for having dynamic client-side behavior with server-side rendering. However, there's only a few cases where that hybrid behavior isn't at the core of how that technology functions. For example, it would be a rare case for someone to use NextJS without the explicit understanding that you naturally have access to client-side React when you need it. But there are some frameworks for which this kind of feature isn't on the front-page of the documentation.

Blazor

If you've found yourself in this position and are lucky enough to already be using C#, .NET, and Razor templates then the good news is that you won't have to stray far to create dynamic components in a technology which you are already comfortable with. Your toolchain has actually already considered this for you in the form of Blazor. Blazor is a technology which gives you access to a superset of Razor templating syntax to support the addition of dynamic client-side behavior. The C# code in Blazor files compiles down to WASM to allow you to use the same tools on the client and server.

<button @onclick="Increment">The count is @count</button>

@code {
    private int count = 0;

    private void Increment()
    {
        count++;
    }
}

A simple Blazor counter component

Considerations for Blazor

If this is your scenario, then you can probably stop reading as Blazor is likely the right technology for your situation.

That being said, it's not a perfect choice. Blazor is slow. It is among the slowest client-side frameworks. This is because the current model for WASM is not intended for building full applications. WASM is intended to offload high-intensity logic from JavaScript to a more performant platform and then communicate the results of that workload back to JavaScript for your application to utilize. For WASM to support a full application and be able to dynamically update the DOM, it must come with a binding-library which exposes all the functions it needs as a bridge back to JavaScript, and that bridge is a major performance bottleneck. As such, Blazor is so slow that I personally can only recommend it in this exact scenario: You already have a .NET application, and you have now realized that you need dynamic client-side components. But I can't in good faith recommend starting a new application with the intention of heavily relying on Blazor.

Phoenix LiveView

There are often features that come with some backend frameworks that give you a bit of prebuilt JavaScript to interface with some more advanced features of the backend framework. If you were making an application using Elixir, Phoenix, and HEEx templates, then you may already have a solution that could work for you in Phoenix LiveView. LiveView is a tool within the Phoenix framework that takes advantage of the BEAM VM's excellent concurrency safety & performance to give stateful updates via a socket connection. This allows the server to own and update UI state which may be exactly what you need to create your dynamic component without having to reach for another tool outside of your current stack.

Challenges & considerations for Phoenix LiveView

LiveView is a great tool for its use-cases but those cases can be limited. And obviously, running stateful updates for dynamic behavior on your server can create some scaling challenges. This also makes your client-side state very limited.

Scenario 2: Your existing stack can be stretched to meet your needs!

HTMX

If you find yourself in the scenario for which your existing architecture can't solve this problem for you, you may want to consider HTMX. HTMX is a single JavaScript bundle that allows you to write dynamic behavior driven entirely by the server. This is similar in concept to Phoenix LiveView but it doesn't entirely rely on sockets or the BEAM concurrency model. In the HTMX model, your server exposes endpoints that act like component templates. These endpoints return HTML fragments, rather than a full document.

server.js

const context = { count: 0 };

on.post("/increment", (request) => {
    context.count++;
    return render(request, "counter.html", context);
});

counter.html

<button hx-post="/increment" hx-swap="outerHTML">
    The count is {{ count }}
</button>

A very primitive counter component implemented in HTMX

Challenges & considerations for HTMX

HTMX can be a very powerful option that allows for a wide range of added capabilities with minimal additions to your architecture. You can continue to use your existing server & templating system as you were before. However, it is not without its drawbacks & complexities. The core challenge of this model can be seen in the above example: all component state must now live on the server, and more ideally in your user's sessions.

In the example above, the context is global to the server. So in that example all users would share one single value for count. In an SPA model, simple pieces of state like this will be naturally segregated to each user's browser environment, whereas the HTMX model forces even simple pieces of dummy state to be maintained by your server. In an SPA, the memory would naturally be dumped when the user navigates or closes the tab/window, whereas in HTMX the server must make a standardized decision for when to stop holding onto that state. And of course where the performance of Blazor DOM updates was limited to the slow WASM to JavaScript bridge, the performance of HTMX updates are limited to the speed of the network. While this means that your initial page loads are faster because your users aren't downloading templates for components that haven't had their state modified yet, this could be considered the worst case scenario for real-time behavior performance.

HTMX has proven to be a fantastic solution in many cases, but it isn't applicable to all scenarios. If you need lightweight interactivity sprinkled throughout a website, then HTMX could be the ideal solution for your use-case. But if you need dense & responsive interactivity, then it may not meet your needs.

Scenario 3: Your stack can't meet your needs.

After reviewing the existing tools available in your stack and considering tools like HTMX, you may still find your options lacking for what you are being tasked to build. So instead, you want to bring in an SPA-like JavaScript component development flow alongside your existing application. So does this mean it's time to bring in React? There are still some more ideal options to consider first. The challenge with React is that there's no natural interface between server-generated HTML and rendering a declared React component. But is there a platform which offers a natural interface between HTML & rendering a declared component? Thankfully yes! There are several frameworks which run on the back of the web-component architecture which provide us with exactly such a model.

Lit

Lit is a framework by Google which allows you to make web-components in a simple and standardized fashion. The key advantage of having your components registered as web-components is that the process of mounting your components happens naturally in the browser's custom elements API. However, the actual development experience is very akin to developing with React class components, rendering using template strings rather than JSX.

import { LitElement, html } from "lit";
import { customElement, property } from "lit/decorators.js";

@customElement("lit-counter")
export class MyElement extends LitElement {
  @property({ type: Number })
  count = 0;

  private _increment() {
    this.count++;
  }

  render() {
    return html`
      <button @click=${this._increment}>The count is ${this.count}</button>
    `;
  }
}

A counter component in Lit

As long as the bundled JavaScript output is added to any & all pages which need these components within your MPA, then all you have to do to add this component is put <lit-counter></lit-counter> in your HTML and the component will be naturally instantiated by the browser. So that means no added script tags which target a <div> with a particular #id on it. Just use your components in a natural fashion.

Stencil

Stencil is a very similar tool to Lit with slightly different design decisions & UX. Stencil aims to be slightly closer to React, and thus uses JSX. However, it also uses the same concept of decorators which denote reactive properties which trigger a re-render.

import { h, Component, Prop } from '@stencil/core';

@Component({ tag: 'stencil-counter' })
export class Counter {
  @Prop() count = 0;

  increment() {
    this.count++;
  }

  render() {
    return <button onClick={() => this.increment()}>The count is {this.count}</button>;
  }
}

A counter component in Stencil

As can be seen from the two examples, Stencil & Lit share very similar architecture both from a DX perspective as well as an implementation perspective. However, Stencil's use of JSX gives it a few advantages. Namely, that Stencil components are actually internally strongly typed with TypeScript. Part of the Stencil compilation process builds out your component tags with their props to the global JSX namespace. So if I wanted to use the above Stencil component in another component and I wrote <stencil-counter count="5" />, I would actually get a compilation error from TypeScript that informs me that property count must be a number and not a string. Comparably, when using the Lit example, <lit-counter count="5"></lit-counter> would actually be the proper syntax, and you could only get a runtime error if you passed in a string which could not be converted to a number.

Challenges & considerations for both Stencil & Lit

Both Lit & Stencil are built to use the web-component architecture, so there are certain integrations that will not work as expected by default. For instance, if your project uses an atomic CSS tool like Tailwind or Bootstrap then it may have some small integration hurdles with these technologies. web-components are defined as a series of browser native features used together. One of those technologies is the shadow DOM, which creates an isolated environment for each of your component instances to run in. That way your components don't have to worry about adding styles which affect the rest of the document and they don't have to worry about being affected by the styles of the rest of the document. This can obviously create a very safe & stable development experience, but if your architecture relied on styles coming from a large shared style sheet, then this could be a major hindrance to you.

Lit & Stencil are both built with the full structure & safety of the web-component architecture in mind, but luckily they both have escape hatches to avoid things like the shadow-DOM. With Lit, you just need to override the createRenderRoot method in your component. Normally this method returns a shadow-root but instead you can have it return this because custom-element classes are, themselves, DOM elements that extend the HTMLElement class. In Stencil, you only need to add the shadow: false option to the configuration object passed to your @Component decorator. This means that with very little work both of these tools can be made to support the architecture of most projects as needed, but they are not without unique considerations.

Scenario 4: You've been prescribed a solution.

We've all been there: Your project manager tells you that another development team has already built the component in React. They even bundled it into a library, so all you need to do is drop in the library right? But you know it's not that easy. You're not using React. You're not even really using a frontend framework. So how can we make this process of utilizing React in your MPA as painless as possible?

Bundling a Vite app into an MPA

There are steps that just can't be skipped surrounding the bundling, building, and exporting process. In my experience, I've found that utilizing Vite is the best way to go these days. It solves the most problems up front, has the fastest performance, and requires very little in terms of configuration overrides.

If you instantiate a Vite application inside your MPA repository, you will naturally get a vite.config.js file. In the Vite config, if you set config.build.rollupOptions.output.manualChunks to undefined and set config.build.rollupOptions.output.entryFileNames to something simple like "app.js", you will remove all chunking & file-name hashing from the build output so that your JS bundle will always be one file with a consistent name. This makes it much easier to link to from your MPA. If you want to invalidate old script builds from cache for users, then you can easily bust the cache by requesting the JavaScript file with an arbitrary param like your latest server start time or time from your most recent build. So in a Node server, for example, the built JavaScript file could be requested from /app.js?cacheBust=${performance.timeOrigin}.

The only other configuration option that must be updated at this point is config.build.outDir which is where you want all output files to be placed. Be warned, that if you've defined a custom config.root then your outDir will be relative to that root. You'll want to make the value of the outDir point to the directory where your backend wants you to place statically hosted files. I would also personally recommend that you put the outDir as its own subdirectory which is marked in your .gitignore so that you can avoid committing build outputs to your actual repository.

Finally, you may want to add a postbuild script to clean up artifacts which you don't want in the output. In the case of Vite, the root of each build is actually the index.html file, so you'll probably just want to delete the copy of that file in your outDir after every build completion.

Once all of these items are complete, then you can just add the Vite installation & testing steps to your existing CI/CD flow and add the build step to fire before your backend build. If some of these steps feel a little bit like a code-smell, they should. This is inherently the twisting of a tool to be what we need it to be. This is not exactly how Vite is intended to be used, but this is the position you will sometimes find yourself in. And from my past experience, I would argue that this makes for better long-term maintenance than fully building out exactly what you need with a custom Webpack configuration.

Exposing React components to your MPA

Now that you have a JavaScript bundle being built and linked to by your MPA, how can we best expose React components to your existing templating? For this step, I recommend looking at all the advantages of the prior technologies and do your best to roll them into this solution. You're not going to find a better solution for exposing JavaScript components to HTML than custom elements, so why not lean into it!

import { createRoot } from "react-dom/client";
import { type ComponentType, useState } from "react";

abstract class ReactMountingElement extends HTMLElement {
  abstract readonly Component: ComponentType;

  readonly #root = createRoot(this);

  connectedCallback() {
    this.#root.render(<this.Component />);
  }

  disconnectedCallback() {
    this.#root.unmount();
  }
}

const Counter = () => {
  const [count, setCount] = useState(0);

  return (
    <button onClick={() => setCount((count) => count + 1)}>
      The count is {count}
    </button>
  );
};

const exposedComponents: Record<string, ComponentType> = {
  "react-counter": Counter,
};

Object.entries(exposedComponents).forEach(([tag, Component]) =>
  window.customElements.define(
    tag,
    class extends ReactMountingElement {
      Component = Component;
    }
  )
);

A React counter component which is exposed as a custom element

The above code shows a very simple pattern for exposing React components as custom elements. Once the above code is added to the bundle loaded into your MPA, you can create instances of the counter component using <react-counter></react-counter>! This gives you the simplest possible interface between React & your backend HTML templating engine. This doesn't solve all challenges, but it's a very simple baseline to build off of. From here, you can make decisions that fit your needs around things like accepting children to your component, managing incoming props, and especially non-children HTML/JSX props. For those solutions, I think utilizing a web-component model using <template> & <slot> elements would be the best approach but that doesn't make it easily solved.

The drawbacks to this solution

The obvious drawback here is in the fact that we are once again bending a tool slightly outside of how it is intended to be used. And for that reason, we have to re-solve problems that are solved by default in the solutions for the previous scenarios. What was supposed to be the easy solution, "just implement an already existing React library," has become an avalanche of custom infrastructure problems that must all be solved (or will likely eventually need to be solved).

To summarize...

There are so many different versions of this scenario that you might find yourself in. You may discover that it was never an issue because you can integrate something like Blazor into your already existing app. Or you might discover that HTMX meets your needs and you can run all the dynamic components out of server session state. But if neither of these cases meet your needs, but you still have full control over the technology used, I cannot recommend enough leaning into frameworks & tools that have this situation in mind like Lit or Stencil. However, if you find yourself needing to integrate a specific frontend framework into your MPA, I hope that this article has shown you that it's within reach.

Sidebar: Protecting yourself from these scenarios in the future

There are a lot of new technologies that are appearing in the web development world that solve these kinds of problems by default. Obviously tools like Next, Nuxt, Angular Universal (or Analog.js), SvelteKit, SolidStart, or QwikCity all exist as meta-frameworks around existing frontend frameworks. However, there are also tools which allow you the simplicity of server-side only rendering, while being well prepared for an escape hatch into the frontend frameworks. At the forefront of this model and a technology that I can personally recommend, is Astro.

Astro is built to default for simplicity by shipping zero JavaScript to the client by default, so it is incredibly fast. However, Astro is also built for adaptability by offering a suite of plugins, adapters, and integrations. Astro uses SSG by default for maximum performance, but it has SSR adapters to allow it to run best however it's deployed. Astro ships no JavaScript by default, but it has plug-ins for every major frontend framework so that components in that framework can be used naturally in Astro's templates. When you mount a React component in Astro, the React plugin will offer you the full control for exactly how you want that component to hydrate via a series of directives.

So if you want to build a web application with maximum foresight for scenarios like this, then picking a meta framework would be a great way to secure yourself when complex business requests come in. But picking a framework agnostic platform like Astro can give you maximum performance and technical simplicity early in a project, while giving you the ability to easily grow your architectural complexity.

]]>https://ashwinsundar.com/blog/compiled/django-front-end.html65a4763d7000a20001d3464cThu, 25 Jan 2024 14:10:30 GMT

"Front-end" in web development refers to the visual appearance of an application or website. In Django, a web development framework, the front-end is created in DTL (Django Template Language), which is a superset of HTML (Hypertext Markup Language).

DTL represents the "what" of the page - the actual contents. The "how" of the page - how the page appears - is defined by CSS (Cascading Style Sheets).

Django template language (DTL) files

Here's an example of a DTL file:

stats-pane.html

{% load static %} <link rel = "stylesheet" href = "{% static 'css/stats-pane.css' %}">
<div class = "stats-pane">
    <div class = "stat-component-A">
        {% include "components/stat-component.html" with title="On Target" numdetail=stats.on_target_count %}
    </div>
    <div class = "stat-component-B">
        {% include "components/stat-component.html" with title="At Risk" numdetail=stats.at_risk %}
    </div>
    <div class = "stat-component-C">
        {% include "components/stat-component.html" with title="Efficiency Index" numdetail=stats.efficiency green=9 yellow=4 red=0 %}
    </div>
</div>

Here is what is happening in this file:

• The file loads a CSS stylesheet called stats-pane.css
• The file establishes scaffolding for three components - stat-component-A, stat-component-B, and stat-component-C
• Each div.stat-component includes a template file called stat-component.html
  • Information is passed to these templates using the with keyword

Here are the contents of the referenced stat-component.html file:

stat-component.html

{% extends "components/generic-square.html" %}
{% block content %}
    {% load static %} <link rel = "stylesheet" href = "{% static 'css/components/stat-component.css' %}">
    <div class = "stat-container">
        <div class = "stat-title">
            {{ title }}
        </div>
        {% if numdetail > green %}
        <div class = "stat-numdetail stat-green">
        {% elif numdetail > yellow %}
        <div class = "stat-numdetail stat-yellow">
        {% elif numdetail > red %}
        <div class = "stat-numdetail stat-red">
        {% else %}
        <div class = "stat-numdetail">
        {% endif %}
            {{ numdetail }}
        </div>
    </div>
{% endblock content %}

• The file extends an existing template called generic-square.html. This means that some pre-existing content has been defined in the generic-square.html template, and this file will override some of the content.
• The file loads a CSS stylesheet called stat-component.css.
• {% block content %} represents the start of the overrideable section. All content from here until {% endblock content %} will override the section in the generic-square.html file of the same name.
• The file defines a framework for containing a single statistic - stat-title and multiple stat-numdetail components, each wrapped in conditional logic.
  • Conditional logic determines whether particular styles shall be rendered. For example, if the numdetail variable is greater than the green variable, the statistic shall be displayed with the class stat-green. This class merely colors the text of the component in green.

Cascading Style Sheets (CSS) Files

CSS describes how the elements of an HTML page appear. Here is the CSS file used by the stats-pane.html file described in the first section:

stats-pane.css

.stats-pane {
  display: flex;
  flex-direction: row;
  gap: 10px;
  justify-content: center;
  align-items: center;
}

• The text before the {}, .stats-pane, is called an element selector. We know this element selector applies to classes because the statement begins with a .. If this applied to a different selector, such as id, it would begin with a different character.
• This definition applies styling characteristics to elements that have an attribute of class = "stats-pane".
  • In this case, the stats-pane is defined as a layout style called flexbox[^flexbox], and the elements of the flexbox should appear as rows. The gap between each element is set to 10 pixels, and finally the contents are center-aligned.

Summary

• Django Template Language (DTL) files represents the "what" of the page:
  • What are the structural elements (i.e. HTML elements)?
  • What is the basic logic for the page (i.e. embedded scripting)?
• Cascading Style Sheets (CSS) files define the "how" of the page:
  • How are elements rendered (i.e. styles)?
  • How do those elements appear on the page (i.e. animations, transitions)?

While DTL contains some Python-esque language features, it only permits a small subset of the Python language to be run. This is a design choice - permitting arbitrary Python code to run can be dangerous. That said, DTL is fairly powerful and extensible via custom filters and tags.

CSS takes some time to become acquainted with and ultimately master. The core specification is constantly improving. This guide is a good way to understand the modern contours and best practices of CSS in 2024.

]]>https://engineering.deptagency.com/prisma-vs-kysely/648370a99cab1e000127ea50Wed, 20 Dec 2023 13:21:21 GMT

So, we set out to find a replacement to better fit our needs. Having worked with Knex before, we were very impressed with Kysely. It's a modern query builder with a great TypeScript focus. As we'll see here, it also performs a lot better than Prisma.

To set the context a bit, the project we're working on uses Nest.js and it has a database with around 30 tables. It has support for localization and several many-to-many relations. It's a backend for iOS and Android apps. We'll use a sample database with a similar size since we cannot share the database schema here.

We spent a couple of days getting acquainted with Kysely before deciding on our approach. Our setup allowed us to refactor from Prisma piece by piece. This is my attempt at documenting my findings and thoughts on why Prisma didn't cut it for us. Fair warning: opinions ahead, your miles may vary, etc.

High-level comparison - Prisma vs Kysely

Prisma sits somewhere between a traditional ORM and a query builder. Prisma uses a single schema file for defining the model. It then uses that to generate database migrations in SQL files. This alone makes Prisma's DX one of the best among ORMs and query builders. Prisma still lacks some common features though, such as compatibility with views, custom column constraints, and handling nullable unique indexes.

Prisma also heavily abstracts SQL, so it's pretty much impossible to optimize the SQL it runs. For example, Prisma's generated SQL prefers executing several queries over using joins. Depending on your use case, this might be a good or bad thing.

In contrast, Kysely is a query builder. The downside (upside) is that you don't get as many batteries included as you would with Prisma. It does have some migration support but is not as well-defined as Prisma's. You also don't get the TypeScript types generated for you out of the box, but there are tools to help you there. For example, you can use kysely-codegen to generate TypeScript definitions from your database. Or use prisma-kysely to reuse a Prisma schema file. Because it's a lot closer to writing raw SQL, it can also be a lot more powerful.

Not having everything included can be a good thing. With Kysely you can pick whichever database driver you need ("dialect" in Kysely's terms). You can use the official pg driver, or you can use the newer postgres.js alternative. We did not notice a considerable performance difference between the two though. Likely because Kysely already does the heavy lifting.

Drizzle deserves an honorable mention here. It is another query builder with features like Kysely. One upside with Drizzle is that their migrations, like Prisma, are only SQL. Also, you write Drizzle's schemas in TypeScript but don't use the up/down approach of Kysely. This post won't cover it in detail, but we'll reference it in some places as an extra point of comparison.

Performance - fetching 1,000 films

This is using the DVD Rental sample database from PostgreSQL Tutorial. I've set up a sample repository for you to run the queries below and play around further with each setup. As a bonus, there's also a Drizzle test included.

The idea is this: fetch the most recent 1,000 films with their categories and actors included. Order films alphabetically if they have the same release year. Also, order categories and actors alphabetically.

Note: Prisma's prisma db pull command does not let you map names to camel-case. Kysely's CamelCasePlugin can do this for you though. If we wanted to achieve the same result in Prisma we'd have to manually add @map(...) and @@map(...) as needed. To save some time, the generated Prisma schema uses snake_case.

const latestFilms = await db
  .selectFrom("film")
  .limit(1000)
  .orderBy(["film.releaseYear desc", "film.title asc"])
  .selectAll("film")
  .groupBy("film.filmId")
  .select((eb) => [
    jsonArrayFrom(
      eb
        .selectFrom("filmCategory")
        .innerJoin(
          "category",
          "category.categoryId",
          "filmCategory.categoryId"
        )
        .where("filmCategory.filmId", "=", eb.ref("film.filmId"))
        .orderBy(["category.name asc"])
        .selectAll("category")
    ).as("categories"),
    jsonArrayFrom(
      eb
        .selectFrom("filmActor")
        .innerJoin("actor", "actor.actorId", "filmActor.actorId")
        .where("filmActor.filmId", "=", eb.ref("film.filmId"))
        .orderBy(["actor.lastName asc", "actor.firstName asc"])
        .selectAll("actor")
    ).as("actors"),
  ]);

const latestFilmsQuery = await prisma.film.findMany({
  include: {
    film_actor: {
      include: {
        actor: true,
      },
      orderBy: [
        { actor: { last_name: "asc" } },
        { actor: { first_name: "asc" } },
      ],
    },
    film_category: {
      include: {
        category: true,
      },
      orderBy: [{ category: { name: "asc" } }],
    },
  },
  take: 1000,
  orderBy: [{ release_year: "desc" }, { title: "asc" }],
});

At first glance, Prisma wins because the "query" looks so simple. But when we consider how Prisma executes the queries, it's not as simple anymore. Yes, it's a bit of a contrived example, but it does highlight some interesting points. I've run into similar queries in various projects, so it's not an unlikely scenario.

The queries are not a perfect apples-to-apples comparison. As mentioned, out of the box, Prisma doesn't do inner joins as our Kysely query does. It runs a separate query for each table it encounters. Prisma will only rely on joins if needed for filters. This is likely fine for smaller sets of data and can even outperform Kysely's query. But once you start fetching many rows, Kysely performs a lot better.

On my machine, I usually see Kysely averages around 50 ms for the above, while Prisma averages 110 ms. Again, the queries differ, but they do give the same core data back.

We also tried Prisma's preview feature relationJoins. While it's closer to what our Kysely query does, it's a lot slower at 240 ms. Finally, Drizzle runs that same query at around 75 ms using the pg driver. Surprisingly, its postgres.js driver was a bit slower.

One can also argue that Kysely's result is easier to reason about. With Prisma, you get all the nested many-to-many types in there too. If we were working on a backend API, then our front-end developers would not be happy if we returned that as-is. With Kysely you just get the data you requested and nothing more.

Migrations - a few options

You could keep using Prisma's migration style and generate types via prisma-kysely. This works well as long as you don't need to do something the Prisma schema does not support. Support for views is coming soon, but based on the preview feature they will be a bit awkward to manage.

Prisma's schema does not work with custom check constraints or multi-column unique indexes. If you customize the generated SQL, Prisma may not do what you might expect.

Another option is to use Kysely's built-in migrations. If you're familiar with Knex this should be trivial and a good approach. I never went this route so I can't say whether it comes with any challenges. One upside is that you just write TypeScript and don't need to learn Prisma's schema oddities.

A third option is to use a third-party CLI for forward-only SQL migrations. You can also write some custom scripts for that. The upside of this approach is that it forces you to understand the underlying schema. And then rely on kysely-codegen to generate the type definitions.

Finally, you could use Drizzle Kit to manage your schema but still use Kysely to write your queries. The Drizzle ORM includes support for that out of the box.

Compatibility - more than you bargained for

Prisma is very heavy to install at over 15 MB. Kysely, in comparison, takes a bit over 2 MB. This makes Prisma not the ideal choice when running code in a serverless context. As we understand it, Prisma relies on a 4 MB WASM file for its query engine, written in Rust. There's nothing wrong with Rust, but Kysely is simply a lot leaner.

Prisma's query engine needs to support PostgreSQL, SQLite, MongoDB, and more. With Kysely, like many other Node.js database libraries out there, you only need to install a driver.

Summary - wrong batteries

We're not the first to sing high praise of Kysely and feel let down by Prisma. Kysely will be a main contender for us to interact with SQL from now on. One takeaway from all this is that you should consider what you need out of a library, but also what you don't need. If not, you will fight against it and lose productivity.

]]>https://engineering.deptagency.com/flexible-composables/6512ebce905b4d0001c27a74Fri, 08 Dec 2023 14:18:49 GMT

Unlike the traditional XML layouts, Compose makes it easy to create UI components that are easily scalable, reusable, and provide a consistent experience for your application's users.

When creating the UI components, remember to follow these rules:

1. Avoid hardcoding any text, states, or values
2. Keep the size of the container scalable
3. Do not hold any logic in your UI
4. Keep it well documented, and add previews to show different ways of usage

Let's dive into it and see some examples of how we can take the Compose approach to create the following layouts:

Identify similarities

Although they might look different at first glance, we can identify the following similarities:

1. Both are contained in a scalable box with rounded corners
2. Both have a box outline of equal width

Secondly, they have the following properties, which can be customized with parameters. These include:

1. The outline color
2. The box background color

Creating a reusable component

Using this information, we can setup a scalable box like this:

@Composable
fun ScalableBox(
    boxColor: Color = Color.White,
    outlineColor: Color = Color.Unspecified,
    boxOnClick: () -> Unit = {},
    modifier: Modifier,
    content: @Composable () -> Unit = {},
) {
    Box(
        modifier = modifier
            .clip(RoundedCornerShape(12.dp))
            .background(boxColor)
            .border(
                width = 2.dp,
                color = outlineColor,
                shape = RoundedCornerShape(12.dp)
            )
            .clickable { boxOnClick }
            .padding(16.dp)
    ) {
        content()
    }
}

This Composable has some fixed properties such as padding, rounded corner radius, and outline width. Further, it has some customizable properties such as box color, outline color, and the content that will be contained inside the box. These are all passed as parameters.

Now, we can enclose our main UI elements in this Scalable Box, passing it the properties we need accordingly.

The code for the yellow-bordered, gray background box:

@Composable
@Preview(showBackground = true, backgroundColor = 0xFFFFFFFF, widthDp = 300, heightDp = 200)
fun ParagraphFieldPreview() {
    Column(Modifier.padding(16.dp)) {
        ScalableBox(
            boxColor = Color.DarkGray,
            outlineColor = Color.Yellow,
            modifier = Modifier
        ) {
            Text(
                text = "Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod " +
                        "tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, " +
                        "quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.",
                color = Color.White
            )
        }
    }
}

And the code for the black bordered, white background box:

@Composable
@Preview(showBackground = true, backgroundColor = 0xFFFFFFFF, widthDp = 300, heightDp = 200)
fun ImageAndTextFieldPreview() {
    Column(Modifier.padding(16.dp)) {
        ScalableBox(
            boxColor = Color.White,
            outlineColor = Color.Black,
            modifier = Modifier
        ) {
            Row(verticalAlignment = Alignment.CenterVertically) {
                Image(
                    painter = painterResource(id = R.drawable.ic_launcher_foreground),
                    contentDescription = null,
                    modifier = Modifier.size(60.dp),
                    colorFilter = ColorFilter.tint(Color.Black.copy(alpha = 0.8f))
                )
                Text(text = "Lorem ipsum dolor")
            }
        }
    }
}

Both Composable uses ScalableBox to configure their container properties and create the enclosed UI.

And just like that, you have created your first flexible UI components with Jetpack Compose!

Wrapping up

That all sounds great, but what if your requirements need you to create a color-changing spiral box? You can still use the same scalable box component!

Check out the full source code below to find out how:
https://github.com/deptagency/JetpackComposeExample

Cheers!

]]>https://engineering.deptagency.com/how-implement-kyc-aml-protocols-nft-marketplace/6549498d880df90001944fbdTue, 05 Dec 2023 14:16:45 GMT

In our previous blog post, we went over what KYC/AML is and why your company may need it: https://engineering.deptagency.com/why-you-need-kyc.

In this article, we will be going over how we approached KYC and AML measures for an NFT collectible marketplace for a major sports organization, a marketplace for digital collectibles on the Algorand blockchain. First, we'll talk about the KYC process, then how we addressed customer support, and finally, what some of the challenges were in building and maintaining this system.

Acronyms

• NFT = Non-Fungible Token
• KYC = Know Your Customer
• AML = Anti-Money Laundering
• SSN = Social Security Number
• PII = Personal Identifiable Information
• PEP = Politically Exposed People

🏅 Background

In the NFT collectible marketplace we built, users can sign up and purchase packs that have an unknown assortment of collectibles, or they can purchase from the secondary marketplace, where individual collectibles are sold for a price determined by the seller. Making the platform accessible to people new to blockchain technologies was a top priority, so accepting credit card payments (in addition to crypto) was important. This, however, increases the risk of fraudulent transactions.

We used Circle for the underlying financial technologies. When purchases are made using credit cards, those funds are converted to USDC-A (USDC on the Algorand blockchain) which are kept in an Algorand wallet that is created per account. Users purchase “credits,” which means loading USDC into the account’s wallet. Circle is a financial institution that has to abide by Anti-Money Laundering (AML) laws, and we entered a KYC reliance agreement with Circle that dictates what KYC and AML processes we need to have in place for compliance.

Our contract with Circle details that we are responsible for enacting fraud detection on their behalf according to the established AML Policy, and therefore, we needed to find a provider to assist with identity verification (KYC) and flagging suspicious or fraudulent behavior. This provider cannot make final decisions; that responsibility is on us, but they can assist in collecting the data needed to make a final decision.

Initially, when trying to find a provider, Trulioo, Jumio, and Onfido were recommended to us, and all seemed to have services for what we needed. See the previous KYC article for advice on what to look for when choosing a provider. For us, the features we needed supported included:

• Identity verification
• Documentary verification
• Biometric verification
• Ongoing monitoring against AML watchlist

All of the mentioned providers appeared to have products to accomplish what we needed, and Trulioo and Onfido seemed the best match for the technologies we were using. We were impressed with Onfido’s related products, and we opted to use them as our KYC provider.

🪪 The KYC Process

When a user goes through the KYC process, they have to enter personal details, provide documentary evidence (such as a passport), and take a selfie to confirm their identity. Checks are run against AML watchlists - sanctions, politically exposed persons or PEP, and adverse media lists. Onfido is used for data collection and analysis; the workflow logic was contained within Onfido Studio workflows, dictating the flow a user goes through after initiating verification and what data is gathered. There is also logic that lives within our application; we have our own user statuses that we choose for an account after comparing Onfido’s results to the criteria that lives within the application.

Since not everyone is going through the KYC verification process, we built gates throughout the application, both on the frontend and backend, which check the user account status when certain functionality is attempted. When the thresholds are reached, a user will be prompted to go through the KYC verification process to continue transacting on the site. Most commonly users are prompted to go through KYC during the checkout flow, or when attempting to make a payout.

Crypto purchases have some additional restrictions, as certain locales are prohibited from making payments using this method, and we are required to check if any crypto addresses used matched sanctions lists. This feature wasn’t supported by our KYC provider Onfido, but was required for compliance with the AML policy, so we opted to use Chainalysis’ free crypto sanctions product, which allows us to check if an address is sanctioned. At the time we were initially looking into a solution Algorand did not have any sanctioned addresses but it was still a requirement to be monitoring for sanctioned addresses in case that changed.

Making programmatic decisions

Initially we thought that we’d be able to make most decisions automatically based on the results from Onfido’s API, but we quickly realized that there are many situations where we could not be confident in the results. In fact, it would be faster to tell you what we could confidently make programmatic decisions rather than what we couldn’t.

• Happy path: A user goes through KYC verification without issue, provides a document they are confirmed as the owner of, is not from a restricted locale, and there’s no matching activity on AML watchlists. This user would be placed into a Clear status in our system and would be able to transact freely on the platform, though we would continue to monitor AML lists.
• Unhappy path: A user is from a prohibited region. We would know if the user is using a restricted locale during the verification workflow when the user provided address data, as well as if they used a form of identification from that region, so we can more confidently restrict based on that data.
• Unhappy path: Discrepancy between the document and selfie provided. This indicates the person is using someone else’s identification method fraudulently, and would result in them being restricted on the platform.

For all other unhappy paths, we place the user into Manual Review. This status requires further exploration by someone on the compliance team. There are regular situations that could lead to an unclear result, such as submitting a blurry photo for the document or providing an unclear selfie. If there is a problem with the data provided, the customer service team can request the user try verification again by placing into the appropriate status for them to retry, and communicating the issue.

The most common reason, however, that someone falsely ends up being flagged during the process is due to AML watchlists. We check AML watchlists to identify if any active users match sanctions, PEP, or adverse media watchlists, all of which would be prohibited. The AML watchlists very often have false matches, with an emphasis on the adverse media and politically exposed persons (PEP) watchlists. Due to the many false positives, this requires human intervention to analyze the record and determine if the user is the one identified by the watchlist. This analysis takes time, as you have to review the data that was flagged by Onfido as well as the user’s provided data initially and make a decision on if that is the same person. Sometimes it’s obvious, but other times it can be difficult to determine. It is also wise to analyze other user interaction data to make sure there’s no suspicious behavior flagged for any users that are looked into as well.

Since further exploration is often needed when users are placed into Manual Review, this had implications for customer support. We knew customer support is a necessary part of the KYC process, but it was underestimated how much of a lift this would be.

💁 Customer Support

Users would often write in with basic questions, some of which the customer service team can be prepared to answer, but the vast majority of the time questions were handed off to those familiar with the AML policies. It was pivotal to have decision maker(s) who could guide the customer service team and help them with what information to share and what not to share, and who can do further analysis to determine what action should be taken and communicate to the development team if any behavior needs to be verified.

In order to support the compliance team, which was doing the more in-depth analysis, we built various tools for analyzing user behavior. We started out using a headless CMS to build out administrative functionality, but we transitioned to using a product called Retool since it made creating these tools much easier.

We found the following tools very useful in our KYC exploration.

💻 Administrative tools

Many tools were built to support the application, but I’d consider the following tools most critical when analyzing fraudulent or suspicious activity. It was also necessary to educate the compliance team on how to use the tools, and to support them by creating new tools as needed to assist in customer support.

1) User Diagnostics

The most widely used application we created in Retool was for user diagnostics. This enabled searching for users using various criteria, and if a user is selected, this would populate information on associated payments, payment cards, transactions, payouts, marketplace listings, login history, notes, and more.

The only action that could be taken in this application is changing the user status and optionally adding notes. This was a requirement when analyzing users in manual review, since a final status needs to be chosen that either enables or restricts their account.

2) Fraud Detection

Circle would periodically send us a CSV with information about refunds and chargebacks so we built a tool that directly ingested this CSV and processed it. This allowed us to respond quickly when users were flagged.

We also built out functionality that would allow us to use IP addresses and user agent information to help identify accounts that are likely owned by the same user. It is not reliable to use IP addresses to indicate the same person logged into an account. IP addresses are shared by more than one household in certain locales, and VPNs add a whole other layer of complication. It can, however, be helpful to analyze in addition to other user behavior.

3) Marketplace statistics

One thing the compliance team needed to look for was items being sold for suspiciously high prices. We built functionality around analyzing marketplace statistics to determine reasonable prices for marketplace items. Minimums, maximums, averages, and medians were collected per collectible in a materialized view in the database and displayed within Retool.

4) Track bot activity

Users would alert us to bots being used on the site, but there were legitimate uses and we were not able to simply ban all bots. Therefore, we built functionality using Google Recaptcha that tracks scores for each purchase event. The score indicates the likelihood of it having been a bot. We also built a tool in Retool for listing purchase events with this score, along with data on the buyer and seller. This allowed us to keep tabs on where bots were being used, so we could analyze each situation more effectively.

5) Process refunds

In certain cases users had started transacting on the site and then were banned. This could happen, for instance, if a user is from a restricted country but that information was not provided on sign up. We needed to build support for refunding users that had loaded funds but were in a restricted state and not able to cash out.

🔔 Notification System

A notification system has been critical for alerting the compliance team and/or customer service team to certain suspicious activity within the application. Often suspicious activity is not clear-cut enough to act on programmatically, so often what we decided to do was to identify the behavior, and then alert the team, as opposed to taking automatic action. Alerts go out when there are:

1) Overly-priced marketplace items

As mentioned, we built a materialized view within our database that compiled data on sales statistics. This was then used to determine reasonable prices for sales on the marketplace and allows us to alert the compliance team whenever a suspiciously high-priced item is sold on the marketplace.

2) Maximum amount of payment cards is exceeded

A user can only have a small number of cards in their account to prevent fraudulent behavior where someone creates an account and then tries to use various stolen cards for purchase. The compliance team is alerted whenever a card is added to a user account, after a certain threshold.

3) Workflow completed with a Rejected or Manual Review result

The user is alerted anytime the verification workflow is completed for their account. If the account is moved to manual review or restricted, then the compliance team is notified. The user diagnostics tool can be used for further analysis, either searching for all users in review, or searching for a particular user.

4) AML watchlist report completed

An ongoing monitoring subscription is made in Onfido for every verified user on the platform. The monitor continuously checks AML watchlists for any matches for that user. If added to an AML watchlist, the user's account will be restricted or moved into manual review, and the compliance team is notified so they can assess the situation.

⛰️ Biggest challenges

There are high stakes in the KYC world. You think you’ve thought of everything, and the fraudsters find a new way! The KYC process and gating of features should be thoroughly tested, and new issues reported by customers should be looked into within a short time frame. We found other users were a big help in finding suspicious behavior, and users will write in with questions which can uncover flaws which need to be addressed. You’ll thank yourself that you’re responsive and looking into each report when there’s a serious issue that needs attention as quickly as possible. Thankfully we were able to resolve each situation quickly, and always made modifications to deter future situations.

Challenge 1: Preventing duplicate accounts

In our situation, we chose to not require KYC verification for every user. There would be reduced complexity if all users went through verification, but that would also deter some users. We opted to only require KYC when certain limitations were reached according to our AML Policy. This makes it easier to sign up, but also adds complexity, as we have to enable/disable functionality for more situations.

Since we are not requiring KYC for every user, we do not know it’s the same user. Users are prohibited from having multiple accounts, but that of course did not deter many from doing just that. We needed to find a way to identify duplicate accounts. We did not have a way of being sure without requiring verification for all, but we did come up with ways to help us determine.

One situation that presented itself was that users being flagged in chargebacks were being found to use the same card with multiple accounts. We built out the functionality to restrict users if they used a card already being used by another account. If users wrote in to dispute, we would explain the requirement and after they agreed to respect that, we would enable their account again, assuming there wasn’t anything else to be suspicious about.

In addition, we started tracking IP addresses and user agents. This is not reliable, as IP addresses aren’t unique per household in every locale, and it’s also acceptable for two people to live together and have an account. It is also easy to get around IP checks by using a VPN, and on the flip side, multiple people could be using the same VPN IP address. It is, however, a helpful piece of information to analyze while analyzing other information. If a user has a matching IP address and user agent associated with multiple accounts, it’s likely they are the same person.

These methods helped us narrow down, but they weren't the silver bullet we were looking for. We assessed fingerprinting services but opted out because we were skeptical about how useful they would be to us. Our team found through testing that it too frequently could not identify duplicates, and the cost was too significant to warrant without being able to use it with more confidence.

The more we searched, the more we felt confident that the only reliable way to find duplicate accounts was through KYC verification. Instead of requiring verification on sign-up, we decided to implement additional gates that would require verification to use additional features. In addition, we planned to require documentary verification for every user instead of having certain locales have the option to provide an SSN in place of a government document. This would allow us to run biometric verification for every user, and Onfido even has a product called Known Faces that would alert us to the same person associated with different accounts. We had not yet implemented this change, so I cannot speak to how effective it is. Still, we hoped to reduce the number of duplicate accounts by requiring KYC verification more frequently and then watching for matching faces between different accounts.

Challenge 2: Reducing fraudulent transactions on the marketplace

The secondary marketplace is where we found the majority of suspicious behavior. Users can sell items they purchased on the marketplace for a predetermined amount. One fraudulent scheme we identified is when a user sells items on the marketplace, and then signs into other accounts to purchase the items using stolen credit cards. Ultimately, fraudulent users are deterred because to withdraw, you have to complete KYC verification. It’s possible someone hacks into an account that has KYC verification completed, which would be the jackpot for a fraudster, but that had not been the case with any situations we came across. We did have one user complete verification in order to gather funds after hacking into a user’s account; we could report that user so they would be flagged for other platforms, but they clearly found the reward to be more attractive than that.

After analyzing the user behavior, we decided to implement a couple features that would help deter similar fraudulent activity on the platform. I’ll go into each below.

a) Restrict the amount of cards a user could have in one account

By restricting the amount of cards a user can have saved in their account, we can cut down on the number of fraudsters that load stolen cards into their account. Most users have one card saved in their account, maybe two. It would be unusual to have a large assortment, which is common with fraudulent situations.

The feature we implemented would check for the number of cards associated with a user when a purchase was made. If over the threshold and KYC verification has not yet been completed, the user will be required to complete verification. If over the threshold and the user has completed KYC verification, the compliance team will be notified for each additional card added. When we released this feature, all user accounts with cards above the threshold were moved to a status that would require KYC verification to continue using the platform.

b) Prevent multiple accounts from having the same card associated

Duplicate accounts are prohibited on the platform, so if there is a card matching numerous accounts, it is suspicious and could mean that someone is trying to load stolen cards into multiple accounts. Our payment provider, Circle, has a fingerprint value for each payment card used. We use this fingerprint value to determine if multiple accounts are using the same payment card and restrict those users.

If the user reaches out to the compliance team, their account may be unblocked if there is a reasonable explanation. One reasonable situation we identified was that some users were using a service that protects the card number, where temporary numbers are generated that will result in the card being charged but hide the actual card number from the payment service. If a user used a service of this type, they would quickly be flagged as having too many cards.

c) Track overly-priced items

To review what was discussed in the Administrative Tools section, a materialized view was created in the database that tracked averages, min, max, and medians for collectibles on the marketplace. These values were then used programmatically to determine if there was a suspiciously-high-priced item on the marketplace that was just sold. Users may accidentally list an item for too high of a price, but if that item is sold, we can assume that was a fraudulent transaction. No user would opt to purchase the much more expensive version of the same item.

d) Track too high of a success rate

A success rate that is too high can indicate fraudulent activity. If a user sold every item they tried to sell, that would be highly suspicious. We planned to build support for tracking this information and adding it to the notification system to alert the compliance team.

Users flagged for too high of a success rate may first be flagged for selling high-priced items, but it's also possible someone games the system selling lower-priced items, so tracking success rate is another important indicator.

Challenge 3: Communicating KYC process to users

Many updates were made to increase clarity for users around the KYC process. These decisions were usually made by the development and administrative teams. It could have been helpful to have more design expertise involved in those decisions, as it is difficult to properly communicate the process to a user. In an ideal world, a KYC process would be user-tested to see where confusion lies.

A confusing KYC process is frustrating to users. Your users must trust your platform to make purchases, and confusion can lead to distrust. There are also really high stakes with KYC since fraudulent users could lead to legal repercussions as well. It’s to everyone’s benefit that the process be smooth and seamless, and adequate attention should be paid to making this process as seamless as any checkout flow.

It does seem the industry is moving towards in-house KYC done by payment providers. This comes with its benefits and drawbacks. Our setup had a lot of flexibility, which may not be the case when KYC is done in-house, since that provider is assuming all that risk. It would greatly simplify several aspects and may reduce the complexity in communicating to users. I’d need to use a service that had KYC in-house before making an effective comparison.

Final Thoughts

Some final thoughts from our experience implementing KYC/AML protocols for an NFT marketplace…

There's no one-size-fits-all solution.

Your specific KYC verification and AML protocols will depend entirely on your business plan and the amount of risk you’re taking. There may be unique requirements provided by service providers as well.

While KYC for every user would have been much simpler, it would have provided barriers to purchase.

And anyone who does funnel optimization will tell you that you should remove barriers to purchase, not add them…

But playing devil’s advocate, if KYC is required for basic functionality that any user would need to use, such as payouts, there may be an argument for providing that barrier earlier on.

It's expensive to build and maintain

This can be a difficult pill to swallow if there isn't steady revenue coming in or if the majority of sales are for low-price tag items. You'll find that services that provide KYC in-house often have a much higher minimum price requirement.

Fiat transactions made KYC verification necessary

Supporting payments and payouts using fiat currencies was a primary goal of this project, but if supporting fiat was not a requirement, KYC verification could have been avoided altogether. Blockchain address verification against sanctions lists might still be necessary, but it is a much lighter lift. This would have greatly simplified the project, but not supporting fiat transactions would have scared away any customers new to the crypto space.

If the service you are building is successful, it only becomes more challenging.

Consider it a compliment if the fraudsters find you! The more popular the service is, the quicker someone is going to find a critical issue or a way to exploit the system. Be responsive in reviewing issues, and regularly and thoroughly test the verification and security protocols so you can find any issues as quickly as possible.

---

I hope our takeaways from implementing a verification system can be useful to you in your project. There is a lot to understand in the KYC/AML world, so if you’re new to the topic, try not to get overwhelmed too quickly. For us, there have been a lot of iterations and a lot of learning over time. It’s also a rapidly evolving industry. If you are implementing for your own project, the best takeaway may be that it’s complicated. Consider the verification process and related security protocols an integral part of your system that requires as much attention as any critical path, even if it is not utilized by every user.

And if you want to implement a verification system for your project and want to work with someone towards that goal, hit us up! We’d love to take that on with you.

]]>https://engineering.deptagency.com/prop-drilling-in-react-solutions-and-trade-offs-2/6566562fffd18400019b45c5Thu, 30 Nov 2023 15:37:33 GMT

Prop Drilling is the act of passing data, in this case, react props, through several nested layers of components before it reaches the component that needs the data.

Is this a problem in and of itself? It depends (which is the best answer for a question). Props are part of the “React way” of managing state, so using them, even passing them through nested components, isn’t necessarily bad. It depends on the scale.​

For small applications, this is not an issue. You can use either of the approaches featured in this article or none. When a React app grows larger, Prop Drilling becomes an issue. It can make things hard to read, and it couples the parent and child components together, making it difficult to reuse components. When does a small app become a large one? Good question, moving on.

​Let's set up a scenario to better understand what's happening. Let's say we have an app that is set up something like this image.

The Data Page component is fetching our data to be used. Part of that data (tags, prices, attributes, etc.) needs to be used by our filter and sort buttons and their respective modal/form/popover/etc. That code might look something like this…

DataPage (){
  const [filterData, setFilterData] = useState({})
  // ...
  return
    (
      <OptionsBar MetaData={filterData} />
      <DataArea />
    )
}

// OptionsBar.jsx
OptionsBar({filterData}){
  // ...
  return (
    <SearchBar />
    <FilterAndSort filterData={filterData}/>
  )
}

// FilterAndSort.jsx
FilterAndSort({}){
  // ...
  return (
    <Filter filterData={filterData} />
  )
}

// Filter.jsx
Filter({filterData}){
  // ...
  return(
    <>
      <Button>Filter</Button>
      <Modal
        content={<FilterForm filterForm={filterData}/>} 
      />
    <>
  )
}

As you can see, the filterData prop gets passed through several intermediate components before it eventually gets used in the filterForm.

---

Enter Context:

One way to solve the problem of prop drilling is to use React’s built-in contextAPI. Creating a context provider allows every child of that provider, regardless of how deeply nested it is, access to the context data. That implementation might look something like this.​

// App.jsx
import { createContext, useContext } from "react";

DataPage(){
  const [filterData, setFilterData] = useState({})
  // Create Context
  const dataContext = createContext();
  
  ...

  return (
    <dataContext.Provider value={{data: filterData}}>
      <OptionsBar />
      <DataArea />
    </dataContext.Provider>
  )
}

// Filter.jsx
  export Filter({}){
    const { data } = useContext(dataContext)
    return(
      <>
        <Button>Filter</Button>
        <Modal
          content={<FilterForm filterForm={data}/>} 
        />
      <>
    )
  }

By using context, we skip over all the intermediate components and access our data exactly where we need it.

Problem solved! Reacts own documentation suggests using context to solve the issue of prop drilling, "Using context, we can avoid passing props through intermediate elements"⁴. With how easy that solution was, it will be tempting to use context everywhere. This is where we can run into some issues. Let's say we add a userContex, themeContext, and a authContext and end up with a render that looks something like this.

```jsx
render (
  <userContex.Provider value={userData}>
    <themeContext.Provider value={theme}>
      <authContext.Provider value={authentication}>
        <dataContext.Provider value={{data: filterData}}>
          <OptionsBar />
          <DataArea />
        </dataContext.Provider>
      </authContext.Provider>
    </themeContext.Provider>
  </userContex.Provider>
)
```

Now, we have contexts inside of other contexts, leading to unnecessary complexity and confusion. Tracking down bugs or making updates can become a chore with a touch of hide and seek. "Was userLanguage in authentication or userData?" It is also important to remember that using context means that any component and/or their children that access the context data are tightly coupled, thus reducing its re-usability.

​There is another caveat to using context. This is especially true when the data in the context provider is an object. If a value in that object gets updated, react will replace the whole object with a new one¹. This will trigger a re-render on all components using that context and possibly their children. So if that update happens to be on a high-level parent component or a root level one, it could potentially re-render the entire page.​

React's documentation offers a warning about the use of context​.

Context is very tempting to use! However, this also means it’s too easy to overuse it. Just because you need to pass some props several levels deep doesn’t mean you should put that information into context.¹

​and again​

Before You Use Context
Context is primarily used when some data needs to be accessible by many components at different nesting levels. Apply it sparingly because it makes component reuse more difficult.>If you only want to avoid passing some props through many levels, component composition is often a simpler solution than context.⁴

---

Component Composition:

As seen above, React’s documentation suggests that "Component Composition" is a better solution. In other words, put your components in areas where they can access the props they need while skipping over the intermediaries. You can do this by lifting components up in the tree and using child components as props.

​There are a couple different approaches to component composition. One way we could do this is by moving our components up the tree and wrapping them inside of the parent component. In the example below, we could move all of our imports up to the DataPage and wrap them inside of each other allowing the filterData prop to be passed directly to the Filter component.

// DataPage.jsx
DataPage(){
const [filterData, setFilterData] = useState({})
  // ...
  return (
    <>
      <OptionsBar>
        <FilterAndSort>
          <Filter filterData={filterData}/>
        </FilterAndSort>
      </OptionsBar>
      <DataArea />
    </>
  )
}
// OptionsBar.jsx
OptionsBar({ children }){
  // ...
  return (
    // ...
    { children }
  )
}
// FilterAndSort.jsx
FilterAndSort({children}){
  // ...
  return (
    // ...
    { children }
  )
}

// Filter.jsx
Filter({filterData}){
  // ...
  return(
    <>
      <Button>Filter</Button>
      <Modal
        content={<FilterForm filterForm={filterData}/>} 
      />
    <>
  )
}

Admittedly, this is an extreme example and could impact the readability, size, and complexity of our parent component. Luckily, we have flexibility with how we compose our components! We could instead move the wrapping one level down into the OptionsBar component to make things more readable.​

// DataPage.jsx
DataPage(){
const [filterData, setFilterData] = useState({})
  // ...
  return (
    <>
      <OptionsBar filterData={filterData}/>
      <DataArea />
    </>
  )
}
// OptionsBar.jsx
OptionsBar({filterData}){
  // ...
  return (
    <Search />
    <FilterAndSort>
      <Filter filterData={filterData}/>
    </FilterAndSort>
  )
}

​Another approach would be to alter our components to have props that take components. This way we can set our component up in the parent, pass our props to it, and then pass that whole component as a prop.​

// OptionsBar.jsx
OptionsBar({filterData}){
  const filter = <Filter filterData={filterData} />
  ...
  return (
    <Search />
    <FilterAndSort filter={filter} sort={...} />
  )
}

// FilterAndSort.jsx
FilterAndSort({filter, sort}){
  return (
    <div>
      { filter }
      { sort }
    </div>
  )
}

Recap:

💡

“There are no solutions. There are only trade-offs” - Thomas Sowell

So, context can solve the issue of prop drilling by allowing data to be accessed right where we need it. But, it comes with issues of making components hard to reuse and unnecessary re-renders. Component Composition is a better solution. We can retain the flexibility, re-usability, and readability of components. But, it can lead to large and complex parent components depending on how you compose your components (see what I did there). Of course, you are not limited to choosing just one or the other. The best solution for your app may be a mix of both.​​

References:

1. https://react.dev/reference/react/useContext

2. https://react.dev/learn/passing-data-deeply-with-context

3. https://legacy.reactjs.org/docs/composition-vs-inheritance.html

4. https://legacy.reactjs.org/docs/context.html

]]>https://engineering.deptagency.com/convert-get-to-post-inline-in-java-with-spring-gateway/65563db62a64e10001c758deThu, 16 Nov 2023 16:53:21 GMT

You have a third-party provider that you need to obtain news information from.

The only API they have for this is a POST with a request body, yet this clearly should be a GET process since the API is simply retrieving data and nothing transactional is happening.

You are trying to keep your internal APIs as standard as possible and so would like the front-end to request this data as a GET with query parameters. If you are using Java and a Spring gateway, one solution is to receive the GET from the front-end and convert it to a POST inline prior to reaching out to the provider with the request.

We can make use of a request decorator in a gateway filter to alter the original request. This decorator extends ServerHttpRequestDecorator which wraps another ServerHttpRequest and delegates all methods to it. Subclasses can override specific methods selectively. We are overriding the getMethodValue and getURI methods. We override the getMethodValue method to return HttpMethod.POST.name() and we override the getURI method to remove the query parameters from the request.

We still have the request body to deal with. In our route definition, we apply the filter above before modifying the request body to ensure that any new headers we added in the filter are preserved. You can use the ModifyRequestBody filter to modify the request body before it is sent downstream by the gateway.

We use the NewsBody class to convert the query parameters into a json string suitable for a request body.

Full route example:

We have successfully received a GET from our front-end and converted it to a POST for our provider. Spring gateway offers many built-in gateway filters for adjusting requests and responses like Retry GatewayFilter Factory and JsonToGrpc GatewayFilter Factory.

]]>https://engineering.deptagency.com/guide-solid-principles/6536976a1ce9e00001a2aae6Thu, 09 Nov 2023 14:43:39 GMT

A design principle empowers you to handle the complexity of the design process more efficiently, serving as a best practice to simplify design. Design Principles don’t provide a direct answer, but they offer a framework for approaching and solving problems.

Introduced by Robert C. Martin in 2000 in his paper on design principles and patterns, let’s first understand the distinctions between design principles and patterns.

Differences between design principles and design patterns:

Aspect	Design Patterns	Design Principles
Purpose	Reusable solutions for common problems	Guidelines and best practices for design and problem-solving
Application	Direct solutions or answers to specific problems	Overarching approaches to follow best practices
Flexibility	Specific and should be applied when needed	Broad and applicable in various design situations
Interaction with others	Can be used in conjunction with design principles	Complement each other and can sometimes conflict
Examples	Singleton, Observer, Factory Method, etc.	SOLID principles, DRY (Don't Repeat Yourself), KISS (Keep It Simple, Stupid), etc.

Through each of these sections, we’ll focus on various design principles, with particular attention to the five SOLID principles, these will be illustrated through examples.

SOLID is a widely embraced principle for Object-Oriented programming languages. It’s an acronym for five principles:

• Single Responsibility Principle (SRP)
• Open-Closed Principle (OCP)
• Liskov Substitution Principle (LSP)
• Interface Segregation Principle (ISP)
• Dependency Inversion Principle (DIP)

Single Responsibility Principle: Writing clean and maintainable code

SRP suggests that classes, modules, and functions should each have a singular focus. The approach involves breaking down these entities into smaller components, each handling a distinct task. This strategy accelerates development and testing, enhancing understanding of each component’s individual role. By adhering to this principle, you can evaluate class scope to ensure it doesn’t breach SRP.

Let’s say we want to create a user class in which we obtain an image for the user and upload it to the S3 bucket. After obtaining the path to the uploaded file, we will update the database, and set the file path as the profile path for the user. So, let’s start with an empty class containing an empty method called updateAvatar

class UserService{
  public async updateAvatar(id: number, file: any) {
    try {
      // connect to S3
      // upload to S3
      // get path
      // find user and update it with given path
    } catch (error) {
      console.error('Avatar update failed:', error);
    }
  }
}

As you can see, we have an empty method for updating the avatar, which receives the path and updates the database. Now, let’s proceed to actually push it to S3.

class UserService{
  public async updateAvatar(id: number, file: any) {
    try {
      const s3 = new AWS.S3();
      const params = {
         Bucket: 'my-bucket',
         Key: `avatars/${this.id}-${Date.now()}.jpg`,
         Body: file,
      };

      const uploadResult = await s3.upload(params).promise();
      const avatarPath = uploadResult.Location;
    } catch (error) {
      console.error('Avatar update failed:', error);
    }
  }
}

Now that we have the upload result, we can update the database.

const user = await this.usersRepository.findById(id);
user.avatarPath = avatarPath;
await user.save();

So, we’re going to end up with a class that looks like this:

class UserService {
  private usersRepository: UserRepository;

  constructor(usersRepository: UserRepository) {
   this.usersRepository = usersRepository;
  }

  public async updateAvatar(id: number, file: any) {
    try {
      const s3 = new AWS.S3();
      const params = {
        Bucket: 'my-bucket',
        Key: `avatars/${this.id}-${Date.now()}.jpg`,
        Body: file,
      };

      const uploadResult = await s3.upload(params).promise();
      const avatarPath = uploadResult.Location;

      const user = await this.usersRepository.findById(id);
      user.avatarPath = avatarPath;
      await user.save();
    } catch (error) {
      console.error('Avatar update failed:', error);
    }
  }
}

As you can see in the example above, both uploading a file to S3 and interacting with the database, specifically writing to it, have been combined within a single method. This means that our updateAvatar method has two responsibilities:

1. Uploading a file.
2. Writing to the database.

This practice contradicts the first principle of SOLID, which is the Single Responsibility Principle. Now, let’s refactor the code:

1. First, we create a “File” class that is responsible for uploading files.
2. Then, we create a “UserUpdater” class, which is responsible for updating user-related information.
3. Finally, we can include the file upload method inside our updateAvatar method to orchestrate these two methods together.

By separating these responsibilities into distinct classes, we adhere to SOLID principles and maintain a more modular and maintainable codebase.

class FileService {
  public async upload(file: any): Promise<string> {
    const s3 = new AWS.S3();
    const extension = this.getFileExtension(file);
    const params = {
      Bucket: 'my-bucket',
      Key: `avatars/${Date.now()}.${extension}`,
      Body: file,
    };

    const uploadResult = await s3.upload(params).promise();
    return uploadResult.Location;
  }

  public getFileExtension(file: any): string {
    const fileName = file.name || '';
    const parts = fileName.split('.');
    if (parts.length > 1)
      return parts[parts.length - 1];
    return '';
  }
}

As you can see, our FileService class now has the sole responsibility of handling files and includes a method to upload a file. Now, let’s refactor the “User” class as well to adhere to the Single Responsibility Principle (SRP).

class UserService {
  private usersRepository: UserRepository;
  private fileService: FileService;

  constructor(usersRepository: UserRepository, fileService: FileService) {
    this.usersRepository = usersRepository;
    this.fileService = fileService;
  }

  public async update(id: number, fields: Record<string, any>) {
    let user = await this.usersRepository.findById(id);
    user = {...user, ...fields};
    await user.save();
    return user;
  }
 
  public async updateAvatar(id: number, file: any) {
    try {
      const avatarPath = await this.fileService.upload(file);
      await this.update(id, { avatar: avatarPath });
    } catch (error) {
      console.error('Avatar update failed:', error);
    }
  }
}

With this approach, we’ll have code that is more testable and reusable. The file uploader can be reused in other services, and the user update functionality can be used to update other fields of the user. However, when used together within the updateAvatar method, they will cooperate seamlessly. This design promotes modularity and maintainability in our codebase.

Open-Closed Principle: Extending your code without modification

The Open-Closed Principle states that objects or entities should be open for extension but closed for modification. This means they should be extendable without altering their core implementation. This principle can be applied in Object-Oriented Programming (OOP) by creating new classes that extend the original class and override its methods, rather than modifying the original class directly.

In functional programming, this can be achieved through the use of function wrappers, where you can call the original function and apply new functionality to it without changing the original function itself.

The decorator design pattern is also a useful tool for adhering to this design principle. With decorators, you can attach new responsibilities or behaviors to objects without modifying their source code, thus keeping them closed for modification and open for extension.

In the example below, we are going to expand the “updateAvatar” method to include validation, preventing any non-image extensions from being uploaded. Let’s begin by modifying the “updateAvatar” method first.

public async updateAvatar(id: number, file: any) {
    try {
        // Get the file extension
        const fileExtension = this.fileService.getFileExtension(file);

        // Check if it's JPG, throw an error if not
        if (["jpg", "png"].includes(fileExtension.toLowerCase())) {
            throw new Error('Unsupported avatar format. Only image is allowed.');
        }

        const avatarPath = await this.fileService.upload(file);
        await this.update(id, {avatar: avatarPath});
        console.log('Avatar updated successfully.');
    } catch (error) {
        console.error('Avatar update failed:', error);
    }
}

As you can see, the code above violates the Open-Closed Principle (OCP), which is the second principle of SOLID, as it requires us to modify our code if we want to add more extensions. Now, let’s refactor this code to adhere to the OCP.

1. We will create a file validator where we can pass a file and the expected extension.
2. Then, our validator will check if the file extension matches the expected extension before allowing it to proceed.

Let’s begin by adding new methods to the “File” class.

class FileService {
    public async upload(file: any, extension: string): Promise<string> {
        const s3 = new AWS.S3();
        const params = {
            Bucket: 'my-bucket',
            Key: `avatars/${Date.now()}.${extension}`,
            Body: file,
        };

        const uploadResult = await s3.upload(params).promise();
        return uploadResult.Location;
    }

    public validate(file: any, supportedFormats: string[]): boolean {
        const fileExtension = this.getFileExtension(file);

        if (supportedFormats.includes(fileExtension.toLowerCase()))
            return true;

        throw new Error("File extension not allowed!");
    }

    public getFileExtension(file: any): string {
        const fileName = file.name || '';
        const parts = fileName.split('.');
        if (parts.length > 1)
            return parts[parts.length - 1];
        return '';
    }
}

As you can see, the old method for uploading a file has not been modified; instead, we have simply added new methods to the File class.

The second step is to add the validation method to our “updateAvatar” function.

public async updateAvatar(id:number, file:any) {
    try {
        this.fileService.validate(file, ["jpg"]);
        const avatarPath = await this.fileService.upload(file);
        await this.update(id, {avatar: avatarPath});
    } catch (error) {
        console.error('Avatar update failed:', error);
    }
}

Although there is still a minor concern — what if we had more supported formats? — you are completely correct. In that case, we would need to modify this method. So, let’s perform another refactoring on the code. We will separate the supported image formats from the “updateAvatar ” method and move them to global variables or application configuration, where we will only manage the constants, not the logic.

// config.js

export const SUPPORTED_IMAGE_FORMATS = ["jpg", "png", "jpeg", "svg", "webp"];

And then we can use these constants in our “updateAvatar” method.

this.fileService.validate(file, SUPPORTED_IMAGE_FORMATS);

Note: The reason we haven’t directly imported this constant into the “Validate” method is to keep it reusable. This way, we can use the “Validate” method for videos, documents, and other extensions as well.

Liskov Substitution Principle: Inheritance and polymorphism done right!

LSP suggests that any superclass needs to be replaceable with its subclasses without breaking the application. This means that if we have class A and class B is extended from class A, and there is a client for class B (a function, module, or anything that uses any property or method from class B), it should be able to use class A instead. So, if you use a superclass instance instead of a subclass, everything should still work correctly.

Let’s take a look at this example of the “UserService”, where we aim to implement new methods for database operations.

class UserService {
    private usersRepository: UserRepository;

    constructor(usersRepository: UserRepository) {
        this.usersRepository = usersRepository;
    }

    public async create(fields: Record<string, any>) {
        const user = await this.usersRepository.insert(fields);
        return user;
    }

    public async get(id: number) {
        const user = await this.usersRepository.findById(id);
        return user;
    }

    public async update(id: number, fields: Record<string, any>) {
        let user = await this.usersRepository.findById(id);
        user = {...user, ...fields};
        await user.save();
        return user;
    }

    public async delete(id: number) {
        await this.usersRepository.delete(id);
    }
}

In this example, the “UserService ”has implemented all the necessary methods from the “UserRepository”, which adds a new layer to our application for performing database operations. These methods not only interact with the database but also allow us to perform additional tasks, manipulate the results, or add validation.

However, let’s say we want to create similar services for other entities, such as “Roles,” where we manage roles and permissions. In this case, we would create a new class for the “Roles” service.

class RoleService {
    private roleRepository: RoleRepository;

    constructor(roleRepository: RoleRepository) {
        this.roleRepository = roleRepository;
    }

    public async create(fields: Record<string, any>) {
        const user = await this.roleRepository.insert(fields);
        return user;
    }

    public async get(id: number) {
        const user = await this.roleRepository.findById(id);
        return user;
    }

    public async update(id: number, fields: Record<string, any>) {
        let user = await this.roleRepository.findById(id);
        user = {...user, ...fields};
        await user.save();
        return user;
    }

    public async delete(id: number) {
        await this.roleRepository.delete(id);
    }
}

As you can see, these services are nearly identical, with the only difference being the injection of a different repository for each service. To reduce code duplication and promote reusability, let’s start by creating a superclass that handles the repository integration. We can then extend this superclass in subclasses for different entities.

class BaseService {
    private repository;

    constructor(repository) {
        this.repository = repository;
    }

    public async create(fields: Record<string, any>) {
        const user = await this.repository.insert(fields);
        return user;
    }

    public async get(id: number) {
        const user = await this.repository.findById(id);
        return user;
    }

    public async update(id: number, fields: Record<string, any>) {
        let user = await this.repository.findById(id);
        user = {...user, ...fields};
        await user.save();
        return user;
    }

    public async delete(id: number) {
        await this.repository.delete(id);
    }
}

Now that we have our “BaseService ”class, we can extend it to create the “UserService ”and “RoleService ”classes. This approach allows us to reuse common functionality and minimize code duplication.

class UserService extends BaseService {
    constructor(repository: UserRepository) {
        this.repository = repository;
    }
}

class RoleService extends BaseService {
    constructor(repository: RoleRepository) {
        this.repository = repository;
    }
}

As you can see, the “UserService ”and “RoleService ”classes no longer contain repository integration; they simply switch the repository they use. This demonstrates the beauty of the Liskov Substitution Principle (LSP), where we can replace these subclasses with their superclass, such as “BaseService”, without causing the application to crash.

Now, let’s take a look at the code where an instance of “UserService ”is created.

const userService = new UserService(new UserRepository());

The code where an instance of “UserService ”is created can be replaced by the superclass.

const userService = new BaseService(new UserRepository());

You can replace it with any other repository to create a new service for different repositories or entities.

Interface Segregation Principle: Flexible interfaces for specific needs

ISP refers to implementing only the methods of an interface that are needed. So, clients of an interface should not be forced to implement all the methods of an interface if those methods are not used. In other words, interface segregation points out the fact that having many small interfaces is more beneficial than having one general interface.

Let’s examine this example where we have a “UserService ”class extended by the “CustomerService ”and “SellerService ”classes. In this design, the “UserService ”has its own interface that enforces the subclasses extended from it to implement methods that may not necessarily be needed.

Now, let’s begin by creating the interfaces for our services.

interface BaseServiceInterface {
  create(fields: Record<string, any>): Promise<User>;

  get(id: number): Promise<User>;

  update(id: number, fields: Record<string, any>): Promise<User>;

  delete(id: number): Promise<void>;
}

interface UserServiceInterface extends BaseServiceInterface {
  getReviews(userId: number): Promise<Review[]>;

  getOrders(userId: number): Promise<Order[]>;

  getSells(userId: number): Promise<Order[]>;

  getShops(userId: number): Promise<Shop[]>;
}

Now, let’s implement the necessary methods that our “UserService ”is missing. This will ensure that the subclasses, such as “CustomerService ”and “SellerService”, can provide their own implementations for these methods as needed.

class UserService extends BaseService implements UserServiceInterface {
  private reviewService: ReviewService;
  private orderService: OrderService;
  private shopService: ShopService;

  constructor(
    repository: UserRepository,
    reviewService: ReviewService,
    orderService: OrderService,
    shopService: ShopService,
  ) {
    this.repository = repository;
    this.reviewService = reviewService;
    this.orderService = orderService;
    this.shopService = shopService;
  }

  getReviews(userId: number): Promise<Order[]> {
    this.reviewService.findAll({userId})
  }

  getOrders(userId: number): Promise<Order[]> {
    this.orderService.findAll({userId})
  }

  getSells(userId: number): Promise<Order[]> {
    this.orderService.findAll({sellerId: userId})
  }

  getShops(userId: number): Promise<Order[]> {
    this.shopService.findAll({userId})
  }
}

As you can see, our “UserService ”has implemented all the methods it needs and will inherit the remaining methods such as “get,” “create,” etc., from “BaseService”.

Now, let’s proceed to extend our “UserService ”into “CustomerService ”and “SellerService”.

class CustomerService extends UserService implements UserServiceInterface {
  //
  //
}

class SellerService extends UserService implements UserServiceInterface {
  //
  //
}

However, there is an issue: a regular customer can’t have sales or a shop, while a seller can’t post a review or have orders. Therefore, we need to override these methods to prevent them from being used.

class CustomerService extends UserService implements UserServiceInterface {
  getSells() {
    throw new Error("User is not a seller");
  }

  getShops() {
    throw new Error("User is not a seller");
  }
}

class SellerService extends UserService implements UserServiceInterface {
  getReviews() {
    throw new Error("User is not a seller");
  }

  getOrders() {
    throw new Error("User is not a seller");
  }
}

The problem has been resolved, and instances from the customer or seller can now only use the allowed methods. However, this example currently violates the Interface Segregation Principle by having one large, generic interface instead of many small interfaces. To rectify this, we can follow these steps:

1. Break the “UserServiceInterface ”into two interfaces: “CustomerServiceInterface ”and “SellerServiceInterface”.
2. Move methods that belong to one of these interfaces but not the other. If there are common methods, they can remain in the “UserServiceInterface”.
3. Instead of implementing the generic interface on “SellerService ”and “CustomerService ”classes, we will implement the specific interfaces.

Let’s begin with these steps.

interface UserServiceInterface extends BaseServiceInterface {
  //
}

interface CustomerServiceInterface extends UserServiceInterface {
  getReviews(userId: number): Promise<Review[]>;

  getOrders(userId: number): Promise<Order[]>;
}

interface SellerServiceInterface extends UserServiceInterface {
  getSells(userId: number): Promise<Order[]>;

  getShops(userId: number): Promise<Shop[]>;
}

So, we have just split our interface into two smaller interfaces. Next, we will move the necessary methods into the related classes only.

class CustomerService extends UserService implements CustomerServiceInterface {
  private reviewService: ReviewService;
  private orderService: OrderService;

  constructor(
    repository: UserRepository,
    reviewService: ReviewService,
    orderService: OrderService,
  ) {
    this.repository = repository;
    this.reviewService = reviewService;
    this.orderService = orderService;
  }

  getReviews(userId: number): Promise<Order[]> {
    this.reviewService.findAll({userId})
  }

  getOrders(userId: number): Promise<Order[]> {
    this.orderService.findAll({userId})
  }
}

class SellerService extends UserService implements SellerServiceInterface {
  private orderService: OrderService;
  private shopService: ShopService;

  constructor(
    repository: UserRepository,
    orderService: OrderService,
    shopService: ShopService,
  ) {
    this.repository = repository;
    this.orderService = orderService;
    this.shopService = shopService;
  }

  getSells(userId: number): Promise<Order[]> {
    this.orderService.findAll({sellerId: userId})
  }

  getShops(userId: number): Promise<Order[]> {
    this.shopService.findAll({userId})
  }
}

As you can see, we no longer have methods and properties that are not used in those classes, and there is no requirement to implement unused methods. This adheres to the Interface Segregation Principle and results in more focused and efficient interfaces for each class.

Dependency Inversion Principle: Building flexible and maintainable software designs!

The Dependency Inversion Principle (DIP) suggests that high-level modules should not have direct dependencies on low-level modules. Instead, both high-level and low-level modules should depend on abstractions or interfaces. Furthermore, abstractions should not rely on implementation details; rather, implementation details should depend on abstractions. By adhering to this principle, the risk of unintended side effects in high-level modules caused by changes in low-level modules is minimized. With the introduction of an abstract layer, dependencies are inverted, reducing the traditional top-down dependency structure.

Let’s examine an example involving a “UserService” and “RoleRepository” class. In this scenario, the “UserService” includes an implemented “getRole” method to retrieve the role of a user.

class UserService extends BaseService {
  constructor(repository: UserRepository) {
    this.repository = repository;
  }
}

In the previous examples, the “UserService” implemented the “BaseService”. However, to add a new method called “getRole”, we will include it in the “UserService” to retrieve the user’s role. Additionally, within the “getRole” method, we will create an instance of “RoleRepository” to access the “RoleRepository”.

class UserService extends BaseService {
  constructor(repository: UserRepository) {
    this.repository = repository;
  }

  public async getRole(userId: number): Promise<Role> {
    const roleRepository = new RoleRepository();
    const user = await this.get(userId);
    const role = await roleRepository.findById(user.roleId);
    return role;
  }
}

With the current approach, we can achieve our desired functionality, but we are also violating the fifth principle of SOLID, which is Dependency Inversion. This is because our “UserService ” is dependent on the details of the “RoleRepository” and creates an instance of it.

To address this issue, we need to inject the “RoleRepository” as a dependency into the “UserService” instead of creating a class instance within it. Furthermore, since “RoleRepository” implements “BaseRepository”, which is an abstract class, we can ensure that the get method exists in that abstract class. This approach helps us avoid relying on implementation details and promotes better adherence to the Dependency Inversion Principle.

class UserService extends BaseService {
  private roleRepository: RoleRepository;

  constructor(repository: UserRepository, roleRepository: RoleRepository) {
    this.repository = repository;
    this.roleRepository = roleRepository;
  }

  public async getRole(userId: number): Promise<Role> {
    const user = await this.get(userId);
    const role = await this.roleRepository.findById(user.roleId);
    return role;
  }
}

In the refactored code, we can easily replace “RoleRepository” with any other repository that handles roles. An improvement in this code is that, instead of using the “RoleRepository” class, we can utilize the “RoleService”. This approach keeps the “RoleRepository” layer isolated within the “RolesModule”, promoting a more modular and maintainable design.

class UserService extends BaseService {
  private roleService: RoleService;

  constructor(repository: UserRepository, roleService: RoleService) {
    this.repository = repository;
    this.roleService = roleService;
  }

  public async getRole(userId: number): Promise<Role> {
    const user = await this.get(userId);
    const role = await this.roleService.get(user.roleId);
    return role;
  }
}

Concluding the understanding of the SOLID Principles

In this exploration of SOLID principles, we've delved into key principles that serve as guiding lights in software design. These principles, namely the Single Responsibility Principle (SRP), Open-Closed Principle (OCP), Liskov Substitution Principle (LSP), Interface Segregation Principle (ISP), and Dependency Inversion Principle (DIP), play a pivotal role in shaping robust, maintainable, and flexible software architectures.

1. Single Responsibility Principle (SRP): Keep classes, modules, and functions focused on a single task. This streamlines development, testing, and code understanding.
2. Open-Closed Principle (OCP): Design code to be open for extension but closed for modification. This allows you to add new features without changing existing code.
3. Liskov Substitution Principle (LSP): Subclasses should be able to replace their superclasses without causing issues. This ensures that new subclasses fit seamlessly into existing code.
4. Interface Segregation Principle (ISP): Create small, focused interfaces rather than large, unwieldy ones. This prevents unnecessary method implementations and keeps code clean.
5. Dependency Inversion Principle (DIP): High-level modules should not directly depend on low-level modules. Both should rely on abstractions, reducing the risk of unintended side effects when making changes.

By applying these principles wisely, software development becomes more manageable, adaptable, and efficient. SOLID principles are valuable tools for building resilient, maintainable, and extensible software systems.

]]>https://engineering.deptagency.com/why-software-development-estimates-are-so-often-wrong/6543dc38bfaa040001b75523Fri, 03 Nov 2023 14:07:21 GMTTLDR: Estimates should be used as a vital sign for the health of the project, not as a deadline.

When starting a new project, one of the first questions that executives and senior managers always ask is, “How long is this going to take?” 

Usually, that question is trickled down through the team’s hierarchy to the engineers tasked with building the thing before the estimate is then bubbled back up to the stakeholders. The stakeholders then treat it like a development schedule and use it to set a rigid deadline. By the end of the project, the estimate is, more often than not, wildly inaccurate and the team embarks on a journey of self-reflection to figure out what went wrong.

Why do we underestimate how long projects will take?

Every team building a new product runs into this issue at some point in the product development process, so why does it still occur so frequently? 

Naivety

One problem with the initial estimate is they’re done when you know the least about the project, its trajectory, and what hurdles you might run into. 

That’s like trying to estimate how long it will take a tree to reach its full height when it’s only a seedling. Things that you can’t predict might affect the growth of that tree (wildfires, pests, drought, disease, thermonuclear war, etc.), so it’s best to use that estimate as a fluid target that could (and should) change when new information about the project becomes available to you.

Hopeless Optimism

When you ask a kid what they want to be when they grow up, they’re going to give you their pipe dream job. Most people like to plan for only the best-case scenario. Engineers are no different.

Software engineers are usually quite accurate at estimating the best-case scenario for how long a project should take. 

The problem is that unforeseen issues that add to the overall development time aren’t usually considered for that initial project estimate. Some examples of those issues might be 

• Bugs
• Dependencies on the schedules of 3rd parties
• Changing project requirements
• Researching and learning a new technology
• Developing a new technology
• Team member turnover and holidays/vacations.

How can you estimate software development more accurately?

Estimating accurately is incredibly hard for a reason: you’re trying to predict the future with limited information. There simply isn't a reliable formula for development. However, here are some things we routinely do to get as close as possible:

More discovery time will lead to a better-defined problem

Ironically, the best way to improve the accuracy of a development timeline is to spend more time. Of course, not just time, but valuable time via a comprehensive discovery or architecture sprint. 

Understand and test the perceived problem against technology restraints and user feedback.

To identify problems, interview current and prospective users, conduct research, and look at what already exists in the market. To map solutions, you’ll need to consider various factors, including market opportunity, cost to build, risks, time to market, and value. 

On top of these factors, it can be hard for engineering teams to objectively assess concepts because teams tend to fall in love with their ideas. To know what to build, an outside perspective is always helpful. 

If you don’t have the luxury of extended discovery time, here are some slightly more clunky ways to create estimates: 

High, middle, and low 

It can be useful to come up with estimates for best-case, worst-case, and most-likely scenarios. This is helpful in a few ways:

1. It gets you actively thinking about issues that might derail a project, and helps you to consider how likely it is that they are to occur.
2. When the project is complete, you can look back at these estimates to see which was the most accurate, giving you some hard data to use for your next estimate. 

Double your estimate

Yes, we’re serious. Come up with a best-case estimate and then multiply that number by two to account for all of the hiccups. If the project includes a lot of features or technologies that your team or vendor does not have experience with, multiply by three. Estimating this way, more often than not, comes close to how long the project will actually take.

What if you have a hard deadline?

For most projects, you have a list of features that need to be completed, and your engineering team estimates how long it will take to complete them all. 

But sometimes, you’re instead given an immovable deadline. In those cases, it’s a matter of determining how many features your team can complete in a given amount of time. The same rules for estimating apply to this scenario, but you’ll estimate each individual feature and then implement them in order of highest priority until you’ve reached your time budget.

If you have a hard deadline and a hard list of features that need to be completed, make an estimate for the list and be ruthless about feasibility. If it looks challenging, your team and stakeholders need to have honest conversations (as soon as possible) about where time can be gained:

• Can you add more designers and developers?
• Can any of the features be simplified?
• Can you forgo manually building any features by pulling in 3rd party SDKs?

Don’t become too attached to your estimate 

Estimates are most valuable when used as a vital sign for the overall health of the project. You shouldn’t treat them as a one-and-done way to set an immovable deadline, especially at the beginning of the project when you know the least. That will only cause tension and resentment between the engineers and the stakeholders when things invariably don’t go according to plan.

At the end of the day, estimates are just a snapshot in time. When you gain more information about the project, you should be using that to reevaluate your estimate. Your estimate should be continually updated to be a reflection of reality. Furthermore, you should be recording that information for use with future estimates. That way, you won’t have to guess, you’ll know.

Anyone who's written a Domain-specific language (DSL) or found themselves accidentally entrenched in a shotgun parser knows the pains involved in processing a grammar. We found ourselves needing to take a predefined Abstract Syntax Tree (AST) and generate a UI from it. The pain we experienced along the way forced us into a disciplined simplicity from which some pretty useful design patterns emerged. In this blog post, we share the lessons we learned from situations where complexity led us into a set of simple patterns.

So, what's Modelica?

Modelica is a class-based language for describing mathematical behavior. Those classes can be composed to create models, and in turn, those models can then be used to simulate the behavior of systems. In our case, we were simulating HVAC systems.

In addition, Modelica is useful across many other engineering disciplines. A key feature of Modelica is that it allows the user to adjust parameters without rewriting code.

Modelica does not have a UI, but through interpretation of its annotation system, we can programmatically generate one.

Our task

Below is a Modelica model with two parameters, hello and allow_hello. hello as an input needs to be conditionally enabled based on the expression: allow_hello == true

within ExamplePackage;
  model TestModel "Test Model"
    parameter String hello="World"
      "Hello"
      annotation (Dialog(enable=allow_hello == true));
    parameter Boolean allow_hello=true
      "Enable";
end TestModel;

From this Modelica model, we are provided with a custom AST representation. We received the model in an AST format (written in JSON) so it could be serialized and sent over the wire to the UI.

In other words, we need to go from this:

{
    "within": "ExamplePackage",
    "class_definition": [
      {
        "class_prefixes": "model",
        "class_specifier": {
          "long_class_specifier": {
            "identifier": "HelloModel",
            "description_string": "Test Model",
            "composition": {
              "element_list": [
                {
                  "component_clause": {
                    "type_prefix": "parameter",
                    "type_specifier": "String",
                    "component_list": [
                      {
                        "declaration": {
                          "identifier": "hello",
                          "modification": {
                            "equal": true,
                            "expression": {
                              "simple_expression": "\"World\""
                            }
                          }
                        },
// ...

To this:

Labels, data types, and initial values seemed relatively simple, but the input 'hello' is conditionally enabled based on the expression:

allow_hello == true

This means we need to be able to resolve variables, and resolving variables means we have to understand scope, assignments, and much more. It's a hard problem.

Compressing the problem: Parsing the AST to a simplified format

Ultimately, we want to arrive at a more simplified representation of the AST that only contains the parameters we care about. Something like the following:

{
  "nodes": [
    {
      "path": "TestModel.hello",
      "type": "String",
      "value": "Hello World",
      "enable": {
        "operator": "==",
        "operands": ["TestModel.allow_hello", true]
      },
      "childNodes": []
    },
    {
      "path": "TestModel.allow_hello",
      "type": "boolean",
      "value": "true",
      "enable": "true",
      "childNodes": []
    }
  ]
}

Ultimately, the flow of information will be as follows:

Generating a simplified structure from a (deeply-nested) AST

Given that we're provided with the AST, our first task is to create a simplified and flat schema structure that can be easily interpreted within a browser to display the UI.

When the parser finds a new type, the AST representation is mapped to the simplified schema. The user-facing text, initial value, type, and enable/disable behavior are extracted into a mostly flat structure.

This map allows us to greatly reduce the amount of data needed. The AST representation of the models was around 40MB while the simplified format was closer to 4MB.

Removing implicit imports and simplifying type references

The parser steps through the AST and, as types and parameters are discovered, it inserts that type and its simplified representation into a dictionary.

{
    {[key: string]: SimplifiedDataType} // 'key' is the absolute path
}

In our 'Hello World' example, we find the String and boolean primitive types. Classes can also be assigned to parameters, e.g. SubFolder.MyCustomClass. Modelica has multiple implicit strategies for finding type definitions. For example, SubFolder.MyCustomClass could imply that MyCustomClass is defined in a file located at ./SubFolder/MyCustomClass.mo. The parser implements these rules, finds the appropriate file, and parses the definition into the dictionary. What was previously a file tree is now a single, easily serializable dictionary with the file tree structure (./SubFolder/MyCustomClass.mo) flattened into a key (SubFolder.MyCustomClass).

Creating a tree representation: Making relations explicit

The AST references different types without direct links. The parser extracts these types into a flat, easily accessible dictionary of types. The components that make up a class definition are explicitly connected through the attribute 'children', a list of absolute paths referencing other types in the dictionary of types.

When iterating through a class definition, the parser builds up a tree of relationships between parameters and types.

Expanding on the original TestModel:

within ExamplePackage;
  model TestModel "Test Model"
    parameter String hello="World"
      "Hello"
      annotation (Dialog(enable=allow_hello == true));
    parameter Boolean allow_hello=subModel.nestedBoolean
      "Enable";
    parameter SubFolder.SubModel subModel(nestedBoolean=true)
      "A model with its own list of parameters";
end TestModel;

The parameter subModel has been added with a type SubFolder.SubModel. A child parameter of SubModel nestedParameter has been assigned false and the value of allow_hello is now an assignment from the instance value subModel.nestedBoolean.

After iterating through the parameter list, the dictionary of simplified types looks like this:

{
  "TestModel.hello": { "type": "String" },
  "TestModel.enable": { "type": "Boolean" },
  "TestModel.subModel": { "type": "SubFolder.SubModel" },
  "SubFolder.SubModel": {
    "type": "SubFolder.SubModel",
    "children": ["SubFolder.SubModel.nestedParam"]
  }, // <-- SubModel class definition
  "SubFolder.SubModel.nestedBoolean": { "type": "String" } // <-- nested param defined in SubModel
}

This allows us to follow the related types from the children of TestModel to build a tree that looks as shown below:

Notice that there are parameters (like TestModel.subModel) and definitions (like SubFolder.SubModel) mixed together. I'll discuss more about this later.

This data structure is helpful for understanding the structure of a given class. No implicit behavior around importing files or class relationships is required to understand the structure of TestModel.

And more

There are additional features that were also parsed and simplified but we will not cover here, including:

• Simplifying dependency injection
• Fitting within the simplified schema the polymorphic behavior that enabled swapping types (imagine a system that allows for selecting a branch to traverse)
• Formatting assigned values (right-hand-side values) into a common shape
• Flattening inherited properties

We didn't need expressive, flexible grammar to get to the features we needed for the UI. The parser simplifies the AST representation into a more explicit, less flexible tree representation that worked for our purpose. Less variability and implicit behavior meant simpler algorithms for interpretation on the front end.

Writing an interpreter

With our simple grammar in place, we can return to the original expression we were attempting to resolve:

allow_hello == true;

To resolve this, a few things needed to be understood:

1. How do you evaluate the operator (==) and it's operands (allow_hello, true)
2. How do you get the value of allow_hello
3. How do you incorporate user input to determine the value of allow_hello

Evaluating expressions (without variables)

Before attempting to solve an expression with variables, we have to deal with the basics of evaluation with constants and operators. How do you handle different operators? How do you deal with expressions like 3 < 5, true && (3 < 4 || 2 > 1)?

We came up with a schema that looked like this:

export type Expression = {
  operator: string;
  operands: Array<Literal | Expression>;
};

This type of structure allows for relatively simple evaluation function:

function evaluate(expression: Expression) {
  switch(expression.operator) {
    case '<': {
	    // reduce the list of operands into a single value
      const accumulator;
      expression.operands.forEach(operand => {
        if (isExpression(operand)) {
          operand = evaluate(operand); // recursive call
        }
        // operator specific reduction with the accumulator
        operatorReduction(accumulator, operand);
      });

      return accumulator;
    }
}

Each operator (e.g. <, ==, etc.) can be separated into a case in the switch statement.

Nested expressions could be handled by making a recursive call.

Resolving variables

With the basics of evaluation in place, we can now address variable resolution.

Like before, there was a goal in mind for what variable resolution would look like:

const context = new Context(userSelections, testModel);
context.getValue(`allow_hello`); // returns 'true'

A model should be able to be loaded (testModel) with a class constructor (Context), and the created 'context' instance should then be queryable like entering variable names in a REPL.

With that in mind, we now have to figure out how to determine the initial values of testModel.

Returning to the tree view of TestModel, if we add the initial value to each node we get the following:

What this view captures is the tree of dependencies that make up our dictionary of variable names and their values. If TestModel is loaded, the following variables and values should be collected in a dictionary that represents our current scope.

{
    "hello": "hello world",
    "enable": true,
    "subModel.nestedBoolean": true
}

Notice that the keys in scope do not match the original node identifiers. Instead we are using 'instance paths'. Instance paths are formed by appending parameter names together. Why use instance paths? Types like SubFolder.Submodel can be reused but the instance paths subModel.nestedBoolean is unique.

Notice as well that the value subModel.nestedBoolean resolves to true even though the node SubFolder.SubModel.nestedBoolean sets the same instance path to false. The SubFolder.SubModel.nestedBoolean node represents the original definition of the class parameter. The TestModel.subModel node is the instance that includes an override of the original value of 'nestedBoolean'. Overrides must take precedent over the downstream definition.

This behavior can be implemented using a pre-order tree traversal. This is a depth-first algorithm that processes the current node before visiting child nodes. In this example, that means getting the values assigned by the node TestModel.subModel before visiting the child node SubFolder.SubModel.nestedBoolean.

And More

The logic around tree traversal and the process of building up 'scope' was slightly more complicated.

• We ignored class definitions when building instance paths. As part of that we had to indicate in the simple schema what nodes were class definitions
• Child node traversal was dependency driven. When a variable is referenced that is not yet in the scope dictionary, we jump to the dependency's branch first. For our example, when the variable subModel.nestedBoolean is reached in the middle branch and not found in scope, a recursive call is made to process the TestModel.subModel branch first.

Generating UI and incorporating user input

With variables resolved and expressions evaluated, UI can now be generated.

We need to map from parameter values and types to a UI component. If the parameter is of type boolean, we need a checkbox component, if it is an enum we need a dropdown.

Like before, a pre-order tree traversal was used to visit each node of the template. This time, child nodes could be visited in order as all nodes were populated with a value. No need to worry about dependency-driven loading.

The same string-building logic that was used to build up an instance path was also used to label and identify the corresponding UI. On user input, the value is stored using that instance path. As the user makes selections, a mirror object to the variables dictionary is built up using those instance path keys.

Because the inputs used the same variable instance names, incorporating these selections into the concept of 'context' was straightforward. When resolving a variable value, always check what's in the selection dictionary before going to the variables dictionary.

Arrival

We have arrived! We can now resolve the expression:

allow_hello == true;

Templates provided as an expansive AST are converted to a paired down 'simple schema' that strips away extraneous detail and features of the Modelica language.

We are able to understand both what to do with the operator (==) and operands (allow_hello and true)

The simple schema can be interpreted to generate all variables and definitions currently in scope, meaning we can resolve the value allow_hello.

Finally, that value can be used to conditionally enable our Hello World UI.

General tips for reducing complexity

Use best practices to think in terms of interfaces, not implementation

They’re best practices for a reason, and using them allows you to spend less time wading through implementation details. Some I’d like to highlight:

Single purpose classes and functions: identify the concept/action/behavior you're trying to encapsulate and make sure your class/function sticks to it.
Modify state deliberately in well known parts of code. Rely on pure functions as much as possible to keep the non-deterministic harder-to-test portions of code limited.
Use types in weakly typed languages

Consistency around these practices allows for predictability. With predictability you can understand more about what a function does just by its signature.

Nurture the mental model

Find a simplified representation to underpin understanding of something complex. A mental model is the internal representation of a problem in your head, and it should be a clarifying force: terse enough to be quick to reason about but accurate enough that it can be used as a heuristic for implementation details. A good way to test the effectiveness of your mental model is to describe it to someone else who is unfamiliar with the project.

Also, take time with the actual language used in the project, both for variable naming and general descriptions. What is the difference between ‘name’ and ‘id’? When do I use the term ‘node’ vs. ‘parameter’ vs. ‘input’? A shared understanding of these terms provide an implicit hint on implementation and usage.

Conversely, the wrong terms can be a near constant source of obfuscation, tripping up discussions by inserting confusion.

Testing offloads thinking

Tests offload thinking by allowing your code contracts and corner cases to be enshrined in nice re-runnable tests. When implementing some new behavior, you can primarily focus on that specific feature, without the background noise of special cases.

Tests are also a good way to keep your headspace thinking in terms of interfaces. Tests are clients of the code you are writing, so you immediately test out the interface you are establishing. Additionally, when an interface needs to be changed, you get to double check that change by updating the appropriate test.

A good test suite allows you to take bigger risks with your code.

Set up your code for play

Make it easy to ‘play’ with your code. A combination of fast tests, quick validation cycles, and sound strategies for reverting a commit can provide a degree of safety that allows for play.

Tests that can quickly run on change can help course correct an implementation strategy.

Having a fast validation cycle (time from code change to result) also allows for fast course corrections. If you are using a weak typed language use the types for clearly defined interfaces. Most editors provide instantaneous feedback on whether or not your new code works.

And in git, make commits with the revert in mind. Like functions or classes, encapsulate the functional change the commit includes. Then if you do have to revert, you can cleanly remove work around one feature without having to revert something unrelated.

All of this is working towards a frictionless, safe development environment. A feeling of safety is important for creativity, and creativity is definitely needed when working through hard problems.

How we bridged Backend with Frontend by converting template syntax and how AI helped.

What is templating?

In web templating, a template acts as a blueprint or a skeleton for the web page. It contains the static parts of the page layout and defines placeholders or variables where dynamic content can be inserted.

I believe every developer is familiar with it, as it comes from the ‘ancient’ web development times and has been around for decades.

The ISML templating

ISML stands for “Intershop Scripting Markup Language” and is a templating language specifically used in the context of Salesforce Commerce Cloud - a cloud platform for creating e-commerce sites. 

No need for a detailed description of the platform, but it uses MVC principles, has JavaScript on the backend, and has a unique templating language called ISML.

You can distinguish it from other templating languages by specific tags that only work with ISML. The tags are responsible for loops, conditions, and the inclusion of other templates. Most templating languages have those, but there are slight differences in syntax.

Compatibility problem

Even though the majority of the templating languages use similar approaches and structure, the syntax is something that varies from one templating to another. 

As an example, let’s see how a single template can look in two different templating languages: one in Nunjucks and another one in ISML:

The structure is similar, but the syntax is different, meaning only one of two templates will work in ISML and one with Nunjucks. That’s quite normal in the development world and usually is not a problem since you don’t need to transform from one template syntax to another. This was a given until we started to see problems with ISML templating - it has to be run on a server only. 

This significantly slows down templating because each change has to be uploaded to a server to become visible. The full template must be uploaded to the server when you change a line, tag name, or HTML attribute. And then, to validate the change, you must reload the resulting page. 

This circle takes around five seconds to execute, which does not sound like a lot until you do it a hundred times. And that is exactly what happens when a template is developed according to a required design. 

So we started looking into seeing the changes locally without using a remote server. That’s when we come across the Nunjuck's templating language. 

Nunjucks and conversion

Nunjucks helps you create templates faster than via ISML because templates don’t have to be uploaded to a server. Setting up a localhost server is all you need, and it comes with a standard Webpack configuration familiar to most FE developers.

But having a template in Nunjucks will not work for SFCC without the obvious syntax change I mentioned earlier. And each Nunjucks template would require at least some manual changes to the syntax. But does it have to be really manual? Of course not!

At DEPT®, we started using the OpenAI platform to convert the templates from Nujucks to ISML. This changed the workflow we had in place for a typical SFCC project. Instead of writing an ISML template, we write a Nunjucks one and then convert it to ISML. This also allowed us to reduce the learning curve for FE developers because they don’t have to learn ISML syntax and SFCC specifications as a new platform. Instead, they only have to work with the more popular and familiar Nunjucks templates.

So our new development process right now is looking like this:

• Frontend team writes templates using Nujucks
• Backend team converts the templates to ISML using the AI
• Backend team makes sure the templates working as expected and commits the changes into the repository

As a result:

• Frontend team saves a lot of time as they don’t have to deal with the server-side rendering and can test the templates locally
• Frontend team does not have to learn ISML templating or SFCC specifications
• Backend team does not spend time on onboarding the Frontend team on the SFCC platform  

Conclusion

So this is one case for us as a company for a template conversion. This is most likely not a very common problem; for some, it does not sound like a problem. However, it’s a good example of how an AI can connect two different expertise: frontend development and cloud-based backend. 

Working with custom shapes has been a pain to work with for a long time in Android development. Yes, Android provides some tools to customize views for developers, but I was never satisfied with the achieved result. Take the triangle button, for instance; a true Material Design triangle button has dynamic shadows and a triangle ripple effect while pressing. Despite several approaches and the ability to see something that resembled a triangle, there were still compromises with mimicking real shadows and rendering with an unacceptable rectangular ripple effect. Eventually, I gave up on the existing tooling.

Now, with the introduction of Jetpack Compose, developers' hands are freed, and they are allowed to customize views in different ways. By overriding the one-function interface I achieved everything I wanted with a minimum amount of code with some basic trigonometry. Let me share my experience with you in this article.

As you can see - it is a triangle with rounded corners. Corner ratios can be set up dynamically. It has real shadows, and they react to pressing. The ripple effect also does not exceed the shape of the button. The size can be of any dimension the UI requires.

In order to render the button, Compose has the @Compose Button function. Among all the parameters, let’s take a closer look at shape: Shape. As you might have guessed, we will provide an instance of the Shape interface here. Despite this parameter having a default value, RoundedCornerShape() is usually provided here as an argument with a radius of the corner. The modifier and elevation parameters allow us to assign a size and an elevation. This is what we have for now so far.

Button(
   onClick = { },
   modifier = Modifier.size(width = 40.dp, height = 30.dp),
   shape = RoundedCornerShape(5.dp),
   contentPadding = PaddingValues(4.dp),
   elevation = ButtonDefaults.elevatedButtonElevation(defaultElevation = 6.dp)
) {
   Text(
       text = "+1",
       fontSize = 8.sp
   )
}

Of course, in order to customize the shape, we will implement Shape interface by overriding its single function fun createOutline(size: Size, layoutDirection: LayoutDirection, density: Density): Outline. Here, we have a size parameter, which will be provided to our implementation with every recomposition. We will utilize this argument in order to fit the view into this size.

class TriangleShape : Shape {
   override fun createOutline(
       size: Size,
       layoutDirection: LayoutDirection,
       density: Density
   ) = Outline.Generic(
       Path().apply {
           val x = size.width
           val y = size.height
           
           moveTo(0f, 0f)
           lineTo(x, y / 2)
           lineTo(0f, y)
       }
   )
}

Outline here is the borders of the view that we want to customize. It is a sealed class with three classes extending it - Rectangle, Rounded, and Generic. What we need to do is create an instance of class Generic(val path: Path) : Outline() by passing an instance of Path. This class actually contains a list of functions that allow us to draw everything we might want.

For now, let’s start with something relatively simple and just implement a rectangular button with no roundRadius. As was mentioned before, will need implementation of the Shape interface. We will use fun moveTo(x: Float, y: Float) and fun lineTo(x: Float, y: Float). x and y here are coordinates in pixels.

A couple of words regarding this new class. Again, we can easily access view width and height via size argument. Here, we need first to land on the top left corner of the triangle, which is the zero point of the coordinate system, by invoking the moveTo function. Then, we draw a line to the middle of the right edge of the view - lineTo(x, y / 2). Finally, we draw a line to the bottom left corner of the triangle - lineTo(0f, y). No need to connect the last point with the first - it will be done for us by drawing a line automatically.

Updated Button function

Button(
   onClick = { },
   modifier = Modifier.size(width = 40.dp, height = 30.dp),
   shape = TriangleShape(),
   contentPadding = PaddingValues(4.dp),
   elevation = ButtonDefaults.elevatedButtonElevation(defaultElevation = 6.dp)
) {
   Text(
       text = "+1".uppercase(),
       modifier = Modifier.offset(x = -(40.0 * 1 / 5).roundRadiusToInt().dp),
       fontSize = 8.sp
   )
}

Here we simply pass a new instance of TriangleShape as shape parameter. It’s worth mentioning that due to the peculiarity of the triangle form, the centered text shifts towards the right corner. Moving it to 20% of width to the left makes it look a bit better: Modifier.offset(x = -(40.0 * 1 / 5).roundRadiusToInt().dp).

In general, we are ready to implement more complicated shapes. Since the majority of our future calculations will involve the right triangle, let me refresh some basics of trigonometry.

A right triangle is described with 5 values - two legs(x and y), hypotenuse(h) and two angles opposite to legs(alpha and beta). If we know the values of any two sides of a triangle or one side and angle, we can calculate the rest of them.
The sine of an angle is the ratio of the opposite leg to the hypotenuse - sin(alpha) = x / h.
The Cosine of an angle is the ratio of the adjacent leg to the hypotenuse - cos(alpha) = y / h.
The tangent of an is the ratio of the opposite leg to adjacent - tan(alpha) = x / y.
Angle alpha is equal to the arctangent of the ratio of the opposite leg to adjacent - alpha = atan(x/y).

Ok, now starts the tricky part. This is the scheme of the button. We need to draw the △ABC but with rounded corners, so the figure is contained in DGPHKJE.

First what we need here is the radius of the circle which will be at the corners of the button. Since we want it to be configurable by the developer let’s add an argument in our class.

class TriangleShape(private val roundRadius: Float) : Shape

Also let’s add top level value which will convert dp to pixels in MainActivity.

private val Dp.float: Float get() = this.value * getSystem().displayMetrics.density

Now we can pass desired roundRadius as parameter.

shape = TriangleShape(4.dp.float),

Besides two previous functions, we will use the additional function of the Path class.

fun arcToRad (
    oval: Rect,
    startAngleRadians: Float,
    sweepAngleRadians: Float,
    forceMoveTo: Boolean
)

To explain the meaning of these parameters, let’s take a close look at the right angle, for example.

• oval: Rect - In our case, an oval is a circle into which a square is inscribed. There are different methods on how to init Rect object, but we will use fun Rect(center: Offset, radius: Float) In our case, center is the coordinates of point N and radius is a roundRadius argument.
• startAngleRadians: Float - Compose considers point P as the starting point for describing angles with clockwise direction. For instance - 6 o’clock - is 𝜋 / 2 radians, 9 o’clock - 𝜋 radians and 12 o’clock - is -𝜋 / 2. Of course, 3 o’clock is 0 radians. In our case - the negative value of angle GNP.
• sweepAngleRadians: Float - is how large the arc is. Value of angle GNH for our example forceMoveTo: Boolean is always false.

So, the information we need to draw this shape - Coordinates of points D, G, K, E, M, N, O and angles ∠GNH, ∠KOJ and ∠EMD.

Let’s wrap it all together and calculate these values.

Before doing that, we will need values of angles of the triangle - ∠EAD and ∠GCH. Values of ∠EAD and ∠KBJ are equal since the triangle is isosceles. Triangle △ACR is a right triangle and we know it sides (width and height / 2) it is possible to easily calculate angle ∠EAD using fun atan(x: Float): Float

In the code, I will deliberately use article notation so it could be easier to follow the logic

val RC = size.width
val AR = size.height / 2


val DAE = atan(CR / AR)

For the next part, here is the top left corner with additional segments for calculations.

Here we need to calculate coordinates of point D. To do it, we need to calculate AQ and QD of △AQD. What else do we know about this triangle? We can calculate the value of ∠DAQ. Since ∠EAQ is right

If we have known the value of hypotenuse AD, the rest of the calculations are trivial. Let’s take a look at another △ADM. We know it’s leg DM - it is the roundRadius argument. We also know that the center of a circle inscribed in an angle lies on the bisector. Consequently:

val DAQ = PI.toFloat() / 2 - DAE
val DAM = DAE / 2
val AD = roundRadius / tan(DAM)

Now we are ready to calculate coordinates of D

val DQ = AD * sin(DAQ)
val AQ = AD * cos(DAQ)

Finally, we are ready to use the first Path function.

moveTo(AQ, DQ)

Let’s take a look at the right side of the triangle.

Next, we need to draw a line to point G. We already know the coordinates of point C. We can derive coordinates of point G by subtracting CT and GT from x and y of point C respectively. These values are legs of the right △CGT.
As in the previous example, we can calculate the legs of the right triangle if we know one of its angles and hypotenuse - ∠GCN and CG in our case.
∠GCN is a half of ∠GCH. Since we know that the sum of angles of a triangle is 𝜋 radians and our triangle is isosceles, and basis angles are equal to DAE

CG is also a leg of another right △CGN with known ∠GCN and leg - roundRadius argument or GN.

Once we know CG we can calculate GT and CT

Coordinates of point G: CR - CT, AR - GT

val GCN = (PI.toFloat() - 2 * DAE) / 2
val CG = roundRadius / tan(GCN)
val GT = CG * sin(GCN)
val CT = CG * cos(GCN)

lineTo(CR - CT, AR - GT)

Now we have to draw an arc. Here, we need the center of the circle - coordinates of N and two angles - ∠CNG and ∠GNH. Since it is one of angle of right triangle △CNG and other angle is known,

For point N we know y coordinate. Let’s now calculate CN.

Now we can draw the arc.

val CN = roundRadius / sin(GCN)
val CNG = PI.toFloat() / 2 - GCN
arcToRad(Rect(Offset(CR - CN, AR), roundRadius), -CNG, 2 * CNG, false)

The coordinates of point K are calculated similarly as we did for point Q. We can reuse precalculated values and apply them here.

val AB = size.height
lineTo(AQ, AB - DQ)

Moving next to the bottom left of the triangle.

We also need coordinates of point O and two angles for arc function - ∠KOU and ∠JOK. In order to calculate BV and OV as x and y coordinates, we need to know hypotenuse BO and value of ∠OBV for right △BOV. It is worth noticing that ∠OBV is equal to the sum of ∠DAM and ∠DAQ because it lies on the opposite side of the isosceles triangle. Also, admit that hypotenuse BO for the △BVR is also hypotenuse for △BKO. Leg KO and angle ∠KBO are equal to roundRadius argument and ∠DAM angle. Let’s calculate BO within sine of angle as we did it before.

Now calculating BR and RO could be done via sine and cosine.

Now we need to calculate ∠KOU. ∠UOV is right. That means

We can easily find ∠KOV by subtracting ∠BOV from ∠BOK. Both of them can be calculated.

Consider the right △BOV first. Here, ∠BOV = 𝜋 / 2 - ∠OBV. And in the same way we can easily calculate for △BOK.

Eventually,

Also, let’s calculate ∠JOK. We know that the sum of angles of any rhombus is equal to 2 * 𝜋. We also know that in the kite BJOK, there are two right angles ∠BJO and ∠BKO by the definition of the circle inscribed in an angle. Also we know ∠JBK - it is equal to ∠DAE.

Now we have all the information for drawing rect.

val OBR = DAM + DAQ
val BO = roundRadius / sin(DAM)
val BR = BO * cos(OBR)
val RO = BO * sin(OBR)

val BOR = PI.toFloat() - OBR
val BOK = PI.toFloat() - DAM
val KOR = BOK - BOR
val KOU = PI.toFloat() / 2 - KOR
val JOK = PI.toFloat() - DAE
arcToRad(Rect(Offset(BR, AB - RO), roundRadius), KOU, JOK, false)

Finally we have to draw left top arc.

We have everything that needs to draw a line to the top left corner.

AE is equal to precalculated AD.

lineTo(0f, AD)

The coordinates of point M can be calculated within BR and RO segments. We also need a starting angle. Since ME is perpendicular to the tangent, we can figure out that the initial angle is 𝜋. Sweeping angle was calculated before and it is equal to ∠JOK.

arcToRad(Rect(Offset(BR, RO), roundRadius), PI.toFloat(), JOK, false)

And this is our final result.

And final code.

class TriangleShape(private val roundRadius: Float) : Shape {
   override fun createOutline(
       size: Size,
       layoutDirection: LayoutDirection,
       density: Density
   ) = Outline.Generic(
       Path().apply {
           val CR = size.width
           val AR = size.height / 2

           val DAE = atan(CR / AR)
           val DAQ = PI.toFloat() / 2 - DAE
           val DAM = DAE / 2
           val AD = roundRadius / tan(DAM)
           val DQ = AD * sin(DAQ)
           val AQ = AD * cos(DAQ)

           //move to point D
           moveTo(AQ, DQ)

           val GCN = (PI.toFloat() - 2 * DAE) / 2
           val CG = roundRadius / tan(GCN)
           val GT = CG * sin(GCN)
           val CT = CG * cos(GCN)

           // line to point G
           lineTo(CR - CT, AR - GT)

           val CN = roundRadius / sin(GCN)
           val CNG = PI.toFloat() / 2 - GCN
           // right arc
           arcToRad(Rect(Offset(CR - CN, AR), roundRadius), -CNG, 2 * CNG, false)

           val AB = size.height

           // line to point K
           lineTo(AQ, AB - DQ)

           val OBV = DAM + DAQ
           val BO = roundRadius / sin(DAM)
           val BV = BO * cos(OBV)
           val OV = BO * sin(OBV)

           val BOV = PI.toFloat() - OBV
           val BOK = PI.toFloat() - DAM
           val KOV = BOK - BOV
           val KOU = PI.toFloat() / 2 - KOV
           val JOK = PI.toFloat() - DAE

           // bottom left arc
           arcToRad(Rect(Offset(BV, AB - OV), roundRadius), KOU, JOK, false)

           // line to point E
           lineTo(0f, AD)

           // top left arc
           arcToRad(Rect(Offset(BV, OV), roundRadius), PI.toFloat(), JOK, false)
       }
   )
}

The final blueprint with all auxiliary segments.

Despite the overwhelming amount of calculations they are all pretty simple trigonometric and geometrical rules. Defining any other shapes won’t be a big issue once it is sorted out for a triangle.

Link to Github. Commits are for every step we have done. I deliberately didn’t simplify calculations or unite any steps to make it as clear as possible. Feel free to reuse this code in the way that fits you best or contact me directly if any questions still arise.

The Problem

Moving from a traditional Content Management System to a Headless CMS, and need to convert a large number of articles in HTML to JSON/Rich Text data. These formats do not translate 1:1, so you would need a custom script to make that transformation.

The Challenge

Even simple HTML does not translate 1:1 to rich text. Something like text that is bold, italic and has a hyperlink may be nested three levels deep, whereas rich text will simply apply three attributes to that text block. Writing a script to convert that HTML into Rich Text when there is NO custom CSS is a challenge all it's own. If the text includes any custom styles, or more complex semantic HTML, this becomes an enormous development task that can take weeks at a minimum or months if there is a greater level of complexity or a large number or entries with unique formatting.

Here's an example of what a simple transformation would look like:

HTML:

<p>
  <b>
    <em>
      <a href="proxy.php?url=https://example.com">Example</a>
    </em>
  </b>
</p>

Rich Text:

[
  {
    "type": "paragraph",
    "children": [
      {
        "type": "text",
        "text": "Example",
        "marks": [
          {
            "type": "bold"
          },
          {
            "type": "italic"
          },
          {
            "type": "link",
            "attrs": {
              "href": "https://example.com"
            }
          }
        ]
      }
    ]
  }
]

If you have custom class names included in your rich text, most, if not all, rich text converters available on NPM will remove those class names when converting the HTML to Rich Text, resulting in a loss of potentially necessary styling data as well as making it impossible to go back through and find those text blocks that need to be styled a certain way.

One common example of this is the use of block quotes and pull quotes. Some CMSs might use the html blockquote element properly; many of them do not. Even if the CMS properly implemented this HTML tag, if you are writing articles that differentiate between a pull quote and a block quote, then you would have needed custom class names to style those quotes accordingly. In this case, any default conversion script will immediately lose that difference, requiring a huge chunk of time to go through and visually compare the original article to the data entered in the new headless CMS. If you were doing this via script, then you potentially need to handle a wide array of versions of the quote. It's not as simple as using the right module in the new headless CMS, but also converting all the text inside of the block properly (remember our bold, nested link above).

While this is one example, some other ones I've run into are integrated graphs where the HTML is using data attributes (these may be lost or require regex to pull out the right values from them), images (and their custom aspect ratios applied by CSS classes), and HTML tables (also known as the bane of rich text's existence). Each of these elements will require a custom script to convert it from the HTML to whichever Rich Text format you're using in whichever Headless CMS platform you've chosen. In some cases, like for tables or graph integrations, this will vary from project to project and may require recursive checks (think anchor tags nested inside of tables with lots of different colspan's) to figure out how to format this into your own flavor of rich text.

The Solution

My favorite, lightly hacky solution for this issue is to simply migrate over the raw HTML into your new headless CMS and then apply some scoped CSS to the content when rendering the article. Use this approach for migrated articles and your fancy new rich text for all articles moving forward.

The Details

The real trick to this solution is to use it for the old articles but not the new ones. The gist of this is to script over all the articles into the new headless cms into a read only text field that's only visible on these old articles. Then use your WYSIWYG editor for any new articles being created.

To do this, I'll create an additional field called something like contentType with two options: "html" and "richText". Then, as I'm migrating over the old articles, I'll set the value to contentType: "html". In the new CMS, hide the contentType field for any new articles being created and default the value of it to "richText". That way, new articles won't even know that HTML is an option, but if a content editor needs to go back and work on an old article, they'll see that it was ported over as HTML. At this point, if they need to edit that old article, they have two options: reach out to a developer to figure out how to edit the HTML, or build the article from scratch. While this may sound needlessly difficult ("Why not just allow them to edit the HTML?" you ask...), this is very much on purpose. We really do not want raw HTML being passed to the site. It's rare that these old articles will need any editing at all, and when they do, the time spent converting them to the new format is worthwhile, as it's likely that any old articles that require an update are ones that are frequently visited and therefore should reflect the new format of the CMS.

On the frontend, where we render these articles, it's as simple as doing a check on the contentType field and then rendering as you would otherwise. If the contentType is "html", then we use dangerouslySetInnerHTML for React, v-html for Vue, or whatever frontend framework you're using.

Sanity.io Example

{
  title: 'Article',
  name: 'article',
  type: 'document',
  fields: [
    {
      title: 'Title',
      name: 'title',
      type: 'string'
    },
    {
      title: 'Content Type',
      name: 'contentType',
      type: 'string',
      options: {
        list: [
          {title: 'HTML', value: 'html'},
          {title: 'Rich Text', value: 'richText'}
        ]
      },
      initialValue: 'richText',
      hidden: ({ parent }) => !parent?.html?.length // Hide this field if the `HTML` field is empty
    },
    {
      title: 'HTML',
      name: 'html',
      type: 'text',
      hidden: ({ parent }) => !parent?.html?.length // Hide this field if the `HTML` field is empty
      rows: 50 // Make it easier to see the HTML when quick scanning it
    },
    {
      title: 'Content',
      name: 'content',
      type: 'array',
      of: [{type: 'block'}] // This is the Sanity name for rich text
    }
  ]
}

React example:

import React from 'react';
import { PortableText } from '@portabletext/react';

export default function HtmlOrRichText({ block }) {
  if (block.contentType === 'html') {
    return (
      <div
        className="utility-class-to-style-raw-html"
        dangerouslySetInnerHTML={{ __html: block.htmlContent }}
      />
    );
  }

  return (
    <PortableText
      value={block.richTextContent}
      components={{
        blockQuote: ({ value }) => (
          <blockquote className="block-quote-class">{value}</blockquote>
        ),
        pullQuote: ({ value }) => (
          <blockquote className="pull-quote-class">{value}</blockquote>
        ),
      }}
    />
  );
}

The Cleanup

The biggest issue with this approach is that, unless you spend an insane amount of time writing super specific CSS selectors (in which case you probably should have just written the data migration script anyway), you're likely going to have to accept a simplified approach to your styling on these old articles. In most cases, this is fine. Often, the majority of these old articles haven't been visited in a while, and it's more about the content they hold for SEO purposes, than it is ensuring that they perfectly adhere to the new branding guidelines. If this is unacceptable, then this approach is not for you.

Given this shortcoming, I recommend simply going through and updating the top performing articles into the new format. With most Headless CMS, this is a fairly painless process that takes at most 10 or so minutes per article (often much less), depending on the length and complexity of the article and the new format. Now you can be sure that the top performing articles are styled in the new format, while also feeling confident that none of the old content has been lost in the other articles that have been migrated over. In some cases, the other, lower performing articles can be converted one by one after launch of the new site. In other cases, it's perfectly fine to leave them as is.

Gotchas & Notes

A few notes for anyone taking up this approach:

• Make sure you convert image URLs. If the images are hosted on the old CMS, the URL may be something like: https://old_cms_url.your_site.com/images/some-random-string. You'll need to port these images over to the new site as part of the migration. This would be necessary whether you port over the article as HTML or convert it to rich text, so you're doing this either way. For this process, just make sure you do a find all and replace of the old_cms_url and replace the full src with the URL from your new CDN.
• It's worth finding some libraries to help with the rendering of the HTML. Here's one that pairs with chakra-ui: https://www.npmjs.com/package/@nikolovlazar/chakra-ui-prose, but there are lots out there to help with this. Remember, the idea of this approach is to move quickly.
• Make sure you're not siloed on how this all works. It's important that other people have context for what you're doing and why you're doing this. Otherwise you are going to have go back and script the whole thing all over again.
• Be aggressive about collecting other data from the posts. Don't rely on this approach for things like title, related articles, or any other type of tracking/SEO data that you need for the article. That should all be ported into your new headless CMS just like it would otherwise. This is just for the CONTENT of the article to make it simpler to render it properly.
• Beware of script tags. If you had users jamming script tags into the old articles, make sure you double check them to be sure they are working, or yank them out and covert those articles manually (if this is tenable). If there are too many script tags to port over manually, then, once again, this process is not for you.
• Save your work. Track your script in a repo so you can see what changes have happened to it over time in case you did something that caused a data loss.
• Save your data. Even though the old data is likely to take up an enormous amount of space, find a way to hang onto it well through the site launch. Use an S3 bucket or something if you have to, but make sure you can go back and grab the original data in case something got lost in translation here.

]]>https://engineering.deptagency.com/rapid-enterprise-development-with-redwoodjs/650b2f0041e0af00019d49c0Mon, 25 Sep 2023 16:46:22 GMT

Redwood quietly entered the framework world in March 2020, with an enterprise first approach as “the framework for Startups” (Next.js, Remix, etc). If you haven’t heard of Redwood it was founded and created by Tom Preston-Warner, co-founder of Github, creator of TOML language, and many other ventures.

Redwood describes itself as, “an opinionated, full-stack, JavaScript/TypeScript web application framework designed to keep you moving fast as your app grows from side project to startup.”

What makes RedwoodJs unique as a framework is that it doesn’t exactly reinvent the wheel, but instead uses industry standard tools that we would already use in an opinionated full-stack framework. It uses creative and smart integrations with boilerplate code that feels like the monolith we never knew we needed. The backend stack runs on Prisma, Graphql, and Node. This is independent from the frontend, but easily integrates with it using "cells”.

A cell in Redwood is a collection of frontend code used to query to API layer that allows you to quickly add boilerplate code you need to get an application to MVP quickly and easily. Just as the backend uses industry standard tools, the frontend follows that same pattern. When scaffolding your component, you also get a full component built out for you including a storybook story already ready to view before you begin development, Jest tests setup and working, and a functional base component.

Let's build a quick to-do app connected to a Postgres database on Railway. We will explore installing and setting up a Redwood app, setup a schema, scaffold our component, set up some routing, and quickly add styles.

Setting up Redwood

Setting up Redwood is quite easy. First we will use the redwood-app package to install our project:

yarn create redwood-app dept-todos --typescript

This will create our project, initialize a git repo, and perform our init commit. Now we can swap to the dept-todos folder created during setup:

cd dept-todos

Next, we will want to set up Tailwind on our project. With Redwood that is also a very simple CLI command:

yarn rw setup ui tailwindcss

Side note before continuing...
If you didn’t want to use tailwind in a project, it is just as easy to setup a sass configuration. Instead of running the tailwind setup above, you could simply run the following command to add a sass setup:

yarn workspace web add -D sass sass-loader

Adding a schema

Redwood uses Prisma as the ORM for the API layer. If you are not familiar with Prisma, they have great documentation and use cases on their website.

After creating a postgres database on Railway and adding the postgres address to our .env file at the root of our directory, we are now ready to set our schema.

We will keep this schema very simple with one model for our todos, located at api/db/schema.prisma.

Replace the contents with our new todo schema:

datasource db {
    provider = "postgresql"	
    url = env("DATABASE_URL")
 }
    
generator client {
    provider = "prisma-client-js"
    binaryTargets = "native"
}

model Todo {
    id        Int      @id @default(autoincrement())
    body      String   @db.VarChar(255)	
    completed Boolean  @default(false)	
    createdAt DateTime @default(now())	
    updatedAt DateTime @updatedAt
}

As you can see we have an id, body, completed status, and a timestamp - a very simple setup for our todos. We now need to migrate and deploy our schema changes to our database. This can be done easily with Prisma.

yarn redwood prisma migrate dev

Prisma will run its migration CLI tool and prompt you to name the migration for your schema changes. We just named this migration todos model.

We have now saved a migration file to our repository and migrated the changes to our database on Prisma.

Now that the database changes are active, let's scaffold our Todos. Scaffolding is easy and should be familiar to those who have worked with Ruby/Rails, we can run the following command to get our todo’s component ready.

yarn rw generate scaffold todo

That created a folder in our components on the frontend, set up the associated routing in our router file, and set the the API files needed for basic CRUD functionality.

As you can see, Redwood set up our todo’s component on the frontend and a CRUD file for the API.

Our Scaffolding created components, pages, services, a todo layout, and more for us with one simple command. This can be a very time consuming step during the initial setup of a traditional application and with Redwood we are able to go from install to running in less than 5 minutes.

Start up your development server and see the todo app in action!

yarn rw dev

The Vite instance and your API layer will start and you should be able to see your site now live at http://localhost:8910/.

While our app is already functional, it’s not really great to use out of the box. In the next section we will style and refactor some of the boilerplate files to improve the user experience and deploy our application!

Time to style

Now that we have our core API setup for an app, let’s create a homepage to view our app. We can use the CLI for this as well.

yarn redwood generate page home /

This adds a homepage route to our Router. We will use this page during our refactor to make our app more usable. It can be found at /web/src/Routes.tsx.

In the Router, you will see our todo model was added under the todos route during the scaffolding process:

import { Set, Router, Route } from '@redwoodjs/router'

import ScaffoldLayout from 'src/layouts/ScaffoldLayout'

const Routes = () => {
  return (
    <Router>
      <Set wrap={ScaffoldLayout} title="Todos" titleTo="todos" buttonLabel="New Todo" buttonTo="newTodo">
        <Route path="/todos/new" page={TodoNewTodoPage} name="newTodo" />
        <Route path="/todos/{id:Int}/edit" page={TodoEditTodoPage} name="editTodo" />
        <Route path="/todos/{id:Int}" page={TodoTodoPage} name="todo" />
        <Route path="/todos" page={TodoTodosPage} name="todos" />
      </Set>
      <Route notfound page={NotFoundPage} />
    </Router>
  )
}

export default Routes

Let’s modify the router so our todos will render on the root path:

import { Set, Router, Route } from '@redwoodjs/router'

import ScaffoldLayout from 'src/layouts/ScaffoldLayout'

const Routes = () => {
  return (
    <Router>
      <Set wrap={ScaffoldLayout} title="Todos" titleTo="todos" buttonLabel="New Todo" buttonTo="newTodo">
        <Route path="/new" page={TodoNewTodoPage} name="newTodo" />
        <Route path="/{id:Int}/edit" page={TodoEditTodoPage} name="editTodo" />
        <Route path="/{id:Int}" page={TodoTodoPage} name="todo" />
        <Route path="/" page={TodoTodosPage} name="todos" />
      </Set>
      <Route notfound page={NotFoundPage} />
    </Router>
  )
}

export default Routes

Excellent! Now you can see our vanilla scaffold functional on our root path.

When we generated the scaffold earlier, we also created a todos folder that contains both our components and our cells, the frontend code that will host our graphql calls and hydrate our components.

First we are going to install Lucide icons to update the generic links. We do this by first navigating into our web directory where react is present:

cd web
yarn add lucide-react

Then we can replace the contents of Todos component at  web/src/components/Todo/Todos/Todos.tsx with the following snippet:

import { FileEdit, XCircle } from 'lucide-react'
import type {
  DeleteTodoMutationVariables,
  EditTodoById,
  FindTodos,
  UpdateTodoInput,
} from 'types/graphql'

import { Form, CheckboxField } from '@redwoodjs/forms'
import { Link, navigate, routes } from '@redwoodjs/router'
import { useMutation } from '@redwoodjs/web'
import { toast } from '@redwoodjs/web/toast'

import { QUERY } from 'src/components/Todo/TodosCell'
import { truncate } from 'src/lib/formatters'

const DELETE_TODO_MUTATION = gql`
  mutation DeleteTodoMutation($id: Int!) {
    deleteTodo(id: $id) {
      id
    }
  }
`

const UPDATE_TODO_MUTATION = gql`
  mutation UpdateTodoMutation($id: Int!, $input: UpdateTodoInput!) {
    updateTodo(id: $id, input: $input) {
      id
      body
      completed
      createdAt
      updatedAt
    }
  }
`

const TodosList = ({ todos }: FindTodos) => {
  const [deleteTodo] = useMutation(DELETE_TODO_MUTATION, {
    onCompleted: () => {
      toast.success('Todo deleted')
    },
    onError: (error) => {
      toast.error(error.message)
    },
    refetchQueries: [{ query: QUERY }],
    awaitRefetchQueries: true,
  })

  const onDeleteClick = (id: DeleteTodoMutationVariables['id']) => {
    if (confirm('Are you sure you want to delete todo ' + id + '?')) {
      deleteTodo({ variables: { id } })
    }
  }

  const [updateTodo] = useMutation(UPDATE_TODO_MUTATION, {
    onCompleted: () => {
      toast.success('Todo updated')
      navigate(routes.todos())
    },
    onError: (error) => {
      toast.error(error.message)
    },
  })

  const onSave = (input: UpdateTodoInput, id: EditTodoById['todo']['id']) => {
    updateTodo({ variables: { id, input } })
  }

  return (
    <div className="flex justify-center px-8">
      <table className="container max-w-4xl">
        <tbody>
          {todos.map((todo) => {
            if (!todo.completed) {
              return (
                <tr
                  key={todo.id}
                  className={`flex items-center p-4 transition-opacity ${
                    todo.completed && 'opacity-25'
                  }`}
                >
                  <td>
                    <Form>
                      <CheckboxField
                        id="completed"
                        name="completed"
                        onChange={() =>
                          onSave({ completed: !todo.completed }, todo.id)
                        }
                        defaultChecked={todo.completed}
                        className="rw-input h-4 w-4"
                        errorClassName="rw-input rw-input-error"
                      />
                    </Form>
                  </td>
                  <td className="flex-1 px-2">{truncate(todo.body)}</td>
                  <td>
                    <nav className="rw-table-actions gap-2">
                      <Link
                        to={routes.editTodo({ id: todo.id })}
                        title={'Edit todo ' + todo.id}
                        className="text-gray-500 hover:text-green-500"
                      >
                        <FileEdit />
                      </Link>
                      <button
                        type="button"
                        title={'Delete todo ' + todo.id}
                        className="text-gray-500 hover:text-red-500"
                        onClick={() => onDeleteClick(todo.id)}
                      >
                        <XCircle />
                      </button>
                    </nav>
                  </td>
                </tr>
              )
            } else {
              return (
                <tr key="notodos" className="w-full text-center">
                  No todos. Please add a todo to create a list!
                </tr>
              )
            }
          })}
        </tbody>
      </table>
    </div>
  )
}

export default TodosList

We can now see our newly designed todos at the root of our application (http://localhost:8910/) as seen below.

And after adding our first todo:

Next steps?

Now it is time to explore the full power of Redwoodjs and explore some of its features. Some fun ideas would be to implement user authentication to keep those todos separate, update the input styles, and possibly a view for seeing completed todos.

To view the demo repo, https://github.com/deptagency/blog-todoapp-example.

Conclusion

While our example was very simple, you can see how quickly it is to get up and running with functional code using RedwoodJs! Now for some caveats, at the time of this writing I am unable to recommend Redwood in its current state for beginners. Redwood’s development started as an more of an enterprise scaling applications more so than consumer usage so if you are not familiar with the underlying tools (Prisma, GraphQL, Node, React, Postgres), I would encourage you to start with gaining fundamental knowledge in those tools before diving in too deep with Redwood.

The future roadmap includes integrating with React Server components as the React core team continues to update React’s core to utilize these more smoothly.

If you still want more Redwood, I would encourage you to view the outstanding documentation on the Redwood homepage.

DevOps Quick Fix are solutions to common DevOps problems, usually encountered during client engagements.

Problem

From Google Cloud Platform (GCP), your outgoing HTTPS connections to third-party REST APIs are slow or dropping.

Solution

Preamble

The cause of the issue is most likely GCP Cloud NAT port exhaustion.

If you are using private network Compute Engine, GKE, or any of the GCP serverless services like Cloud Run, your outgoing connections are going through Cloud NAT.

Cloud NAT needs a unique 5-tuple [source IP, source port, destination IP/port/protocol] to make an outgoing connection. If all the 5-tuples are used, the connections are slowed or dropped. This is known as NAT port exhaustion.

Turn off Endpoint-Independent Mapping

You can read the gnarly technical details about Endpoint-Independent Mapping conflicts, but bottom-line, EIM is more trouble that it is worth. Turn EIM off and increase the "Minimum ports per VM instances" setting (2048 is a good starting option).

Steps:
1. In GCP console, go to Cloud NAT
2. Click your Cloud NAT Gateway link
3. Click the "Edit" button on top, then click "Advanced Configuration"
4. Uncheck "Enable Endpoint-Independent Mapping" and increase the "Minimum port per VM instance" value
5. Click the "Save" button at the bottom

Optional: Enable Dynamic Port Allocation

Dynamic Port Allocation will scale up the number of VM ports when it senses port exhaustion. The main issue is connections can drop during scaling. I usually prefer to turn it off and set the "Minimum ports per VM instance" to a high number.

If you prefer to turn on "Dynamic Port Allocation", set the minimum port high to reduce the chance of dropped connections.

Optional: Assign more IPs to Cloud NAT

If you are using static reserved IPs for Cloud NAT because of third-party API firewall allow-lists, you can manually add more reserved IPs to reduce port exhaustion.

Steps:
1. In GCP console, go to Cloud NAT
2. Click your Cloud NAT Gateway link
3. Click the "Edit" button on top, then go to the "Cloud NAT mapping" section.
4. Click the "+ ADD IP ADDRESS" button to add more IPs to Cloud NAT
5. Click the "Save" button at the bottom

Are your Docker builds slow in GitHub Actions? Here's how to speed it up with the built-in GitHub Actions cache.

Official Docker Action

Docker published an official GitHub Actions cache integration along with an official GitHub Actions plugin.

Here's how to use both and turn on the GitHub Actions cache to speed up your Docker builds. The trick is to set cache-from and cache-to to type=gha

      -
        name: Build Tag and Push Docker image
        uses: docker/build-push-action@v4
        with:
          file: nextjs-blog/docker/Dockerfile
          context: nextjs-blog
          tags: ${{ steps.dockermeta.outputs.tags }}
          labels: ${{ steps.dockermeta.outputs.labels }}
          push: true
          cache-from: type=gha
          cache-to: type=gha,mode=max

A complete working example is here at
https://github.com/deptagency/engineering-blog-github-actions/blob/main/.github/workflows/docker-nextjs.yml

How do you know that the cache is working? The Docker build output will tell you. Your first Docker build will be slower because it's populating the cache. For subsequent builds, you will see a Docker output similar to:

#10 [deps 1/4] RUN apk add --no-cache libc6-compat
#10 CACHED

See sample output here
https://github.com/deptagency/engineering-blog-github-actions/actions/runs/5895631079/job/15991811696#step:7:176

Alternatives

GHA local cache

If you want more control over your GitHub cache, you can use Docker local cache. Just set cache-from and cache-to to type=local .

      -
        name: GitHub Actions Cache Docker layers
        uses: actions/cache@v3
        with:
          path: /tmp/.buildx-cache
          key: ${{ runner.os }}-buildx-${{ github.sha }}
          restore-keys: |
            ${{ runner.os }}-buildx-

      -
        name: Build Tag and Push Docker image
        uses: docker/build-push-action@v4
        with:
          file: nextjs-blog/docker/Dockerfile
          context: nextjs-blog
          tags: ${{ steps.dockermeta.outputs.tags }}
          labels: ${{ steps.dockermeta.outputs.labels }}
          push: true
          cache-from: type=local,src=/tmp/.buildx-cache
          cache-to: type=local,dest=/tmp/.buildx-cache-new,mode=max

A complete working example is here at
https://github.com/deptagency/engineering-blog-github-actions/blob/main/.github/workflows/docker-nextjs-actions-cache.yml

Registry cache

You can use the container registry, such as GitHub Container Registry or AWS ECR, to store the image build cache.

For this example, we will use the AWS ECR by setting the cache to type=registry

      -
        name: Build Tag and Push Docker image
        uses: docker/build-push-action@v4
        with:
          file: nextjs-blog/docker/Dockerfile
          context: nextjs-blog
          tags: ${{ steps.dockermeta.outputs.tags }}
          labels: ${{ steps.dockermeta.outputs.labels }}
          pull: true
          push: true
          cache-from: type=registry,ref=${{ env.ECR_FULL_REPO }}:dockercache
          cache-to: type=registry,ref=${{ env.ECR_FULL_REPO }}:dockercache,mode=max,image-manifest=true

A complete working example for GHCR is here at
https://github.com/deptagency/engineering-blog-github-actions/blob/main/.github/workflows/docker-nextjs-registry-cache-ghcr.yml

A complete working example for AWS ECR is here at
https://github.com/deptagency/engineering-blog-github-actions/blob/main/.github/workflows/docker-nextjs-registry-cache.yml

One of the most common things to do in unity is create a character controller so that you can take input and make something move and interact with its environment. There are lots of options for this along with tutorials on how to create your own, but what is often overlooked is how to make something that is maintainable and can scale as your game becomes more complex. Most tutorials produce code like this which is fine for understanding the basics of locomotion in Unity, but not something you would want to replicate for a character controller that will eventually have dozens of states.

public void Update()
{
	...

    if (wantingToSprint && areWeGrounded && !areWeCrouching)
    	currentSpeed = sprintMoveSpeed;
    else if (!areWeCrouching && areWeGrounded)
    	currentSpeed = walkMoveSpeed;
    if(wantingToCrouch && jumpCrouching)
    	crouch = true;
    else
    	crouch = false;
    if (areWeGrounded)
    	coyoteTimeCounter = coyoteTime;
    else
    	coyoteTimeCounter -= Time.deltaTime;
    if (wantingToJump)
    	jumpBufferCounter = jumpBuffer;
    else
    	jumpBufferCounter -= jumpBuffer;
    if (coyoteTimeCounter > 0f && jumpBufferCounter > 0f && jumpCoolDownOver)
    {
    	characterController.velocity = new Vector3(characterController.velocity.X, 0f, characterController.velocity.Z);
        characterController.Move(transform.up * jumpForce);
        jumpCoolDownOver = false;
        areWeGrounded = false;
        jumpBufferCounter = 0f;
        currentSpeed = jumpMoveSpeed;
        endJumpTime = Time.time + jumpTime;
        Invoke(nameof(jumpCoolDownCountdown), jumpCooldown);
    }
    else if (wantingToJump && !areWeGrounded && endJumpTime > Time.time)
    	characterController.Move(Vector3.up * jumpAcceleration);
    
    ...
}

My goal in this article is to help you get started and apply some general C# coding best practices along the way to help you create something that’s scalable from the very beginning and minimizes spaghetti code like the previous example.

Prerequisites

You can download the source code from GitHub and here’s a video demo of what we are going to create: