Unintentionally, I’ve been mostly quiet on here since then. In some ways 2025 felt like a year of growing up, both for the business and for me.

But before I start waxing poetic about running Purview Labs, we can run through the numbers. I know that’s what everyone wants to know.

Money, Clients & Projects:

Profit Margin is going down, and according to my adjusted calculations, clocks in at ~65%. This is notably different from my last review at a respectable ~85%.

Revenue went up. In fact, this was another year where the top line was at least 1.5x the previous year. This is not exponential growth. But it is sustained.

I wrapped up long standing engagements with two clients. One engagement lasted nearly 2 years. The other almost 10 months. At a certain point, you just know when to move on.

I worked on projects for 12 different clients in 2025. 4 of those clients were repeat customers.

Technologies

I’ve seen a lot of the usual suspects over the last year:

• AWS, GCP

• Snowflake, BigQuery, Postgres

• Dagster, Airflow

• FastAPI, Flask, Django

• dbt, Dataform

Only 2 projects did not involve writing code.

There are plenty of other tools and technologies to add to that list but those details are really not that important.

Can You Deliver?

I still don’t have a logo and the Purview website is little more than text on a screen.

The only thing that matters is whether your client believes you can deliver what you say you can deliver.

Substitute client with stakeholder, boss, CEO, manager, interviewer, whoever. That is the only thing that matters.

And the way to instill that belief is to:

1. Know what you’re taking about

2. Have proof you’ve done it before

It’s a simple formula. But simple is not easy, as I always say.

Where Is The Constraint?

Business is filled with constraints. And only one matters at any given time.

If I get all Alex Hormozi on you, that constraint is either a demand constraint or a supply constraint.

It could be time, mental capacity, cost, staffing, sickness, AWS outages, who freaking knows. There are a million things that affect your ability to deliver but only one of them matters at any given time.

As a mostly one-person business, I started out the year as the constraint. And, I remained the constraint for most of the year. Unfortunately, the business in it’s current form would not survive without me.

But I am working to unwind that.

Getting contractors to work alongside me has been the first step in that direction. I have 3 of them right now. This has been transformative for the business. But there is still A LOT that can be improved.

I think of myself as a fairly intentional and purposeful decision maker. But I still hem and haw about mundane details. So, I talked to some coaches. I mostly wanted to gut check that what I am doing is not foolish. It’s easy to look at a post like this and think that things are trending in the right direction, but on a daily basis, that is not obvious.

It’s a big game of trial and error. You try some things. They either work or they don’t. Then your try some other things. The cycle continues.

And as it turns out, sometimes that “other thing” is just doing more of what you’re already doing. It’s not about doing something new; it’s just about doing more of what works.

It Doesn’t Have To Be Hard

My network has grown significantly since I started this business. Some of the people I talk to want to go out on their own.

I tell most of them some version of the same thing: if you actually know what you’re talking about, it really doesn’t have to be that hard.

I used to feel guilty when saying that out loud. Concerned I would come across as arrogant. But I don’t feel that way anymore.

Doing hard things is interesting and cool. It tells a good story.

But you don’t have to manufacture a challenge.

The challenges already exist. And by virtue of trying to do something a little different from everyone else (ie working on your own business), you will be challenged.

But, that doesn’t mean it has to be hard.

There are endless ways to make money. There are limitless clients looking for help. And they all have countless problems that need to be solve.

If you know your stuff as well as you think you do, the “hard” part isn’t doing the work.

This defies everything we’re seeing from proponents of “9-9-6” and all those hustle-culture warriors online. A past version of me could’ve gotten behind that mindset to some degree; but today? No thanks.

Effort and value are not the same thing.

The Inevitable Outcome

This business has a lifespan.

Ironically, the more the business grows, the more I recognize this to be true.

If AI automates away all value-add tech work, so be it. At that point, money won’t mean the same thing anyway.

More realistically, however, this business will run its course long before that happens.

In fact, I know it will. I tell people all the time that I am going to capitalize on this opportunity as best I can, while I can.

I have no idea what that timeline is, but I assume I smack in the middle of it right now.

I believe one of two things will occur:

• The company grows to have full-time employees. We ride the wave as long as makes sense

• Purview in its current form no longer exists and I join forces with a client (future or current, who knows) to build something else

Either would be a welcome next professional “season”.

A More Personal Note

I don’t share much personal news online these days.

But two big things happened in 2025 that I need to mention.

First, I got married in February.

And second, we bought a house towards the end of the year.

2025 was a year of upgrades.

If you want to talk data, engineering, entrepreneurship, or anything related to those topics, give me shout.

You can find me on LinkedIn and (very) occasionally on X.

Joe

It applies to everything in work and life.

If you are here and you want to get there, how do you make that happen?

In personal cases, it’s relatively easy to figure that out. You search online. Read some blogs. Talk to some friends or acquaintances. Maybe you got to therapy.

Identifying the how is straightforward, but the doing is not.

In professional cases, it’s different. You’re limited by the breadth and depth of the knowledge of your immediate team, of your current system, of the established processes. You need to support the old way while pushing forward into the new.

Getting from here to there in a professional or business context needs to be incremental. It needs to be iterative.

You’ll likely need to overcome the sunk cost fallacy, do a cost-benefit analysis, and produce a number of other one-off justifications. Those initiatives are worthwhile and necessary within reason. But they are incomplete.

When you break the established frame of good, you change the equation. Suddenly the old option is no longer sufficient.

Sticking with what is familiar can work, but it can also stifling progress. Especially as new requirements get introduced into the mix.

Good in this context may look markedly different from what good looked like in the past.

This holds true for evaluating new tools and systems, designing systems, and vetting consultants.

What’s familiar is not necessarily what is good.

I’ve been delaying posting this update for a few months. Life and work have have kept me pretty busy.

But, I know you’re here for the numbers first and my commentary second, so let’s get those out of the way.

Here’s where we (the Royal we, of course) clocked in at the end of a 2nd year full time operating on my own.

Money:

Profit Margin: 85%

I 2x’d my 2023 revenue. Not bad for a one man show.

I did not bill 2x the hours of 2023. This is a good thing.

Clients:

I had 9 clients throughout the course of the year.

Of that total, 3 were repeat customers. And 2 were referrals.

1 client found me.

I was ghosted by a handful (3 I think?) prospects after talking multiple times, proposing a plan of attack, and getting into the weeds on pricing.

Projects/ Engagements:

I worked on 10 different engagements throughout 2024. Some took up less than 1 full day of billable time. Others lasted the entire year.

3 projects did not include writing code.

4 of the projects involved Airflow, with 3 of them requiring HEAVY usage of it.

2 projects involved Dagster.

5 of the projects used dbt to some degree. The implementations varied wildly in terms of maturity and complexity.

Ok, now it’s time for the commentary.

Crossroads

I offer a service that generates some decent cashflow. I sometimes hesitate to call it a business. Some of you might think that’s a dumb distinction, and you might be right.

There is a myth that some people tell themselves when they do independent consulting. The myth that you’ll work on a handful of projects, find the common thread across some problem-space or industry, and build a product from it.

I do not know a single person who has pulled this off.

But I know plenty of people who have lasted a few seasons trying to be a contractor, or a fractional Head of X, or an independent consultant. Then they stop.

Maybe they can’t sell, maybe they can’t market themselves as well as they thought, or maybe their personal lives changed and they need more stability. More predictability.

Maybe they just got bored.

The challenge with transitioning from being a consultant to building a product is that if you’re any good at solving problems, you’ll be making a fair amount money. And switching gears from doing that to building a product is hard.

Not to mention expensive.

And this is the crossroads where I find myself.

Professional Attention Span

This might ruffle some feathers.

And I may run the risk of coming off the wrong way with this comment, but my professional attention span has plummeted.

You might be able to re-label it as professional patience or even tolerance, but the short of it is that I have learned to value my time, energy and attention at a much higher level than when I was not working for myself.

Illusions (Delusions?) of Grandeur

I used to post on LinkedIn regularly. I stopped a couple years ago because my feed was turning into an echo chamber of self-aggrandizing and unhelpful commentary all centered around data.

Actually, my feed is still an echo chamber.

I stopped posting on Twitter (or even logging in for that matter) for mostly the same reason. Similarly, my attempts at BlueSky lasted all of a few months.

That’s not to say that I was gaining little from these platforms.

They provided value at times.

But they are tiring.

Not everyone can be an expert. That’s just reality. If everyone is an expert, no one is.

I’m fairly knowledgeable about the world of data and analytics. I do not consider myself “an expert”.

The breadth and depth of knowledge needed to be considered an expert is prohibitive to most - there are too many nuances, too many subcategories, too many parts to the whole.

But whenever I log into LinkedIn, it sure seems like there are a lot of experts.

And they all know how that problem - your problem - should be solved. How to do it the correct way.

There are many known solutions to common problems we see as engineers of any discipline - data engineers, backend engineers, frontend engineers, even machine learning engineers.

But things aren’t quite so cut-and-dry as many online want you to believe.

Problem solving is an art. The application of specific solutions is the science.

Middle of the Pack

At the risk of sounding overly simplistic, there are two types of people I see everywhere online.

One is the previously mentioned expert; sometimes they have a strong personal brand. They have the answers. They know their stuff!

The other is the newcomer. Wet behind the ears, and figuring things out. They have questions. They want to learn.

As you could probably guess, I consider myself to be neither of these, but I can relate to both.

But, the group I most closely identify with professionally are also not found in either bucket. And they’re really hard to find online.

Am I doing $1M annually? Not yet.

Am I struggling to pay the bills? Also no.

Those two “profiles” are all over the internet, especially in the “solopreneurship” and independent consulting communities. Where are the people in the middle?

Turns out, they’re just getting shit done and living their lives.

If this is you, it can feel like being in the middle of the pack. But that likely means you’re comparing yourself against the wrong group.

Because the people really crushing it are often the ones you’ve probably never heard of.

On Operating Solo

I spend a lot of time thinking.

Too much time, in fact.

Past a certain point, this is one of the most dangerous places to spend time when working alone.

I’m not talking about the instances when you’re in a “flow state”. I’m talking about an mundane Tuesday when there is not a ton going on.

The biggest downside of working independently? The opportunities for serendipity are lacking.

But the opportunities for over-analysis and preoccupation abound.

Sure, I talk to client teams every day. And work closely with them.

But that’s not the same as working with your immediate team. I do miss some of those things.

What I’ve come realize is that my ability to understand clients needs, execute on the technical components, and communicate with clarity are the things that have helped me along this journey.

Those are inherently social activities.

But my current setup is not particularly conducive to fostering the social aspects of professional life, at least not on a daily basis.

Which brings us full-circle to the crossroads I mentioned towards the top of this post. Funny how that works.

Unprecedented Times! Uncertainty! Fear!

The current state of the market - globally, in the US, and more specifically within tech - is volatile. There is no question.

But, volatility breeds opportunity.

I have no idea how things will play out, but I am operating under the assumption that I can take advantage of the current market conditions.

There really isn’t much of an alternative.

At this point in my career, I strongly prefer to be making the decisions than being told what decisions were made.

Regardless of which direction the economy may head, these things will always remain true:

• Companies will still have problems they do not know how to solve

• Knowledgable and trustworthy outside experts will be the fastest way to address those problems

Does that mean it will always be smooth sailing?

Of course not.

But it does mean that knowing how to solve problems is how you weather the storm. And if the storm never comes, you keep on solving those problems just the same.

If any of this resonated with where you find yourself professionally, I’d love to connect.

If you’re doing it on your own or part of a small team, that’s even better.

You can find me on LinkedIn (even if I’m not posting regularly).

It requires less commitment, less resolve.

You can hide your business’s lack of clarity (or your own) with confusing terms and countless stakeholders and unclear requirements.

The result? An abstract mandate from an exec to “figure out our data strategy”.

But if you ask 10 different people what constitutes a data strategy, you’ll get 10 different answers.

The quality of your strategy has nothing to do with the size of you business. But “good” data strategies generally follow the same themes:

1. A data strategy is not made up of tools.

2. A data strategy has less to do with “the data” than you think.

3. A data strategy is never actually finished.

4. An incomplete data strategy does not mean it is an ineffective one.

Good data strategies are not complicated.

Hell, good strategies are not complicated.

Effective data strategies:

1. Meet consumers where they live

2. Provide a consistent experience

3. Extend to all

Simple, but not easy.

Meet consumers where they live

If your account team spends their time in Salesforce, you’re not going to have a good time telling them to use Metabase or Omni or some other tool. Don’t push for something new. Put the data where they spend their time.

Do your customers want to export data for their own use? Data sharing can take many forms, from warehouse native integrations to cloud storage. The methodology is a one part of the a larger whole.

The entry points you enable are arguably as important as the data you provide. Keep this in mind; we’ll come back to this one in a minute.

Don’t lose sight of the bigger picture - the most important factor is whether or not the data you share actually provides value to those who consume it.

Provide a consistent experience

Consistency means a few thing in this context.

Behavior needs to be consistent across systems.

Data needs to be consistent across access points - for instance, those customer exports and that Omni dashboard. Is data a core component of your in-app experience? Great, you better make sure everything jives across these various points of access. The only way to do this is to ensure they are logical and well-maintained.

The “single source of truth” label everyone likes to throw around is a bit of a misnomer; there is seldom a singular place to go to get all the data you want. Instead, a well-structured and well-understood set of resources (ie tables, datasets, etc) are what you need.

A data platform is an effective way to eventually make this “single source of truth” a reality, but it can often become just a formal entrypoint into an ugly mess behind the scenes.

Beware the “lipstick on a pig” phenomenon. You still need to establish standards and adhere to them.

Remember, tooling does not make a strategy.

Data platforms are also never quite finished, even when you think they may be. The velocity of change may slow and shrink in size and scope, but there will always be change.

It’s simply the reality of providing one centralized point of entry into a much broader and heterogenous set information.

Iterations of work to build and harden your data platform for many use cases are a requisite. Don’t be afraid of this part.

Those iterations also serve another purpose - refining your data strategy.

Extend to All

It might seem odd to use platform development to inform your strategy. Shouldn’t your strategy inform your development work?

If you are an established business with stable revenue, stable tech, and well understood growth levers, sure. If you are not, prepare for a lot of iteration.

But one thing is for certain - your data strategy defines how everyone interacts with data at your business. This means your customers, internal users and the many teams across the organizations.

They may have different SLAs or require different sets of data. But your customers - both the ones who pay you and your coworkers on different teams - can be serviced by the same strategy.

If you got this far and found yourself thinking “ this is not an exhaustive list of criteria for a data strategy”, you’re right.

Itemizing every single thing that can fall under the umbrella of “data strategy” results in more noise than value.

There is an elegance in simplicity. However simplicity does not suggest oversimplification.

Some data problems are hard. They require a lot of work, take up many resources, and have many moving parts.

An effective data strategy does not need to be hard to design or implement.

In fact, it should not be.

The simpler your data strategy, the easier it will be for your team to deliver on it.

The people at the corner store know you. They’ve seen you at your best and your worst. They know you enjoy that one weird drink flavor that no one else buys, and often ask what you’re getting up to.

They know your breakfast order. And they make a joke when you change it up from your usual.

You don’t get that type of attention from the wholesaler.

The wholesaler spends their time analyzing margins. Forecasting fulfillment and renegotiating contracts. Our delivery utilization is below the norm, they say in long internal email chains.

There are 8 people CC’d on your emails with their team. Your recent call with the wholesaler had even fewer people engaged than the last one. A video call filled with the gray squares of disabled laptop cameras and disabled microphones.

Your usual restock is one of dozens, even hundreds that they have lined up this week.

This is not a story about local grocers and suppliers.

This is a story about consulting.

Consulting firms are either the corner store or they are the wholesaler. They cannot be both, at least not at the same time.

I run what I consider to be a corner store. I am hired to solve problems for specific teams. Those problems happen to be related to systems processing and producing data. I provide a specialty set of services and tailor those services to the company’s needs.

Its current setup will not scale to tens of millions in revenue. And that is ok. It’s not intended to right now.

But a wholesaler would not be ok with this.

The wholesaler is a necessary force. They operate at a scale that some businesses require. But many do not.

You could argue that most do not.

Even those with some venture funding in the bank often don’t need wholesalers. They need corner stores.

They need someone to solve their problems. To anticipate what comes next. To actually care about the outcomes.

You lose that level of detail when dealing with wholesalers.

I inherited a project built by a wholesaler a few years back. It had cost the business somewhere around $300k over the course of the build. It followed the well-established formula for what “should work”.

But it didn’t solve the problems we faced well enough. It was cumbersome. Fragile. And obviously expensive.

The business didn’t need a wholesaler. They needed a corner store.

They just didn’t realize it.

This post was inspired by the work that Jacob Matson, Pete Fein and I have done helping companies make use of their data lakes.

If you’d rather read the code, you can find it all in Github: Lean Data Engineering.

But make sure to check out the Implementing This Pattern in the Real World section before giving it a go.

Data lakes are nice because you can throw anything you want in there and move on. This is also why they cause a lot of trouble.

But, we like quick wins. We also like a short time-to-value. So, we’re going to cover a quick-ish pattern for taking what’s in your data lake and making it useful. Limited fancy tech. Limited infrastructure.

Just python, SQL and a couple tools you may have already heard of.

We’re talking lean data engineering.

There are plenty of different - and difficult - ways to get data from one place to another. But, we’re keeping this simple.

Believe it or not, Dagster, DuckDB and a warehouse of choice (we’re using Snowflake) will take you very far. 

AWS and GCP have made it especially easy to create systems designed for high-volume data. In the time it takes to read this post, you could launch your streaming pipelines with AWS Kinesis or GCP PubSub. And landing that event stream in S3 or GCS is trivial.

But, it’s only half the battle. 

The last mile - bringing that data into your data warehouse and actually using it - is the real challenge. 

Data lakes are convenient, but they reframe traditional data engineering challenges. Technical projects always have some governance and cost components. Within the changing context of a data lake, though, they can quickly become challenges. 

They present a set of data engineering problems not found in traditional pipelines. They change the narrative. 

There is no friendly package of data + compute. Instead, it becomes ““here is all the data; provide your own compute”.

It’s common to see companies dumping data into their data lake, ingesting it into their warehouse, and processing it repeatedly. It’s a classic trap: a system that is easy to create and hard to maintain.

The Data team might use the data appropriately, but this system leaves your Analytics Engineers, Data Analysts and Data Scientists heading towards a slippery slope. The opportunity to deploy needless dbt models won’t go away. And with that comes ever growing cloud bills. Let’s not forget that you’ve also created a security risk. 

PII and other sensitive information will land in your warehouse even when you don’t want it to. It’s one unfortunate side effect of this design.

We’ve all seen it happen.

Even the most cost-conscious teams slip up from time to time.

But we can side-step these issues by returning to an ETL pattern of old. By applying aggregations to our data lake and pushing that data to our warehouse.

And with some specific design patterns, we can build a lean pipeline that fits our current infrastructure.

Plus, we’ll get the data where it needs to be faster and cheaper.

Data Lake + Dagster + DuckDB

There are plenty of vendors who will copy your data lake into your warehouse for a fee. But, we’re not heading down that path. We’re keeping it lean. 

You’ll hear a lot of data engineers say to “push it down to the database”, meaning let the DB do the heavy lifting. We’re going to do something similar, but with lighter weight tooling. 

We can’t aggregate data directly in the data lake, because we don’t have access to a compute engine. 

We don’t want to push the aggregations into Snowflake. That requires we copy everything to Snowflake first, and would be far too costly. 

Instead, we’re going to aggregate partitions of data in-process using DuckDB, and dump that to a location accessible by Snowflake. 

Anything downstream of that (dbt models, BI reports, etc) are derived from our aggregated data. We’re using our pipeline to perform work we’d be doing anyway. 

In theory, you could forgo dbt altogether and treat DuckDB like a data modeling layer, but might be going too far. We still want some separation of concern between data available for use and data ready for consumption (reporting). 

We can break our path forward into a few steps:

1. Pre-processing and Aggregation

2. Staging and Loading

3. Reporting and Presentation 

We’re going to use Dagster to execute the first 2 steps above. If we opt for layering some lightweight dbt models on top of our data, we can orchestrate that work with Daster, too. But, the method you choose for that section of the pipeline doesn’t matter too much. 

We’re already in a good place with data capture - the Engineering team is dumping an event stream into s3. We don’t need to rely on Webhooks from vendor systems, and we’re capturing more data per event than we will  use for analytics purposes. With Kinesis, a new file lands in s3 approximately every 15 minutes. A typical payload looks like this: 

{
  "a_unique_identifier": 1234, 
  "another_id": "xyz-abc",
  "event_type": "signup",
  "occurred_at": "2024-03-01T15:00:00.141626",
  "source": "data-producer-app", 
  "raw": {"some": "nested", "json": "data"}
}

The files are timestamp partitioned JSON blobs, and there are a lot of them. 

But, before we do anything we need to handle two important requirements: 

1. Granularity at a daily grain

2. Removing PII

Daily granularity is easy enough - we need to run this pipeline at least once a day, and aggregate our output data on at least a per-day basis. 

We can fine-tune our controls of PII data as it’s getting processed by DuckDB. A vendored solution would be too costly, especially one that requires this level of configuration.

In years past, you might have opted for Pandas, or some generic CSV processing. But we can minimize a lot of boilerplate code by opting for DuckDB. This also helps us reduce our in-process memory footprint (more on this later). 

And, this whole thing is ephemeral - we don’t need to persist our DuckDB instance. We don’t need to maintain state in the pipeline, so we can more easily scale horizontally.

That also means we can re-run jobs with ease.

Our pipeline needs to read and write to s3, make it easy to convert one file format to another, and have a friendly API. DuckDB handles all these requirements with some readily available tooling. 

We’re primarily interacting with it via SQL, and it will work with limited domain knowledge. Not to mention we can drop it into Dagster without having to explicitly integrate the two tools. 

From there, we produce some parquet files and drop them into an s3 bucket. We wrap it up by having Dagster trigger a load from s3.

This might sound a bit abstract, but it’s conceptually pretty simple. There are a few moving pieces though, so let's look at some code.

DuckDB.build()

DuckDB is doing the majority of the work in this pipeline. Since Dagster is python based, we’re going to package up our DuckDB interface as a python object, as well. Dagster provides a native integration with DuckDB by way of Resources, but we don’t need that here. 

Our class has a .build() method that spits out a DuckDB instance loaded with a specific configuration.

Now we can read json, interact with s3, and handle memory spillage predictably.

Incremental Processing with Partitions

Since our data lake will be growing over time, we need to logically partition our source data. This is especially important given our memory constraints. 

We don’t have access to any metadata indicating the size of the file we’re about to ingest. We don’t control the data producing system. And since we’re processing gzipped CSVs, we could have a wide range of compressed files.

Incrementally processing our data at specific intervals is the easiest path forward. And it allows enough flexibility to tweak things as needed.

We can use Dagster’s Multipart Partitions to manage the partition keys we’ll use to filter, parse and process only a subset of files at a time. This is the basis for our incremental processing. 

We’re processing events from multiple producers. So, we’re factoring time and origin into our partitions. 

We’re going to use this definition in a few places so we’re only ever handling a subset of data at a time. We have some helper functions and utilities to make filtering the s3 files easier, but they are not that interesting. 

Within Dagster, two functions make up the vast majority of the pipeline so far - finding some files in s3 and handing them to DuckDB. 

Those functions are tidy, too. They find files we need to process and execute queries against them.

In one case, we cast JSON to Parquet. In another, we do a rollup.

With the hive partitioning and file type conversion, executing a daily rollup would be much more difficult.

By converting the JSON data to Parquet, we can now just point DuckDB to those files, aggregate them, and ship them off to another location in s3.

And, it can happen without having to give DuckDB a predefined schema.

Ship it off to Snowflake

We’re 80% of the way. 

We already did most of the heavy lifting in-transit with DuckDB and Dagster. The final step is loading our pre-processed data into Snowflake. 

We want to keep things relatively streamlined in Dagster and in Snowflake, so we’re defaulting to CREATE IF NOT EXISTS for a Parquet file format, Rollup Stage and Rollup Table. 

Dagster’s Resource pattern makes it easy to inject interfaces into your databases and cloud storage into your job context. 

Instead of managing all of our CREATE IF NOT EXISTS statements with inline sql, we’re going to set up a ConfigurableResource as a “public interface” into our data warehouse. This is how we’ll execute the commands we need to take the staged Parquet files, and ensure they land where we want in Snowflake. 

The last remaining piece? Copy our staged data into the rollup table using our SnowflakeResource, and we’re done. 

From here, anyone with a connection to Snowflake can make use of our newly minted rollups.

And backfills are easy, too, thanks to the partitions.

Quick Aside: If Money Were No Object

If we weren’t considering cost, Snowflake and BigQuery offer plenty of options for getting your data from where it lives to where you want it.  

Snowflake Dynamic Tables and Tasks work well for transferring files from s3 into your warehouse for additional processing. In BigQuery, the Data Transfer Service provides similar tooling. 

But, you’re still incurring storage and processing cost at multiple points. You’d still be copying the “raw” data in your warehouse where you’d have to do more transformation. Adding dbt into this mix quickly takes you from “data-driven” to “cost-center”. 

Implementing This Pattern in the Real World

Examples and walkthroughs are nice, but in the real world, there are always challenges. Here are some “gotchas” that came up when implementing this pattern for a real-world Marketing SaaS.

Your DuckDB configuration matters, and it depends on the platform you’re using

More specifically, your DuckDB Memory Limit and Temp Directory configurations matter. If you’re running this pipeline on an EC2 instance with comically large RAM, you won’t have to worry too much about memory. If you’re running it on a small instance, or through Dagster Cloud, you will have memory limits.

And those memory limits are important. 

Since our data is loaded as gzipped JSON and written as Parquet, we don’t know the initial memory requirement until the filetype changes. The collection of gzipped files may be small or large. Because of this, you need to be able to spill to your local disk since your RAM will not be enough for in-memory processing. We settled on 10GB of RAM since Dagster Cloud defaults to 16GBs. 

On high-volume days, we saw activity skyrocket. As a result, we needed to plan for the edge cases.

Just use Parquet

Speaking of JSON and Parquet, DuckDB will cooperate much better with Parquet files than Gzipped JSON. No surprise that an in-memory database will work better on files with attached metadata, so if you have the opportunity to update your Kinesis pipeline to write your event batches as Parquet, you should do it. 

Preserve a lower level grain in the pipeline when you can

Any experienced data engineer has been here. The requirements expect daily aggregations. But if you can process things on an hourly basis with minimal cost or performance impact, it’s probably worth doing so from the start. 

By presenting daily aggregations to the user, but keeping hourly rollups under the hood, you keep things flexible. You can change the granularity of reporting in the future, but you don't need to make major changes to do it.

Build once, share twice

We dump this data Snowflake in this scenario, but there’s no reason to stop there. Other systems (including the production application!) can use the aggregates that land in s3. Making use of the data you have does not just mean it gets shared via BI report.

Backfill configuration is trivial, but execution is not

I also happened to crash our Dagster Cloud deployment because of backfills. Pair intermittent high-volume periods with a large number of backfill jobs, and the result is poor UI performance. Unfortunately, I caused issues for other Dagster customers, too. 

Simply put, we were creating so many "JobRun" objects that the UI failed to load. And, other Dagster Cloud customers were seeing degraded performance as a result.

Dagster was quick to solve that issue for everyone. Props for moving fast.

But, this brings up 2 changes I'd make next time.

First, change the way DuckDB aggregates a single batch of data. Rather than running 1 job per hour, we could run one job per day , and write local Parquet files partitioned by hour. You maintain the same granularity, but significantly reduce the number of jobs tied to your Dagster instance.

Second, use a hybrid deployment rather than a strictly serverless design. At least this way, we could manage the RAM and CPU settings ourselves.

If you have thoughts or comments, give me a shout on Twitter or LinkedIn.

Brief, yet informative.

The last 4 months have been surprisingly eventful from a professional sense, and it continued all the way through the New Year.

In fact, it’s continued into the new year. I was not expecting that.

Here is a snapshot of my almost first calendar year working for myself and consulting in the data engineering space.

Rather than waxing poetic about life as a solopreneur, I’ll try to keep it pithy.

Projects

~35% Greenfield/ New

~55% Brownfield/ Existing

~10% Non-Development

I’m not surprised by this mix, but the way these project play out differ greatly.

Greenfield projects are seldom truly greenfield. Someone internally has run a POC, convinced management, or decided that XYZ thing needs to get done.

I expect a larger chunk of Brownfield projects in 2024 - companies have no shortage of problems to solve. And coming off the ZIRP-era heyday for data tools, I expect many companies will realize that there is plenty of low hanging fruit with regards to cost-optimization and performance improvements.

In case you didn’t know… you don’t need to throw more money at your problem to solve it. Sometimes, you just need someone who’s seen it before.

Customers

~20% Referrals

~20% Inbound

~20% Repeat

I can’t share all the details here for obvious reasons (read: contractual) . But of the wider range of customers I worked with this year, I’m very happy that 1) people within my network are willing to vouch for me, 2) things I say on the internet are taken seriously (by some) and 3) customers recognize I can help solve their problems.

You know what they say - the easiest customer to sell to is one who's already paid you.

50% of my clients are Venture-backed businesses, all providing some SaaS solution.

The other 50% were bootstrapped or spinout companies.

Timing

Shortest Duration Project: 3 Weeks

Longest Duration Project: 6 Months

Shortest Time to Close: 1 Day

Longest Time to Close: 4 Months

If I had to guess, the typical project duration was 3 months. Not bad.

I’ve learned I much prefer to come into an organization with a specific problem to solve over doing a “staff aug”-like engagement where I am embedded on a team.

Project based engagements still mean working closely with the existing team. But, the markers of success (and sense of urgency) when working as an outside expert are much more concrete than when you’re a hired-hand picking up To-Do’s from the Kanban board.

Money

Price Increases: 2

Profit Margin: 91% (Lol)

Yes, bootstrapped service businesses have low overhead. Yes, 91% is ridiculous.

There are only 2 options to scale this business - prices and people. While I’ve managed teams before, I am not confident the economics work when it comes to hiring additional hands. Sub-contracting, however, is a different story.

By the time I am officially one year into this ride, I will eclipse my previous full-time salary by a meaningful margin. I still have a couple months until I reach that milestone.

A More Personal Note

The biggest news this year wasn’t starting Purview Labs.

We got a dog - a rescue from a farm in South Texas. His name is Phil. He sits next to my desk when he wants pets, runs in circles when he’s excited, and sometimes looks goofy in photos.

I tend to keep my business persona separate from my personal one, at least when it comes to my online presence.

I don’t think that’s totally feasible anymore. Nor is it necessarily good for business.

I may be sharing a bit more about this side of life in 2024. Maybe.

Misc. Observations

• I work almost every day in some capacity. I don’t wear myself thin, but it’s part of the journey right now. I’m not sure this approach is for everyone, and I’m not sure this is something I want to do for the long-term.

• Long-term consulting work is still not the end goal. I find myself debating the merits of selling a product vs selling services.

• Both can be wildly successful and, if done right, complementary offers.

• I’ve expanded my network more in ~ 10 months of working for myself than most of my W-2 roles. Conversations are different when you come into them as your own brand rather than someone who does XYZ thing at Some-VC-Funded-Startup.

• Content works.

• Creating content is time consuming.

• Creating useful content even more so.

• Content about tools and technologies get significantly more readership and engagement than content about business. The data world has perpetual shiny object syndrome.

• Tools like ChatGPT are not going to replace engineers. But they do help flatten the learning curve of new things. Mastery is only ever achieved through practical application of knowledge, though.

• The difference between the Airflow and Dagster dev experience is night and day. I discounted Dagster for too long.

• I haven’t been Duck-pilled, but DuckDB has its place in specific workflows. It is not a magic bullet, but it is a useful open source technology.

• The broader data ecosystem is following a similar arc to that of the “internet marketer”. Lots of online course. Lots online personalities. It can be tiring to sort through the noise.

• I still don’t have a company logo or branding. And I’m not sure I need one

If you made it this far, give me a shout on Twitter or LinkedIn. One of my favorite parts of the last year has been talking to new people.

At first glance, open source software sounds like a utopian ideal - useful products packaged up for anyone to use. Of course, they require a base level of technical acumen, but if you’re already interested in using open source tooling, you probably have some chops.

Recently, there has been a strong push for open-core businesses - companies whose core offering comes in two flavors: a do-it-yourself version (open source) or a managed-for-you-version (hosted).

Is your team too small to afford a data engineer or someone else to run your infrastructure? Pay us for this hot new managed service and we’ll handle it for you. It’s that easy.

Never miss the data tool hype-train again!

There is a certain mystique about open source and open-core businesses. They appear to be exemplars of an one-for-all and all-for-one mindset where the people who build the product aren’t just shilling it for profit. They are here to help the average man or woman.

But that’s a naive and simplified view on something far more complex.

The boom in data tools over the last 5 - 10 years has made this exceedingly obvious. Never before has the Github Stars to VC-backed startup pipeline been stronger.

But, does the idea of open-core even make sense for businesses in today’s market? The answer depends on who you talk to.

Either way, let’s take a look at the concept of open-source and what it means in the world of data.

What’s Open Source?

Fundamentally, Open Source (OSS) is a means of releasing software that allows the end user to distribute, extend, or otherwise alter that software for their own use. It’s the backbone of software today (not just data-related tooling), and you’d be hard pressed to find an engineer who has not worked with a variety of open source tools. I’d venture to say finding such a person would be impossible.

Almost every popular programming language is open-source. In fact, every language has an eco-system dedicated to the distribution of these OSS packages:

javascript —> npm

python —> pip

rust —> cargo

java —> maven/ gradle

… there are many other languages and package managers, but you get the idea

Are you reading this post on a Firefox browser? That’s open source. You can introduce changes to the Firefox project if you really want to.

Chrome is not open source. But, a lot of the stuff under-the-hood, namely the Chromium web browser that powers all instances of Chrome, is an open-source project.

Confused? Welcome to the world of modern software. There’s layers to this game.

There is often a difference between the OSS version of a piece of software, and the commercially available version of it. The vast majority of Chrome’s features come from the open source Chromium browser, but there are some which are only supported by Chrome - not Chromium.

You could install Chromium instead, but you’ll lose some functionality. For instance, you’ll need to manually install every update.

It does get more complicated. Google offers Chrome under a free license; so, despite you not paying for the features they add on top of Chromium, it’s a closed-source project. For our purposes, we don’t need to focus on licensing, though it does have a meaningful impact at scale.

OSS typically follows this pattern - a free, community supported version of a product, and a commercialized cousin with some additional, often very useful, features. It’s like the current version of you and that other version of you that hits the gym, strictly eats whole foods, and doesn’t endlessly scroll social media before going to bed.

This works really well when multi-national mega-corps like Google are incubating the product and issuing free licenses for its use. But many of the OSS tools you’re likely to use in the data world fall under the category of open-core, where a managed version is available if you’re willing to pay the price.

They also operate at a very different scale of business.

The Economics of Open Source

To the average person in another industry, this model probably makes no sense. The incentives on both sides appear misaligned.

Build software that anyone can use.

Let them use it for free.

Pay your own people to maintain it.

Users can contribute to any open project available in the wild.

This model breaks the mold of being compensated for providing value, at least with regards to financial compensation. Often, resources are dedicated to projects that generate no revenue, or in many cases, average people are contributing to projects that are assets of a company with which they are in no way officially affiliated.

That second scenario is a really interesting positive side-effect of open-core businesses: your features are sometimes built by your own users. It is literally free labor and free product development. Quite amazing.

By design, OSS projects rely on contributions (of code) by the public. The alternative path is that the company spends its own resources to develop and maintain a software package that others can use for free.

Can you name an industry other than tech where this would work? Non-profits don’t count.

The closest corollary may be a loss-leader for e-commerce or retail, but even that is not an entirely apt comparison.

Don’t get me wrong - OSS is fundamental to modern software.

In the early days of OSS, this approach offered an alternative to lock-in with tools built by large tech companies. In the mid-1980s, Postgres - one of the most popular database options today - was born (in part) as a response to the marketing aggression and growing market dominance of Oracle and its relation database service.

There is something incredible about choosing to build a free, widely-available alternative to a technology just because you don’t like the incumbent.

But OSS has grown significantly since then.

As engineers and developers have collectively gravitated towards marketplaces like GitHub, the cycle time between creating some software and others finding that software has sped up. There is no doubt platforms like GitHub have contributed to the rise in OSS - more engineers can contribute to these project, and it’s easier than ever for OSS projects to be found.

And this has lead to a partial change in what OSS represents.

The MIT Tech Review and ThoughSpot suggest that OSS might be as much an effort in corporate branding as it is a popular business model. The modern OSS community has built an eco-system where many projects aspire to be spun-off into venture-backed businesses, especially in data.

In many cases, there is nothing wrong with this. The initial contributors set off to solve a problem, grew distribution, identified a market, and created a business.

At face value, that’s great. It’s the exact model that you find with digital marketing gurus and Twitter/ LinkedIn influencers: build an audience first.

But the reality of the “OSS to VC-backed company” pipeline might not be as sexy as it seems.

Open Source and Open Core

The open-core SaaS model is pretty simple:

1. Build a thing that solves a problem

2. Get people to use it

3. Get some portion of those users to pay you to manage the thing for them

That’s fundamentally the OSS to product-business pipeline. Maybe you throw some professional services in there, or you start pricing by usage (Everyone’s favorite pricing model). The formula is well-defined, with many software products fitting this mold.

Take a small slice of popular data tool vendors, and you’re likely to find more than a handful of open source/ open-core products - dbt, Dagster, Astronomer, Grafana, Snowplow, Starburst. There are plenty more.

Even big names like Snowflake use open open source technologies as fundamental components to their offer. Databricks’ own website states their Data Lakehouse product “is underpinned by widely adopted open source projects Apache Spark™, Delta Lake and MLflow”.

This is a new kind of leverage.

It’s a distribution network, with free labor, that becomes a business.

It’s the repackaging of well-distributed, widely available software into new products.

Sounds great, right?

All of the companies mentioned above are venture-backed, and all of them - save Snowflake - are currently privately held.

With the exception of Grafana, who is probably in a somewhat favorable position due to San Francisco Partners’ recent acquisition of NewRelic and the never-ending chatter about DataDog’s outrageous pricing model, you’d be hard-pressed to find a contender for a future public offering.

Yet the pipeline remains strong. Curiously so, if you ask me.

Point-Solutions and Systems-of-Record

OSS exists on a spectrum. On one end you have tiny single use packages - some might even call them toys. On the other, you have large scale enterprise technologies which historically have grown out of big tech companies.

The distinction may seem obvious with the examples above; they are clearly extremes. JavaScript’s LeftPad fiasco is internet lore at this point, while technologies like Hadoop and Spark are some of the most popular and impactful OSS data tools released in the past two decades. But it still illustrates the broader point - where the OSS lands on the spectrum matters.

Hadoop and Spark were incubated at Yahoo! and AMPLab of UC Berkeley, respectively. They are fundamental components of technology for thousands of businesses and technical organizations.

These two tools have spawned multiple commercially viable businesses, and have been co-opted by platform providers like AWS.

Will the typical OSS data tool available today reach this scale? It’s unlikely in my opinion, and that’s mainly a function of where there new tools exist on the spectrum of available OSS.

On one end, you have single-use-case tooling and utilities for “end users” (aka analysts and other reasonably technical roles). They solve specific needs, often tied to existing workflows within the business. On the other, you have databases, infrastructure-as-code, and system observability. These are wide-reaching and flexible tools which (in some cases) act as the system-of-record.

Tech businesses can survive without point solutions; they cannot function without systems-of-record.

Over the last 5-10 years and in the data space specifically, it seems as though there have been fewer lower-level technologies following the OSS-to-product-business formula than one-off utilities and point solutions packages. The exception to this statement is likely Voltron Data, from the team behind PyArrow (and Pandas, back in the day).

And even if that is not the case, there is no denying that point-solutions have been marketing to practitioners aggressively in an attempt to drum up FOMO. You can make a case that tools like dbt, Dagster, Prefect, and many others fall into this camp.

dbt, itself an point-solution on our OSS spectrum, illustrates another unique phenomenon for this market. While individuals may have gripes about its implementation details - namespacing of models, excessive YAML configurations, lack of governance - dbt Labs has been able to plug its own product gaps reasonably well with many of the surrounding tools in the ecosystem.

Is this a product moat?

You could make that case. Though a more realistic assessment is sort of. These complementary tools are, after all, separate businesses and separate concerns. And down the line, they’ll likely be fighting for mind-share and market-share.

It’s tough to say how defensible this positioning is over the long term. If your product encourages users to actively supplement its feature set with additional tooling, the market will figure out how to do this efficiently. It’s just one of the second order effects of the open-core model.

Second Order Effects

Some vendors - for instance, Clickhouse - have apparently reduced their OSS focus in favor of proprietary, closed-source feature development. Their CTO even weighed in publicly on the matter:

It's good to have a small, limited number of modifications exclusive to ClickHouse Cloud, but only those that do not compromise the features or operation in self-managed usages, but in the same way, are crucial and distinguishing for the Cloud.

This shouldn’t be surprising. There is a rich history of large tech platforms like AWS and others releasing managed offerings of these open source options. AWS, specifically, is comically guilty of taking OSS projects and releasing some managed version of it - Managed OpenSearch (ElasticSearch), Managed Airflow, Managed Flink, Managed Kafka (MKS), the list goes on. That is a only a small subset.

You may have gathered that AWS itself has a less than favorable relationship with the broader OSS community - they are arguably the tech giant with the least notable contributions to OSS packages. Or, at least those packages they can’t readily commercialize.

All this is to say the business of open source software is tricky, and the path to commercialization is often tied to a brand.

And the branding aspect of OSS cannot be understated, especially for products that exist closer to the point-solution end of our spectrum.

Apache Airflow - a very popular and equally hated orchestration tool - is now primarily maintained by employees of the Astronomer, a company whose primary offer is a SaaS for managed Airflow deployments. They are not the exception here.

This relationship between OSS tool and well-branded corporate sponsor has spawned an entirely new category of role in tech - Developer Relations .

Your Friends, DevRel

In other industries, it’s called Community Building or Community Marketing. At one point, they were known as “Evangelists”. But today, it seems tech companies prefer to use different terminology in order to hide the fact that they are, after all, selling another product.

I am sure that many businesses have seen both improved traction and improved brand awareness from DevRel efforts, but don’t be fooled. These roles are about driving the bottom line, and that rings especially true for your VC-backed point-solutions. Until the recent boom in open-core businesses, DevRel was primarily associated with only a few companies, many of whom are quite large - Apple, Google, NewRelic, Twilio.

Today, this function has become commonplace among small to mid-sized tech companies building developer tools. If you work in data, I guarantee you’ve come across some blog posts, YouTube videos, or Slack channels where the brand’s Developer Advocate or Head of Developer Experience is the person leading the charge.

Software Re-bundling

The OSS-induced spread of point solutions has lead to software sprawl at many companies. Pair that with the end of the ZIRP era, and it’s likely that many businesses are going to be shortening their list of accounts payable; for many, it’s already happening.

Businesses can push back against software sprawl by cutting licenses and opting to not renewal contracts. But, how do vendors, especially those who may have strong ties to these open-core tools, play this game?

Re-bundling.

This is a positive externality, at least with regards for the broader market. The average consumer (a business adopting OSS tooling) benefits from another organization packaging up various tools into one offer; it’s an efficiency gain for the end customer, and the re-bundler is able to play labor arbitrage by packaging up OSS tools.

It is an all-around fantastic model.

We are already seeing this happen in the data space with companies like Datacoves, Paradime.io, and Y42. There are others in the market, as well, including some service-heavy players like 5X Data.

Other single-platform tools like Mozart Data and Keebola also fit this mold.

Almost all of these businesses use dbt Core, the open-core version of dbt Labs’ product. You have to admire when independent businesses make open-core software a transparent component within their own offering.

These re-bundled offers are a win for everyone involved. The open-core business gains contributors to - and users of - their OSS tooling, the end customers get a friendlier experience thanks to another company handling all the tedious leg-work, and the re-bundlers fill a market need by reducing customer-facing complexity.

Community Alternatives

The final side-effect of open-core businesses is the creation of 100% open solution. And, it builds on the re-bundling/ re-packaging of tools like mentioned above.

Think of it like knowledge sharing; a callback to the true spirit of OSS.

Projects like MDS In a Box, a 100% open-source codebase for running the “Modern Data Stack”, embody this. They include all the features and components of an analytics toolkit, but remain accessible to everyone.

Similarly, we’ve seen releases of fully open-source playbooks on how to mimic dbt Cloud behavior without incurring additional cost related to a recent pricing change from dbt Labs’ hosted cloud service.

Are these options are robust as a fully-managed service? Hard to say, but that does not discount the fact that the option remains.

The very tooling that started as OSS and evolved into venture-backed open-core businesses is now being independently co-opted by other open-source projects. All to illustrate that you don’t need to pay for a managed service.

Perhaps we’ve come full circle.

It’s clear that as long as OSS tools are around, there will always be an alternative to the paid service.

So, Where Does This Leave Us?

The open source path is attractive in some cases, and for good reason. For personal and professional recognition, it’s hard to beat.

But as a means to build a business, that path is not as friendly.

Building a product business is already difficult. Transitioning from a open-source positioning appears to be even more difficult. Not only are you developing the technology, but you’re also managing the perception and opinion of the broader community - the very people helping you develop your product.

The current focus on GitHub Stars feels akin to the Dot-Com era of measuring eyeballs on a webpage. No mention of profit, of building sustainable revenue streams. Just get users; growth-at-all-costs.

Is open-core the best way to build a business in the data space?

Unlikely, when you consider the wide range of tools that fall into the world of “data”.

Is open-core the best way to leverage the broader community, both in terms of distribution and product development?

Yes.

But, it would be naive to think that the open-core path is a guaranteed win. If anything, past experience and recent market headwinds show that scale matters, product category matters, and the open-core model introduces competition in some very clever and unexpected ways.

Think I missed the mark, or want to share your thoughts?

Find me on LinkedIn or Twitter.

For some of you, that headline might not mean much. That’s ok, and this one might not be for you. I’m mixing in some technical posts every once and a while.

This is the first of the series.

If you’re reading this, I’m going to assume you know what DBT is. You don’t need to know what SQLMesh is, or have an opinion Semantic Data Fabric, but the more familiar with the world of data warehouses and data modeling, the better.

This is going to be a side-by-side comparison of 3 tools:

1. DBT (https://www.getdbt.com/)

2. Semantic Data Fabric (https://www.sdf.com/)

3. SQLMesh (https://sqlmesh.com/)

Some guidelines:

• I have no direct affiliation with any of these tools other than being a user

• We’re using the same raw data set for each tool

  • Specifically, some dumbed-down NFL Play By Play data available for download using either R or Python (Here or here)

  • You can see how we munge/ clean the raw data prior to loading to the DB here

• The schema of the Play-by-Play data can be found below:

  • Data Models

• All the code (and some other stuff) can be found in this repo:

  • NFL Pipeline Repo

• While we could easily add Snowflake or BigQuery into the mix for any of these tools, we’re going to use DuckDB so we don’t have to deal with cloud resources, associated costs, and additional configuration

• We’re keeping things pretty simple, and we’re doing that on purpose

  • Simple project setup, simple table definitions, simple outputs

  • While most of these tools offer non-SQL modeling options (ie. Python), we’re sticking to SQL for this walkthrough

• Simple also means we’re not testing/ comparing every feature of these tools. There is just too much to discuss across all 3 pieces of software

The tools outlined here are going to be evaluated based on a few categories:

• Computes and Utilities

• Speed of Development

• Quality of Life

There is no score. There is no winner. This is only a comparison that you can hopefully use to decide which tool fits best with your workflow.

The Setup

I exported all the play-by-play data to some Parquet files that are stored in S3; this data was slightly cleaned in a pre-processing step, that took the raw Play-By-Play data from the Python package linked above and stripped out malformed characters, and dropped some columns. You can find the simple pipeline in Github.

We’re using DuckDB on a local machine to access this data, and each of the three tools mentioned above are configured to use DuckDB to access the data.

You won’t find the DuckDB “instance” in the repo (it’s gitignored), but the pipeline will look for a local DB and create one if it doesn’t exist.

The (large) schema of the our raw data can be found in the repo; just know that each row of data represents a single play. We can move throughout the course of a single game to rewind or fast-forward to a certain game state.

In our can case, we’re using this play-by-play data to do some simple aggregations.

We’ll be replicating the same tables in each tool, using native patterns for that tool.

If you’re a Kimball die-hard, married to Data Vault, or an Inmon purist, you might be a bit annoyed at what you see. These data models aren’t 100% by-the-book; they’re only meant to illustrate how different tools use different patterns.

And, as a result, produce different developer experiences.

DBT

There are plenty of tutorials about setting up a DBT project, how to do X with DBT, and much more online. I don’t want to focus on that.

Instead, we’re going to talk about the experience of using DBT.

Compute and Utilities

The decision to use DuckDB was driven by DBT, even though the other tools support a “cost-less” local developer experience. I simply do not want to pay for every execution of this project. The other tools in this write up enable us to entirely bypass the compute costs of designing and tweaking a data warehouse. Unfortunately, DBT is not such a tool.

There is effectively no way to validate the outputs of your DBT models without running them. This means that in the case of a single model change - even as simple as casting a value to another type - results in an incremental cost.

You also cannot evaluate your tests without incurring some compute overhead.

The local development experience is solid enough - there are a lot of useful extensions in the ecosystem, baked-in utilities to construct a graph of model dependencies, and a well-documented project configuration. But, it comes at a price.

Take the relatively simple `base_games.sql` model below. We’re basically just joining play-by-play table to itself, and doing some minor aggregations to create a single record of the “final” outcome of the game.

with plays_home as (
    select
        game_id,
        home_team as team,
        count(distinct play_id) as total_plays
    from {{ ref('pbp_all')}}
    where posteam = home_team
    {{ group_by(2) }}

),
plays_away as (
    select
        game_id,
        away_team as team,
        count(distinct play_id) as total_plays
    from {{ ref('pbp_all')}}
    where posteam = away_team
    {{ group_by(2) }}

),
plays as (
    select
        game_id,
        plays_home.team and home_team,
        plays_home.total_plays as total_plays_home, 
        plays_away.team as away_team,
        plays_away.total_plays as total_plays_away
    from plays_away
    inner join plays_home
        plays_away.game_id = plays_home.game_id
        
)

select
    distinct game_id,
    week, 
    season_type,
    home_team,
    away_team,
    md5(plays.home_team) as home_team_masked, 
    md5(plays.away_team) as away_team_masked,
    total_away_score,
    total_home_score,
    total_plays_home,
    total_plays_away,
    case 
        when total_away_score > total_home_score then away_team
        else home_team 
    end as victor
from {{ ref('pbp_all') }}
inner join plays
    on pbp_all.game_id = plays.game_id

To validate the output of this model, we would run it on our local machine, which would connect to a database, potentially start up a server/warehouse, and execute that logic in our sandbox schema (as is DBT convention).

If we deploy this model to staging, run some tests on it, and then promote it to production, we’ve now incurred compute charges on at least 4 separate instances.

In our small `base_games` table, that is relatively insignificant, but when you multiply this across 100s or 1000s of production models, you introduce a significant multiplier effect.

Speed of Development

This is easily where DBT shines - once you know the patterns and the configuration framework, you can introduce new models as quickly as you can throw together some SQL.

But, this is a double-edge sword.

Anyone who works in the space knows this. It’s easy to introduce new models, but that often leads to messy projects, duplicate logic, and database objects that persist long after their utility (or accuracy) has expired.

I’ve designed, redesigned or contributed to well over a dozen DBT projects. You can make things happen quickly, but that flexibility is also what contributes to many of the anti-patterns that have become prevalent in the data ecosystem.

Quality of Life

There is a big ecosystem surrounding DBT - packages, SaaS vendors, blogs, whatever else.

When it comes to quickly covering ground in a warehouse project, you’ll probably be able to move very fast with this tool. That is of course assuming you have the appropriate supporting tools and permissions.

In fact, many of the quality-of-life components of DBT are enabled by surrounding tools. Especially the third-party packages.

So, while we all have a few ideas about what could be improved with DBT (name-spacing, for one), the quality of life when using it is quite high. The challenges come when building at scale - duplicate logic, inconsistent definitions, poor input data.

Some of those pieces are out of the reach of DBT entirely; others are not.

In my experience, anyone who says that DBT doesn’t enable them to move quickly is doing something wrong or blocked by organizational constraints.

A quick comment on lineage - DBT kind of shows lineage, but only at a high level. That is, table to table. The tool’s rise to popularity almost certainly helped spawn a suite of supplemental tools, or at least helped them also rise to prominence. This is feature - rather, lack thereof - is a drawback of DBT at scale.

SDF

SDF stands for Semantic Data Fabric, a new-comer product with roots at Meta. I need to caveat everything I mention below because SDF is still in beta. So, as of this post, some features are not yet available.

Of the tools in this post, this one feels most different. And there are some clear reasons why.

First off, it’s written in Rust, which means you install a binary. There is nothing to `pip install`, no package extensions to include, and no virtual environment to maintain.

This lends nicely to development speed, as you don’t have to battle with package dependencies, but that also means it might feel foreign to many less experienced engineers and analysts.

Python is undoubtedly the language of choice in business analytics (sorry, R), and you’re bound to find candidates who are much more familiar with Python than Rust. That said, you’re not writing any Rust code when you’re using SDF.

You just… write SQL.

No macros, no ref statements, no jinja templating (Again: They’re in Beta).

But, just like DBT, there is a fair amount of YAML-based configuration. I did not notice a significant difference in YAML when setting up DBT and SDF, but like most things in software, it depends.

Most of the YAML that SDF relies on is not generated manually - it’s created when you run SDF. In fact, for our small NFL project, the system generated YAML exceeds 6600 lines.

YAML - the workhorse of modern data tools.

Compute and Utilities

Here’s the thing - what SDF might lack in convenience utilities it makes up for in development experience.

You can “run” your production warehouse from your local machine, complete with propagating changes throughout downstream models without incurring a cent of compute cost.

This is different from both the DBT approach and the SQLMesh approach (see below), which require you to run a SQL command against your database or execute a query against it without generating a materialization, respectively.

SDF essentially keeps a cache of your current project definition in a hidden folder on your machine. Change a column or datatype? You can apply those changes to your local project state without incurring compute cost.

You might think of it as something like `terraform plan`, if you’re familiar with IAC patterns.

That DBT model I shared above? In SDF, `base_games.sql` it looks like this:

with plays_home as (
    select
        game_id,
        home_team,
        count(distinct play_id) as total_plays
    from pbp_all
    where posteam = home_team
    group by 1, 2

),
plays_away as (
    select
        game_id,
        away_team,
        count(distinct play_id) as total_plays
    from pbp_all
    where posteam = away_team
    group by 1, 2

),
plays as (
    select
        plays_home.game_id,
        plays_home.home_team,
        plays_home.total_plays as total_plays_home, 
        plays_away.away_team,
        plays_away.total_plays as total_plays_away
    from plays_away
    inner join plays_home
       on plays_away.game_id = plays_home.game_id
        
)

select
    distinct plays.game_id,
    week, 
    season_type,
    plays.home_team,
    plays.away_team,
    md5(plays.home_team) as home_team_masked, 
    md5(plays.away_team) as away_team_masked,
    total_away_score,
    total_home_score,
    total_plays_home,
    total_plays_away,
    case 
        when total_away_score > total_home_score then plays.away_team
        else plays.home_team 
    end as victor
from pbp_all
inner join plays
    on pbp_all.game_id = plays.game_id
;

Looks pretty familiar, right?

It’s just SQL without the `{{ ref() }}` and `{{ group_by() }}` macros.

Some of you might not like this - you lose the convenience utilities we all know with DBT. Others might look at this and declare that SQL-based data models are the way things are supposed to be. I’ll let you decide.

I did find myself pleasantly surprised by how clean working with SDF felt.

But, I also found myself mulling over the fact that if I were to do something complex in SDF, I would be writing much more code.

If I were to translate some DBT projects I’ve seen into SDF models, I know it would be a challenge.

However, that speaks more about the state of data modeling at many businesses today than it does SDF.

That thought alone might be just one reason why a tool like SDF is a strong choice for companies with strong domain-driven data models. After all, there is clearly a reason behind the naming choice of “semantic data fabric” (see: what is data fabric).

The SDF team did build in some nice utilities and tooling to help wrangle the inevitable sprawl of data models, though. `Classifiers` provide an SDF-native mechanism to apply categorical tagging to your columns and their child columns throughout your project.

By pairing `Classifiers` and `Function` definitions, you can reclassify data points from sensitive to non-sensitive as needed, and have that change propagate across all relevant models.

For example, we’re naively hashing the team names in order to simulate what would normally be done through some data masking mechanism. In our case, we’re just applying an md5 hash to a string.

So, something like `Dallas Cowboys` becomes `5a40b7cccc2a230895881416edda4622`.

--- workspace.sdf.yml
classifier:
  name: MASKER
  labels:
    - name: hash_me
    - name: it_was_hashed

--- src/functions.sdf.yml
function:
  name: md5
  reclassify:
    - from: MASKER.hash_me
      to: MASKER.it_was_hashed

--- base_games.sql
select
    distinct plays.game_id,
    week, 
    season_type,
    plays.home_team,
    plays.away_team,
    md5(plays.home_team) as home_team_masked, 
    md5(plays.away_team) as away_team_masked,
    <... more query stuff happening here ...>

And when running SDF, we see that we’ve now reclassified our data from something needing to be hashed (aka hash_me) to a clean value (now flagged as it_was_hashed).

This reclassification follows our home_team and away_team values to every model that this data is used.

Schema nfl_sdf.pub.base_game
+------------------+-----------+-------------+----------------------+
| column_name      | data_type | is_nullable | classifier           |
+------------------+-----------+-------------+----------------------+
| game_id          | varchar   | true        |                      |
| week             | int       | true        |                      |
| season_type      | varchar   | true        |                      |
| home_team        | varchar   | true        | MASKER.hash_me       |
| away_team        | varchar   | true        | MASKER.hash_me       |
| home_team_masked | varchar   | true        | MASKER.it_was_hashed |
| away_team_masked | varchar   | true        | MASKER.it_was_hashed |
| total_away_score | double    | true        |                      |
| total_home_score | double    | true        |                      |
| total_plays_home | bigint    | true        |                      |
| total_plays_away | bigint    | true        |                      |
| victor           | varchar   | true        | MASKER.hash_me       |
+------------------+-----------+-------------+----------------------+

Speed of Development

I’ve never set up an SDF project before. It’s only been publicly available for a few weeks.

And while setting this thing up had a few hiccups, it was nothing beyond what you’d expect for a tool in Beta. First attempts at standing up a new tool can take some getting used to. For what it’s worth, SQLMesh also took me for a little ride.

I will give the team credit, though; they responded to my slacks and clarified questions QUICKLY. Even to the point of sending me resources multiple times over.

It’s clear that this approach is different from DBT.

It’s perhaps less about the convenience methods and more about building fast, stable software. In SDF’s case, that means fast run times, building a native graph of dependencies without the need for Python and jinja wrappers, and evaluating changes against your local project cache.

Writing and running models with SDF gave me the sense I could feel the underlying Rust framework - a markedly different experience than working with a Python-based tool like DBT.

If you’re not familiar with using statically typed languages with strong type-checking, it might be the right time for you to try something new.

Quality of Life

There are some things about SDF that stand out as nice quality of life additions. While the feedback cycle of making changes locally and “running” them against your project is great, I found that small details like syntax errors that include references to your bad code are super useful (thanks, Rust compiler). This beats the nondescript errors you often find in many databases, handsdown.

error: Ambiguous column 'home_team' found, available are 'pbp_all.play_id', 'pbp_all.game_id', 'pbp_all.old_game_id', 'pbp_all.home_team', 'pbp_all.away_team', ...
  --> src/base/base_game.sql:42:4

While it might not yet have the robust ecosystem that DBT has developed, I get the impression that right now, your ability to use SDF effectively depends mostly on your ability to effectively model data using SQL.

In the future, I’d expect that Polars, PySpark, and other tools can be plugged into the mix. Similarly, some of that YAML configuration will likely be supplemented by Python or Rust.

Finally, when it comes to lineage, SDF excels. Simple as that. Between the query parsing and Classifiers, SDF makes it really easy to find where a column originates, and how it may be modified throughout your project.

I obviously only scratched the surface of SDF, but so far, I like what I’ve seen.

SQLMesh

I was intruiged when SQLMesh was released; to me, it was a meaningful step away from the DBT paradigm.

After having read through the docs and setting up the project, I initially found myself thinking of SQLMesh as a new tools build on top of DBT patterns.

But that’s not entirely correct.

Opinionated tools are generally a good thing (even if you don’t agree with their opinions). In the case of SQLMesh, there are no shortages of opinions:

• Pipeline definitions should be in SQL, rather than configured via YAML

  • Python is an option, too

• `SELECT` expressions are expected to follow a certain convention

  • unique column names (this is a given)

  • explicit types

  • explicit aliases are recommended

• Inline models configurations provide metadata, as well as control (orchestration)

• “Backfilling” is a core concept within the SQLMesh workflow

If you look at these requirements independent of other tools, they make a lot of sense. Even things like explicit types and explicit aliases are generally just good practice.

But when you put DBT and SQLMesh side-by-side, you begin to get the sense that these choices were very intentional.

YAML is everywhere in DBT; from docs to project configs to tests to packages. You can’t escape it. That’s not the case with SQLMesh, and it’s nice to see a modeling tool that is lighter on the markup.

This does mean properties that would typically be defined in YAML are now pushed down to individual models, though.

There are at least 14 different model-level configuration options for “full-refresh” models alone; incremental models add another 3 options. And, interestingly, those are independent of the `model_defaults` section in the project-level `config.yaml` or `config.py`file.

Of course, many of these properties are optional, but it does add overhead to the project configuration. For a larger-scale project, these are probably quite useful. For smaller deployments, it feels excessive.

Where some tools require that you write SQL, SQLMesh almost asks you to swap some out for Python. I can’t speak to the friendliness of the API, but this certainly fits the toolset of today’s data teams.

Compute and Utilities

Much like SDF, SQLMesh keeps a local cache of your project. But, it’s used different - specifically for Plans, or what you can think of as a project diff.

Environments and Plans add additional complexity to SQLMesh projects.

Under the hood, there is a robust state machine tracking changes of your models and their dependencies. The “state” of your project - aka Plan - gets applied to an Environment. This get increasingly complex as the scope of your changes is determined.

I am all for tracking changes and enabling teams to understand how their tables have changed overtime. But, SQLMesh seems to take this to the extreme:

Every time a model is changed as part of a plan, a new variant of this model gets created behind the scenes […]. In turn, each model variant's data is stored in a separate physical table.

Each model variant gets its own physical table while environments only contain references to these tables.

This sounds like an interesting feature that enables teams to rewind model changes or fast-forward to a designated model state. But, I’m curious how teams would use this in a real-life production setting outside CI/CD.

Given that databases like Snowflake enable Time-Travel - also something commonly used in CI/CD - I’m curious when one would be used over the other. If you’re not using Snowflake, then maybe a feature like this has more utility. Regardless, this part of SQLMesh adds significant complexity.

One other thing about SQLMesh that stands out is the Virtual Update feature; I initially equated it to something like React’s Virtual DOM - a representation of the DOM (Document Object Model) that is used to determine what needs to change on a webpage as a user interacts with it and changes state - but the docs indicate that “the process of promoting a change to production is reduced to reference swapping”. So, it’s not like the Virtual DOM at all.

In fact, it appears more like an internal mechanism for blue-green deployments.

The more I read through the docs for SQLMesh, the more it appeared that many features of the tool were inspired by existing paradigms in other tools - DBT, Airflow, various pipeline vendors.

Testing within SQLMesh is done through two mechanisms - Tests and Audits. SQLMesh Tests effectively act as unit tests, checking some logical output is produced as expected. Audits, however, allow you to attach assertions to your models; these are more akin to data quality checks. Much like DBT, these are SQL-based assertions, designed to assert that no rows are returned.

The distinction between the two components of the project make sense, and follow with the established DBT pattern of testing the data and custom logic tests.

SQLMesh’s implementation feels more robust, possibly because of the wide range of Audits available by default. These Audits seem to blend the default testing patterns of DBT with those available in packages like DBT-Expectations, Calogica’s testing toolkit inspired by Great Expectations.

It is unclear if there are other mechanisms to test Python-based models (beyond Python-native testing utilities, of course), given that Tests and Audits are SQL-based.

You can see that an Audit is included in the SQLMesh `base_game.sql` model:

MODEL (
  name base_sqlmesh.base_game,
  kind FULL,
  grain game_id,
  audits [assert_game_scores_positive],
);

with plays_home as (
    select
        game_id,
        home_team,
        count(distinct play_id) as total_plays
    from raw.pbp_all
    where posteam = home_team
    group by 1, 2

),
plays_away as (
    select
        game_id,
        away_team,
        count(distinct play_id) as total_plays
    from raw.pbp_all
    where posteam = away_team
    group by 1, 2

),
plays as (
    select
        plays_home.game_id,
        plays_home.home_team,
        plays_home.total_plays as total_plays_home, 
        plays_away.away_team,
        plays_away.total_plays as total_plays_away
    from plays_away
    inner join plays_home
       on plays_away.game_id = plays_home.game_id
        
)

select
    distinct plays.game_id,
    week, 
    season_type,
    plays.home_team,
    plays.away_team,
    md5(plays.home_team) as home_team_masked, 
    md5(plays.away_team) as away_team_masked,
    total_away_score,
    total_home_score,
    total_plays_home,
    total_plays_away,
    case 
        when total_away_score > total_home_score then plays.away_team
        else plays.home_team 
    end as victor
from raw.pbp_all
inner join plays
    on pbp_all.game_id = plays.game_id
;

And of course, our Audit `assert_game_scores_positive.sql`:

AUDIT (
  name assert_game_scores_positive,
);

SELECT *
FROM @this_model
WHERE total_away_score < 0 OR total_home_score < 0
;

Speed of Development

I found setting up my first working SQLMesh project to be as challenging as the SDF project. It wasn’t necessary complicated, just something with which I was not familiar.

SQLMesh’s Environments and Plans is a good example of this. Conceptually, they are not difficult to grok, but the way they interact with table changes, Virtual Updates, and “Backfilling” takes some getting used to.

SQLMesh lets you spin up models as quickly as an other tool, but making use of its advanced features requires following specific paradigms within the tool.

For many, these will be new concepts.

Quality of Life

One small, though impactful thing that SQLMesh provides is the ability to configure your project in different - declaratively with YAML or dynamically with Python. The `config.yml` file you need to set up your connections and projects specs can instead be a `config.py` file, enabling you to do more complex pathing logic, loading env vars, and more.

This project uses a `config.py` in order to access a DuckDB instance located at the project root. Using YAML would mean a hard-coded value, effectively making the project useless across computers.

SQLMesh feels like a lot of software you find in the Python-based data tool category - there is the quick-and-dirty way to get things working, and then there is the idiomatic approach.

The idiomatic approach always unlocks a new toolkit, but it can be challenging to embrace.

I encourage everyone to look at SQLMesh Macros as an example of something that may be very powerful when done right, but requires a foreign syntax to get there.

Final Considerations

DBT, SDF and SQLMesh solve the same problem in different ways.

You can’t say that one is better than the others, mostly because there are parts to each of them which support different workflows. Plus, there are many features to each tool, so it’s quite difficult to say you should only use X in some situations but only Y in others.

I stand by my earlier comment that using the idiomatic approach (read: tool-specific) will unlock new methods to solve your problem, but it’s not always straightforward to get to this point.

Especially when you or your team is accustomed to solving that problem in a specific way.

Perhaps a wildly under-appreciated fact is that these 3 tools can only take you so far. Speed of development and easy of use does not make up for data-producing systems that lack governance and well structured access patterns.

Data modeling as a practice existed longer before DBT, Cloud Warehouses, and compute credits. Some companies have apparently forgotten this, or worse yet, never knew this was the case.

At the end of the day, you’re not going to go wrong by using DBT, SDF or SQLMesh - in fact, choosing one is almost certainly better than not using any of them. Yes, there are tradeoffs in choosing one vs the others, but that’s the nature of software.

Some things should not factor into your decisions, mostly because they are table stakes:

• Speed of getting started

  • You can get a fully functional project running in well under an hour no matter which tool you’re using

• Support for specialized systems and platforms

  • If you’re using such a specialized toolset that none of these tools fit what you need, you’re probably looking into the wrong category of solutions altogether

• Deployments

  • The options for deploying data tools like these are virtually ENDLESS. They all going to support a paid deployment mechanism in one way or another, but prudent data teams know you don’t need to rely on a vendor for this. In fact, the most cost-conscious of them will purposely look elsewhere.

Instead, your decision should come down to:

• Incremental cost of delivery

• How well the tool fits into your existing systems and development workflow

• Whether you can support this tool within your workflow for a long time

Do they have 100% feature parity? No. But, chances are they will in short order. This category - who saw new entrants to the market even as I was writing this post - is on a path towards feature commoditization.

Over the next next 12-18 months, I expect most tools in the space to spend significant resources attempting to further differentiate. We’ll see how it plays out.

Think I missed the mark, or want to share you thoughts? Give me a shout on LinkedIn or Twitter.

I wasn’t going in completely blind (or without work lined up), but I wasn’t exactly battle-hardened, either.

I was moonlighting for 9 months as a consultant while working full-time before I decided to go off on my own. I had known I would do this for many years, but the time finally had arrived.

A spike in interest rates, a tightening of budgets, and an increased scrutiny on vendor spend seemed to be the right scenario for increased demand for outside expertise.

So far, it’s been an enjoyable ride working with a number of clients on data engineering and data architecture problems, but it’s only been about 6 months.

We don’t hear many people in the space talk about running a solo shop and the dynamics you deal with along the way. That might be because:

1. Many people do consulting as a side gig, not full-time

2. There are different approaches to service delivery within consulting/ contract work (ie. design-only, staff augmentation, short-term freelance projects, etc)

3. They haven’t figured out how to scale predictably, or they may not want to

I certainly have not figured out the predictability component, but I have noticed a few constraints in the market. You can find them below.

If you’re more interested in commentary on project scopes, scaling a solo business, and other topics related to running a consulting business, feel free to jump ahead.

Constraints on a one-person business

Constraint 1: You can only work a certain number of hours in a day.

For the sake of my sanity, and in order to maintain high quality output, it’s not feasible to burn the midnight oil and wake up at 5am to start the day. This puts a natural limit on my billable hours; this applies to everyone. Even the internet gurus.

You have limited leverage. Even with project-based pricing, you are essentially working on a hourly basis. Yes, there is money to be made when you know how to solve a problem, but consulting does not result in paid conversions while you’re sleeping or middle-of-the-night upgrades like you’d find at a self-service SaaS business.

There are way around this - content, communities, courses - but those would be by-and-large more distracting than beneficial for me right now.

Constraint 2: Your clients dictate the market rate of your services

This might be obvious, if you work with startups, you’re going to be a different price point than if you’re working with Fortune 500 organizations. I’ve been keeping up with Twitter, Reddit and other places online in order to stay as plugged in as is reasonable (and beneficial), and it shocks me that some people forget this.

Even the sharpest GTM expert would be hard-pressed selling services to pre-product-market fit companies - they just don’t have money to burn. But people still try to do just that. A fool’s errand if you ask me.

You need to go where the money is - figuratively, of course. This can take many forms, and applies to technology as much as any other specialty.

In the world of data and engineering, going where the money is might mean:

1. Expanding revenue channels for a customer by productizing some underutilized or underdeveloped component of their platform

2. Reducing cloud expenses

3. Working on their core product lines

That last bullet is probably the most important; many consultants in the space focus on the Modern Data Stack, or some flavor of it. In my opinion, that has become a commoditized offer.

If you’re not driving more revenue for clients or reducing their technology expenses, you are also just another expense. You’re fighting an uphill battle.

Set your sights on solving bigger problems.

Constraint 3: The projects dictate your earning potential

Startups may be willing to pay less on an hourly basis than Mega Corps, but the scale of the projects you work on matter. 300+ hour projects for a smaller client will always be more meaningful (both to the client AND to your bank account) than a 20 hour project at Mega Corp. Personally, I will take the longer-term, higher impact work over a single brand name.

But, if you are looking for short duration, high-volume freelance engagements, then the Mega Corp logo might be a prudent thing to add to your portfolio.

It depends on your expertise, your goals, and your ability to actually deliver what you say you’ll deliver.

Naturally, this ties directly with point 3 above. Projects that are core to your client’s product lines will almost always be the longer-term, higher impact work.

Just don’t forget - your ability to deliver on the projects you commit to is perhaps the most important piece of the puzzle.

Especially if you try to scale.

Thoughts on Scope, Scale, and Market Equilibrium

Many consultants present themselves in a coaching or advisory capacity - they provide suggestions, guidance, and general know-how related to a domain. The may sub-contract the hands-on work, or they may not be involved in that capacity at all.

In the world of data engineering, designing the system and building the system are very different skillsets.

But, until you reach a certain scale of business (we’ll talk about this in a bit), you need to be able to do both.

After a few years of running teams, I happen to like working on implementations (aka writing code) again. It’s nice to be hands-on solving problems once more. Will I keep doing this for a long time? Hard to say, but designing systems AND building them has been fun so far.

The real challenge is looming, though. That is how to surpass the point at which your obligations to your book of business exceed your individual capacity to deliver. All while maintaining your standard of quality, of course.

If the market roughly dictates the price a client is willing to pay, and you have reached your serviceable capacity, you could sub-contract. But, like we discussed in Constraint #2, your clients dictate the market rate of your services (at least to some degree). So, if you want to scale via sub-contracting without losing money, you need to play your cards right in terms of target client and pricing.

It takes a fair amount of trial-and-error to find a good balance between the “right” type of customer, the number of simultaneous customer projects, and your rate.

Balancing the growth of your business with your ability to service your clients is not difficult to do well, but it’s incredibly easy to screw up.

And, as a one-person business, screwing it up means there’s only one person to blame.

Miscellaneous thoughts that no one tells you about operating a business of one

• Closing projects is not that difficult if you know your area of expertise as well as you think you do. But, finding those projects can be.

• You need to deliver quick wins to buy breathing room on longer term projects (this applies to W-2 life just the same as solopreneur life, actually)

• Time to business value is arguably more important in a client-contractor relationship than for an employer-employee

• The admin work associated with a new client takes a meaningful amount of time, as do invoicing and billing

• You’ll think a lot about leverage, efficiency, and what you consider “good enough”

• “Niche-ing down” a technology-focused service business is challenging, arguably more so than a product business

• Writing/ creating content is incredibly time consuming when you have real deliverables to produce

If you’re building a single person consultancy, or just generally operate in the data and engineering space, I’d love to connect. Give me a shout on LinkedIn or Twitter.

Mike Evans, cofounder of Grubhub, released a memoir in 2022. It’s called Hangry. It’s an interesting story of a startup with a marketplace business model. He gleans over a lot of the dirty work building a business of that variety, but he repeatedly comes back to the concept of mutual embiggening.

Strangely comical term, but altogether sound business principle.

The idea is pretty straightforward - as a business, you grow as your customers grow. In the case of Grubhub, the more orders placed through their platform, the more sales restaurants are making, and in turn, the more revenue Grubhub generates from those restaurant. Everyone wins. Mutual embiggening.

This idea and general business model is not new. It exists in many verticals.

But it is a rare sight in the world of data.

What is the purpose of data?

Plenty of tech and non-tech companies alike have data teams of some flavor. Analytics, Machine Learning, Business Intelligence, Decision Science - they’re all flavors within the wide spectrum of "“data teams”.

But, what is the general purpose or mandate of a Data team?

Knee-jerk reaction might be “to drive better decision-making” or something filled with more meaningless jargon (ie. “to uncover actionable insights!”… please excuse my virtual eye roll).

Really, there are only 3 things a “data team” of some flavor are meant for:

1. Improve operations of the business through objectivity

2. Provide an alternative mechanism for monetization

3. Act as a signal to outsiders that this company “knows what they’re doing” even if they actually don’t

Those might seem too generic - where does the data science team fit in? Why are we ignoring data engineering?

In reality, those teams also fall into these categories. If you’re building machine learning pipelines, you’re either working to improve some existing inefficiency in the business, or your producing a monetizable feature that powers some user experience. No need to complicate things.

What do these flavors of data team work look like? #1 usually takes the form of dashboards, reports or some automation of an existing workflow. It might mean sending data from one tool to another, delivering some metrics on a regular schedule to the weekly Exec meeting, or just dumping a spreadsheet for an analyst. This is work that is meant to highlight improvements in other parts of the business.

The second is what you’ll hear referred to as a “data product”. This is often the goal for most teams, but getting to this point can be a challenge. Sometimes a company may think they’re on the path to building a data product only to fall into the trap of internal business intelligence reporting.

Or, they may have ambitions of monetizing their internal datasets only to be derailed by a lack of internal data management, poor data modeling, and plenty of unexpected work to make things usable.

Data products are the best opportunity for mutual embiggening, though. But we need to talk about billing before we can get there.

On Billing in Today’s Data Ecosystem

For the last 5-10 years, a wave of optimism has washed over the tech industry. Data has been viewed as a “more is better” commodity - a phenomenon that has produced some wildly popular products in the space (Snowflake, BiqQuery, Fivetran, Census, etc, etc… the list goes on).

FirstMark’s ML/AI/Data Landscape now lists an apparent 1400+ companies, and there are plenty missing. Each one of those businesses are battling for your budget.

And while there are plenty of tools that provide real value in specific scenarios, looking at their pricing pages and billing models should leave you with one significant takeaway - these companies by-and-large show no alignment between your usage and the value your derive from the tool.

Usage-based billing has taken over. OpenView Partners says it themselves - usage-based billing has nearly doubled within B2B SaaS over the last 5 years.

And guess what? SaaS companies in the data space are all B2B; let’s be real (Side note - you could make the case a product like Whoop is a data product, but that is the exception).

At a high level, this is fine. These companies are making money charging for a product that people want.

But, what’s shocking is how foreign Mike Evan’s mutual embiggening concept is to the data ecosystem. Businesses are designed to make money, but why has this entire industry turned away from a model that serves so many other businesses incredibly well? Call me crazy, but it can feel like a cash-grab at times.

The cost-value alignment of their pricing models in other industries are obvious. The businesses fortunate enough to fit the “mutual embiggening” paradigm seem to do pretty well, regardless of size - Grubhub, Postscript (a former employer of yours truly), and ConvertKit are good examples.

The prevalent theme? They make money as their customers make money - delivery orders, ecommerce, and newsletter subscriptions, respectively.

And this is the core problem.

Many of the companies on the Firstmark MAD Landscape are service providers or internal tools, not revenue drivers. And without that alignment, you can’t embrace mutual embiggening. Even data SaaS business who claim cost savings are still just an expense at the end of the day. The economics are tough.

The Chosen Few

There is another way to look at this mutual growth business model - it has to do with consumables.

We need to be careful here since it’s easy to conflate consumption with usage, but for this conversation, they have to be viewed differently. Consumption-based are valued and priced based on the outcomes they provide'; in our B2B SaaS scenario, that usually is tied to some other application’s making use of the product. Usage-based products provide an interface into a service of some sort, and charge based on the activity on the platform. There is no concept of “consumption” in another application context.

For instance, a software product that provides observability into errors in your running code (ala Grafana or DataDog) is a usage-based product. They are providing a service (introspection and observability), not a consumable.

On the other hand, a company like SafeGraph is a consumable - they provide data to be consumed and processed by their customers. Coincidentally, their website explicitly says they offer “No pay-per-use policies”.

SafeGraph also happens to embody a mutual embiggening paradigm. Their data sets offer clear revenue generation opportunities, which in turn drives more consumption. It’s a win-win.

The fact is many SaaS businesses within the Data space just don’t provide opportunities for revenue generation and monetization for their customers. The select few that do are data-as-a-product companies who offer datasets for enrichment purposes. Clearbit is another such example.

Data Teams are constantly pushed to move towards data products and monetization, but the reality is that the incentives often don’t align. Even within the broader data industry, pricing models overwhelmingly lean into a “we charge you for using our product” position rather than one of mutual benefit.

And until mutual embiggening becomes a more common practice for both vendors and data teams alike, the unspoken truth is that these vendors and teams will remain as costs centers.

Subscribe now

What is the cost of doing business?

There is no single answer to that question. For some, it’s paid ads, salaries paid for content marketing, or the dollars they shell out on SaaS tools. We tend to look at receipts as the “cost” of things - how much is a new Salesforce license, how long has this Ad Set been running on Facebook, is this new tool actually worth the $30,000 upfront annual contract price tag?

This view of “cost” is accurate from an accounting perspective, but it is too narrow a view when it comes to the productivity cost of a team.

Technical roles - engineers, data scientists, analysts or anyone that is expected to write code - have an inherently higher cost than many other roles. If you’re working in sales, perhaps your total compensation is up there, but you’re offsetting your own price tag by the deals you bring in. This is fairly straightforward.

What is not obvious, though, is that for some of these technical roles to even do their job, you’re looking at tens of thousands of dollars in overhead. And that is on the conservative side.

Here are the average base salaries for the most common data team roles by market, according to Indeed:

| Role           | New York, NY | San Francisco, CA | Austin, TX |
| -------------- | ------------ | ----------------- | ---------- |
| Data Analyst   | $84,376      | $95,171           | $75,288    |
| Data Scientist | $135,280     | $140,487          | $119,553   |
| Data Engineer  | $138,676     | $147,978          | $120,578   |

You don’t need to read into the fact that Analytics Engineer is not included; it’s too nascent of a title for Indeed to report.

These salaries are not obscene, but don’t forget they are only the base and skew lower than many actual salaries as these are averages among all level of experience. These roles are unlikely to get large cash bonuses, and far more likely to receive equity top-ups, so the total cash compensation is fairly accurate here.

But, let’s not forget - this is just to show up.

Tack on a minimum 10% for payroll and insurance, as well as whatever miscellaneous perks you choose, and we’re now looking at a sizable chunk of change.

A “standard” data team in today’s startup ecosystem is anywhere from 3-8% of headcount. For a Series A business of 30ish employees, we’re looking at roughly 1-2 people. By the time you’re at Series B or C, you’re definitely north of 150+ employees, and probably a team of around 5.

That’s a combined annual payroll of between $250,000 and $650,000 if we use a conservative $130,000/team member.

For a company who is lucky to be hitting $15M+ in revenue at Series A and $30M+ in revenue by Series C, these numbers are meaningful.

The cost of software, tools and systems

There is no escaping the price tag associated with salaried employees; those salaries are supposed to be a fair exchange for work performed, anyway. But these employees - charged with “making the business data-driven” or “uncovering insights” - need tools to do their jobs.

(Side note: Don’t use those phrases when hiring. They are complete corporate fluff. But, we digress)

Many of those tools are expensive by default, and are designed to make it easy for your teams to inadvertently drive your monthly cost higher. Go figure.

The Warehouse and Pipelines

Let’s focus on Snowflake here, since it is arguably the fastest growing option in the cloud warehouse space. Sure, Amazon’s Redshift has an edge in marketshare (22.16% vs 19.73% as of 2022), but talk to anyone in the data space and they’re likely moving off of Redshift to some newer option.

I work with Snowflake a lot. I like the tool. But I’ve found there are plenty of companies who have made the move to Snowflake who start off just fine, only to wake up one quarter and realize their spend has topped $100k ARR. In some cases I’ve seen Series C startups spending well north of $300k annually with minimal monetization efforts related to that infrastructure. There are plenty of examples of $1M+ ARR Snowflake users, as well. I know of multiple who went through layoffs in 2022.

But, Snowflake and other cloud warehouses are useless without data in them. And their utility decreases as the “freshness” of that data wanes. Stale data = less useful for the business. Coincidentally for Snowflake, stale data also means a lower ARR.

As a business you can do only a few things.

Option 1 - point some data engineers at the problem and let them build pipelines for you. This may take weeks, or it may take months depending on your maturity as a business and technology org. This is also an expensive undertaking, both in terms of absolute dollars and opportunity costs for the team.

Option 2 is a more sane choice - pay for more tools to make it easier to get data from your applications (mobile, web, whatever) into Snowflake. This, too, has a price, but it’s well worth the increase in your team’s velocity.

So, you adopt a pipeline vendor. Fivetran, Airbyte, or something else. They have their own costs, often obscured by the fact that they charge based on some bespoke “credit” methodology. Maybe 1 credit is 100MB of data, maybe it is 400K monthly updated rows, maybe it is something else altogether.

You start small. You want to transfer some of your production application data from Postgres to Snowflake. You decide to stick with only a handful of tables at first. Things are stable... $2500 per month seems reasonable right?

By this point we’ve passed the $300k mark and are inching closer to a $500k total cost for a single team to do it’s job. And that is on the conservative side with a small team.

I don’t care how much money you’ve raised. That is a lot of upfront investment for something that your exec team has been told will improve the business.

Will it, though? That depends.

The hidden drivers of cost

Before we get into the softer side of business/ data alignment and how that rears its ugly head in terms of cost, let’s talk about the way you go from a “reasonable” operating cost for data activities to something that scares your CFO.

It has three words - Usage-based Billing.

You might hear people call it consumption-based billing, too. This business model has become the de-facto standard for many, if not all, notable data vendors today. And it happens to be the biggest driver of unexpected costs for data teams and their finance counterparts.

The economics are pretty simple - you pay for what you use. The “gotcha”, however, is that the vendors commonly used by data teams, especially within the data warehouse and data pipeline categories, are designed to drive higher and higher usage.

It’s a genius business model for the vendor, but potential black hole for your cash as a customer. At first, the time-to-value is great - your team configures Fivetran and next thing you know, data shows up in your Snowflake instance.

You can talk your way through the initial usage bills since a lot of your initial data is loaded for free. But over time, scope creep sets in, and your once $500/ month usage now has passed $1000/month. Due to some inevitable misalignment with the business, a cool $2000+/month is not soon out of reach.

You started with a small scope of only “the essential” tables from production. Then the BizOps team asks for a slight twist on an existing analysis; this requires new tables. You set up Fivetran to transfer more tables. This increases your usage.

You eventually realize that the Engineering team didn’t include a nice updated_at field in some of the new tables you need, so you’re going to have to do full-table loads until they deploy a change.

Congratulations, you got got.

As a product-led motion, this is the goal. A customer signs up, see immediate value, and comes back for more. But what’s missing on the customer-side of the equation is knowledge. Too many teams - many of whom are young and inexperienced, or following rote “best practices” shared by other influential data vendors - wind up inflating their SaaS vendor bills because it’s so freaking easy to do.

They literally do not know how to make this workflow happen in other ways.

You almost can’t blame the teams doing the implementation here - they’re doing what they’re supposed to be doing. What they’ve been told to do.

But this is the pattern that leads from a reasonable spend for business and data analytics infrastructure to something that balloons significantly over time.

And this is business-as-usual for many companies today. Really makes you wonder.

The Compute Challenge

Ok, by this point it’s pretty clear how usage-based billing can get you in a hole you never anticipated, but we have only scratched the surface. Compute is the other part of the puzzle.

In Snowflake, you pay for storage (more often than not a relatively small part of the total bill) and for compute. Compute is basically just the time some server is running and executing the commands you tell it.

The Modern Data Stack lives off of compute credits. Every piece of the stack - from loading data, to processing that data, to running tests, to introducing new code via pull-request, to ad-hoc queries, to dashboards, to understanding lineage - requires compute.

Some of those activities incur higher compute costs than others, but the point remains. They all need a server to run some commands. And doing that costs money.

You might get pushback from your Looker account rep that they cache query results and try to minimize the need for querying the database, but the fact remains you’re still paying > $100k in platform fees and generating a larger cloud warehouse bill every time someone at your company hits refresh.

Don’t get me wrong. It’s not wrong to use Looker, or any of the tools in the categories I mentioned. But it’s important that you know the “hidden” costs associated with the tools your teams adopts.

The kicker to our earlier Fivetran debacle? You get hit with the one-two punch of a Fivetran bill for loading data, and Snowflake dings you for the compute to process the loading.

This is one thing you can categorize as “the cost of doing business”. There is no way around it.

The Alignment Problem

The final nail in the coffin for your warehouse and pipeline costs have little to do with the absolute cost of the technology, and more about the business problems dependent on their outputs.

Analytics as a function is intended to improve business operations in some regard. It doesn’t matter if you’re working on business intelligence questions, product analytics questions, or building a predictive model of some sort. If you’re working on some problem set within a business, and you call yourself a data or analytics person, your job is ultimately to help the business perform better.

But, that’s really hard to do.

Plenty of companies wind up over-indexing on solving problems that really don’t matter, or changing focus so rapidly that whatever effort went into one problem is lost on the next. This is one part of how you accumulate tech debt, but that’s another topic.

Misalignment between teams does not just look like frustrated coworkers and incorrect deliverables. It also translates to increased costs, and in our case, those costs are borne by the data team.

Remember that BizOps request from earlier? We can use any team in their place. Without alignment in terms of analytical focus, metric definitions, or even problem areas, your data team will be eating up compute credits in an effort to catch up with the needs of the rest of the org.

You won’t hear many teams or companies talk about this, but misalignment is an implicit driver of cost. It requires work and rework. And it often happens over and over.

So, what’s the alternative?

We’ve successfully hit a theoretical run rate over somewhere between $500K and $1M for a single team to do it’s job. Pretty wild.

Frankly, I don’t see how many companies can justify this kind of spend. Especially when many early data hires are young and inexperienced. Much of the cost incurred with today’s data stacks accumulates over time by way of design decisions and tooling selection.

An easy way to trim off a sizable chunk of your future warehouse and pipeline bills?

Bring in someone who’s done it before.

Not as a full-time hire; just to stand things up so your infrastructure cost does not scale faster than the value you’re generating from it.

We do it all the time at Purview Labs.