My last day at Dagster has wrapped up, and after two years of building and running developer marketing, I thought I’d take the time to answer some questions I often get about DevRel and developer marketing.

The timing feels right to share what I’ve learned.

It seems like DevRel has had a resurgence lately. I’ve seen more discussions on it on X, and many more job postings and recruiter calls trying to fill these roles. I sense that it’s related to the challenges of breaking through the noise within the latest AI/LLM craze that has been building over the past few years.

Google Search/SEO is dead; the cost of producing mediocre automated content is essentially zero, and so the question on many people’s minds is How do we reach developers?

The answer often defaults to “hire DevRel,” and often, I think people have not thought about it more deeply than that.

This post is a reflection on many things I’ve learned running DevRel, and hopefully can provide a framework and some guidance on answering questions I hear often.

DevRel at Dagster

When I first joined Dagster, it was to run the DevRel team, and one of the first things I quickly realized was that no one really knew what the goal of the DevRel team was. Everyone had an opinion on what DevRel should do at Dagster, but no one agreed on that definition.

The result was a lot of little tasks being completed, but without an overall story or narrative for it to fit under. This inevitably led to people wondering what the point of DevRel even was.

This is not unique to Dagster at all, and it’s something I’ve heard many people tell me, both founders and developer advocates have suffered some version of this story.

Over the next 2 years, I helped transform the team into one of the best teams I’ve ever had the pleasure of working with. You’d be hard-pressed to find a single person at Dagster who wouldn’t sing the praises of the devrel team, whether they were producing engaging content that brought new leads in, working directly with the community, bringing product feedback back to the R&D team, helping reduce friction points through excellent documentation and educational materials, or just being hilarious online.

One of the first things I did was have an open-ended discussion at the leadership level about what everyone believed the role of DevRel was at Dagster. The goal was to let the room do the talking and make it apparent to everyone that we don’t have alignment on the purpose of the team. I wrote a big list on a whiteboard of everything the team was doing and worked with the leadership team to identify the most impactful pieces of work on that list.

The items fit nicely in my framework for building DevRel teams, which I call the three pillars of DevRel: awareness, product/community, and education.

With alignment on what matters, I set out to build a team and put in place the processes and routines to help make sure we can execute at a high pace.

The transformation of the DevRel team was so visible that I was later asked to run the marketing team as well. This turned out to be a pivotal moment. The more I worked on marketing, the more I realized the fundamental truth of DevRel: that DevRel is just marketing.

Combining the DevRel and marketing teams was not something I was convinced would work, but after having done it, there was a real (dare I say?) synergy between the teams that, in hindsight, was the absolute right decision.

A strong DevRel team that understands the product and the consumer, paired with domain experts in marketing who can run campaigns, events, and other marketing functions, is the missing glue that can take a DevRel team from a nice-to-have to a core revenue-generating part of the business.

With that out of the way, let’s get into it.

What is Developer Relations?

DevRel, or developer relations, goes by many names: DevRel, developer advocates, developer marketing, evangelists, digital prophets, and so on.

I don’t think there’s universal agreement on what it all means, but here’s my general framework:

Developer Advocates: the people you hire who advocate on behalf of your product. Also sometimes called a DevRel Engineer.

Developer Relations: the overall practice of engaging with your developers along their developer journey. This could be a team, a department, or just an ephemeral idea within the organization. Developer Relations, to me, is a subset of developer marketing.

To understand what Developer Relations actually encompasses, it helps to visualize the full developer journey. I particularly like James Parton’s map from his book that describes this journey. To me, Developer Relations covers the full range of the journey from discovery to scaling.

Developer Marketing: a marketing function that is focused on marketing to developers. Some people believe this is just marketing. I do think developer marketing requires a unique set of skills that differentiates itself from ‘traditional marketing’. Some might argue this is just marketing. While Developer Relations is a large part of developer marketing, many associated functions within marketing don’t belong there: marketing ops, event logistics, partner programs, marketing campaigns, and so on.

The controversial part of this definition is that it presupposes that DevRel is, in essence, marketing. I believe the sooner you accept that, the easier it is for everyone.

In some ways, the name “DevRel” is marketing a marketing role to engineers who would never consider a marketing job.

The Three Pillars of DevRel

As I think of building DevRel teams, I think along three pillars and how to ensure that each is well-served. The pillars are awareness, product/ community, and education. There are other models as well: swyx, for example, has three.

The pillars are not individuals. You might find one hire does well in some but not others, and you might choose to ignore some to focus on another. I do think the pillars have a natural order of importance. If no one is aware of your product, then education isn’t going to be much of a concern. You need awareness before education can have meaningful impact.

Awareness

This is generally the first reason people hire developer advocates. They have a product for developers, and they want more people to know about it. Awareness, when done well, looks like engaging content created for developers about things they care about. Awareness, when done poorly, looks like astroturfed Reddit comments that get you banned from /r/somecategory.

I could write a whole book on awareness, but awareness for DevRel is not that different from general marketing awareness. The reason we hire DevRel for this is that you want people from the community who understand the product and the audience, who can find engaging ways to interact with them.

The goals of awareness are fairly straightforward: increase the number of people who know what your product is, and once they know of you, help them become aware of what your product does.

Don’t become preoccupied with a single channel. You need multi-channel content that reaches developers where they are. You must first understand your audience and who they actually are. What do they do for fun? What problems do they have at work? Where do they look up information? Marketing and Positioning 101 stuff.

At Dagster, our audience was on Reddit, X, and LinkedIn. They were also at conferences and meetups. So that’s where we showed up.

Don’t make the mistake of thinking you can create awareness by talking about how great you are. No one cares. Show up in ways that show how helpful you are. Create content no one else is creating. Everyone is creating the easy stuff. “What is a database?” Who cares. “I spent 10 hours testing the top three database vendors, here’s my deep dive”. Hell yes. More of that, please.

Create content that goes ahead of your product. What do people care about before they learn about what you do? At Dagster, data engineers decide to buy Dagster only after they’ve decided to build a data platform. So we created an ebook about building data platforms. It became one of our top-performing assets. Why was it successful? Because it wasn’t easy to do.

We printed it and brought it to conferences, and it flew off the shelves. Colton, who co-authored the book, even did a book signing at one of our conferences. You would not believe the lineup.

You should almost always start with an awareness-focused hire, and they should be spending half their time showing people how they can solve problems with your product, and the other half creating content about problems people face before using your product.

Product & Community

I group product and community because they are so intertwined. Product is all about contributing to the development of the product, whether through deep and thoughtful feedback, or through actual code or integrations. Community is all about staying close to your users, whether free or paid, to maintain a pulse on what the broader community vibes are, and feeding this information regularly back to both R&D and leadership.

This impactful type of work of giving frequent and in-depth feedback on the product, means your developer advocates need to be able to understand the product and be given enough time to use it in their day-to-day job.

A great exercise for a new hire is to have them run through onboarding and create a friction log of everywhere they got stuck. Chances are, your engineers have forgotten how difficult it is to use the product, given how intimately familiar they are with the code. Odds are they never need to pull up the docs to understand what feature X is or how to do Y.

This is why it’s so critical to hire DevRel from the community. I’ll talk more about finding DevRel later on in this post.

But product is not just about feedback; great DevRel hires go deeper. They contribute to the product itself. Especially with AI assistance now, the ability for developer advocates to work like an R&D team means added capacity to make the product better. One of the highest impact things our team had done at Dagster was create integrations that were requested by the community: you can see our impact on things like the dagster-sling library or Airlift.

Airlift is a particularly great example. I built a throwaway proof-of-concept of being able to peer into an Airflow instance from Dagster, and the POC was so compelling that an engineer took it and made it into the powerful integration it is today.

Beyond integrations, DevRel teams can also create additional tooling that helps solve common user pain points. This is often work that an engineering team can’t justify, but can be a great way for a developer advocate to stay current on their technical skills, while also contributing back to the product and community. A win-win.

Education

Education is the final pillar, and it requires a specific mindset and persona that is hard to find but life-changing when you do. Often, this person might be a technical writer, or maybe they create online educational courses. In our case, we have both.

I cannot over-emphasize the importance of good documentation. It is how developers learn about what your product actually does. We developers are trained to skip the marketing headlines and go straight to the docs to see what this product actually looks like, and what it can do. Anytime I go to a product’s website and view the docs and see mostly unfinished content, placeholders, and boilerplate, I just know they’re bleeding traffic.

Your engineers are not good at writing docs. They don’t want to do it, and they don’t know how. They will write docs if you tell them to, but they do not think about the overall information architecture that makes a complex product easier to understand.

Hire a good technical writer.

If your product is complex, documentation may not be enough. You may also need structured online courses, cookbooks, how-to guides, code examples and snippets, tutorials, example projects, and more. Keeping all of these up-to-date and deprecating where possible is a full-time job. Hire for it.

Should I hire a Developer Advocate?

I’m so glad you asked. Most don’t. You should start by asking why you want to hire one. It’s the same argument Claire Carroll makes in her post “How to build a community’ so just read that and replace community with DevRel and you’ll get the same experience.

You should really know why you are hiring for this role, what you want them to accomplish, and how you will enable them to be successful. That is your job.

I’ve seen way too many cases of someone hiring a junior person to ‘do devrel’; they get very little guidance on what that means, and before you know it, the person is let go, and devrel is deemed a failure.

Here are some great questions to ask yourself first:

• Is my product ready for a DevRel hire? Do I already have some product market fit?

• Will I hire a senior person or a junior person? If I hire a senior person, will I give them clear guidance on what I want them to accomplish? If I hire a junior person, I am prepared to meet with them regularly to help guide them on what they should be working on.

• What part of the business needs the most help? Do we need more people to know about our product? Or do we have a lot of complaints about how complicated our product is to use?

• Am I willing to invest in this for the long term? Or am I going to hire one person to do everything, burn them out, and wonder why they quit after six months?

Where do I hire DevRel?

Assuming you decided you want to hire one, how do you go about finding someone to hire?

The number one advice I give here is to avoid looking for career devrel people. It is so much better to hire people in your actual developer community who have been doing the actual work of developers, but that show the signs of a promising devrel talent. They may share their projects online, they might have created great blog posts, they may just be really funny on X. It’s so much better to pull from the community than to look for someone with 5 years of DevRel experience.

I say this as someone with several years of DevRel experience, too. Some red flags to look for:

• They spend most of their public persona talking about themselves, and not the product or problem.

• They can’t code their way out of a cardboard box.

• They don’t come from the same domain as your product. Don’t hire someone who’s been doing front-end devrel for your security product.

Some green flags to look for:

• A few blog posts that go deep on a topic

• A history of talks, presentations, and other public speaking roles

• GitHub profile with actual projects that pass the sniff test, not just random forks.

• Named Pedram

Where do you find these magical devrel people? Don’t go looking in “DevRel” communities, or lists of Top 100 DevRel of 2025. Look in your community. Find users of your product or users of your competitor’s product. Find a blog post someone wrote that you really like.

How do you convince them to join? Reach out to them, personally. Tell them who you are, how you found them, and if they’d consider working for you. My best hires were from directly reaching out to candidates with promising potential.

What does success look like? Why does DevRel get a bad rap? Why is it so hard?

These are all related questions, and I think they boil down to a common theme. It is really hard to measure DevRel. In fact, before combining the DevRel and Marketing teams, the only good measurement I had was vibes.

There were all sorts of metrics on productivity, and I even maintained a Notion database of everything we did, from blog posts to webinars to YouTube videos. But I try not to focus too much on engagement metrics. 100 highly qualified readers of a post can be 10,000 low-quality engagement bait readers.

However, once we brought marketing and devrel together under one roof, it did become easier to measure the effectiveness of the team through marketing campaigns.

Not everything DevRel does falls under a marketing campaign, but many things do: webinars bring in leads that drive pipeline, social posts drive awareness and engagement that help us build lead lists, and new integrations bring new users who previously weren’t qualified.

It’s still not a perfect measure. I don’t think you can measure the impact on the pipeline of your docs, for instance.

I think the measurement questions are largely a red herring. The reason I think DevRel teams fall apart isn’t because they can’t measure their success through metrics, but because they haven’t defined the goals of the team up front.

If DevRel fits into the broader GTM strategy, and there’s clear alignment on how DevRel supports the overall company’s goals, then the only question is whether or not the team is executing well. You don’t measure engineers by lines of code written, but by how much the CEO loves the engineering org. I think the same can be said of DevRel. Does everyone at the company love working with them? Could they imagine not having them around?

A good DevRel team feels indispensable. A bad DevRel team fights over attribution of leads on a blog post. The problem is usually not the metrics, but something deeper.

Proving the Value of the Investment

Mehdi asks: What I realized is that devrel takes time to build. Even if you have the right people. It’s hard to get short-term/mid-term results. How do you convince higher-ups that it’s worth the investment?

I think of this the same way I think of joining a new company or running a marketing team. You have to start with quick wins to build trust, and only then will you earn the right to work on things that take time to build.

The good news is that it’s not that hard to get quick wins within DevRel. Something as simple as hosting a biweekly/monthly webinar, writing a newsletter, and getting a few hundred subscribers, creating a 3-part YouTube series on the product, are all short-term, achievable tasks that can demonstrate your ability to perform.

Once you’ve built that trust, it’s about selling the story of your big idea. I often talk about how, in the age of AI, anyone can produce something easily. Undifferentiated content might as well be no content.

Developers want deeply technical, well-produced, engaging content. If your goal is to bring in a new audience, you need big bets to hit that next step function of success. Incremental improvements alone won’t get you there.

My advice is to plan out a quarterly roadmap, reserve 35-65% of the time for a big-bet project, and the rest for smaller quick wins throughout the quarter. Get alignment up front, and try to address concerns ahead of time. Three weeks in, when the exec team forgets, remind them that you’re cooking something that’s going to be great. Find a way to funnel new requests so that they feel heard without thinking every request gets prioritized ahead of what’s in flight.

When 3.5 came out, I remember everyone would send anyone who would listen a poem, a rap, or some other trite garbage written by a clanker. While that novelty wore off quickly, what hasn’t changed is that people are still sharing much more mundane chat’s with each other.

I am sure we have all been a victim of the emotional burden of being sent someone else’s research, analysis, discussion, or meandering thought. I know every time I get one of these, I can immediately feel my blood boil, and I’ve narrowed down the cause.

The cost of thinking has been offloaded to the consumer, not the producer.

Let’s say you had some project at work. It used to be that you would do your analysis and research, capture your thoughts, synthesize information, and then send that work out for feedback. But more and more, people are simply throwing the problem and data into a model, and sending the model’s results (either via a link, or just raw copy and paste) to others. The idea is that it should be their job to synthesize information.

It’s the offloading of emotional labor to the person who is receiving information, not the person producing it. And with AI, the cost of doing more things has fallen to essentially zero. It’s my belief that things having some cost is actually a feature of a system, not a bug. It’s easy to come up with ideas, it’s doing something about that is actually hard.

Everyone wants a piece of the pie. Nobody wants to bake.

People have been primed to be rewarded for accomplishing what feels like the task: getting analysis complete. But the real hard work, the work where value is actually created, was never done.

The smartest intern

In many ways, the very nature of how we build and train AI systems is to create the worlds smartest intern. There is no experience in these models, the data consists of the entirety of written data available to it. But when we think of why we hire experienced engineers, product managers, and staff it’s because we know they’ve learned through experience what could not be learned in books.

What we are moving toward though, is a new playing field where people rely not on their experience and judgement, but on the average next token, to drive decisions. I think in this environment, more than ever, the value of good taste will rise. It takes a certain annoying type of person, one with good taste and the confidence to assert it, to push back against AI slop dominated work. The forces are often against them, as AI-produced work feels fast, feels important, feels useful.

And this is not to say AI is not a useful assistant in day-to-day knowledge work. I use it all the time in brainstorming, outlining, reviewing, and essentially acting as a much better search engine.

But what I don’t do is take AI-output and present it to others as if I had something worth sharing.

Even me, the vibe-coding lover, who was one-shotting with Claude, find myself frequently deleting more code and writing it myself. I created applications that almost work but are impossible to debug or understand. Largely because the way they were structured doesn’t make sense to me, even if they do make sense to a clanker.

Who knows, I could be wrong. Maybe the future is offloading all our thinking to the robots, and I’m the one that is going to be left behind.

I know some of you believe that AI will come for your jobs and replace you, and while I don’t share that view, I can see how, if you extrapolate the progress we’ve made over the past twenty years, it seems inescapable.

I am not great at that type of long-term vision, but I can see what might be around the corner in the short term. I begged for dbt but better a few years back, and then dbt acquired SDF. I wrote about the rise of Data Platform Engineering and how it requires a framework approach to enable teams, and then the team at Dagster shipped components.

My barometer for the future is loosely based on what I need today that I don’t see great solutions for, rather than extrapolating a world fundamentally different from what we have today. In that vein, I’m going to achieve progress through more complaints.

It’s the interface, stupid

First, I’ve seen firsthand the potential of LLMs to fill in the gaps of lower-priority work that never seems to get done. Work that often required a little technical and domain expertise, but not a lot. Work that was just annoying enough to defer until later, not important enough to prioritize strategically, but still valuable in its own right.

One of the most used things I’ve built is exceedingly simple: Help our sellers self-serve collateral for their sales calls using ChatGPT’s GPTs feature. I used to think the GPTs feature was a hokey thing with no real value, but I’ve realized that its superpower is its extremely simple and easy-to-use interface.

A chat interface in a web app remains the easiest way for less technical people to interact with AI tooling. Even Slack, which seems like a great way to chat, does not make a great interface for LLMs. The responses are too long, and the Canvas feature in ChatGPT and Claude is a necessity. Of course, you can build your own chatbot, or use Slack, or a terminal, or API calls, or a sidebar in your code editor, or create agents with Aider, or a million other possibilities, but ChatGPT’s interface remains the simplest way to work with an LLM.

Context is king

The interface is half the equation. The other missing piece is providing relevant and curated context. ChatGPT’s GPTs feature fails miserably here, and very few tools I’ve used so far provide a good way to manage this.

For LLM adoption to really take root in organizations, people like me will try to build tooling for others. I’ve found that I need control over the context I feed into these tools for there to be any chance of success.

One of the first things I’ve built was our AskAI Slack bot, which Scout powers. It’s been a powerful tool for helping our community self-serve by feeding an otherwise useless LLM relevant context that completely changes how it operates.

Using a plain LLM to ask it a question about Dagster inevitably leads to wrong answers, hallucinations, and deprecated APIs. Feeding that same LLM our docs, GitHub Issues, and Discussions completely transforms it into a valid and sometimes correct assistant.

In this specific case, the context problem is fairly straightforward: scrape the website daily, run a daily job that fetches issues and discussions updated on the last day, and update the document store in Scout. Feed the 10 most relevant documents as context to the LLM and call it a day.

Where this model doesn’t work as well is the rest of the knowledge created at companies. I have a positioning guide in Notion that I regularly update, we have case studies in a Google Drive folder, our website has the latest information on pricing and features, and there’s a Google Slides presentation someone once shared that I can never find that has our public-facing roadmap. And right next to all that helpful information is hoards of information I don’t want anything to do with.

I can upload PDFs to GPTs but this means I need to constantly remember to update them as I update our documentation. Apps like Slack and Notion and Zoom seem to promise to solve this, but most everything-apps end up being everything-sucks-apps.

Curating this knowledge meaningfully with tooling that can help me easily create interfaces to an LLM that everyone at a company can interact with seems simple, but remains unsolved.

The problem remains the N integrations that need to be built. I talked about documents and emails, but there’s also a repository of knowledge in systems of truth.

Imagine a sales rep wants to craft an email to a prospect who recently attended our event. I’d want to give that rep access to not only our differentiated messaging and positioning, but also context from Salesforce and Hubspot on the account, the campaigns and events, previous conversations, recent activity, and other internal knowledge. I might also want to run an enrichment against Common Room and Clay to understand better how this person interacted with us on GitHub, Slack, and LinkedIn, or to enrich their profile with relevant details from their company’s website or public filings.

Right now, I am stitching together a half-dozen pieces of tooling to make this work, and the interfaces I have are fairly poor. ChatGPT’s GPTs don’t work well with this model, so I am forced to use Slack bots and duct tape. I don’t want to build this from scratch, because every use case is slightly different.

I am sure that eventually, we’ll see AI workflow cloud platforms that bridge this gap. And when that happens, I’ll be sure to take credit for their success.

The better I was at being right, the better I was at my job. Many people can get stuck in this, and I’ve seen too often people frustrated with the fact that simply being right is not sufficient to propel organizational change.

As you progress in your role, you will be responsible for a greater impact on a business with ever-increasing complexity. So today, I want to explore all the other aspects of your career that are important to a business.

First, note how I speak about career progression. Often, we talk about career progression in a vacuum: “What do I have to do to get promoted?” What I find helpful, instead, is to frame your goals alongside the goals of the organization. Instead of asking how to get the next promotion, ask yourself, “Why would an organization promote me?”

For one, promotion can be a means of retaining top talent. More often, promotion and growth are how an organization recognizes your ability to have a greater impact on the business, so you earn a title and salary commensurate with that increased impact.

While being right does have some marginal impact on a business—it is important for a business that the forecasting of models be accurate and that the systems that are built be resilient—these are not the primary goals of an organization.

As organizations grow in complexity, the needs of the organization aren’t that people be right. If you hire well, many people may be right about the same thing. The hard problem of a business is affecting change, and the bigger the organization, the harder it is to affect that change.

This—being effective—is what separates a senior engineer from a lead or staff. It is the reason that being a lead, a staff, or a manager is hard. It is hard to affect change in organizations. You are fighting competing interests and priorities, varying levels of optimism and engagement, difficult personalities and entitlements. Even something as fundamental as communication is hard and only getting harder with remote and hybrid work.

I’ve seen too often many people who all agree that a particular system needs an upgrade. Being right about that is not a fundamentally useful feature for the system. Agreement about some fact does not result in a prioritization exercise; it does not communicate the value of that fact in terms of business outcomes: are we losing customers, are we not able to sign new customers, are we losing revenue, are we losing customer goodwill?

It takes a lot to affect change. First, you must understand the business objectives and how a particular change will help drive better business outcomes. Second, you must be able to navigate resourcing and prioritization. There are a million potential projects. What makes this the right one to do? Third, and most importantly, you must be accountable and own the change.

Ownership means many things to many people. I have seen a naive view of ownership that believes ownership means deciding who gets to do what within a particular domain. This is the least interesting version of ownership.

Actual ownership is a double-edged sword. I warn everyone who asks for ownership and accountability to be careful what they wish for. Ownership means being accountable for the success and failure of what you own. It means you set goals that are measured by metrics, delivered through milestones, and regularly reviewed and reported on.

Ownership does not necessarily mean you are writing the code or changing the system. You can own a project without deploying a single line of code. But ownership means refusing to allow the ‘that’s not mine’ line of thinking to affect your responsibility to see things through. It means committing to a change and seeing it through.

If this sounds like a lot of work, that’s because it is. This is where I’ve seen frustration, especially from engineers, who have seen their ability to be correct about something as the thing that drives their success until it isn’t. If being correct was all that mattered, the world would be ruled by pedants. Instead, their frustration boils over into arguments about a workplace being “political,” which is the naive way of admitting you lack interpersonal skills.

You build political capital by being right, but you also build trust and influence. These are harder to measure, but it helps to think of who people want to work with: kind, empathetic people are far more fun to work with, especially if they are ruthlessly effective.

Earning trust is slowly won and quickly lost but can often be recovered. It means delivering on your commitments. It means having good taste, which is the only way (other than good vibes) that I can describe the intangible character that makes some people’s decisions better than others. It is not something I have ever been able to teach, but I can always find it in those who have it.

If I were to summarize all of this, I would say that it doesn’t hurt to be right, but it is critical that you be effective.

Of course, this entire essay is predicated on the fact that you are working at a somewhat functional organization. There are many dysfunctional organizations out there where the only thing that matters is saying yes to leadership, being right be damned. If you work there, you will need therapy more than my advice.

Soon, companies realized that a team of 20 Data Scientists couldn’t be effective with access to good data, and the role of the Data Engineer was brought to the forefront. Data Engineers would be responsible for the ingestion and transformation of data and the platform that enables data scientists, while the data scientists would become consumers of that data.

While on paper this seemed like a great division of labor, engineers famously do not want to write ETL pipelines. So far back as 2016 Jeff Magnusson at Stitch Fix suggested that engineers build platforms, services, and frameworks and not ETL pipelines.

This largely did not happen.

Back in 2016, Hadoop clusters were still status quo. Spark and the JVM were the best we had. Scala was cool. What soon changed wasn’t that data scientists and data engineers ended up listening to Jeff, but a new breed of software was born. Cloud Data Warehouses were just becoming a natural replacement for the existing data systems. Instead of requiring a team of dedicated infrastructure engineers to scale your data requirements, you just needed a dedicated credit card.

Instead of tasking data scientists with writing ETL pipelines, we gave that task to Fivetran, Stitch and other cloud providers. The birth of the Modern Data Stack was just around the corner with Snowflake’s IPO in 2020.

Meanwhile, Data Scientists sat unhappy that they were using their PhDs to create dashboards. Consultants were picking up the slack until a little company called Fishtown Analytics open-sourced a tool they were using for transforming data in the warehouse. dbt was born and exploded in popularity, giving rise to the Analytics Engineer role. This role supplanted the Data Scientist, and soon the Data Scientists were freed of the chains of answering Yet Another Stakeholder Question (YASQ) and were able to move on to more important work, like creating flashcards, founding startups, and getting into fights on Twitter.

Data Engineers, however, kept writing ETL pipelines. Sure, you could pay Fivetran to sync your Salesforce data, and maybe Stripe had a native Snowflake connector, but there was no escaping the long tail of data needs. Cost constraints meant that more and more companies were looking to bring some of the offloaded work back in-house. It was harder and harder to justify spending your pennies on every row that changed in a database.

As the dust settled, and interest rates rose, and VCs got bored of data and moved on to AI, we finally moved toward some sense of normalcy in data. Instead of hot takes, the data people continued to do the work it took the help make a business operate. We came to terms with the fact that Data Work is often just Blue Collar Work.

Instead of hiring 20 data scientists and asking them to ‘find insights’, we had smaller more focused teams that worked against delivering actual value to different lines of business. From building data models that made it easier to self-serve using modern BI tools, to creating recommendation models or predicting churn, the bread-and-butter stuff continued.

As teams matured and the frenzy of SaaS died down, we’ve started to return to the dilemma posed by Magnusson back in 2016. What should Data Engineers be working on?

The Second Coming of the Data Platform Engineer

I believe we’ve passed the trough of disillusionment and are entering the plateau of productivity. We’ve made a lot of progress in the last ten to fifteen years in data. The tooling is better than it has ever been, and it’s possible to do so much more with much less. DuckDB on a laptop is replacing MS Access on a corporate desktop. This is a good thing.

With that rise of productivity among data professionals of all kinds, from ML Engineers to Analytics Engineers, to Data Scientists and beyond, pressure is starting to build on Data Engineers.

There are two ways to react to that pressure. The easiest is to hire more data engineers to support your business, but we are fortunate that we live in a high-interest rate era.

High interest rates cure all ailments.

Instead, Data Engineers are coming back to the original sin of Data Engineering, building bespoke custom pipelines for your downstream consumers, and they’re solving it the same way we were trying to solve it 10 years ago: building platforms, frameworks, and services.

Part of the problem, I think, is the title Data Engineer simply beckons you to build pipelines. The next evolution of the role is more akin to a Data Platform Engineer.

This is someone who is tasked not with building ETL pipelines, but with making it possible for their various consumers to build any pipeline they need without having to resort to a complex higher language.

How to do that well is still not a solved problem: whether it’s custom bespoke yaml-to-pipeline factories, or something more purpose-built remains to be seen. But what I am seeing is more and more companies starting to move toward a framework approach to data platforms. It’s the only way to scale the demands of a data platform without scaling up the number of data engineers supporting your analysts.

What I like the most about this is that it finally gives Data Engineers something to look forward to. Career progression for Data Engineers often felt like it was simply bigger data and more complex pipelines, but most Data Engineers I know prefer software engineering to data analysis, and pipeline building is by its very nature closer to data analysis than building software.

While building pipelines will never go away, being able to see some light at the end of the tunnel is sometimes all we need.

It’s been a while. I’m afraid I haven’t had many thoughts to lead with, and so I’ve stopped writing for a few months, but I have missed you all.

A lot has happened since we last talked. I quit posting on the site formerly known as Twitter, although I occasionally open it to see if I’ve missed anything.

In some ways I have: whether it’s Charlie Marsh talking about another Ruff milestone, or Simon Willison posting about prompt injections, there’s lots of really interesting work I’m missing, but at the same time, I just don’t have the heart for misinformation and the muskification of my internet. For what it’s worth, I’m on Threads posting to all 3 of you who read my content, and on LinkedIn, trying more and more unhinged forms of posting to see what I can get away with.

But the real reason I left the site formerly known as Twitter is that it was a site that made me feel bad. I would scroll until I got upset, and then scroll some more, and then would feel worse after I was done. And once I stepped away from it, I realized, something that makes you feel bad all the time is probably bad for you. It’s good to listen to your feelings.

While something is lost, I’m hopeful that something new will emerge, and while we’ve tried three forms of recreating Twitter, all without success, maybe there will be some other place where we can all get together and share what we’re working on.

Until then, I’d like to ask you to do something for me. Send me an email with what you’re working on. I’d love to hear from all of you still out there. It can be about data, a project, a deal, a hobby, or even a home renovation project. Whatever it is, I’d like to hear about it, and let me know if you’re comfortable with me sharing it, because I’d like to start writing not just about what I work on, but on what you all do too.

While I no longer have the reach of millions of potential likes, retweets, quotes, and replies, I think we have something a little more intimate. I think there’s something to be said for writing slower, engaging more thoughtfully, and chasing a good connection over a hot take, so indulge me.

In that spirit, here’s what’s been going on in my life:

Orchestrators Everywhere

If you haven’t been following me closely, you might have missed it, but I’ve also recently joined Dagster to do data things. It’s not every day you get to join a company that is building a tool purpose built for you, and I consider myself really lucky to be part of such a highly talented group of people.

Orchestrators are funny things. At first, they seem relatively simple: a scheduler, a task runner, a webserver, and some glue. And while orchestrators like Airflow and Dagster at first glance seem a natural place for data pipelines to run, when you look closer you start to see orchestrators everywhere.

Github Actions, CircleCI, Airbyte, Meltano, Fivetran, dbt Cloud, are all operating as orchestrators too. I’d argue not by desire, but by necessity.

Consider a simple extract-load pipeline where you fetch data from a database and load it into a data warehouse. Airbyte, Meltano and Fivetran all offer this capability. But being able to extract data from one system and load it into another alone isn’t sufficient to build a product.

You also need to schedule that task, you need to be able to monitor it for failures, retry when it doesn’t succeed, and look at logs to understand what went wrong. You may even need to create a dependency, allow for configurations, and different environments. Quickly what seems trivial becomes complex.

Part of what I’ve been thinking about and working on is wondering what a world might look like if we didn’t have to reinvent orchestrators for every job we wanted to accomplish? What if we had simple tools for extracting and loading data, or for all the other data concerns we seem to have: data quality, anomaly detection, cataloging?

Some of that work has resulted in a simple little idea I call dagster-embedded-elt. That’s what I’ve been up to over the past little while. I’d love to hear what you’ve been doing.

Coalesce 2023

Last year at Coalesce, someone whose identity I will protect, made the claim that 2023 would be the last good year for that conference. I think what he was getting at is that dbt labs would be forced to grow up and start running conferences like real companies do: customer stories, product showcases, proof of enterprise-readiness.

While in some ways, that was always the point of Coalesce, it has definitely become more true this year than any prior year. Attendance seemed lower, probably due more to the economy than anything else, but production value remained high as was the bar for talks. Talking to practitioners, it seemed the general consensus was pretty positive, and so while we didn’t have the marching bands and parties and free rides of yesteryear, I didn’t get the impression that this year disappointed.

Personally, I was stuck in my hotel room for 3 days fighting a brutal bought of exhaustion, and must’ve slept 15 hours a day each, so for me, this was the most beautiful of all conferences ever. Never had I slept so much, so peacefully, so quietly. If you can pull it off, I highly recommend going to a conference to sleep for 3 days straight.

Next year, Colaesce goes to Vegas, completing dbt Labs’ transformation from small, scrappy startup to Enterprise-Ready (TM).

Did you go to Coalesce? If you didn’t sleep for three days straight, I’d love to know your thoughts.

Everything Else

I’ve started therapy, once a week, for the past few months. I don’t know that I like it, or enjoy it, but I have to believe that it is good for me, given how much I am spending on it. I believe they call that cognitive dissonance in the biz.

After getting jealous of Taylor Murphy’s sim racing life, I decided I had to jump in. I got a wheel, pedals, and a stand. It’s equal parts fun and embarrassing.

I’ve been playing with LLMs non-stop, trying to better understand them. I’ve trained models on my desktop, built RAG pipelines using Llama Index, and am working on a support bot trained on Github Issues, Discussions, and Docs.

I am [——] this close to try NixOS. Something about it seems really interesting, but there’s also something really compelling about NOT spending three days setting up a new Linux environment.

I desperately need a new desk chair. This one from West Elm, is so squeaky it is driving me insane. Please give me chair recommendations

If you do any of the above, or anything else that’s fun, please write in and tell me how it’s going for you.

All the best,

Your Friend.

pedram

The rollout was odd: no announcement, no email. Just a blog post quietly published that someone in the community saw, and posted in dbt’s Slack channel. A community that somehow, inconceivably, has 54,000 people. (If a Slack rep is looking for the deal of a lifetime, see if you can get dbt to start paying for that workspace, and drop me 1% of your commission.)

Even though this was only affecting new users, and existing users would have something like a year before this hits their pocketbooks, a minor revolt started. You know things are bad when the CRO has to step in to a Slack thread. Somewhere in the madness, people realized that you could avoid materializing views, and so quietly, without notice, dbt dropped the number of models included in their base package from 20,000 to 15,000. The people revolt, the CRO tries to calm down the peasants and pitchforks, time is a circle.

People are tossing around back of the napkin math. Costs are going up 300% for some. One person complained that they would have to pay a whopping $200 a month. Others would have been hit far worse. Smelling an opportunity, the blogposts and Twitter threads cropped up seemingly overnight.

• Top 10 Sexiest Ways to Deploy Dbt (sic)

• How One GitHub Action Made Me a Senior Analytics Engineer

• PROOF SQL IS THE ACTUAL DEVIL

And so on. The knives were barely sharpened before they came out.

This isn’t a story about a botched pricing strategy, or one about the powerful lessons of price elasticity though.

I want to talk about something else that’s happening: Data just isn’t fun anymore.

This Used to be Fun

There was a time, not too long ago, where the data community used to be really fun. There were many like-minded people, and I made friendships from that community. Real friendships.

People I meet in real life, who I text with on the phone, who refer me to their therapists and show me pics of their new dogs. Real people I truly love, and all through this weird thing known as the ‘data community’. I have group chats with the boys, I check-up on my friends after a California Hurricane (some light rain), I message some less regularly but still with them always in my heart and in my head.

The community used to be fun. We had the same job: “something to do with data” and we used some of the same tools and we all suffered the exact same problems: data problems that were actually people problems.

There was even an exciting open-source library released by a small consultancy shop that was solving a real problem real people had. We bonded over the shared trauma that was looking at Hadoop and Spark error logs, trying to decipher Java stack traces, looking each other in the eye and saying “No. Never again. We will not forget.”

We’d make tweets and some of them would go viral, usually the worst ones, because the plebs have horrible taste. We knew the best tweets got 12 likes, and the worst ones got 12,000. We tried to pretend we didn’t laugh at puns, but deep down we did. Because we were happy.

Because this used to be fun.

But this isn’t fun anymore. Maybe it’s being harassed, both publicly and privately. Maybe it’s just the trouble with becoming a public figure, in that people you don’t know suddenly feel entitled to the same closeness you reserve for those that’s you do know privately. Maybe it was the sudden infusion of billions of dollars in capital without due diligence that brought in so many people looking to make a quick buck on whatever the flavour of the day is.

But this used to be fun.

Now it’s knives out for a company that can’t figure out how to monetize something, after taking round after round of money without a clear path to a valuation that makes little sense once the drunkenness of ZIRP gives way to the ruinous sobriety of revenue streams and capital expenses.

Maybe it’s a narcissist with a weak ego that took over a platform that was once bad, but not this type of bad.

It’s the death of dbt this, the death of the modern stack that, it’s gotchas and I-told-yous. It’s venomous toxicity, and the slow disappearance of people I used to know from a platform I used to love.

Did I mention, this used to be fun? A familiar feeling. Another platform I once loved was also destroyed.

I used to write a lot in a little place called Livejournal. There too I made friends, and we wrote for ourselves as much as for each other, while reading what others wrote for us and leaving comments. A social network before it had a name. A platform that brought people together, until eventually it drove us all apart. If it wasn’t the Russians buying the platform, it was the harassment from the same people who would later become neo-nazis.

This used to be fun.

Everyone Wants a Piece of the Pie, Nobody Wants to Bake

In many ways, we have it better than ever before. dbt still exists, and it still remains open-source, for now. There are countless free and open-source tools that we would’ve killed for a decade ago. DuckDB is truly magical. Dagster, (a company I just joined) is a labor of love of from someone who just wanted to build something that not only fixed pain, but did so elegantly, although don’t worry, I recognize it is a VC-backed company too. There’s a wealth of tooling out there, and yes, some of it is of questionable quality, but much of it is great, and there’s so much we can do today that was difficult to do before.

And while we continue to get more than we will ever deserve from the good-hearted Dutch, we also benefit from the capital that venture-backed companies provide in order to hire the engineers that build the products that we use. The problem with good software is that it requires labor to build, maintain and improve, and there’s still no substitute for a bunch of people motivated by a common goal and passion in building anything of sufficient complexity.

But there is something about the data space that feels wrong, like something is missing. There’s a spirit that we once had of building and giving away that doesn’t seem so prevalent anymore. For all the millions of dollars you raised off the backs of the jinja, how much have you donated back? For all the hot takes and blog posts, how much time have you spent maintaining open-source projects? How much have you given away for all that you have taken?

Yes, data is dumb, and VCs ruin everything, and this company sucks and that one sucks too, yes it’s all a big scam, and everyone is a marketer, and the only person who has all the right answers charges you by the hour. These things are all true, but they are not fun truths. They are sad truths, and I don’t care for sad truths.

There’s plenty of good truths too. There are people who are still giving away knowledge because they believe knowledge should be shared, and there are people building things because they enjoy the act of building, and there are people solving problems not for profit, but for fun.

If you haven’t spent some time mentoring people and watching them grow over the months and years, if you haven’t spent too many thankless hours of your life trying to maintain and contribute to an open-source tool that everyone uses and no one cares to improve, if you don’t build things to solve a pain you’ve had in the hopes that it’ll solve someone else’s, if you don’t give away your hard work for free, then kindly, please, shut the fuck up.

Because this used to be fun.

With this, I’ve started to develop three theories that I will share today on what I think the future holds for data: how ops will learn from data, the multiplication of semantic layers, and the single biggest problem data teams will continue to face.

Let’s dive in.

Ops Teams Finally Get Some Love

For the past five or so years, Data Teams have proudly learned lessons from software engineering, much of this kicked off by dbt’s introduction to data teams. We’ve learned about version control, CI/CD, configuration as code, automation, commit checks, and more. As a result, data teams have become better at managing the complexity of operating on data. Great job, everyone. I’m so proud.

Now, it’s time for ops teams to learn from data. Unfortunately, I often think of ops as the forgotten stepchild of data. In many ways, ops teams are just data teams with worse tools and more chaos. Yet, in the best organizations, I’ve seen ops and data teams work closely together to enable each other.

Ops teams work across various functions, including marketing, sales, revenue, finance, and facilities. What they all have in common are workflows driven by data that are primarily manual, annoying, and slow. Work occurs predominantly in spreadsheets, extracts to CSVs, in the Salesforce UI, and manual uploads and overrides. None of it is validated, verified, or version-controlled.

This is starting to change. Emilie Schario is building Turbine, which offers a fresh take on procurement, inventory, and supply chain management. Savant Labs is creating a cloud-native automation platform for analysts. Zamp is automating sales tax compliance. Latern is automating forecasting and churn reporting to enable CS teams to understand their customers better.

What these and other companies all have in common is a focus on bringing the best of data capabilities to teams that are traditionally underserved.

What’s clear to me is that now everyone is a data person. Whether it’s someone in success, marketing, finance, or sales: everyone is working with data, and so much of the data work in companies is happening outside data teams.

While data teams are still invaluable in building data assets and products, the work does not end there. Those data products become the fundamental building blocks on which other teams understand, forecast, and grow their business. I’m excited about what the future holds for these teams.

One Semantic Layer to Rule Them All

Just kidding. That won’t happen. To be clear: I want one semantic layer, and conceptually I love the idea of a semantic layer living close to my transforms and only having to define revenue once, in one place.

A place where every downstream tool can ingest capital-T Truth and finally fulfill all my dreams: a single-source-of-truth and self-serve analytics.

I can sip margaritas on the beach knowing that my VP of Finance will never report a wrong number again.

While the dream is beautiful, we need to consider that the one product that cares the most about semantic information is not dbt but BI tools. Tools like Looker, Holistics, or Lightdash all have a semantic layer not because they want one but because they need one.

The unlock of a semantic layer is clear: you can scale out a data team by enabling self-serve analytics. The problem is that these tools, and other BI tools, depend on the semantic layer to build their product and roadmap.

While some tools may be happy to put their hat in the dbt Cloud Semantic Layer, I think there’s an obvious issue: 1) it limits their customers to only those who are on dbt Cloud and 2) it puts their product roadmap in the hands of another company.

If you had a BI solution and had to rely on another company approving a PR to ship a feature, would you be okay with that?

This is, I think, the fundamental problem of semantic layers.

There are, however, some interesting alternative approaches to this space. For example, Honeydew is building a semantic layer middleware that can be consumed directly from the warehouse, obviating the need for integrations with another tool’s semantic layer. Any application that can read from the warehouse can read semantic information.

Julian Hyde’s work on Apache Calcite aims to bring metrics directly into SQL, with the long-term hope of standardizing how we express metrics and storing them in the warehouse outright. Malloy is an alternate take, a new open-source language for expressive data modeling.

Will any of these solutions win? If I were to bet, even five years from now, they’ll all be around, along with other half-dozen new ideas, and we’ll be no closer to a single standard than we are today. Sorry.

The Single Biggest Problem Data Teams Face Today

It does not know how to measure data teams. It’s not managing various vendor contract renewals and negotiating discounts. It’s not trying to figure out how to deploy that Python script. It’s certainly not something I’ve seen any solution or vendor offering in this space, but it’s pervasive, all-encompassing, and full of toil and dread.

It’s business logic.

Maintaining and understanding is the most challenging part of any data pipeline. We attempt to document it in yaml or a data catalog, but the code is the only accurate documentation of business logic.

Take a concept as simple as ‘lead source.’ All you need to know is, for a lead, where did it come from. Well, first, we need to define a lead. Marketing might have one definition, and sales might have another. Once you have a lead, you need to identify its source. Again, the rules vary depending on who is asking.

Marketing might have rules based on priority and a rolling window. Did you get the email from an event, webinar, or ebook? What if they did two activities? Which one counts? Sales might have their questions: did it come from a partner? Was it part of an outbound email campaign? An SDR? A business card found on the ground that you cold-called?

The rules for how you classify this lead are constantly changing. In addition, the data you collect to classify it is never clean. They are removing Gmail accounts, cleaning up fake data, and stitching pieces together from various source systems. All these data flow issues are the core parts of data pipelines we seldom discuss.

They’re also the core models that drive all the metrics and measures downstream.

When someone asks you to explain how we calculate churn, you could direct them to documentation that explains how churn is calculated, but the true definition of churn isn’t in the documentation; it’s in the DAG of transforms, filters, conditions, aggregations and more that occur step-by-step to take a sequence of events, invoices, subscriptions, payments, and constants into a monthly view of reported churn.

This complexity is why column-level lineage demand and appeal are ever-growing. Understanding how this number came to me means understanding the web of inputs that led to it.

Over time the cruft adds up. There’s data from the previous migration, the random spreadsheet from an experiment that needs to be accounted for, test accounts, wrong ids, and corrections upon corrections.

What do we have to manage this complexity? At best, some modularity, but the fundamental complexity hasn’t been managed; it’s only been transformed.

Data mesh is, in some ways, the admission of defeat in the face of complexity. The demands of teams are so complex that we must break apart the whole thing into smaller, more manageable chunks. Sales get their metric, and Marketing gets theirs. When someone asks why the numbers don’t match, we tell them that they don’t match because they are different.

Reconciliation is impossible. The quantum theory of data states that you may not understand and measure a metric with perfect accuracy. You may understand but not measure, measure but not understand, or measure and sort of understand. These are the only states we know.

The other option is forcing stakeholders to make tough decisions about simplifying their requests; that’s a dream we can all have. We can stare into the abyss and demand that the pit of despair leave us be; that the dread and anguish of staring into 500 lines of SQL that somehow define what customer is be ridden from our lives. We may reject the Sisyphean rock-pushing of model-building for now, but the rock will remain steadfast in its place. We may decide to quit data altogether and see what those software engineers are up to—they seem much happier.

But we wake from our dream. The data remains a mess. The stakeholders remain impatient. The work never ends, and we press on—one reconciliation at a time. The numbers are different because they are not the same. The numbers are different because they are not the same—the numbers.

The numbers.

The horror.

Until next time, happy model building.

I’ll explore the first part of every data journey: getting data out of a system. The story is likely to be a familiar one to many of you. Data about something you care about exists in some system, and you want to extract that data and store it somewhere. How hard can it be?

I’m going to skip over the pleasantries of why you might want to do this and pretend that we all understand it’s something that needs to be done. There are many types of systems that have data that we might want, but for our example, we will cover a common use case: application databases.

You can’t extract data without also putting it somewhere so we’ll also discuss the merits of two different strategies: saving the data in some structured format like a CSV or a Parquet file, or writing the data directly into a Data Warehouse.

Let’s get started

Querying Data

Given some source system, we will need to query that system to retrieve some subset of data. If our source system is a SQL database, then naturally, we’ll use SQL.

In the brute force method, we could extract all data from all tables that we’re interested in, ignoring system load and storage costs. This is a stateless process that is often overlooked but is rather simple. It might look something like this in Postgres:

\copy customers TO './customers.csv' CSV DELIMITER ','

We might soon realize while this is a simple method, it comes with some costs: namely it can be expensive to run. Our backend database may not appreciate the load. (You are running this against a replica, right?)

One solution is to only fetch data that has changed since your last run. This is known as an incremental data load.

We have two options:

1. Use Change Data Capture: Subscribe to a log of all database events and fetch all events that occurred after our last load.

2. Use a column that updates every time a row is updated, and only fetch records that have been updated since the last load.

The first option tends to be more accurate, but also a bigger pain to set up. It requires configuration of the database server, possibly even a restart. To really understand it, you’ll need to understand the WAL or write-ahead log in Postgres.

A quick detour on databases

The WAL is a mysterious place within a database.

While on the surface, a database appears to be a collection of tables that mirror a spreadsheet with many tabs, under the hood what is going on are sequences of events. The WAL is the ledger that keeps track of these events. Anytime you insert, update, or delete a row, a record of that transaction is kept just like it would be in an accounting ledger.

The balance of all these transactions, much like the bank balance you have, is the collective sum of all these events. While the WAL is used to ensure that in the event a database goes down, a record of everything that happened since the last backup is persisted, the WAL can also be used to sync data other systems, such as a backup replica database, or even, your silly little ETL job.

A plugin, such as wal2json, can translate these events into something a little more manageable, as we see here.

{
        "change": [
                {
                        "kind": "insert",
                        "schema": "public",
                        "table": "inventory",
                        "columnnames": ["id", "item", "qty"],
                        "columntypes": ["integer", "character varying(30)", "integer"],
                        "columnvalues": [1, "apples", 100]
                }
        ]
}
{
        "change": [
                {
                        "kind": "update",
                        "schema": "public",
                        "table": "inventory",
                        "columnnames": ["id", "item", "qty"],
                        "columntypes": ["integer", "character varying(30)", "integer"],
                        "columnvalues": [1, "apples", 96],
                        "oldkeys": {
                                "keynames": ["id"],
                                "keytypes": ["integer"],
                                "keyvalues": [1]
                        }
                }
        ]
}

Every change to a database emits an event, and nearly every database has their own way of emitting these events. There is no single standard for CDC so it is up to the downstream implementations to handle the varying logic. Now that you know what a WAL is, let’s pick an option.

Incremental Options

The WAL, while appealing, comes with many complexities. We’d have to store JSON data and process each row.

Instead, we’ll opt for using the updated_at column.

These are typically maintained by the database and automatically update whenever a row changes. Be cautious, sometimes they don’t update, and this can cause inconsistencies.

This simplifies our future queries quite a bit. Now we can filter on data updated since our last sync.

select * from customers
// if incremental run
where updated_at >= {{last_sync_date}}

Data Storage and Encoding

Once the data has been queried, it needs to be saved. The key decision here is whether to save the data in a binary or text format. You’re already familiar with text formats: CSV and JSON are the most popular. The benefits of text formats are that they are easy to read for humans, but this comes at the expense of efficiency and ambiguity around data types.

If you’ve ever had to parse the text “01-03-12” into a date format, you’ll appreciate the horrors of ambiguity. The structure of JSON also makes for large files as keys are repeated for every row.

Binary formats solve these issues by encoding data in a machine-readable format. Binary formats encode the data’s schema and types and offer more efficient storage of data. Parquet, for example, offers techniques like column-level compression and bit-packing. But nothing is every easy when it comes to data.

Parquet has a few well-defined standard types. Your typical types such as INT and FLOAT along with strings, dates, and timestamps are all well supported. The problem is your database may not have the same type system. For example, Postgres has the inet and cidr types which are ways of encoding network addresses.

Here you may wish to resort to trusty strings for unknown data types by default, allowing downstream to cast these as needed or simply use them as is.

To extract the data into parquet, you could use a wonderful little like pg2parquet which can take either a table or a query and output a parquet file, or even take advantage of DuckDB’s Postgres scanner.

With pg2parquet, you can extract an entire table in a single line:

 pg2parquet export 
     --table businesses \
     -H $POSTGRES_HOST \
     -U $POSTGRES_USER \
     --password $POSTGRES_PASSWORD \
     -o ./businesses.parquet \
     --dbname can_i_haz_replica

You’ll want to save this to a persistent storage through your cloud provider, such as S3 or GCS for safe-keeping. Don’t let your hard work go to waste!

Skipping Storage

Pedram, you might say, why store data twice? I could simply write each row to my data warehouse right away!

And you are correct, you could do this! In fact, if you think this is a good idea, I’d encourage you to try it. What you will eventually run into is a failure, and failures are never fun.

If you chose to write directly into the warehouse and something in the process failed, you’re left scratching your head. How do I debug this issue? What was the row that caused the issue? Can I even see what the data looked like? Why did I choose this career?

Instead, if you chose to write the data to an intermediate layer, you have some more options available to you. You can take the exact file that failed and inspect it, load it into a dev environment, scan for gremlins, and hopefully address them so that your pipeline becomes more robust to failures in the future. What a happy data engineer you’ve become!

Loading Data

Loading data can be fraught with difficulties. You will need to insert new rows, updating existing rows, and delete removed rows. All of these actions require a primary key.

The MERGE DDL command allows you to perform all of the above in one go.

MERGE dataset.Inventory T
USING dataset.NewArrivals S
ON T.product = S.product
WHEN MATCHED THEN
  UPDATE SET quantity = T.quantity + S.quantity
WHEN NOT MATCHED THEN
  INSERT (product, quantity) VALUES(product, quantity)

Issues arise however whenever schemas change. New columns in your source data means tables must be altered before an insert. Even worse, if a source data type changes then you may have a bigger problem on your hands.

It can often be helpful to add helper columns as you load data, such as a timestamp of when the current batch was loaded, in case rollbacks are needed.

Now do it all again

Let’s pretend this wonderful journey proceeded smoothly for you. The next question is, can you do it again tomorrow? Will you know if it ran successfully?

Subscribe now

Doing a task once is much easier than doing a task every day. In my next post, we will look at orchestration and scheduling, and all the other bits that come with.

Until next time!

What we’ve seen over the past few years has been a number of companies formed to help solve real problems data teams face. The first wave of companies solved broad, large problems where the value trade-off between engineering effort and vendor spending was clear: building and maintaining ETL pipelines can require a team of data engineers to handle ingesting, changing APIs, monitoring, logging, and the rest. But as the more obvious parts of the stack became dominated by one or two vendors, smaller and smaller pieces of the data pie were left for others to build against.

Subscribe now

There was also an eagerness to believe that solving a problem at one company was a universal problem that could be easily solved at many companies. I’ve seen many attempts at bringing a well-designed solution at a larger tech company to market with the hopes that others could soon benefit from a scaled-out solution. Still, I think we are starting to see that large-company problems are often not the same as small-company problems and, even more so, that tech-company solutions don’t always work well for non-tech companies.

All this to say that it’s no surprise that we’re seeing some pushback against the complexity of the past few years.

Compounded by the unfortunate market reality that companies need money to survive. Yes, I understand this can come as a surprise, but it is a new reality we all try to navigate. With that, there’s a minimum bar for any vendor below which a customer is simply not worth acquiring. Conversely, for every company looking to buy a data product, there’s a bar above which the solution isn’t worth the cost.

I fear the gap between these two bars is increasing, which is perhaps causing an overdue re-examination of what we need and how we get there.

At the danger of waxing nostalgic, I’d like to take a look at how we solved data problems prior to having access to endless vendors eager to solve problems for us.

I’ll explore common tasks like ETL, orchestration, and moving data between systems. I’m not a purist here; some things are not worth doing yourself — BI comes to mind. And the goal isn’t to demonstrate the cheapest possible stack. We will still use cloud vendors, such as Google Cloud or AWS, but the emphasis will be on building in-house with an assortment of custom code and open-source tooling.

We will consider what it will take to go from proof-of-concept to development to production and discuss the key concerns we want to keep in mind as we build systems: from maintainability to reliability and performance.

If this sounds interesting to you, smash that subscribe button, and leave a comment about what part of the stack you’re most interested in learning about. The next post comes out in about a week’s time.

See you then.

Let’s explore the fantasy together.

DSLs over Templated Code

One thing I really enjoy about using LookML is the ability to know when I made a mistake instantly. Because LookML is a domain-specific language, it has many nice features, such as type safety, validation, and autocomplete.

There’s a richness to LookML that you instantly start to miss when you’re working in templated languages like yaml. While you can certainly provide a certain level of validation through JSON Schemas which dbt does provide, there’s still something missing when you’re writing templated SQL or model configurations in yaml.

This bifurcated state of Jinja-templated-SQL on the one hand and yaml-configs on the other in two separate files becomes painful over time as model complexity grows.

In a world where we don’t care about backward compatibility, I’d like to see a whole new language that dbt uses to provide model metadata and describe the SQL model itself simultaneously. All in one file.

Maybe it would look something like this. It could come with its own language server protocol that would have a rich context-aware understanding of your source database schema to help with autocomplete and typos.

Notice how the code that describes the model and the SQL is contained in one file? This would reduce cognitive interruptions that frequently occur today when building models. How often have you updated a column in your model and not bothered to edit the schema file because it was too far away?

Language servers are very powerful tools that most programmers rely on when building software. They also work across various tools, so the same language server in VS Code would also work in Vim.

Imagine a world where you could rename a column in a base model. The language server could rename all symbols associated with that column in the model configuration and downstream models in one go.

You could also leverage code actions. For example, after explicitly naming your columns, a Code Action could generate boilerplate names and descriptions for all columns as part of your model config. You could even go so far as to have generative AI infer the column descriptions and tests. Why shouldn’t the computer already know that your id columns should be unique and not null?

Debuggers

Analytics Engineers can have nice things too. In software engineering, debuggers can be essential. When something goes wrong, it can be nice to interrupt the flow of a program to understand the state of the system right before things go bad.

In Python, you can drop into the python debugger by adding a `breakpoint` function call anywhere in your program. When running tests, you can ask pytest to fall into the Python debugger whenever a test fails, or even at the start of a test to trace through the program.

What if the same were possible when building SQL models? Suppose you had an incremental model that just wasn’t quite working properly. Maybe defined like so:

I could imagine a trace mode, where dbt would first preview both the compiled SQL and the results of each CTE, starting with sessions. Next, other_sessions would preview, and finally the last statement.

What if you dropped into a live debugger that allowed you to run SQL against your warehouse and displayed the results right there in the editor where you work? No more jumping between editing code to your Snowflake UI and back.

(dbg) > ${events}
-> dim_events
> ${if incremental} 
-> true
> ${incremental_condition}
->  event_time > (select max(event_time) from dim_events
> ${sessions}
-> sql: select * from events where event_time > (select max(event_time) from dim_events
-> rows: [event_time, event_type, event_id]
         [2022-01-01, cart, 1234]
         [2022-01-02, cart, 2456]
... press m for <More Rows> or w to quit.

Unit Tests

Imagine if simple unit tests were easy to write and run. One could imagine a world where you provide a small sample dataset to a query and ask dbt to run it, perhaps through an alternative adapter like DuckDB, in order to validate pieces of logic.

Now I recognize there are a million reasons why this wouldn’t work, but again, this is me dreaming. Why can’t analytics engineers have nice things too?

As I keep going down this path, more and more things seem possible now. With a proper language in place, column-level lineage is achievable. Code generation becomes easier too. Why should I ever have to write a base staging model when the database is right there?

dbt Packages, while decent enough, could be vastly improved if you could import specific modules from them. Today it feels cumbersome to pick and choose a few tables from a package or even to pass in variables to them. But what if you could import logic from packages instead?

Relationships can also be expressed independently of metrics. Downstream BI tools could leverage this for self-serve analytics. Who knows? Anything is possible.

Anyway, a boy can dream.

Remember Zoom? A pleasant way to run virtual meetings. A clean interface, good video and audio quality, and an easy way to create, schedule, and share meetings. But was that enough? No.

Instead, there are Limited Time Offers and Apps for things no one has ever heard of, like twine (advanced breakouts), or Sesh (a virtual agenda) or W (AI Business Cards). I have no opinion on the usefulness of advanced breakouts or AI Business Cards. Still, I do know that seeing these pop-up in my Zoom meetings automatically, unwarranted is not something I would ever want.

I’m not alone; there are dozens of us out there. And this isn’t strictly a Zoom phenomenon. It happens everywhere, and it’s so pervasive I would argue it’s a driving force for many new products. Old products get so bad that people leave in frustration for anything that stops treating them so poorly.

Remember Gmail? This is what it looks like today.

There’s Mail, Chat, Spaces, Meet on one sidebar. A whole different sidebar on the right for Google Calendar, Keep, Tasks, and Contacts. Not to mention Active? Whatever that is. Gmail became so annoying that people started paying for an email client to avoid having to login to that nightmare.

Microsoft Windows 11 is so bad that entire websites are dedicated to helping people remove advertisements from your operating system. Only Adobe could perhaps rival Microsoft in this domain. Something as simple as saving a file is now a complex feature-parity contest.

It’s a clear and pervasive trend, but the question is why is it happening and why is Pedram writing about this in a data newsletter? Is he going to try and weave some tangental thread through these as if there’s some type of natural connection with his central thesis, and pretend that this isn’t just an excuse to complain about things that annoy him?

Absolutely not.

Simpson’s Paradox, Local Maximums and Other Tangential Threads

In statistics, Simpson’s paradox is a confusing trend where what appears to be a correlation in one direction may actually be a trend in another direction. What the model failed to do was account for confounding variables. What appeared to be a positive trend may actually have been a negative trend for each subgroup.

Somewhat related is Anscombe’s famous Quartet of identical datasets when explored using summary statistics, but with obviously different distributions. On the surface, looking at a high-level summary, the data all looks the same, but when plotted, or analyzed more carefully, it turns out there are four very different models at work.

There’s also the oft-used (and abused) idea of local maxima, where what seems like the best decision given the current context is far behind the globally best decision given full context. It’s a common pushback against experimentation-culture, where the critics say that small incremental changes are unlikely to lead to the large, big-bet changes that help build lasting companies.

These are well-understood entry-level concepts within statistics. But they all point to a common trend I think we see in some of the products I described above and many others.

What starts as a good product, gains adoption. The fast pace of growth eventually slows due to the laws of physics. Rather than update models and assumptions, the company hires a growth team; they ‘experiment’ and each experiment in isolation seems like a good idea. They might move the metrics in the right direction. Maybe they generate marginal increases in revenue. But the overall experience suffers. And soon, all these marginal improvements end up creating a bad product, which inevitably leads to the downfall. New players sense this weakness and dissatisfaction, and competition enters the field to serve your market better than you did.

This same model plays out in the data ecosystem. Vendors who build something really good at first can quickly find that they’ve tapped their initial market. Faced with this precipice, they look for ‘quick wins’. But quick wins is code for minor optimizations.

What they should have been doing was preparing for this scenario months ago. The best founders are always paranoid and never satisfied. They’re less focused on how to win today, but on what to win next.

As you’re building out products and gaining success, I’d encourage founders to ask themselves that same question. You have an entire team dedicated to helping you win today. The question you should be always asking yourself is what’s next?

Last month I got a sneak preview of Striim’s new developer experience that makes it easy to get started with CDC using BigQuery or Snowflake. If you missed my thread about that, check it out. In this post, I’ll look at how you can leverage Striim, Parquet, and DuckDB for real-time data ingestion with fast data analysis.

Data pipelines have traditionally been batch, and batch pipelines are usually easier to reason about. Data comes in once a day. I run all my transformations and load them sometime between 12:01 AM and 7 AM UTC (or was it PT? Timezones are hard.) Views and tables get updated, people look at data from yesterday, they get the answers to all their questions, and life is good. Life is simple. 

Unfortunately: the good old days are dead. Now we run operational workflows off data constantly fed into the data warehouse. We need to reduce the lag of all the various components that ingest, digest, transform, and reform our data as much as possible. For example, we have personalized workflows that send automated emails to prospects and customers who expect us to understand only every interaction they’ve had with us but also anticipate every human desire they could conceivably have in the next fifteen minutes. 

We’ve started pushing batch to the limits of streaming. Some of these batch tools can run as often as every 5 minutes, pushing the boundaries of what is and isn’t streaming anymore.

This is what got me interested in the streaming and CDC space in the first place. I wanted to know if there was a better way. After a dizzying stroll down Debezium Lane, and a confusing jaunt through Kafka Caverns, I received a nice demo from the fine folks at Striim. 

Striim is an enterprise-grade CDC platform, and I am but a lowly developer with toy examples, and it works just as well for me. For my first attempt in the tweet above, I set up a simple Postgres instance, piped data into it, and watched as Striim fed my BigQuery tables with change capture data every few seconds. 

But BigQuery is old news. Today, I wanted to see if I could get our lord and savior, DuckDB, to work with Striim. The setup was simple: use a GCP Writer to save streaming data to Parquet. Then, use DuckDB’s HTTPFS extension to read data from Parquet files in bulk. Write queries. Enjoy streaming.

Let’s dive in. 

Striim Setup

I decided to use one of the built-in data generators to get started quickly. These data generators are great for sketching out ideas since they let you avoid the messy parts of connecting data systems, such as permissions and IP allow-lists.

The ContinousGenerator can be set to various types of throughput. Low Throughput sends about ten messages per second, Medium for hundreds per second, or Spike for variable traffic with high spikes which can be handy for testing the resiliency of your pipelines.

Next up, I used a Query cell, which operates on an incoming stream and allows you to do transformations as data is produced. This can save lots of expensive compute in your warehouse by shifting the transformations left, closer to the data source. You can also do data-masking as data arrives, to make staying compliance easier. I wrote a simple query that takes the generated data, masks sensitive information, and outputs the results to a GDPR stream. 

Finally, to be able to analyze this data in DuckDB, I elected to write the data in Parquet format to Google Cloud Storage, although S3 would also work just as well. To do that, I used the GCP Writer Target. After creating a Service Account in Google Cloud, I setup a few basic settings such as the path to the bucket and format I’d like the files saved in.

One setting to be aware of is the Upload Policy, which determines how frequently (and conversely, how large) the files are. Finding a good balance here is important, as too many files or too few can both hinder performance.

I set the Upload Policy to write every 100,000 events or every 1 minute. I set the ParquetFormatter as the output option.

With the setup complete, all that is needed is for the app to be deployed and started. You even have a preview feature to watch data as it is fed through the system. You can see I’m fetching about 900 messages every second, and after about a minute the data will write to GCP. 

What’s neat is that Striim even displays the total End to End Lag so you can have insight into how delayed your pipelines are. In my case, the lag was about 30 seconds from creation to write.

After running for a while, the Parquet files are loaded in GCP and now it’s time to analyze the results with DuckDB.

Analyze with DuckDB

There are many ways to use DuckDB given that it’s a small portable binary. The CLI is a great place for simple prototyping but I prefer using Datagrip for writing queries. 

After creating a new DuckDB connection and enabling single-session mode, I added a small startup script to ensure that every time I connect to DuckDB my GCP credentials are entered.

The docs on setting up S3 or GCS access are pretty straightforward. A few simple SET commands and then you’re ready to query!

INSTALL httpfs;
LOAD httpfs; 
SET s3_endpoint='storage.googleapis.com';
SET s3_access_key_id='MY_ACCESS_KEY';
SET s3_secret_access_key='MY_SECRET';

To start, I ran a simple query to see how many records we have in each file. By using the filename=TRUE command, DuckDB returns the filename as a column in the table, which I use for aggregation.

SELECT
 
filename,
COUNT(1) AS n_records
 
FROM parquet_scan('s3://my-duckdb-bucket/striim-out.*', filename=TRUE)
GROUP BY filename
ORDER BY 1;

In about 7 seconds, DuckDB scanned 760,000 records across 14 files with 55,000 records each to generate a count of records by file. And the best part is there’s no Spark cluster to maintain. You can see below that using the filename to do a group by makes it easy to get a sense of how many records were written in each file.

We can even do fast text processing. In 6 seconds, I can categorize all products by whether they have Heavy or Lightweight in their name and aggregate across both dimensions.

SELECT
    product_name LIKE '%Lightweight%' AS is_lightweight,
    product_name LIKE '%Heavy%' AS is_heavy,
    COUNT(1) AS count_products
FROM parquet_scan('s3://my-duckdb-bucket/striim-out.*')
GROUP BY 1, 2;

The best part is this data is constantly updated by Striim. Every minute a new batch of 55,000 records arrives. 

That was fun, but we needed to go faster. Just for fun, I cranked up the generator to see how it would handle a higher rate and set the Upload Limit to 25,000 records per file. I easily hit 20,000 messages per second, and the end-to-end lag was just a few seconds. Striim had no problem with the throughput. In just a few minutes, I had 60 Parquet files ready for DuckDB to process.

With now 60 files to process, DuckDB took just under 30 seconds to count every record in every file. The product name query now took 23 seconds on 1.45 million records.

As one final test, I decided to push some regex and aggregates down to see the impact on performance, and DuckDB held up well. This query took under a minute to query all 1.45 million records, and I didn’t have to store a single file locally. (And if you were wondering, the average of the last 4 digits of a phone number is 5001).

SELECT
    date_trunc('minute', CAST(TIME AS TIMESTAMP)) AS DATE,
    avg(CAST(regexp_extract(Phone_Number, '\d+') AS NUMERIC)) AS avg_number
FROM parquet_scan('s3://my-duckdb-bucket/striim-out.*')
GROUP BY 1

Wrapping Up

I hope this was a helpful exploration of how you can use Striim and DuckDB to process real-time analytic queries quickly and easily. Gone are the days of Kafka, Zookeeper and Debezium. In less than 30 minutes you can get a CDC stream setup, write to a cloud bucket location, and query with DuckDB for blazing-fast analytics. 

If you want to give Striim a try, you can sign up here with my referral code tAlaDngxjQ.

But I soon learned that knowledge isn’t a requirement for predictions. We make all kinds of assumptions and predictions without any basis in fact all the time, and maybe that’s okay

So then I started to come up with a laundry list of predictions I had for the future, and thought, why not make this into a blog post, as I’ve been horrible at keeping up with this Substack? Perhaps I can trick these gentle readers into thinking I have actual content when all I have are screenshots of moments where I was funny.

But, no, that felt wrong too. I have more than just screenshots to share with the world; I also have unfinished drafts. What if I could surreptitiously sneak in a few drafts and random screenshots together, would the reader even notice? Unlikely unless it was made incredibly obvious to them by some dim-witted narrator who wasn’t too careful with his (or her) words.

And so, kind reader here is every thought I’ve had in 2022, gathered from drafts, from notes, from todo’s, from tweets, from slack, from the withered remains of my memory.

If there’s one thing I can say about 2022 is that it certainly did happen, and it is almost, without a doubt, nearly over. And I have the data to prove it.

On Building a Brand

Here’s the ugly truth. Being public and writing publicly has done more for my career than anything else. Never mind the books I read, the courses I took, the systems I built, the systems I destroyed, the resumes I wrote, the cover letters I didn’t, none of it has had the impact of being public and writing about myself.

Not to say none of that stuff isn’t necessary, but (in a language you nerds will understand) it isn’t sufficient.

Marketing, to some, is a dirty word. I’ve learned to embrace it. I am still in the unfortunate position of caring about job security, and being seen as someone who knows things is more important than knowing things.

Networking, to some, is a dirty word. I’ve learned to rephrase it. Networking, to me, is nothing more than being friends with people who work in the same industry as you. I’ve made a lot of friends this way. I’ve also made a few enemies.1

There’s no magic trick, secret cabal, or mysterious meetings. I talk to people I’m genuinely interested in, meet them in person when I can, and treat them as well as I can. If we like each other, we become friends, and if we’re friends, we’ll help each other. That’s all networking is.

Source: Substack post called “How I Learned to Stop Worrying and Love My Job”.
Status: Draft has now been deleted.

On Asking for Help

Learn to ask for help. This is somewhat related to the previous topic. Strangers don’t owe you anything, so if you’re asking them for help, please, learn to ask for help.

Here are some simple steps to follow:

1. Acknowledge that you’re asking for someone’s time and mental energy. You may not get anything back, and that is okay. It is not a reflection on you or them.

2. Keep it succinct. Do not send a four-page essay to someone. If they have follow-up questions, they will ask you. Make it easy for them to say yes.

3. Show you’ve done the work. Have you thought carefully about why you’re reaching out to this person? Have you tried answering it on your own? Here’s a great example I made up just now: “Hey, Pedram, I’ve been thinking about switching from data scientist to data engineering, and you’ve talked publicly about having made that career switch before. I was wondering if you wouldn’t mind answering a few questions I had. I can send them here or by e-mail if that’s easier for you. Thanks for your time!’”

I honestly don’t mind helping people. I’ve been very fortunate in life and my career, and I want others to have that too. But please, make it easy for me to help you. Help me help you.

Source: Substack post called “How to Ask for Help and How to Give It: A guide for everyone who's ever messaged me out of the blue”.
Status: Draft has now been deleted.

On Hiring Your First Data Role at a Startup

I don’t think we have a good answer for this yet. Data is one of the loneliest positions at startups. Up there with Finance. A team of 1 for far too long, no one to talk to, no one to bounce ideas off of.

Whatever you do, don’t hire someone too junior. Data roles aren’t about data. They’re about negotiating between varying teams about who gets credit for success and who gets blamed for failures. They’re about trying to get an organization aligned on what matters. How you measure something is more about the process than it is about a data pipeline. Finding holes in the process will be all you ever do.

Data roles can be extremely isolating, even in the best of times. Apart from a stack under constant evolution, data teams are often a single-person show, while their peers in engineering typically have multiple people they can rely on for everything from code reviews, to mentorship and just ranting at each other about the state of Javascript.

Not so with data, which is perhaps why communities like dbt and Locally Optimistic are so large and vibrant. When you don’t have anyone inside the company who can truly feel the pain of incompatible schema changes that weren’t communicated, a community of others who can understand your pains becomes very valuable. But having a community is no replacement for a leader, and that’s where I think most frustrations data practitioners in their role feel, especially in early-stage companies as the first data hire.

Source: Substack post called “Hiring Your First Data Hire”.
Status: Draft has now been deleted.

On Talking About The Work

When we talk about data, we talk about frameworks, mental models, designing data teams, bundling, unbundling, rebundling, databases, databases, and more databases.

What we don’t talk about is the work because we can’t. Data work is private, secret, and sometimes legally-protected. Engineers get to write blog posts and release open-source software for the work they build. Data people get to talk about a tool they used, maybe a method without context, if you’re lucky. But never the journey that got us there. Just imagine this talk at a conference:

“We’re a mortgage company, and our processing rate was down 15% this quarter. We ran an analysis to identify the causes and found that it was because we were chronically understaffed during the summer months; however, after doing a cost-benefit analysis, we found that it was cheaper to have longer processing times than to hire additional staff to cover peak hours, so we decided to settle for reduced service levels.

In this talk, I will discuss how we discovered our findings and how I negotiated to present these results to our CFO without upsetting our partners in product and processing.”

I would love to hear that talk, but it will never happen. So instead, you get a talk on how we enabled self-serve analytics by buying a tool.

Source: Substack post called “We Need to Talk about Data (but can't)”.
Status: Draft has now been deleted.

On Smelly Code

Data Modeling Code Smells is my term for the stuff you write; as you write it, you tell yourself…this stinks.

Here’s a non-exhaustive list of smelly code when data modeling:

1. Duplicated code

2. Too many lines

3. Too many columns (very wide models)

4. Excessive comments

5. Clean-up code in marts

6. Too much jinja

7. Inconsistent naming

8. Casts and Coalesces

9. Right Joins

10. Functions in Joins

11. Magic Variables

Source: Substack post called “Data Modeling Code Smells”.
Status: Draft has now been deleted.

On dbt Cloud’s Pricing Changes

I used to work at a bank, where I did compensation modeling. Once you see the world through the lens of incentives, your brain breaks, and you cannot see anything but incentives everywhere you look.

Pricing and incentives are among the most interesting parts of running any business. The pricing model you choose can make or break you. dbt Cloud’s decision to go with a seat-based pricing model for their cloud text editor/scheduler made sense at the time but locked them into a corner.

The best thing you can do is align your pricing structure with the value you create for your customers. If you can’t do that, then you might as well just make up numbers and call it enterprise pricing. This is a perfectly valid way of pricing your product; call it a platform fee.

Seat-based pricing is rarely a high-growth strategy. Unfortunately, when you raise VC Capital, you’re expected to have high growth until you exit (at which point you can cease growing altogether).

Consumption-based pricing is better, when it can work, and when it can be understood. Fivetran is one of the few MDS companies that makes money because it’s easy to get started and rack up a $100k annual spend without blinking. Snowflake isn’t hitting 166% NRR on seat-based pricing.

When your SaaS product costs the average company less than they spend on toilet paper, you’ve got a Shit Pricing Model™. When money is free, pricing doesn’t matter, but when money costs something, the Shit Pricing Model ™ needs a redo. It’s no surprise dbt decided to increase their prices; what is surprising is:

1. They did it with very little notice at the end of the year.

2. They claimed that they were doing it because we asked them to. Just own up that you’re a business trying to make money; it’s not a crime. It’s an easier story to believe.

3. It came with no real increase in value to the customer.

4. It is relatively easy to trade their subpar experience for a home-grown subpar experience.

Again, it all comes down to incentives. People who had no incentive to roll their own mini-scheduler now had a major incentive to switch off dbt Cloud. Going from $50 to $100 a month doesn’t really impact most teams (and doesn’t really generate any real revenue for dbt). But going from $5k a year to an enterprise contract because you have 9 analysts, well, now we have the incentive to try and build it in-house. Yikes!

I know I am not a CEO (wait, I am now), and it’s easy to criticize from the sidelines (but when has that ever stopped anyone?), so in the end, what I say doesn’t matter or mean much. But using what limited info I have (here comes that data-talk again), I would have used levers like metrics or dbt Server to push companies into enterprise rather than just seats, especially since those are much harder to build internally. Oh well, what do I know!

Source: Twitter threads ,and substack post called “Let’s Talk about Incentives Baby”.
Status: Draft has now been deleted.

100 Ways to Align with Business Outcomes

Now that money isn’t free, it’s time to prove your data teams are worth something. Here are 100 ways to align your data team with business outcomes.

Actually, I only came up with 16 before running out of ideas. Sorry.

1. Optimize marketing spend to increase conversions

2. Identify marketing channels to drop due to inefficient spend

3. Run A/B analysis on email campaigns to identify better messaging 

4. Forecast end-of-quarter pipeline to identify sales targets 

5. Calculate the true customer acquisition cost by channel and recommend ways to reduce expensive channels 

6. Analyze inventory orders to identify wasted opportunities 

7. Identify factors that lead to backlogs during peak demands season at processing centers 

8. Create an LTV model for customers and build a process for continually updating it. Identify best sources for high LTV

9. Analyze and forecast infrastructure cost as a function of customer growth and recommend ways to prevent linear or higher growth.

10. Analyze funnel data to identify drop-off points and make recommendations on how to improve the experience to increase retention 

11. Analyze support tickets for common themes and recommend product improvements to reduce tickets 

12. Figure out how long it takes for leads to get to a first meeting and identify ways to highlight quality leads earlier to reduce that length 

13. Create a lead scoring model and identify commonalities in top-scoring leads and recommend a nurture campaign to get them into conversations

14. Analyze cloud costs and identify underutilized resources to reduce cloud spend 

15. Identify sales pipeline drop-off and break it down by factors to identify leaks in the sales process and how to plug them.

16. Analyze experimentation results to identify perverse incentives that may have happened through gamed metrics  and recommend guardrail metrics to protect against them 

Source: LinkedIn and Notes
Status: Notes draft has now been deleted.

And last but not least, from my Notes Draft

Here comes the dag again
Failing all my jobs like a memory
Head in my hands like a new emotion
I want to work in the open source
I want to talk like lovers do
I want to dive into your data 
Is it activating with you
So baby talk to me
Like data do
Talk to me
Like lovers do
Talk to me
Like data do

Source: Notes
Status: Notes remains in case I ever produce this song

If you prefer video, I gave a talk on Entity Resolution at dbt's office hours 2 years ago. If you prefer books, there's no other book I'd recommend more than Peter Christen's Data Matching. If you prefer my Substack, you’re in the right place.

Let's dive in.

What is Entity Resolution?

Entity resolution is all about combining multiple records of things. There are two parts to entity resolution: first is the entity, and second is the record of that entity in some database.

The entity can be anything from a person to a company to a physical product. I'll use companies as examples here, but the underlying logic applies to any entity you want to dedupe. 

A record of that entity might exist in a spreadsheet, a database, or across multiple databases. 

What's important is that there is no unique identifier representing that entity. If you were trying to dedupe people and had their Social Security Number or another national identification number, then the problem would be relatively easy. However, absent a single unique indicator, if we want to match or dedupe these records, then we need a way to resolve them to a single entity: hence, entity resolution.

An Illustrative Example

Let's say you work at a small B2B company with data in various systems of record: your production database, your Salesforce instance, and several spreadsheets of data with leads captured at various events. 

Anyone can sign in to your product by providing their company name and email address. Your Salesforce instance has accounts with company names, locations, websites, and contact information. The leads spreadsheet has similar information hand-captured by a person running the event.

It might look something like this:

How do you decide whether or not these different records are part of the same underlying entity? You might keep it simple and decide that two entities are the same if they share the same website, but even websites change over time. While a simple solution may be sufficient, you're entering the realm of entity resolution if you're not satisfied with that.

Share Pedram's Data Based

The 5 Steps to Entity Resolution

This deep dive will go through the five keys to entity resolution. There is much more nuance and depth beyond this post, but this should be enough to get us started.

1. Pre-processing

2. Indexing

3. Comparing

4. Classifying

5. Merging

Pre-processing

Before we embark on our journey, starting with a good foundation is essential. As much as possible, we want to clean our underlying data. Everything from trimming extra whitespaces to lowercasing all the characters, removing stop-words, or even stemming and lemmatizing is possible here. The specifics are highly context-dependent, and always an iterative process. As you perform these steps on data samples, your appreciation for what you need to do to improve the quality of matches will increase, and you will refine your pre-processing.

You'll want to abstract the pre-processing steps as much as possible. Regular expressions can be convenient here, and knowing how to use them correctly can increase the performance of your system. For example, in Python, compiling your regular expression before using them will improve performance, and taking advantage of a cache to avoid repetitive computations can save significant time as you process millions of rows.

If you're using dbt, macros are helpful to reduce code duplication, and as you find incremental improvements, you only have to apply them in one place.

Some common pre-processing steps I've seen are:

• Making everything lowercase and removing whitespaces

• Splitting an email into user and domain

• Cleaning company names to remove stop words such as ... Inc. ... LLC, The .., A ...

• Converting words such as null, na, n/a to actual NULLs

• Filtering out demo/test/internal users

• Parsing and cleaning websites and addresses

  

Indexing and Blocking

Once you have cleaned your data, the next step is to index the data to improve performance. Consider this example: you have 100,000 records in Database A and 10,000 in Database B, with no common indicator. How many comparisons are you performing if you look at the website and name?

The formula for this is (m * n) * p, where m and n are the number of records in each table, and p is the number of indicators. So we get 2,000,000,000 or two billion comparisons if we do the math, and that's on relatively small tables. You can see how this will not scale beyond a million records. 

Our only hope is to reduce the search space. Of course, we can't compare every single record against each other, but we can compare within a subset with a three-step process of blocking, indexing and reverse-indexing. 

Blocking is taking your entire record and chunking it into smaller blocks to avoid comparing every value with every other value. So the obvious question is, how do you block records together if you need to know how to match them? 

In the simplest example, you might block by comparing records from the same country, state, or zip code, or you could look at the first letter of a name. 

There are also algorithmic options. If your entities are names of people or companies, you could use various functions to reduce the search space by compressing information. 

Soundex is an example of an algorithm, developed a century ago for use on names. It encodes similar-sounding names, can be used as a blocking key and is supported in many data warehouses, including Snowflake.

select

soundex('pedram') as pedram,

soundex('pedrum') as pedrum,

soundex('peter') as peter,

soundex('pedro') as pedro

> PEDRAM PEDRUM PETER PEDRO 

   p365 p365 p360 p360

You can see that Peter and Pedro are blocked together, and Pedram and Pedrum are as well.

There are many different blocking techniques, and this survey paper reviews many of them, but the principle behind them is essentially the same. 

You could also use multiple blocking keys to improve accuracy. For example, you might run Soundex on first and last names and compare similar blocks across either first or last names. But, of course, the trade-off is always between performance and accuracy. 

Once you've defined your blocking function, you can apply it to every database record. This step is called indexing. For example, below, suppose we ran every name through a Soundex function. Each record has a Soundex associated with it.

Record 1 - D130
Record 2 - D130
Record 3 - F235
Record 4 - F235
Record 5 - D130

Next, we combine all records with the same blocking key into a subgroup for comparison purposes. To do this, we rely on a reverse index: for every blocking key, identify all the rows that belong to that block.

D130: {1, 2, 5}

F235: {3, 4}

In doing so, we can efficiently work on a block of similar records, and can even distribute this work in parallel. Once we have our reverse index, we are ready to proceed to Comparing.

Comparing and Classifying

I group comparing and classifying into one topic here because they are interrelated. Comparing is the act of summarizing the similarity between two records, and classifying is the act of deciding whether two records are 'similar enough.' 

There are many ways to compare two records. First, you can look at equality for any column, which is the most straightforward comparison.

if(a.name = b.name) then 1 else 0

On strings, you can look at how similar they are using similarity functions such as the edit distance or the Jaro-Winkler similarity score.

You can compare numbers by absolute or percent differences. You could look at dates, ages, times, or geographies. 

To classify, you can average the results from above to generate a score for each record. You can weigh individual fields, for example, giving greater weight to the last name over the first name. You might decide on a threshold that balances false positives and negatives or lean on machine learning or other more advanced techniques for classification.

There's no perfect way to compare and classify, but the goal is always the same: to create a list of tuple pairs of matched records.

One complication: records A and B might match, and records B and C might match, but records A and C might not. Therefore after matching, you need to process all the records to perform a merge. 

For example, suppose you have seven records and have compared them with some arbitrary matching formula. You end up with the following tuple pairs of matched records.

• {1, 2}

• {3, 4}

• {5, 6}

• {1, 7}

• {2}

• {6, 5}

• {7, 5}

If we graphed these pairs, it would look something like this:

However, since some pairs share a common record, we need to connect these pairs. So how do we do that? We use the aptly named connect components algorithm!

Using this algorithm, we reduce the above example to two distinct entities:

We identified two entities using those seven records and are tantalizingly close to the end: our last step is merging records.

Merging Records

Given two or more records that refer to the same underlying entity, we must decide how to merge the information. For example, if the names are different, what do we keep?

The first approach is randomly picking one as the master record and using all the information in that one. This approach is the easiest and least subtle, but it can often be sufficient for our needs.

A better approach is to rank sources on a per-record or per-field basis. So, for example, we might pick first and last names from Database A but use addresses from Database B. 

Both methods result in a loss of information, so another approach is called the Union Set. Essentially we keep all distinct elements across all records. At the very least, we want to keep the union set of table primary keys for better debugging. 

Suppose Database A has a record with primary key 123 and Database B has a record with primary key 456; we might merge these two records such that the primary key field is now {A: 123, B: 456}

Another option is to use ranges. If we are merging company information and have two different sources for the number of employees, we might include them as a range. If Record A had 100 employees, Record B had 250 employees, and Record C had 175, we might merge these two records as [100, 250]. 

You can imagine many other ways to merge records, but the goal is to preserve the right level of detail for your particular use case. 

Wrapping Up

Once merged, your work is largely done. You have created a set of candidates, classified them, linked records, and merged them together, but your journey has just begun. There is much more in this field to learn. Decisions on blocking keys, classification methods, and supervised/unsupervised learning, just to name a few.

You may also want to check out some libraries and products in this space, such as the Python Record Linkage library and the many available research papers on this topic.

Hope you enjoyed this deep dive, if there’s any topic you’re interested in, reach out and let me know!

The Servicing Framework Gap

"I'm the sole data person and consistently feel like I'm playing defense in service of countless requests. I don't have a great framework for answering questions as they come in."

This is a common refrain I've heard, especially as data teams are small, yet stakeholders seem never to stop growing. Part of the problem is that the framing of the question implies that your job is to answer requests. When you're the only data person at the company, answering questions should only be a small part of your job.

Forget everything you know about data, and pretend for a moment that you are a senior executive. What would you spend your days doing? If you're an executive worth keeping, your job is to invoke maximum impact on the organization, given your limited availability. You should be laser-focused on providing maximum leverage. A VP of Sales doesn't spend all day doing outbound cold calls, even though they might be quite capable of closing deals. When they are on the phone, it's to get a big deal over the finish line right before the end-of-quarter.

If you're the VP of Marketing and the product team came to you and said they need a post in two days on some feature that is about to launch, and this is the first you heard of it, what would you say? If you were told to send an outbound email and were given the subject line and content, would you take that well?

"But I'm not a VP of Data. I'm a data engineer. I'm a data analyst. I'm the manager of analytics."

Get over it. A title is what you list on your resume. Your job is what you make of it. Stop taking tickets if you don't want to be seen as a ticket-taker.

Okay, now that I've stepped off my soapbox. Here's some actionable advice.

Train Your Stakeholders

If you're a victim of unreasonable requests, you first need to work on the supply side. I find that stakeholders don't know how to ask data teams for help. They'll do everything except what you want them to do, whether it's a quick question for a number without context or a detailed description of how to pull data along with every column required.

Kelly Burdine, in a thread in Locally Optimistic, shares her team's Acceptance Criteria for requests. These fields in an intake form are the first step in teaching people how to ask data teams for help.

In many ways, you need to act like a PM. I asked the "so what problem are you trying to solve" question many times, and I've gotten nicknames based on it. It's tireless work, but it's a continuous reminder to teams that your job isn't to take orders; it's to help make impactful decisions using limited resources.

If a team can't articulate why they want access to data or what decisions they will make with the data, or if there's no clear indication that the request supports a goal or KPI, then it's time for the requestors to step back and spend more time thinking about what they want before coming to you. I know that sounds scary, but Hallowe’en is around the corner, so get comfortable with being spooked.

As your company and team grow, some type of intake form is inevitable. Still, you might be fine with a Slack channel for free-form question-asking or even email in the early days. A dedicated Slack channel for asking for help with data is a great place to generate awareness amongst your peers of the types of questions people ask. With Slack workflows, you can create simple templated forms to help structure the requests.

I recommend starting with the least amount of questions possible and increasing them over time. Striking the right level of friction is more art than science, although you can't go wrong with this question from Caitlin Hudon.

Prioritization

Inevitably, you will get more requests than you can answer with the time you have. You also need to realize that your job has more facets than time spent responding to questions. A prioritization framework can become complicated, often looking at multiple dimensions, such as urgency and impact. But, in the early days, prioritization can be simpler: focus on one executive you're going to support and ignore the rest.

It all comes back to the role you're in. You're here to have maximum impact. Start with the CEO and work your way down. If you can align your work with the CEO, then there's little anyone can say or do to attack your priorities. That's not to say you can't help the rest of the organization, but the more you can align your work with helping the CEO make better decisions, the better. In doing so, making a case for scaling your team will become easier. More on that soon.

Failures in Thinking Strategically

"How do I think strategically? I never have the time."

Hopefully, I've convinced you that you need to act like an executive, even if you aren't one. If so, then this next argument will be even easier for me to make.

If you are not spending time at work thinking strategically, you are not doing your job.

What does strategic thinking mean? Everyone has their definition of what strategy is. For me, a data team's strategy is thinking about the approach you will take to address the company's needs over the medium to long term.

It starts with thinking. If you do not have time to think, then you will not think strategically. Block time on your calendar. Get a notebook and leave your phone behind. Sit down and think for an hour about how well or poorly you are serving the company's goals and how you will serve them as it scales and grows. What is the vision for the company? What do the founders and executives talk about at all hands? What are the headwinds, and where are the pain points? Are we concerned with growth, retention, expansion, sales, churn, hiring, revenue, and costs?

Your strategy is both what you will focus on and what you will say no to. Be explicit, write it down, and share it with your boss. Get alignment.

Here's an example scenario.

We have been growing successfully over the last two years. Still, sales cycles have increased over the past quarter, and we missed our revenue target for the first time. There's a clear focus in the market on cost containment. We are planning on raising funding over the next 12 months. Our focus as a company over the next year will be to shift from logos to revenue, improving our sales efficiency and reducing spend where possible.

My strategy will be to align with our VP of Sales to get her everything she needs to hit our revenue goals. Our VP of Marketing will be my next closest ally. I will work with the two of them to help sharpen our understanding of our inbound leads through enrichment, optimize our pipeline by analyzing sales rep performance, and help figure out which channels drive the most revenue and which channels are underperforming. I may need to invest in certain tools or even outsource some help, especially when it comes to marketing channels.

As I won't have time to service the needs of the rest of the organization, I'll need to hire an analyst or double down on self-service capabilities.

When your strategy is set, you've solved many downstream problems for yourself. I'll often get a question like "what do I build first," "how do I plan for the future," or "how do I divide my time between answering questions and building scalable systems," but these questions are all symptoms of a lack of strategy.

Start with strategy, then the answers to these questions become more manageable. Without it, there's no framework for answering them well.

Loneliness is such a sad affair

And I can hardly wait to write SQL again.

I can't count the times I've looked longingly at engineering teams as a sole data hire. Yet, they are reviewing each other's PRs and discussing trade-offs and architectural decisions. Meanwhile, I'm commenting on my GitHub issues, trying to act like I have friends.

Loneliness is the one thing no one prepares you for as you build a career in data. It's why I think communities like dbt and Locally Optimistic flourish. We had to expand our reach to find just one other person who feels like we do. It's what makes Data Twitter, despite its flaws, so great. It's why I started a LinkedIn group for data practitioners.

My only advice here is to engage in the communities and create your own. I wouldn't be where I am today if I wasn't relentless in the questions I ask. I learn more from others than I hope to learn on the job.

There's only so much time I have available. I made myself available to three people for mentorship, and there were 27 others I had to say no to.

That's why my last request is if you've benefitted from a community and have the time to dedicate to even one person once a month, reach out and let me know. Hopefully, we can find more mentorship for the other 27.

Coming Up

That's it for Part 1. In Part 2, I'll cover three more common questions: Scaling a team, building a data culture, and the ever-nagging question: "am I doing this right?"

In January 2021, Base Case (an investor in what was then the headless BI version Supergrain) explored the future of headless BI as a solution to unbundling metrics from BI. In April 2021, Benn Stencil made a case for the metrics layer. In October that year, Drew opened an issue that generated more discussion. In December 2021, the Metrics Layer achieved keynote status at dbt Coalesce (with a long journey through the history of standardization).

Since then, Supergrain pivoted from a headless BI to a marketing tool (ok, a warehouse-native approach to customer engagement). Transform, which was a metrics engine, is shifting toward self-serve BI.

dbt continues development on the metrics layer, later renamed the semantic layer. In October at Coalesce, we’re likely to hear more on the metrics layer, although we’ve gotten the occasional update.

To date, Amit Prakash has done the best job exploring the metrics layer on Thoughtspot’s blog. In it, he describes six classes of metrics and three solutions for what a proper semantic layer could look like. I won’t go into all the details since the post already does a great job, and his writing is clear and approachable.

Instead, we’ll go one step closer to code and look at three implementations of the metrics layer and what a world without it looks like.

The Activation Metric

For the rest of this post, we’ll look at a metric that I think shows the true power of a well-defined metrics layer.

Pretend we’re a B2B SaaS, where users can sign up for our product and belong to one or more workspaces. Each workspace has one or more users. A workspace is active if they perform some activation event within 24 hours of workspace creation.

We’ll call the metric of interest activation rate, and we’ll define it as so:

The activation rate is the ratio of active workspaces to all workspaces over a certain period.

More concretely, every day, we have a list of all workspaces and a flag for whether that workspace was active on that day or not. The count of all workspaces on that day is the total number of workspaces. The count of all workspaces where the flag is true is the count of active workspaces.

We may want to report on the activation rate daily, weekly, or monthly. In addition, we’ll want to know the change in the activation rate over time.

First, in SQL

Let’s define everything in SQL to get a baseline. We’ll start with a basic table:

select reporting_day, workspace_id, is_active from workspace_details;

####

reporting_day | workspace_id | is_active
--------------|--------------|----------|
2022-07-04    | 100          | true
2022-07-04    | 101          | false
...

So far, so good. Now let’s count workspaces:

select

reporting_day,
count(distinct workspace_id) as n_workspaces,
sum(case when is_active then 1 else 0 end) as n_active_ws

from workspace_details
group by 1

####

reporting_day | n_workspaces | n_active_ws|
--------------|--------------|------------|
2022-07-04    |       2      |         1
...

Now, if we want to know the activation rate, we divide active over the total. For simplicity, we’ll pretend we’re using Snowflake, which allows us to refer to columns created in the same select statement.

select

reporting_day,
count(distinct workspace_id) as n_workspaces,
sum(case when is_active then 1 else 0 end) as n_active_ws,
n_active_ws / n_workspaces as activation_rate

from workspace_details
group by 1

####

reporting_day | n_workspaces | n_active_ws | activation_rate
--------------|--------------|-------------|---------------
2022-07-04    |       2      |         1   | 0.5
...

So far, so good. We could take this SQL, create a dbt model, and then use any reporting tool to visualize the activation rate over time. We can even start looking at change over time. But, first, let’s make the activation rates easier to read with some formatting.

select

reporting_day,
count(distinct workspace_id) as n_workspaces,
sum(case when is_active then 1 else 0 end) as n_active_ws,

round(100 * (n_active_ws / n_workspaces), 2) as activation_rate,

activation_rate - lag(activation_rate) over(order by reporting_day) as abs_change,
round(100 * abs_change / lag(activation_rate) over(order by reporting_day), 2) as pct_change


from workspace_details
group by 1
order by 1

####

reporting_day | n_ws | n_active | a_rate|abs_change|pct_chg
--------------|------|----------|-------|----------|-------
2022-07-04    |  2   |      1   | 50.   |    -     |   -     
2022-07-05    |  3   |      2   | 66.6  |  +16.6   | +33.3%   

...

We’ve made a ton of progress and haven’t needed to touch a metrics layer, so what’s the big deal? The real pain comes when your stakeholder now asks you for these numbers at a weekly, monthly, and quarterly aggregate. Pain is imminent.

What’s worse is if your end-users don’t understand how these measures are defined, they might start doing silly things like this:

select

date_trunc('month', reporting_day) as reporting_month,
avg(activation_rate) as avg_activation_rate

from...

Instead of finding the average over a period by adding the individual components and calculating the rate, they might average a ratio and end up with incorrect measures. We don’t want that.

Enter the Metrics Layer

A metrics layer solves these and other problems. Let’s look at how Looker approaches this.

Looker

In Looker, we define metrics in LookML files. We have a view, which represents a model of data, usually built from an existing table or view in the data warehouse. Here’s what that might look like:

view: workspace_activation {
  sql_table_name: "METRICS"."WORKSPACE_ACTIVATION"
    ;;

  dimension_group: date {
    type: time
    timeframes: [
      raw,
      time,
      date,
      week,
      month,
      quarter,
      year
    ]
    sql: ${TABLE}."REPORTING_DATE" ;;
  }

  dimension: is_active_workspace {
    type: yesno
    sql: ${TABLE}."IS_ACTIVE_WORKSPACE" ;;
  }

  dimension: workspace_id {
    type: string
    primary_key: yes
    sql: ${TABLE}."WORKSPACE_ID" ;;
  }

  measure: count_workspaces {
    type: count_distinct
    description: "# of Workspaces"
    sql:  ${workspace_id} ;; 
    filters: [workspace_name: "!='Demo Workspace'"]
  }

  measure: count_active_workspaces {
  type: count_distinct
  description: "# of Unique Workspaces Active within 1 Day"
  sql:  ${workspace_id} ;;
  filters:  [is_active_workspace: "yes"]
}


  measure: activation_rate {
  type:  number
  sql:  ${count_active_workspaces} / ${count_workspaces} ;;
  value_format_name: percent_1
}


}

A lot is going on here, but the thing to notice is that we are defining measures as formulas, not as fully-formed SQL tables. We also specify what formatting to use on measures, how to drill into the details when going from aggregate views to detailed views, how to aggregate measures, and all the ways we want to break down our reporting date.

From this code, Looker can dynamically generate the SQL needed without us having to worry about different granularities.

We could go further and start defining joins between this table and other tables, for example if we wanted to break down workspaces by paid vs. not-paid or attribution category.

Without a metrics layer, we’d have to anticipate and perform all these joins upfront. With a metrics layer, we can specify relationships between tables and let the BI tool join as needed, only on the columns the user requests. As a result, our users never need to consider what types of joins to use.

dbt Metrics

Let’s try and do the same with the dbt metrics layer to understand better what we’ve got. The big caveat is that dbt metrics are not yet complete and are undergoing active development. So things may change, and rough edges might need time to polish.

We’ll first define our metrics in the dbt yml file:

metrics:
  - name: count_workspaces
    label: '# Workspaces'
    model: ref('active_workspace')
    type: count_distinct
    sql: workspace_id

    timestamp: reporting_day
    time_grains: [day, week, month]
    filters:
      - field: workspace_name
        operator: '!='
        value: 'Demo Workspace


  - name: count_active_workspaces
    label: 'Active Workspaces'
    model: ref('active_workspace')
    type: count_distinct
    sql: workspace_id

    timestamp: reporting_day
    time_grains: [day, week, month]

  - name: activation_rate
    label: 'Activation Rate'
    type: expression
    sql: " 100.0* {{ metric('count_active_workspaces') }} / {{ metric('count_workspaces') }} "
    
    timestamp: reporting_day
    time_grains: [day, week, month]

And we’ll create a model that allows us to select from these new metrics:

select * from 
  {{ metrics.calculate(
   [metric('count_workspaces'), metric('count_active_workspaces'), metric('activation_rate')], 
    grain='week',
  )}}

There are a few key things to note here. In the Looker model, the organizing principle was a View, which contained dimensions, measures, and time-grains all within one namespaced View object. dbt took a different approach: metrics are self-contained units. Each metric must specify which dbt model it should run against, the timestamp column, and which time grains to support.

This approach is already leading us to some duplicated code for the three metrics above.

Another point to consider is the expression metric. We can’t refer to a metric there but need to wrap it in jinja, leading to jinja in YAML, which can be a parsing nightmare without a good IDE. While Looker’s IDE can parse, highlight, and show errors within your LookML code and expression, we don’t have that level of tooling for dbt.

You’ll also note that I am defining how to query my metrics within dbt using the dbt_metrics macro. For now, there’s no support for reading these metrics outside of dbt itself, although dbt has partnerships with BI tools, and I expect they’ll be announcing better ways to interact with dbt’s metrics layer soon enough.

Filtering is more clunky. In Looker, we provide an array of expressions to filter on, while in dbt we build our filters as yaml, explicitly defining what operator to use.

One final observation: there is no support for joins. In Looker, you can define relationships between different tables and explicitly define which related views should be available to a user within an Explore. Until support for joins arrives in dbt, it’s hard to see any value in an isolated semantic layer.

Lightdash

Lightdash is a BI tool that is tightly integrated within the dbt ecosystem. It offers two ways of expressing metrics: the first uses the native dbt metrics layer we discussed above. But beyond that, it also has a metrics implementation that you can leverage, which has some added benefits, such as joins and formatting.

Like dbt, you add your metrics implementation directly in your dbt yml file. However, the structure is a little different. The metrics are specified in a meta tag underneath the column. You can define multiple metrics related to a column in-line, and there is no need to duplicate dimensions across metrics.

version: 2
models:
  - name: active_workspace
    columns:
      - name: reporting_day
        description: "Day of report"
        meta:
          dimension:
            type: date
      - name: workspace_id
        description: "The Id of the workspace"
        meta:
          metrics:
            count_workspaces:
              type: count_distinct
            count_active_workspaces:
              type: count_distinct
              sql: "case when is_active then workspace_id else null end"
            activation_rate:
              type: number
              sql: (1.0 * ${count_active_workspaces} / ${count_workspaces})
              round: 2
              format: percent

We also have convenient helpers for rounding and formatting numbers. The templating is simplified, and there’s no reliance on dbt ref macros. We can directly specify a metric using the ${metric_name} format.

There are some downsides to the integration with dbt. Namely, any change to your metrics requires a full dbt refresh, which can be slow. There’s also the question of where a metric belongs: not all metrics should live under a particular dbt column definition; perhaps a separate metric definition file could be more maintainable long-term.

That said, the reporting is simplified quite a bit. It’s easy to query the metrics using the Lightdash UI, and there’s no need to write custom code to fetch a metric. But then, your metrics are only accessible within Lightdash, although this could be alleviated with APIs that make metrics more accessible beyond just Lightdash. Given the open nature of the product, I wouldn’t be surprised if metrics became more accessible over time.

What it all means

All three tools have different trade-offs, and their strengths and weaknesses tell of the challenges a metrics layer faces. Looker deeply integrates its metrics layer within the Looker ecosystem. Dimensions and measures are defined within the same application, and Looker’s semantic understanding of LookML allows for a rich parsing and developer experience. Looker can write to Git for version control, but most development occurs within the Looker ecosystem.

Despite its strength, there are also pitfalls. Measures defined within Looker are not easily accessed. While Looker exposes an API, we haven’t seen it become a standard metrics layer across the data stack, perhaps because the high entry price makes it prohibitive for smaller companies.

That said, a well-configured Looker instance can reduce the burden on data teams. Providing access to views your end-users can query without relying on data teams whenever you need just one more column can be powerful. That power has led to increased interest in a universal metrics layer solution.

With dbt, it’s clear that they are trying to stake their place within the data ecosystem as a natural fit for a universal metrics layer. Much of the modern data stack already integrates with dbt, and dbt is widely adopted and available to nearly any data team. However, dbt is also moving toward a cloud-based and server-based model, and full adoption of the metrics layer will likely involve some subscription requirements.

Pricing aside, the real challenge with dbt is delivering an ergonomic and performant solution. The current jinja/yaml-based definition of metrics, the lack of any significant development tooling, and a gap in features that would make it broadly applicable are still outstanding questions. 

Since it’s been announced, there has been very little news, although there’s still active development. Just last week, dbt changed the API by renaming some fields. Unfortunately, this active development also makes it difficult to recommend. Without stability, data teams will not likely want to develop against it.

Lightdash is in an exciting place as well. In some ways, they are trying to integrate with dbt and find a way to develop their own metrics definitions apart from it. Too much reliance on dbt can bring challenges, especially as there’s no clear roadmap on where the metrics layer will be going. On the other hand, saving your metrics definition next to your dbt code can have a lot of ergonomic benefits. The outstanding question is whether other apps can leverage the metric definitions. If not, Lightdash may approach Looker-status, another BI silo for metrics.

So the real question I have is this: Can a metrics layer be universal enough to gain applicability across the data stack yet still be designed in such a way to be relevant to BI tools?

We are still ways off from having an answer to that question, but I’m excited to see how we get there.

Get 20% off for 1 year

A Short History of Orchestration

Apache Airflow is part of a class of tools called an orchestrator, but to understand what it is and why people use it, we need to travel back a little bit to its origin and Airbnb.

Airflow was created in 2014 and released in 2015 at Airbnb. The original blog announcing the release is still up and is a good resource for reminding ourselves of where Airflow came up and what the world was like then.

At Airbnb, data engineers used tools like Apache Hive as a data warehouse, with much of their infrastructure built on Hadoop and Spark. There were many problems to be solved and jobs to be done: data extraction, cleaning, quality checks, and long-term storage.

Airbnb was also performing a lot of computation. They needed to know everything from how guests felt about their accommodations to how their hosts felt about their guests. They needed to understand how well their recommendations were doing and whether their experiments were working well. They needed to compute sessions from all the clickstream data on both their app and the web.

Read more

When Two Become One

Let’s say Rachel visited your site over the past few months. For simplicity, she was kind enough to persist cookies, use the same device across both sites, and generally be friendly toward your site tracking. We are using something like Rudderstack, Segment, Amplitude, Mixpanel, Jitsu, or Snowplow for event tracking.

Rachel clicks the giant, blinking, iridescent ✨sign-up✨ button your growth team so thoughtfully placed in the middle of your website. She signs up with her email address and creates a password. Somewhere, a growth marketer wakes from her dreams. Success.

If your engineering team was kind and generous, they also instrumented the sign-up event and the subsequent sign-in, and now you have three types of events, and they might look like this.

Our tracked events table, (let’s call it tracks), looks like this:

anonymous_id | event_name | event_time | user_id  | source
123          | viewed_page| 2022-06-01 | null     | web
123          | signed_up  | 2022-08-01 | aaaa-111 | nodejs
123          | signed_in  | 2022-08-01 | aaaa-111 | nodejs

A few things to note:

1. Your old rows don’t get updated when new information arrives. In June, we didn’t know the user id of our visitor, since they had not signed up. But in August we did have that information.

2. Events take place in different contexts. The first event was emitted from your marketing website on the front-end. The second and third were server-side events from the backend, directly to your event-tracker. The first event may not always fire, depending on ad-blocking, network blips or browser behaviour.

These nuances will make the lives of your data practitioners hard, so it’s important to have lots of sympathy and moral support for them when they inevitably start working on sessionization. Help is available, and they are not alone.

What Can We Do With Events?

Given just the three events, we can ask many different types of questions:

Attribution

What are the leading sources of user sign ups?

For people who signed up, what was the first page they visited on our marketing site? Or the last?

Adoption

How many people signed-up for my product each day?

How long does it take for an average visitor to sign-up?

What percent of visitors end up signing up for our product, and how does that change over time?

Engagement

How many people who sign-in to our product every day are new users?

How many of them are existing?

How many users stopped signing in?

How many users came back after a break?

Stitching User Events

Before we can start chipping away at our newly formed backlog of questions we still have to solve the fundamental problem of user stitching. We want to associate every event we have with the user id, even if the user was not known until later.

Given the simplified example above, we can create a mapping of anonymous_id → user_id by using a window function.

If you’re new to window functions, this can look daunting. Think of a window as a slice of a table. We want to operate on every row that has the same anonymous_id. In each slice, apply a function to get a result, and add it as a new column. In this case, we’re applying the last function, which finds the last row in that window.

Here’s a little illustration of how a window function might work. Start by taking the partition highlighted in yellow, then within each partition order by timestamp, and then take the last value in that partition, and use that as the result that fills each row.

Once the events have been stitched, we have our desired output: a mapping of anonymous id to user id.

Conversely, we also know that any anonymous id that isn’t mapped to a user id has not signed up, or cannot otherwise be identified.

anonymous_id | user_id 
123          | aaaa-111
124          | aaaa-111
125          | aaaa-111
234          | bbbb-222 
789          | NULL      <- this person has never signed up

With the above, we can now start to chip away at our questions from before.

Attribution

What are the leading sources of user sign ups?

For people who signed up, what was the first page they visited on our marketing site? Or the last?

What we know:

• all the visitors to our websites and how they got there (from Counting Users, Part 1)

• which visitors eventually became users

So naturally, we can look at where visitors came from before signing up to understand where our sign-ups come from.

There are many models for attributing sign-ups to visits (all of them bad, some of them useful). The simplest ones look at either the first, or last thing someone did before they performed a conversion. Consider the following events:

anonymous_id| path   | utm        | referrer | event_time
123         | /      | cpc-google | google   | 2 days ago
123         | /blog/ | NULL       | bing     | yesterday

We start by joining anonymous_id above to a stitching table we created in the previous step, and let’s say we find that the anonymous user 123 is actually user aaaa-1111.

Pretend we also have a table that tells us when the user signed-up, and it was today. We can either give credit to a cost-per-click advertisement (first-touch attribution) or to our blog (last-touch attribution).

We can get more complex if we wish. Maybe we only want to look back a certain number of days. For example, does it make sense to give credit to a paid ad from 18 months ago if someone signs up today?

We might want to categorize different types of web traffic according to some rules by bucketing similar traffic together, such as social media sources. (There’s a great dbt seed file in the segment package that helps with this)

We may want to go beyond attributing conversion events, to better understand what brings new visitors to our site for the first time. For every user, we could look at the first page they visited and categorize that traffic to understand ‘landing pages’.

Hard to believe, but the answers to every single one of these questions starts with just a few events and stitching.

Adoption

How many people signed-up for my product each day?

How long does it take for an average visitor to sign-up?

What percent of visitors end up signing up for our product, and how does that change over time?

Understanding adoption is also made possible by the same types of events we used for attribution. If we look only at the sign-in events, we can count how many people visit our site like so:

select 
date_trunc('day, timestamp) as event_day,
count(distinct user_id) 

from tracks
group by event_day

This query counts the number of distinct users within a specified time period. We use count distinct because a single user often has multiple events a day.

If we want to know how long it takes for a visitor to sign-up, we can look at the time elapsed between their first visit, and their sign-up event.

with conversions as (
  /* Assume one signed-up event per user for simplicity */
    select
    user_id,
    timestamp as signup_date

    from tracks 
    where event_name = 'signed-up'

)

select distinct

user_id,
first_value(timestamp) over(partition by stitched.user_id) as first_event_date,
signup_date,
datediff('days', first_event_date, signup_date) as days_to_signup

from tracks 
join stitched using(anonymous_id)

We use a window function again, this time to get the first event. We count the days between the first event and the conversion to see how long it takes for someone to sign up.

We can also perform a very rudimentary funnel analysis by counting the number of new visitors and sign-ups each day.

To help us, let’s imagine a helper column called blended_user_id. It is the user id if it’s known, or the anonymous id if not.

We find the first event ever for a particular blended user id, and then find the first sign-up event for each user. Count the number of times each of those events happen, every day, and get a funnel count of visitors → users.

with visitors as (
    select 

    date_trunc('days', timestamp) as day,
    count(distinct blended_user_id) as new_visitors

    from stitched_tracks
    group by day
    qualify row_number() over(partition by anonymous_id order by timestamp) = 1
),

signups as (

    select 

    date_trunc('days', timestamp) as day,
    count(distinct blended_user_id) as new_signups

    from stitched_tracks
    where event_name = 'signed-up'
    group by day
    qualify row_number() over(partition by anonymous_id order by timestamp) = 1
),

select 

day,
new_signups,
new_visitors

from visitors
full join signups using (day)

Engagement

How many people who sign-in to our product every day are new users?

How many of them are existing?

How many users stopped signing in?

How many users came back after a break?

We can even start to get into some fun churn and retention analysis. One really simple (and not useful) way to measure churn might be to count:

• Anyone who signed in today that signed in yesterday (retention)

• Anyone who signed in yesterday that didn’t sign in today (churn)

We’re using some really fun SQL now, by joining a single table to itself and offsetting the day in the join condition.

with daily_activity as (
  select distinct
    date_trunc('day', timestamp) as day,
    user_id
  from tracks
  where user_id is not null
),

retained as (
select
  today.day,
  count(distinct today.user_id) as retained
from daily_activity today
join daily_activity yesterday
  on today.user_id = yesterday.user_id
  and today.day = yesterday.day + interval 1 DAY
group by today.day
),

churned as (
select
  yesterday.day + interval 1 DAY as day,
  count(distinct yesterday.user_id) as churned
from daily_activity yesterday
left join daily_activity today
  on today.user_id = yesterday.user_id
  and today.day =  yesterday.day + interval 1 DAY,
where today.user_id is null
group by 1
)

select 
day,
coalesce(retained, 0) as retained,
coalesce(churned, 0) as churned

from retained
full join churned using (day)
order by 1

This example was taken with great inspiration from the Sisense blog, so feel free to give it a read to really understand what’s going on. Don’t sweat if this one makes your head hurt, the goal here is really to show you how much you can do with just a couple events.

I hope this was a useful foray into the depths you can go to with event streams. The world only gets more complicated from here as you try to do things like tie ad spend to revenue by connecting Salesforce Accounts to Product Signups through intermediary tables. Yuck! Let’s pretend we never spoke of such things.

Did you enjoy this post? Do you have ideas for future metrics to cover? Maybe you think Cohort Analysis is something you’ve always wanted to learn more, or you think there’s nothing hotter than a well-defined activation metrics. Well, leave a comment or drop me an email!

It’s a question as old as time: how many users we do we have? Well, that depends on what you mean by ‘users’, and ‘we’, and ‘have’.

If I were to press you to define them, you might have a few definitions for each. For example,

A user is:

• Someone who visited our website, or

• Someone who has logged in to our application

• Any account or customer within our CRM

We might mean:

• All the teams at our company

• Anything that the marketing team is responsible for

• Anyone that sales knows about

Have could mean:

• The number of users we have today

• The number of users we’ve ever had

• The number of users we’ve had on a given day, at that point in time, and subsequently into the future

Let’s dig into the first one for now. We’ll return to the second one later.

How many people visited our website?

Let’s take the first one: how do you know when someone visits our website? Well, we have event tracking, so our event tracker can tell us when someone views any of our pages. But what data does an event tracker provide? Let’s take Rudderstack’s standard schema and explore it further. When you save their data to your warehouse, you get something like this:

• anonymous_id: The user’s anonymous ID

• event: the name of the event

• context_ip: The IP address of the device

• context_<props>: Additional properties on the event

• id: the event’s unique id

• url / path: the URL and path where the event was captured

• timestamps: various timestamps with slight nuances that don’t matter here

It seems we’re in the clear. If we want to know how many people visited our website, we can just1 count the distinct number of anonymous IDs. This data stuff is easier than you think!

But, let’s push our curiosity out a little more. What..is an anonymous id? Well, we don’t have to go very far to find out. Rudderstack is open-source so we can find out for ourselves.

In here, storage refers to the device’s local storage where cookies are saved. If the ID already exists, then your anonymous id is what it was previously. But if not, then Rudderstack will generate one for you and save it to your cookies.

Interesting, so if you clear your cookies, then you get a new anonymous id. If you use a different browser, you get a new anonymous id. If you use private mode, you get a new anonymous id. If you use a different device, like your laptop, your work laptop, or your phone, then you get a new anonymous id. And with the war on cookies from Safari and Firefox, this problem is getting worse. Turns out one person can have many different anonymous ids.

Well, what about the IP address? Couldn’t we just use that to dedupe? Let’s think a bit more about that one too. How do devices get an IP address? Let’s not dive too deep, but from a router. But multiple people connect to the same router. Especially at work, or at school, or at the airport, or on public wifi. We could have many, many different people all on the same IP address.

Wow, maybe counting things isn’t so easy after all?

So what do we do? Well, in the absence of the right answer, we often have to make do with a good enough answer. Let’s say that when we count visitors to our website, we will count the distinct anonymous ids, knowing full well that that number over-inflates the true number of people visiting our website.

Our final code might look something like this

select count(distinct anonymous_id) from events;

How many net new people visited our website every week?

Okay, so we know how many ‘people’ visited our website. But that’s not actionable data. Is 20,000 good? Is 50,000 bad? What we almost always want with analytics is understanding change over time.

The first question to ask is what time-grain should we break down our data over time? We can count the number of users every minute, hour, day, week, month, or year.

Picking the right time-grain is context-dependent, but the choice revolves around having enough time for the noise to level off but not so much time that the results are not actionable. Also consider how often you will look at the data. Looking at data daily or more frequently than that is not healthy, or good for the soul, and I’m all about making sure you’re living a healthy, happy life.

Daily data is subject to fluctuations based on weekends and holidays. Monthly data is smoother, but it lacks immediacy. Do you want to wait 20 days to find out your website broke on the 10th? Let’s go with weekly, it smooths out the weekends and provides a nice balance.

The simple approach to counting people by week might look something like this:

select 

date_trunc('week', timestamp) as week, 
count(distinct anonymous_id) as visitors 

from events
group by 1

What we end up measuring here is the number of unique visitors to our website, every week. If Pedram and Claire both visit the website every week, but no one new shows up, well have a steady rate of 2 weekly users. Fine, but not exciting enough.

What we’re interested is how many new people are we bringing into our website. We want new people joining so we can create a healthy top-of-funnel pipeline to drive our marketing and sales motions. Without new people visiting, we’ll run out of sales, our company will die, and we will be sad forever. We’re all about happiness here.

So instead, let’s find out when we first saw a user:

select

date_trunc('week', timestamp) as week,
anonymous_id,
row_number() over(partition by anonymous_id order by timestamp) as event_date_index

from events;

This cute little row_number function does nothing more than count from 1 all the way down until there are no more rows. But, the magic is in the partition. A partition is nothing more than a group, so we’re asking our little function to count the number of times every user visited our website, from the 1st time, to the last time (we ordered from oldest visit to newest, but could also have done it in descending order with order by timestamp desc)

Now we can do something fun. We can find the first time a user visited our website by filtering that previous query.

with numbered_events as (
  select

  date_trunc('week', timestamp) as week,
  anonymous_id,
  row_number() over(partition by anonymous_id order by timestamp) as event_date_index

  from events
)

select 

week, 
count(anonymous_id) as new_visitors

from numbered_events
where event_date_index = 1
group by 1

We use a CTE2 because we can’t filter on row_number using WHERE or HAVING3

So we answered our first question and dug into it a little bit. We have a better sense of the limitations of the data, and why it’s hard, but that doesn’t mean it’s not useful. Every week we can keep an eye on our overall users and see how it is trending. We might even take our models further and use things like the referrer and UTM parameters to better understand not just how many users we have, but where they come from! On to our next question.

Next Time: How Many People Use Our Product?

Now that we’ve solved the first question, in our next one we’ll dig into our product itself. There, our users have authenticated, so counting should be easier. But we might end up with some more interesting questions, like how many of them use our product every day?

Did you enjoy this post? Do you have ideas for future metrics to cover? Maybe you think Cohort Analysis is something you’ve always wanted to learn more, or you think there’s nothing hotter than a well-defined activation metrics. Well, leave a comment or drop me an email!