I’ll explore the first part of every data journey: getting data out of a system. The story is likely to be a familiar one to many of you. Data about something you care about exists in some system, and you want to extract that data and store it somewhere. How hard can it be?

I’m going to skip over the pleasantries of why you might want to do this and pretend that we all understand it’s something that needs to be done. There are many types of systems that have data that we might want, but for our example, we will cover a common use case: application databases.

You can’t extract data without also putting it somewhere so we’ll also discuss the merits of two different strategies: saving the data in some structured format like a CSV or a Parquet file, or writing the data directly into a Data Warehouse.

Let’s get started

Querying Data

Given some source system, we will need to query that system to retrieve some subset of data. If our source system is a SQL database, then naturally, we’ll use SQL.

In the brute force method, we could extract all data from all tables that we’re interested in, ignoring system load and storage costs. This is a stateless process that is often overlooked but is rather simple. It might look something like this in Postgres:

\copy customers TO './customers.csv' CSV DELIMITER ','

We might soon realize while this is a simple method, it comes with some costs: namely it can be expensive to run. Our backend database may not appreciate the load. (You are running this against a replica, right?)

One solution is to only fetch data that has changed since your last run. This is known as an incremental data load.

We have two options:

1. Use Change Data Capture: Subscribe to a log of all database events and fetch all events that occurred after our last load.

2. Use a column that updates every time a row is updated, and only fetch records that have been updated since the last load.

The first option tends to be more accurate, but also a bigger pain to set up. It requires configuration of the database server, possibly even a restart. To really understand it, you’ll need to understand the WAL or write-ahead log in Postgres.

A quick detour on databases

The WAL is a mysterious place within a database.

While on the surface, a database appears to be a collection of tables that mirror a spreadsheet with many tabs, under the hood what is going on are sequences of events. The WAL is the ledger that keeps track of these events. Anytime you insert, update, or delete a row, a record of that transaction is kept just like it would be in an accounting ledger.

The balance of all these transactions, much like the bank balance you have, is the collective sum of all these events. While the WAL is used to ensure that in the event a database goes down, a record of everything that happened since the last backup is persisted, the WAL can also be used to sync data other systems, such as a backup replica database, or even, your silly little ETL job.

A plugin, such as wal2json, can translate these events into something a little more manageable, as we see here.

{
        "change": [
                {
                        "kind": "insert",
                        "schema": "public",
                        "table": "inventory",
                        "columnnames": ["id", "item", "qty"],
                        "columntypes": ["integer", "character varying(30)", "integer"],
                        "columnvalues": [1, "apples", 100]
                }
        ]
}
{
        "change": [
                {
                        "kind": "update",
                        "schema": "public",
                        "table": "inventory",
                        "columnnames": ["id", "item", "qty"],
                        "columntypes": ["integer", "character varying(30)", "integer"],
                        "columnvalues": [1, "apples", 96],
                        "oldkeys": {
                                "keynames": ["id"],
                                "keytypes": ["integer"],
                                "keyvalues": [1]
                        }
                }
        ]
}

Every change to a database emits an event, and nearly every database has their own way of emitting these events. There is no single standard for CDC so it is up to the downstream implementations to handle the varying logic. Now that you know what a WAL is, let’s pick an option.

Incremental Options

The WAL, while appealing, comes with many complexities. We’d have to store JSON data and process each row.

Instead, we’ll opt for using the updated_at column.

These are typically maintained by the database and automatically update whenever a row changes. Be cautious, sometimes they don’t update, and this can cause inconsistencies.

This simplifies our future queries quite a bit. Now we can filter on data updated since our last sync.

select * from customers
// if incremental run
where updated_at >= {{last_sync_date}}

Data Storage and Encoding

Once the data has been queried, it needs to be saved. The key decision here is whether to save the data in a binary or text format. You’re already familiar with text formats: CSV and JSON are the most popular. The benefits of text formats are that they are easy to read for humans, but this comes at the expense of efficiency and ambiguity around data types.

If you’ve ever had to parse the text “01-03-12” into a date format, you’ll appreciate the horrors of ambiguity. The structure of JSON also makes for large files as keys are repeated for every row.

Binary formats solve these issues by encoding data in a machine-readable format. Binary formats encode the data’s schema and types and offer more efficient storage of data. Parquet, for example, offers techniques like column-level compression and bit-packing. But nothing is every easy when it comes to data.

Parquet has a few well-defined standard types. Your typical types such as INT and FLOAT along with strings, dates, and timestamps are all well supported. The problem is your database may not have the same type system. For example, Postgres has the inet and cidr types which are ways of encoding network addresses.

Here you may wish to resort to trusty strings for unknown data types by default, allowing downstream to cast these as needed or simply use them as is.

To extract the data into parquet, you could use a wonderful little like pg2parquet which can take either a table or a query and output a parquet file, or even take advantage of DuckDB’s Postgres scanner.

With pg2parquet, you can extract an entire table in a single line:

 pg2parquet export 
     --table businesses \
     -H $POSTGRES_HOST \
     -U $POSTGRES_USER \
     --password $POSTGRES_PASSWORD \
     -o ./businesses.parquet \
     --dbname can_i_haz_replica

You’ll want to save this to a persistent storage through your cloud provider, such as S3 or GCS for safe-keeping. Don’t let your hard work go to waste!

Skipping Storage

Pedram, you might say, why store data twice? I could simply write each row to my data warehouse right away!

And you are correct, you could do this! In fact, if you think this is a good idea, I’d encourage you to try it. What you will eventually run into is a failure, and failures are never fun.

If you chose to write directly into the warehouse and something in the process failed, you’re left scratching your head. How do I debug this issue? What was the row that caused the issue? Can I even see what the data looked like? Why did I choose this career?

Instead, if you chose to write the data to an intermediate layer, you have some more options available to you. You can take the exact file that failed and inspect it, load it into a dev environment, scan for gremlins, and hopefully address them so that your pipeline becomes more robust to failures in the future. What a happy data engineer you’ve become!

Loading Data

Loading data can be fraught with difficulties. You will need to insert new rows, updating existing rows, and delete removed rows. All of these actions require a primary key.

The MERGE DDL command allows you to perform all of the above in one go.

MERGE dataset.Inventory T
USING dataset.NewArrivals S
ON T.product = S.product
WHEN MATCHED THEN
  UPDATE SET quantity = T.quantity + S.quantity
WHEN NOT MATCHED THEN
  INSERT (product, quantity) VALUES(product, quantity)

Issues arise however whenever schemas change. New columns in your source data means tables must be altered before an insert. Even worse, if a source data type changes then you may have a bigger problem on your hands.

It can often be helpful to add helper columns as you load data, such as a timestamp of when the current batch was loaded, in case rollbacks are needed.

Now do it all again

Let’s pretend this wonderful journey proceeded smoothly for you. The next question is, can you do it again tomorrow? Will you know if it ran successfully?

Subscribe now

Doing a task once is much easier than doing a task every day. In my next post, we will look at orchestration and scheduling, and all the other bits that come with.

Until next time!

Last month I got a sneak preview of Striim’s new developer experience that makes it easy to get started with CDC using BigQuery or Snowflake. If you missed my thread about that, check it out. In this post, I’ll look at how you can leverage Striim, Parquet, and DuckDB for real-time data ingestion with fast data analysis.

Data pipelines have traditionally been batch, and batch pipelines are usually easier to reason about. Data comes in once a day. I run all my transformations and load them sometime between 12:01 AM and 7 AM UTC (or was it PT? Timezones are hard.) Views and tables get updated, people look at data from yesterday, they get the answers to all their questions, and life is good. Life is simple. 

Unfortunately: the good old days are dead. Now we run operational workflows off data constantly fed into the data warehouse. We need to reduce the lag of all the various components that ingest, digest, transform, and reform our data as much as possible. For example, we have personalized workflows that send automated emails to prospects and customers who expect us to understand only every interaction they’ve had with us but also anticipate every human desire they could conceivably have in the next fifteen minutes. 

We’ve started pushing batch to the limits of streaming. Some of these batch tools can run as often as every 5 minutes, pushing the boundaries of what is and isn’t streaming anymore.

This is what got me interested in the streaming and CDC space in the first place. I wanted to know if there was a better way. After a dizzying stroll down Debezium Lane, and a confusing jaunt through Kafka Caverns, I received a nice demo from the fine folks at Striim. 

Striim is an enterprise-grade CDC platform, and I am but a lowly developer with toy examples, and it works just as well for me. For my first attempt in the tweet above, I set up a simple Postgres instance, piped data into it, and watched as Striim fed my BigQuery tables with change capture data every few seconds. 

But BigQuery is old news. Today, I wanted to see if I could get our lord and savior, DuckDB, to work with Striim. The setup was simple: use a GCP Writer to save streaming data to Parquet. Then, use DuckDB’s HTTPFS extension to read data from Parquet files in bulk. Write queries. Enjoy streaming.

Let’s dive in. 

Striim Setup

I decided to use one of the built-in data generators to get started quickly. These data generators are great for sketching out ideas since they let you avoid the messy parts of connecting data systems, such as permissions and IP allow-lists.

The ContinousGenerator can be set to various types of throughput. Low Throughput sends about ten messages per second, Medium for hundreds per second, or Spike for variable traffic with high spikes which can be handy for testing the resiliency of your pipelines.

Next up, I used a Query cell, which operates on an incoming stream and allows you to do transformations as data is produced. This can save lots of expensive compute in your warehouse by shifting the transformations left, closer to the data source. You can also do data-masking as data arrives, to make staying compliance easier. I wrote a simple query that takes the generated data, masks sensitive information, and outputs the results to a GDPR stream. 

Finally, to be able to analyze this data in DuckDB, I elected to write the data in Parquet format to Google Cloud Storage, although S3 would also work just as well. To do that, I used the GCP Writer Target. After creating a Service Account in Google Cloud, I setup a few basic settings such as the path to the bucket and format I’d like the files saved in.

One setting to be aware of is the Upload Policy, which determines how frequently (and conversely, how large) the files are. Finding a good balance here is important, as too many files or too few can both hinder performance.

I set the Upload Policy to write every 100,000 events or every 1 minute. I set the ParquetFormatter as the output option.

With the setup complete, all that is needed is for the app to be deployed and started. You even have a preview feature to watch data as it is fed through the system. You can see I’m fetching about 900 messages every second, and after about a minute the data will write to GCP. 

What’s neat is that Striim even displays the total End to End Lag so you can have insight into how delayed your pipelines are. In my case, the lag was about 30 seconds from creation to write.

After running for a while, the Parquet files are loaded in GCP and now it’s time to analyze the results with DuckDB.

Analyze with DuckDB

There are many ways to use DuckDB given that it’s a small portable binary. The CLI is a great place for simple prototyping but I prefer using Datagrip for writing queries. 

After creating a new DuckDB connection and enabling single-session mode, I added a small startup script to ensure that every time I connect to DuckDB my GCP credentials are entered.

The docs on setting up S3 or GCS access are pretty straightforward. A few simple SET commands and then you’re ready to query!

INSTALL httpfs;
LOAD httpfs; 
SET s3_endpoint='storage.googleapis.com';
SET s3_access_key_id='MY_ACCESS_KEY';
SET s3_secret_access_key='MY_SECRET';

To start, I ran a simple query to see how many records we have in each file. By using the filename=TRUE command, DuckDB returns the filename as a column in the table, which I use for aggregation.

SELECT
 
filename,
COUNT(1) AS n_records
 
FROM parquet_scan('s3://my-duckdb-bucket/striim-out.*', filename=TRUE)
GROUP BY filename
ORDER BY 1;

In about 7 seconds, DuckDB scanned 760,000 records across 14 files with 55,000 records each to generate a count of records by file. And the best part is there’s no Spark cluster to maintain. You can see below that using the filename to do a group by makes it easy to get a sense of how many records were written in each file.

We can even do fast text processing. In 6 seconds, I can categorize all products by whether they have Heavy or Lightweight in their name and aggregate across both dimensions.

SELECT
    product_name LIKE '%Lightweight%' AS is_lightweight,
    product_name LIKE '%Heavy%' AS is_heavy,
    COUNT(1) AS count_products
FROM parquet_scan('s3://my-duckdb-bucket/striim-out.*')
GROUP BY 1, 2;

The best part is this data is constantly updated by Striim. Every minute a new batch of 55,000 records arrives. 

That was fun, but we needed to go faster. Just for fun, I cranked up the generator to see how it would handle a higher rate and set the Upload Limit to 25,000 records per file. I easily hit 20,000 messages per second, and the end-to-end lag was just a few seconds. Striim had no problem with the throughput. In just a few minutes, I had 60 Parquet files ready for DuckDB to process.

With now 60 files to process, DuckDB took just under 30 seconds to count every record in every file. The product name query now took 23 seconds on 1.45 million records.

As one final test, I decided to push some regex and aggregates down to see the impact on performance, and DuckDB held up well. This query took under a minute to query all 1.45 million records, and I didn’t have to store a single file locally. (And if you were wondering, the average of the last 4 digits of a phone number is 5001).

SELECT
    date_trunc('minute', CAST(TIME AS TIMESTAMP)) AS DATE,
    avg(CAST(regexp_extract(Phone_Number, '\d+') AS NUMERIC)) AS avg_number
FROM parquet_scan('s3://my-duckdb-bucket/striim-out.*')
GROUP BY 1

Wrapping Up

I hope this was a helpful exploration of how you can use Striim and DuckDB to process real-time analytic queries quickly and easily. Gone are the days of Kafka, Zookeeper and Debezium. In less than 30 minutes you can get a CDC stream setup, write to a cloud bucket location, and query with DuckDB for blazing-fast analytics. 

If you want to give Striim a try, you can sign up here with my referral code tAlaDngxjQ.

If you prefer video, I gave a talk on Entity Resolution at dbt's office hours 2 years ago. If you prefer books, there's no other book I'd recommend more than Peter Christen's Data Matching. If you prefer my Substack, you’re in the right place.

Let's dive in.

What is Entity Resolution?

Entity resolution is all about combining multiple records of things. There are two parts to entity resolution: first is the entity, and second is the record of that entity in some database.

The entity can be anything from a person to a company to a physical product. I'll use companies as examples here, but the underlying logic applies to any entity you want to dedupe. 

A record of that entity might exist in a spreadsheet, a database, or across multiple databases. 

What's important is that there is no unique identifier representing that entity. If you were trying to dedupe people and had their Social Security Number or another national identification number, then the problem would be relatively easy. However, absent a single unique indicator, if we want to match or dedupe these records, then we need a way to resolve them to a single entity: hence, entity resolution.

An Illustrative Example

Let's say you work at a small B2B company with data in various systems of record: your production database, your Salesforce instance, and several spreadsheets of data with leads captured at various events. 

Anyone can sign in to your product by providing their company name and email address. Your Salesforce instance has accounts with company names, locations, websites, and contact information. The leads spreadsheet has similar information hand-captured by a person running the event.

It might look something like this:

How do you decide whether or not these different records are part of the same underlying entity? You might keep it simple and decide that two entities are the same if they share the same website, but even websites change over time. While a simple solution may be sufficient, you're entering the realm of entity resolution if you're not satisfied with that.

Share Pedram's Data Based

The 5 Steps to Entity Resolution

This deep dive will go through the five keys to entity resolution. There is much more nuance and depth beyond this post, but this should be enough to get us started.

1. Pre-processing

2. Indexing

3. Comparing

4. Classifying

5. Merging

Pre-processing

Before we embark on our journey, starting with a good foundation is essential. As much as possible, we want to clean our underlying data. Everything from trimming extra whitespaces to lowercasing all the characters, removing stop-words, or even stemming and lemmatizing is possible here. The specifics are highly context-dependent, and always an iterative process. As you perform these steps on data samples, your appreciation for what you need to do to improve the quality of matches will increase, and you will refine your pre-processing.

You'll want to abstract the pre-processing steps as much as possible. Regular expressions can be convenient here, and knowing how to use them correctly can increase the performance of your system. For example, in Python, compiling your regular expression before using them will improve performance, and taking advantage of a cache to avoid repetitive computations can save significant time as you process millions of rows.

If you're using dbt, macros are helpful to reduce code duplication, and as you find incremental improvements, you only have to apply them in one place.

Some common pre-processing steps I've seen are:

• Making everything lowercase and removing whitespaces

• Splitting an email into user and domain

• Cleaning company names to remove stop words such as ... Inc. ... LLC, The .., A ...

• Converting words such as null, na, n/a to actual NULLs

• Filtering out demo/test/internal users

• Parsing and cleaning websites and addresses

  

Indexing and Blocking

Once you have cleaned your data, the next step is to index the data to improve performance. Consider this example: you have 100,000 records in Database A and 10,000 in Database B, with no common indicator. How many comparisons are you performing if you look at the website and name?

The formula for this is (m * n) * p, where m and n are the number of records in each table, and p is the number of indicators. So we get 2,000,000,000 or two billion comparisons if we do the math, and that's on relatively small tables. You can see how this will not scale beyond a million records. 

Our only hope is to reduce the search space. Of course, we can't compare every single record against each other, but we can compare within a subset with a three-step process of blocking, indexing and reverse-indexing. 

Blocking is taking your entire record and chunking it into smaller blocks to avoid comparing every value with every other value. So the obvious question is, how do you block records together if you need to know how to match them? 

In the simplest example, you might block by comparing records from the same country, state, or zip code, or you could look at the first letter of a name. 

There are also algorithmic options. If your entities are names of people or companies, you could use various functions to reduce the search space by compressing information. 

Soundex is an example of an algorithm, developed a century ago for use on names. It encodes similar-sounding names, can be used as a blocking key and is supported in many data warehouses, including Snowflake.

select

soundex('pedram') as pedram,

soundex('pedrum') as pedrum,

soundex('peter') as peter,

soundex('pedro') as pedro

> PEDRAM PEDRUM PETER PEDRO 

   p365 p365 p360 p360

You can see that Peter and Pedro are blocked together, and Pedram and Pedrum are as well.

There are many different blocking techniques, and this survey paper reviews many of them, but the principle behind them is essentially the same. 

You could also use multiple blocking keys to improve accuracy. For example, you might run Soundex on first and last names and compare similar blocks across either first or last names. But, of course, the trade-off is always between performance and accuracy. 

Once you've defined your blocking function, you can apply it to every database record. This step is called indexing. For example, below, suppose we ran every name through a Soundex function. Each record has a Soundex associated with it.

Record 1 - D130
Record 2 - D130
Record 3 - F235
Record 4 - F235
Record 5 - D130

Next, we combine all records with the same blocking key into a subgroup for comparison purposes. To do this, we rely on a reverse index: for every blocking key, identify all the rows that belong to that block.

D130: {1, 2, 5}

F235: {3, 4}

In doing so, we can efficiently work on a block of similar records, and can even distribute this work in parallel. Once we have our reverse index, we are ready to proceed to Comparing.

Comparing and Classifying

I group comparing and classifying into one topic here because they are interrelated. Comparing is the act of summarizing the similarity between two records, and classifying is the act of deciding whether two records are 'similar enough.' 

There are many ways to compare two records. First, you can look at equality for any column, which is the most straightforward comparison.

if(a.name = b.name) then 1 else 0

On strings, you can look at how similar they are using similarity functions such as the edit distance or the Jaro-Winkler similarity score.

You can compare numbers by absolute or percent differences. You could look at dates, ages, times, or geographies. 

To classify, you can average the results from above to generate a score for each record. You can weigh individual fields, for example, giving greater weight to the last name over the first name. You might decide on a threshold that balances false positives and negatives or lean on machine learning or other more advanced techniques for classification.

There's no perfect way to compare and classify, but the goal is always the same: to create a list of tuple pairs of matched records.

One complication: records A and B might match, and records B and C might match, but records A and C might not. Therefore after matching, you need to process all the records to perform a merge. 

For example, suppose you have seven records and have compared them with some arbitrary matching formula. You end up with the following tuple pairs of matched records.

• {1, 2}

• {3, 4}

• {5, 6}

• {1, 7}

• {2}

• {6, 5}

• {7, 5}

If we graphed these pairs, it would look something like this:

However, since some pairs share a common record, we need to connect these pairs. So how do we do that? We use the aptly named connect components algorithm!

Using this algorithm, we reduce the above example to two distinct entities:

We identified two entities using those seven records and are tantalizingly close to the end: our last step is merging records.

Merging Records

Given two or more records that refer to the same underlying entity, we must decide how to merge the information. For example, if the names are different, what do we keep?

The first approach is randomly picking one as the master record and using all the information in that one. This approach is the easiest and least subtle, but it can often be sufficient for our needs.

A better approach is to rank sources on a per-record or per-field basis. So, for example, we might pick first and last names from Database A but use addresses from Database B. 

Both methods result in a loss of information, so another approach is called the Union Set. Essentially we keep all distinct elements across all records. At the very least, we want to keep the union set of table primary keys for better debugging. 

Suppose Database A has a record with primary key 123 and Database B has a record with primary key 456; we might merge these two records such that the primary key field is now {A: 123, B: 456}

Another option is to use ranges. If we are merging company information and have two different sources for the number of employees, we might include them as a range. If Record A had 100 employees, Record B had 250 employees, and Record C had 175, we might merge these two records as [100, 250]. 

You can imagine many other ways to merge records, but the goal is to preserve the right level of detail for your particular use case. 

Wrapping Up

Once merged, your work is largely done. You have created a set of candidates, classified them, linked records, and merged them together, but your journey has just begun. There is much more in this field to learn. Decisions on blocking keys, classification methods, and supervised/unsupervised learning, just to name a few.

You may also want to check out some libraries and products in this space, such as the Python Record Linkage library and the many available research papers on this topic.

Hope you enjoyed this deep dive, if there’s any topic you’re interested in, reach out and let me know!

In January 2021, Base Case (an investor in what was then the headless BI version Supergrain) explored the future of headless BI as a solution to unbundling metrics from BI. In April 2021, Benn Stencil made a case for the metrics layer. In October that year, Drew opened an issue that generated more discussion. In December 2021, the Metrics Layer achieved keynote status at dbt Coalesce (with a long journey through the history of standardization).

Since then, Supergrain pivoted from a headless BI to a marketing tool (ok, a warehouse-native approach to customer engagement). Transform, which was a metrics engine, is shifting toward self-serve BI.

dbt continues development on the metrics layer, later renamed the semantic layer. In October at Coalesce, we’re likely to hear more on the metrics layer, although we’ve gotten the occasional update.

To date, Amit Prakash has done the best job exploring the metrics layer on Thoughtspot’s blog. In it, he describes six classes of metrics and three solutions for what a proper semantic layer could look like. I won’t go into all the details since the post already does a great job, and his writing is clear and approachable.

Instead, we’ll go one step closer to code and look at three implementations of the metrics layer and what a world without it looks like.

The Activation Metric

For the rest of this post, we’ll look at a metric that I think shows the true power of a well-defined metrics layer.

Pretend we’re a B2B SaaS, where users can sign up for our product and belong to one or more workspaces. Each workspace has one or more users. A workspace is active if they perform some activation event within 24 hours of workspace creation.

We’ll call the metric of interest activation rate, and we’ll define it as so:

The activation rate is the ratio of active workspaces to all workspaces over a certain period.

More concretely, every day, we have a list of all workspaces and a flag for whether that workspace was active on that day or not. The count of all workspaces on that day is the total number of workspaces. The count of all workspaces where the flag is true is the count of active workspaces.

We may want to report on the activation rate daily, weekly, or monthly. In addition, we’ll want to know the change in the activation rate over time.

First, in SQL

Let’s define everything in SQL to get a baseline. We’ll start with a basic table:

select reporting_day, workspace_id, is_active from workspace_details;

####

reporting_day | workspace_id | is_active
--------------|--------------|----------|
2022-07-04    | 100          | true
2022-07-04    | 101          | false
...

So far, so good. Now let’s count workspaces:

select

reporting_day,
count(distinct workspace_id) as n_workspaces,
sum(case when is_active then 1 else 0 end) as n_active_ws

from workspace_details
group by 1

####

reporting_day | n_workspaces | n_active_ws|
--------------|--------------|------------|
2022-07-04    |       2      |         1
...

Now, if we want to know the activation rate, we divide active over the total. For simplicity, we’ll pretend we’re using Snowflake, which allows us to refer to columns created in the same select statement.

select

reporting_day,
count(distinct workspace_id) as n_workspaces,
sum(case when is_active then 1 else 0 end) as n_active_ws,
n_active_ws / n_workspaces as activation_rate

from workspace_details
group by 1

####

reporting_day | n_workspaces | n_active_ws | activation_rate
--------------|--------------|-------------|---------------
2022-07-04    |       2      |         1   | 0.5
...

So far, so good. We could take this SQL, create a dbt model, and then use any reporting tool to visualize the activation rate over time. We can even start looking at change over time. But, first, let’s make the activation rates easier to read with some formatting.

select

reporting_day,
count(distinct workspace_id) as n_workspaces,
sum(case when is_active then 1 else 0 end) as n_active_ws,

round(100 * (n_active_ws / n_workspaces), 2) as activation_rate,

activation_rate - lag(activation_rate) over(order by reporting_day) as abs_change,
round(100 * abs_change / lag(activation_rate) over(order by reporting_day), 2) as pct_change


from workspace_details
group by 1
order by 1

####

reporting_day | n_ws | n_active | a_rate|abs_change|pct_chg
--------------|------|----------|-------|----------|-------
2022-07-04    |  2   |      1   | 50.   |    -     |   -     
2022-07-05    |  3   |      2   | 66.6  |  +16.6   | +33.3%   

...

We’ve made a ton of progress and haven’t needed to touch a metrics layer, so what’s the big deal? The real pain comes when your stakeholder now asks you for these numbers at a weekly, monthly, and quarterly aggregate. Pain is imminent.

What’s worse is if your end-users don’t understand how these measures are defined, they might start doing silly things like this:

select

date_trunc('month', reporting_day) as reporting_month,
avg(activation_rate) as avg_activation_rate

from...

Instead of finding the average over a period by adding the individual components and calculating the rate, they might average a ratio and end up with incorrect measures. We don’t want that.

Enter the Metrics Layer

A metrics layer solves these and other problems. Let’s look at how Looker approaches this.

Looker

In Looker, we define metrics in LookML files. We have a view, which represents a model of data, usually built from an existing table or view in the data warehouse. Here’s what that might look like:

view: workspace_activation {
  sql_table_name: "METRICS"."WORKSPACE_ACTIVATION"
    ;;

  dimension_group: date {
    type: time
    timeframes: [
      raw,
      time,
      date,
      week,
      month,
      quarter,
      year
    ]
    sql: ${TABLE}."REPORTING_DATE" ;;
  }

  dimension: is_active_workspace {
    type: yesno
    sql: ${TABLE}."IS_ACTIVE_WORKSPACE" ;;
  }

  dimension: workspace_id {
    type: string
    primary_key: yes
    sql: ${TABLE}."WORKSPACE_ID" ;;
  }

  measure: count_workspaces {
    type: count_distinct
    description: "# of Workspaces"
    sql:  ${workspace_id} ;; 
    filters: [workspace_name: "!='Demo Workspace'"]
  }

  measure: count_active_workspaces {
  type: count_distinct
  description: "# of Unique Workspaces Active within 1 Day"
  sql:  ${workspace_id} ;;
  filters:  [is_active_workspace: "yes"]
}


  measure: activation_rate {
  type:  number
  sql:  ${count_active_workspaces} / ${count_workspaces} ;;
  value_format_name: percent_1
}


}

A lot is going on here, but the thing to notice is that we are defining measures as formulas, not as fully-formed SQL tables. We also specify what formatting to use on measures, how to drill into the details when going from aggregate views to detailed views, how to aggregate measures, and all the ways we want to break down our reporting date.

From this code, Looker can dynamically generate the SQL needed without us having to worry about different granularities.

We could go further and start defining joins between this table and other tables, for example if we wanted to break down workspaces by paid vs. not-paid or attribution category.

Without a metrics layer, we’d have to anticipate and perform all these joins upfront. With a metrics layer, we can specify relationships between tables and let the BI tool join as needed, only on the columns the user requests. As a result, our users never need to consider what types of joins to use.

dbt Metrics

Let’s try and do the same with the dbt metrics layer to understand better what we’ve got. The big caveat is that dbt metrics are not yet complete and are undergoing active development. So things may change, and rough edges might need time to polish.

We’ll first define our metrics in the dbt yml file:

metrics:
  - name: count_workspaces
    label: '# Workspaces'
    model: ref('active_workspace')
    type: count_distinct
    sql: workspace_id

    timestamp: reporting_day
    time_grains: [day, week, month]
    filters:
      - field: workspace_name
        operator: '!='
        value: 'Demo Workspace


  - name: count_active_workspaces
    label: 'Active Workspaces'
    model: ref('active_workspace')
    type: count_distinct
    sql: workspace_id

    timestamp: reporting_day
    time_grains: [day, week, month]

  - name: activation_rate
    label: 'Activation Rate'
    type: expression
    sql: " 100.0* {{ metric('count_active_workspaces') }} / {{ metric('count_workspaces') }} "
    
    timestamp: reporting_day
    time_grains: [day, week, month]

And we’ll create a model that allows us to select from these new metrics:

select * from 
  {{ metrics.calculate(
   [metric('count_workspaces'), metric('count_active_workspaces'), metric('activation_rate')], 
    grain='week',
  )}}

There are a few key things to note here. In the Looker model, the organizing principle was a View, which contained dimensions, measures, and time-grains all within one namespaced View object. dbt took a different approach: metrics are self-contained units. Each metric must specify which dbt model it should run against, the timestamp column, and which time grains to support.

This approach is already leading us to some duplicated code for the three metrics above.

Another point to consider is the expression metric. We can’t refer to a metric there but need to wrap it in jinja, leading to jinja in YAML, which can be a parsing nightmare without a good IDE. While Looker’s IDE can parse, highlight, and show errors within your LookML code and expression, we don’t have that level of tooling for dbt.

You’ll also note that I am defining how to query my metrics within dbt using the dbt_metrics macro. For now, there’s no support for reading these metrics outside of dbt itself, although dbt has partnerships with BI tools, and I expect they’ll be announcing better ways to interact with dbt’s metrics layer soon enough.

Filtering is more clunky. In Looker, we provide an array of expressions to filter on, while in dbt we build our filters as yaml, explicitly defining what operator to use.

One final observation: there is no support for joins. In Looker, you can define relationships between different tables and explicitly define which related views should be available to a user within an Explore. Until support for joins arrives in dbt, it’s hard to see any value in an isolated semantic layer.

Lightdash

Lightdash is a BI tool that is tightly integrated within the dbt ecosystem. It offers two ways of expressing metrics: the first uses the native dbt metrics layer we discussed above. But beyond that, it also has a metrics implementation that you can leverage, which has some added benefits, such as joins and formatting.

Like dbt, you add your metrics implementation directly in your dbt yml file. However, the structure is a little different. The metrics are specified in a meta tag underneath the column. You can define multiple metrics related to a column in-line, and there is no need to duplicate dimensions across metrics.

version: 2
models:
  - name: active_workspace
    columns:
      - name: reporting_day
        description: "Day of report"
        meta:
          dimension:
            type: date
      - name: workspace_id
        description: "The Id of the workspace"
        meta:
          metrics:
            count_workspaces:
              type: count_distinct
            count_active_workspaces:
              type: count_distinct
              sql: "case when is_active then workspace_id else null end"
            activation_rate:
              type: number
              sql: (1.0 * ${count_active_workspaces} / ${count_workspaces})
              round: 2
              format: percent

We also have convenient helpers for rounding and formatting numbers. The templating is simplified, and there’s no reliance on dbt ref macros. We can directly specify a metric using the ${metric_name} format.

There are some downsides to the integration with dbt. Namely, any change to your metrics requires a full dbt refresh, which can be slow. There’s also the question of where a metric belongs: not all metrics should live under a particular dbt column definition; perhaps a separate metric definition file could be more maintainable long-term.

That said, the reporting is simplified quite a bit. It’s easy to query the metrics using the Lightdash UI, and there’s no need to write custom code to fetch a metric. But then, your metrics are only accessible within Lightdash, although this could be alleviated with APIs that make metrics more accessible beyond just Lightdash. Given the open nature of the product, I wouldn’t be surprised if metrics became more accessible over time.

What it all means

All three tools have different trade-offs, and their strengths and weaknesses tell of the challenges a metrics layer faces. Looker deeply integrates its metrics layer within the Looker ecosystem. Dimensions and measures are defined within the same application, and Looker’s semantic understanding of LookML allows for a rich parsing and developer experience. Looker can write to Git for version control, but most development occurs within the Looker ecosystem.

Despite its strength, there are also pitfalls. Measures defined within Looker are not easily accessed. While Looker exposes an API, we haven’t seen it become a standard metrics layer across the data stack, perhaps because the high entry price makes it prohibitive for smaller companies.

That said, a well-configured Looker instance can reduce the burden on data teams. Providing access to views your end-users can query without relying on data teams whenever you need just one more column can be powerful. That power has led to increased interest in a universal metrics layer solution.

With dbt, it’s clear that they are trying to stake their place within the data ecosystem as a natural fit for a universal metrics layer. Much of the modern data stack already integrates with dbt, and dbt is widely adopted and available to nearly any data team. However, dbt is also moving toward a cloud-based and server-based model, and full adoption of the metrics layer will likely involve some subscription requirements.

Pricing aside, the real challenge with dbt is delivering an ergonomic and performant solution. The current jinja/yaml-based definition of metrics, the lack of any significant development tooling, and a gap in features that would make it broadly applicable are still outstanding questions. 

Since it’s been announced, there has been very little news, although there’s still active development. Just last week, dbt changed the API by renaming some fields. Unfortunately, this active development also makes it difficult to recommend. Without stability, data teams will not likely want to develop against it.

Lightdash is in an exciting place as well. In some ways, they are trying to integrate with dbt and find a way to develop their own metrics definitions apart from it. Too much reliance on dbt can bring challenges, especially as there’s no clear roadmap on where the metrics layer will be going. On the other hand, saving your metrics definition next to your dbt code can have a lot of ergonomic benefits. The outstanding question is whether other apps can leverage the metric definitions. If not, Lightdash may approach Looker-status, another BI silo for metrics.

So the real question I have is this: Can a metrics layer be universal enough to gain applicability across the data stack yet still be designed in such a way to be relevant to BI tools?

We are still ways off from having an answer to that question, but I’m excited to see how we get there.

Get 20% off for 1 year

A Short History of Orchestration

Apache Airflow is part of a class of tools called an orchestrator, but to understand what it is and why people use it, we need to travel back a little bit to its origin and Airbnb.

Airflow was created in 2014 and released in 2015 at Airbnb. The original blog announcing the release is still up and is a good resource for reminding ourselves of where Airflow came up and what the world was like then.

At Airbnb, data engineers used tools like Apache Hive as a data warehouse, with much of their infrastructure built on Hadoop and Spark. There were many problems to be solved and jobs to be done: data extraction, cleaning, quality checks, and long-term storage.

Airbnb was also performing a lot of computation. They needed to know everything from how guests felt about their accommodations to how their hosts felt about their guests. They needed to understand how well their recommendations were doing and whether their experiments were working well. They needed to compute sessions from all the clickstream data on both their app and the web.

Read more

When Two Become One

Let’s say Rachel visited your site over the past few months. For simplicity, she was kind enough to persist cookies, use the same device across both sites, and generally be friendly toward your site tracking. We are using something like Rudderstack, Segment, Amplitude, Mixpanel, Jitsu, or Snowplow for event tracking.

Rachel clicks the giant, blinking, iridescent ✨sign-up✨ button your growth team so thoughtfully placed in the middle of your website. She signs up with her email address and creates a password. Somewhere, a growth marketer wakes from her dreams. Success.

If your engineering team was kind and generous, they also instrumented the sign-up event and the subsequent sign-in, and now you have three types of events, and they might look like this.

Our tracked events table, (let’s call it tracks), looks like this:

anonymous_id | event_name | event_time | user_id  | source
123          | viewed_page| 2022-06-01 | null     | web
123          | signed_up  | 2022-08-01 | aaaa-111 | nodejs
123          | signed_in  | 2022-08-01 | aaaa-111 | nodejs

A few things to note:

1. Your old rows don’t get updated when new information arrives. In June, we didn’t know the user id of our visitor, since they had not signed up. But in August we did have that information.

2. Events take place in different contexts. The first event was emitted from your marketing website on the front-end. The second and third were server-side events from the backend, directly to your event-tracker. The first event may not always fire, depending on ad-blocking, network blips or browser behaviour.

These nuances will make the lives of your data practitioners hard, so it’s important to have lots of sympathy and moral support for them when they inevitably start working on sessionization. Help is available, and they are not alone.

What Can We Do With Events?

Given just the three events, we can ask many different types of questions:

Attribution

What are the leading sources of user sign ups?

For people who signed up, what was the first page they visited on our marketing site? Or the last?

Adoption

How many people signed-up for my product each day?

How long does it take for an average visitor to sign-up?

What percent of visitors end up signing up for our product, and how does that change over time?

Engagement

How many people who sign-in to our product every day are new users?

How many of them are existing?

How many users stopped signing in?

How many users came back after a break?

Stitching User Events

Before we can start chipping away at our newly formed backlog of questions we still have to solve the fundamental problem of user stitching. We want to associate every event we have with the user id, even if the user was not known until later.

Given the simplified example above, we can create a mapping of anonymous_id → user_id by using a window function.

If you’re new to window functions, this can look daunting. Think of a window as a slice of a table. We want to operate on every row that has the same anonymous_id. In each slice, apply a function to get a result, and add it as a new column. In this case, we’re applying the last function, which finds the last row in that window.

Here’s a little illustration of how a window function might work. Start by taking the partition highlighted in yellow, then within each partition order by timestamp, and then take the last value in that partition, and use that as the result that fills each row.

Once the events have been stitched, we have our desired output: a mapping of anonymous id to user id.

Conversely, we also know that any anonymous id that isn’t mapped to a user id has not signed up, or cannot otherwise be identified.

anonymous_id | user_id 
123          | aaaa-111
124          | aaaa-111
125          | aaaa-111
234          | bbbb-222 
789          | NULL      <- this person has never signed up

With the above, we can now start to chip away at our questions from before.

Attribution

What are the leading sources of user sign ups?

For people who signed up, what was the first page they visited on our marketing site? Or the last?

What we know:

• all the visitors to our websites and how they got there (from Counting Users, Part 1)

• which visitors eventually became users

So naturally, we can look at where visitors came from before signing up to understand where our sign-ups come from.

There are many models for attributing sign-ups to visits (all of them bad, some of them useful). The simplest ones look at either the first, or last thing someone did before they performed a conversion. Consider the following events:

anonymous_id| path   | utm        | referrer | event_time
123         | /      | cpc-google | google   | 2 days ago
123         | /blog/ | NULL       | bing     | yesterday

We start by joining anonymous_id above to a stitching table we created in the previous step, and let’s say we find that the anonymous user 123 is actually user aaaa-1111.

Pretend we also have a table that tells us when the user signed-up, and it was today. We can either give credit to a cost-per-click advertisement (first-touch attribution) or to our blog (last-touch attribution).

We can get more complex if we wish. Maybe we only want to look back a certain number of days. For example, does it make sense to give credit to a paid ad from 18 months ago if someone signs up today?

We might want to categorize different types of web traffic according to some rules by bucketing similar traffic together, such as social media sources. (There’s a great dbt seed file in the segment package that helps with this)

We may want to go beyond attributing conversion events, to better understand what brings new visitors to our site for the first time. For every user, we could look at the first page they visited and categorize that traffic to understand ‘landing pages’.

Hard to believe, but the answers to every single one of these questions starts with just a few events and stitching.

Adoption

How many people signed-up for my product each day?

How long does it take for an average visitor to sign-up?

What percent of visitors end up signing up for our product, and how does that change over time?

Understanding adoption is also made possible by the same types of events we used for attribution. If we look only at the sign-in events, we can count how many people visit our site like so:

select 
date_trunc('day, timestamp) as event_day,
count(distinct user_id) 

from tracks
group by event_day

This query counts the number of distinct users within a specified time period. We use count distinct because a single user often has multiple events a day.

If we want to know how long it takes for a visitor to sign-up, we can look at the time elapsed between their first visit, and their sign-up event.

with conversions as (
  /* Assume one signed-up event per user for simplicity */
    select
    user_id,
    timestamp as signup_date

    from tracks 
    where event_name = 'signed-up'

)

select distinct

user_id,
first_value(timestamp) over(partition by stitched.user_id) as first_event_date,
signup_date,
datediff('days', first_event_date, signup_date) as days_to_signup

from tracks 
join stitched using(anonymous_id)

We use a window function again, this time to get the first event. We count the days between the first event and the conversion to see how long it takes for someone to sign up.

We can also perform a very rudimentary funnel analysis by counting the number of new visitors and sign-ups each day.

To help us, let’s imagine a helper column called blended_user_id. It is the user id if it’s known, or the anonymous id if not.

We find the first event ever for a particular blended user id, and then find the first sign-up event for each user. Count the number of times each of those events happen, every day, and get a funnel count of visitors → users.

with visitors as (
    select 

    date_trunc('days', timestamp) as day,
    count(distinct blended_user_id) as new_visitors

    from stitched_tracks
    group by day
    qualify row_number() over(partition by anonymous_id order by timestamp) = 1
),

signups as (

    select 

    date_trunc('days', timestamp) as day,
    count(distinct blended_user_id) as new_signups

    from stitched_tracks
    where event_name = 'signed-up'
    group by day
    qualify row_number() over(partition by anonymous_id order by timestamp) = 1
),

select 

day,
new_signups,
new_visitors

from visitors
full join signups using (day)

Engagement

How many people who sign-in to our product every day are new users?

How many of them are existing?

How many users stopped signing in?

How many users came back after a break?

We can even start to get into some fun churn and retention analysis. One really simple (and not useful) way to measure churn might be to count:

• Anyone who signed in today that signed in yesterday (retention)

• Anyone who signed in yesterday that didn’t sign in today (churn)

We’re using some really fun SQL now, by joining a single table to itself and offsetting the day in the join condition.

with daily_activity as (
  select distinct
    date_trunc('day', timestamp) as day,
    user_id
  from tracks
  where user_id is not null
),

retained as (
select
  today.day,
  count(distinct today.user_id) as retained
from daily_activity today
join daily_activity yesterday
  on today.user_id = yesterday.user_id
  and today.day = yesterday.day + interval 1 DAY
group by today.day
),

churned as (
select
  yesterday.day + interval 1 DAY as day,
  count(distinct yesterday.user_id) as churned
from daily_activity yesterday
left join daily_activity today
  on today.user_id = yesterday.user_id
  and today.day =  yesterday.day + interval 1 DAY,
where today.user_id is null
group by 1
)

select 
day,
coalesce(retained, 0) as retained,
coalesce(churned, 0) as churned

from retained
full join churned using (day)
order by 1

This example was taken with great inspiration from the Sisense blog, so feel free to give it a read to really understand what’s going on. Don’t sweat if this one makes your head hurt, the goal here is really to show you how much you can do with just a couple events.

I hope this was a useful foray into the depths you can go to with event streams. The world only gets more complicated from here as you try to do things like tie ad spend to revenue by connecting Salesforce Accounts to Product Signups through intermediary tables. Yuck! Let’s pretend we never spoke of such things.

Did you enjoy this post? Do you have ideas for future metrics to cover? Maybe you think Cohort Analysis is something you’ve always wanted to learn more, or you think there’s nothing hotter than a well-defined activation metrics. Well, leave a comment or drop me an email!

It’s a question as old as time: how many users we do we have? Well, that depends on what you mean by ‘users’, and ‘we’, and ‘have’.

If I were to press you to define them, you might have a few definitions for each. For example,

A user is:

• Someone who visited our website, or

• Someone who has logged in to our application

• Any account or customer within our CRM

We might mean:

• All the teams at our company

• Anything that the marketing team is responsible for

• Anyone that sales knows about

Have could mean:

• The number of users we have today

• The number of users we’ve ever had

• The number of users we’ve had on a given day, at that point in time, and subsequently into the future

Let’s dig into the first one for now. We’ll return to the second one later.

How many people visited our website?

Let’s take the first one: how do you know when someone visits our website? Well, we have event tracking, so our event tracker can tell us when someone views any of our pages. But what data does an event tracker provide? Let’s take Rudderstack’s standard schema and explore it further. When you save their data to your warehouse, you get something like this:

• anonymous_id: The user’s anonymous ID

• event: the name of the event

• context_ip: The IP address of the device

• context_<props>: Additional properties on the event

• id: the event’s unique id

• url / path: the URL and path where the event was captured

• timestamps: various timestamps with slight nuances that don’t matter here

It seems we’re in the clear. If we want to know how many people visited our website, we can just1 count the distinct number of anonymous IDs. This data stuff is easier than you think!

But, let’s push our curiosity out a little more. What..is an anonymous id? Well, we don’t have to go very far to find out. Rudderstack is open-source so we can find out for ourselves.

In here, storage refers to the device’s local storage where cookies are saved. If the ID already exists, then your anonymous id is what it was previously. But if not, then Rudderstack will generate one for you and save it to your cookies.

Interesting, so if you clear your cookies, then you get a new anonymous id. If you use a different browser, you get a new anonymous id. If you use private mode, you get a new anonymous id. If you use a different device, like your laptop, your work laptop, or your phone, then you get a new anonymous id. And with the war on cookies from Safari and Firefox, this problem is getting worse. Turns out one person can have many different anonymous ids.

Well, what about the IP address? Couldn’t we just use that to dedupe? Let’s think a bit more about that one too. How do devices get an IP address? Let’s not dive too deep, but from a router. But multiple people connect to the same router. Especially at work, or at school, or at the airport, or on public wifi. We could have many, many different people all on the same IP address.

Wow, maybe counting things isn’t so easy after all?

So what do we do? Well, in the absence of the right answer, we often have to make do with a good enough answer. Let’s say that when we count visitors to our website, we will count the distinct anonymous ids, knowing full well that that number over-inflates the true number of people visiting our website.

Our final code might look something like this

select count(distinct anonymous_id) from events;

How many net new people visited our website every week?

Okay, so we know how many ‘people’ visited our website. But that’s not actionable data. Is 20,000 good? Is 50,000 bad? What we almost always want with analytics is understanding change over time.

The first question to ask is what time-grain should we break down our data over time? We can count the number of users every minute, hour, day, week, month, or year.

Picking the right time-grain is context-dependent, but the choice revolves around having enough time for the noise to level off but not so much time that the results are not actionable. Also consider how often you will look at the data. Looking at data daily or more frequently than that is not healthy, or good for the soul, and I’m all about making sure you’re living a healthy, happy life.

Daily data is subject to fluctuations based on weekends and holidays. Monthly data is smoother, but it lacks immediacy. Do you want to wait 20 days to find out your website broke on the 10th? Let’s go with weekly, it smooths out the weekends and provides a nice balance.

The simple approach to counting people by week might look something like this:

select 

date_trunc('week', timestamp) as week, 
count(distinct anonymous_id) as visitors 

from events
group by 1

What we end up measuring here is the number of unique visitors to our website, every week. If Pedram and Claire both visit the website every week, but no one new shows up, well have a steady rate of 2 weekly users. Fine, but not exciting enough.

What we’re interested is how many new people are we bringing into our website. We want new people joining so we can create a healthy top-of-funnel pipeline to drive our marketing and sales motions. Without new people visiting, we’ll run out of sales, our company will die, and we will be sad forever. We’re all about happiness here.

So instead, let’s find out when we first saw a user:

select

date_trunc('week', timestamp) as week,
anonymous_id,
row_number() over(partition by anonymous_id order by timestamp) as event_date_index

from events;

This cute little row_number function does nothing more than count from 1 all the way down until there are no more rows. But, the magic is in the partition. A partition is nothing more than a group, so we’re asking our little function to count the number of times every user visited our website, from the 1st time, to the last time (we ordered from oldest visit to newest, but could also have done it in descending order with order by timestamp desc)

Now we can do something fun. We can find the first time a user visited our website by filtering that previous query.

with numbered_events as (
  select

  date_trunc('week', timestamp) as week,
  anonymous_id,
  row_number() over(partition by anonymous_id order by timestamp) as event_date_index

  from events
)

select 

week, 
count(anonymous_id) as new_visitors

from numbered_events
where event_date_index = 1
group by 1

We use a CTE2 because we can’t filter on row_number using WHERE or HAVING3

So we answered our first question and dug into it a little bit. We have a better sense of the limitations of the data, and why it’s hard, but that doesn’t mean it’s not useful. Every week we can keep an eye on our overall users and see how it is trending. We might even take our models further and use things like the referrer and UTM parameters to better understand not just how many users we have, but where they come from! On to our next question.

Next Time: How Many People Use Our Product?

Now that we’ve solved the first question, in our next one we’ll dig into our product itself. There, our users have authenticated, so counting should be easier. But we might end up with some more interesting questions, like how many of them use our product every day?

Did you enjoy this post? Do you have ideas for future metrics to cover? Maybe you think Cohort Analysis is something you’ve always wanted to learn more, or you think there’s nothing hotter than a well-defined activation metrics. Well, leave a comment or drop me an email!