Overview

At Pine, we have a wide variety of data ranging from personal finance information, internal and external communications and real estate data including Multiple Listing Service (MLS) data from across Canada. In this post, we’ll explore how this data is generated and stored, and how it enables smarter business decisions and AI-powered oversight of thousands of communications.

Data Creation

There are many ways data is created within our system. The first one is through our mortgage platform. As users apply for a mortgage, they are prompted to enter data such as their name, income and property information. Users can also provide their SIN number and give permission to pull their credit report. All of this data is stored within an Postgres16 Database hosted on AWS RDS.

The flow of this data can be represented the following:

The next set of data we store is client communications. This data includes emails through Gmail, phone conversations, and text messages.

Each week we process over 6,000 emails through a webhook powered by Front. We save both full HTML and raw text copies of each email along with the recipient and thread ID. This enables us to fully rebuild email threads for use in LLMs and internal systems.

Calls and texts follow a similar flow. When a text is sent or received, or a call is completed, Dialpad triggers a webhook that sends the event to our backend services. Call recordings are saved as MP3 files in an S3 Bucket. This powers our internal communication feedback product.

Once an email, call, or text is stored in Postgres, we publish an asynchronous event to our communications monitoring system. For calls, this system retrieves the MP3 file from S3 and transcribes it using OpenAI’s Whisper model:

Our communications monitoring system performs a range of asynchronous checks across all inbound and outbound communications. Two features of this system that we’ve noticed high impact from are:

• Customer Sentiment Monitoring: We use sentiment analysis on transcribed calls, emails, and text messages to detect when a customer may be upset. When flagged, these interactions are escalated to our internal management team for review and resolution.

• Follow-up Monitoring: Given our high volume, this monitoring functionality searches for clients who are missing a response according to Pine’s SLA. To reduce noise, we leverage LLMs to assess context, for example, a simple “thank you” reply won’t trigger a follow-up alert.

This system has vastly improved our Customer Satisfaction (CSAT) and Net Promoter (NPS) Score.

Product Improvements

To improve product usability, we use Amplitude to track user sessions and interaction events across our platform. These events are carefully instrumented throughout the product and pushed into Google BigQuery, where they form the basis for deeper analysis around engagement, conversion, and friction points. This data helps us iterate quickly and make informed decisions. We’ll revisit how we use BigQuery in more detail later.

Real Estate and MLS Data

The last set of data (and by far the largest) is MLS property data. As a licensed real estate brokerage, Pine has access to both active and historical listings across the country. This dataset powers Pine Homes, our real estate discovery platform.

For MLS data, we leverage MongoDB as the application store. We chose MongoDB for its flexibility and scalability. When it comes to MLS data, schemas can be different depending on the regional real estate board it comes from.

We leverage MongoDB’s change streams to sync changes from our application layer to BigQuery on an hourly basis. As part of this pipeline we also normalize parts of the data into the proper schema for ease of querying by our data analytics team.

Moving to Big Query

When we first began exploring internal analytics, we used Zoho Analytics. At the time, it was a natural extension of our CRM setup, which was also based in the Zoho ecosystem.

Our biggest issue with Zoho Analytics was its ability to scale. It’s a great tool for light analytics but struggles to hold up against more robust applications. This is how we landed on BigQuery. Given BigQuery’s easy set-up and severless design, we don’t need to worry about handling the infrastructure side of our analytics database which is valuable to a growing team.

Most of our data is synced into BigQuery using postgres’ publication slots. As writes happen within our database, data is streamed to these publication slots. We use Google Datastream to sync the data from the publication slot into BigQuery.

One of the most meaningful outcomes is our ability to view the customer journey end-to-end. By connecting mortgage application data with communication patterns and property interests, we can deliver more personalized, timely support at every stage of the homebuying process.

Note: We’ll be releasing a deep dive into our switch from Zoho to BigQuery, If you’re deciding between these two platforms make sure you’re following to catch it.

What’s Next

As we continue to scale, our next priority is activating this data across the team. This includes:

• Expanding our use of LLMs to increase Customer Satisfaction (CSAT)

• Develop data-driven systems to improve product pricing models

• Train models to more accurately predict home buying and valuation data

We’re going to continue investing into making this data easier to action in the hopes of moving faster on thoughtful, data-driven decisions.

Acknowledgements: I’d like to give special thanks to Jonathan Shih and Jasper Hui for their help on this project.

Subscribe now