The Airbyte playbook: why data sprawl leads to short-sighted decisions

If your data isn’t accessible and actionable, you’ll never fully understand what fuels your company’s growth.

No one understands the importance of data better than the team at Airbyte, the creators of the fastest-growing, open-source data integration platform. Airbyte’s ELT platform addresses all data movement needs — whatever the scale. 

We sat down with Michel Tricot, Co-Founder and CEO of Airbyte, to unpack the platform’s core functionality: enabling data movement from any source to any destination.

Our conversation dives into topics like:  

1. How the pandemic inspired a major (and beneficial) pivot for Airbyte
2. Why mitigating data sprawl is crucial to overall company health
3. Doing the work to understand your end user’s pain points

“We observed a pain point where people had data across operating systems — without interoperability. Airbyte solves this by making your data interoperable.”  

How COVID Forced Airbyte to Pivot for the Better

The Airbyte platform was initially called Dataline. But, after a massive brand overhaul during early COVID, it transformed into the ELT platform we know today. 

The team could pivot so quickly because they took a long, hard look at its metrics — and were honest with themselves about the early shortcomings in the numbers. 

The original idea before Airbyte was an integration product for marketing teams that demonstrated decent numbers and revenues. But the moment COVID hit, the traction plummeted as marketing budgets decreased due to economic uncertainty.

In response, the team refocused on newer ideas and pivoted to an open-sourced product: Airbyte. The adoption went really fast, and now, their Cloud offering is helping them progress even further.

Ultimately, they were able to pivot and transform their product for the better because: 

1. They were solving a real problem, and the market forced them to grow.
2. They had the capital to execute change.

Let the Data Guide You — Not the Other Way Around

The markets during that era of COVID were cutthroat, but they provided Michel and his team with much-needed clarity on their growth trajectory. 

They realized that if they continued on their path, they’d end up wasting five years on an ultimately mediocre product. This wake-up call began once COVID did; the market shakeup across the board rendered all of their previous metrics irrelevant.  

The team had to stop and take stock. How could they take a good long look at their company's health and big-picture value? They asked questions like: 

• Are people installing and actually using the product? 
• Does the product have a revenue-generating paid component? 

“If your metrics are weak: Be honest with yourself and change your operation. Honesty is key because you can make the numbers tell you whatever you want.” 

Identify Systemic Pain Points through Bottom-Up Research

In return to Entrepreneurship 101, Airbyte ID’d their target pain point through bottom-up research, speaking to as many industry members as possible. 

They called data engineers, data analysts, marketers, and everyone in between. Through these conversations, they noticed that technical workers kept raising the same issue: 

Despite existing solutions, bringing data into warehouses was still a massive struggle. 

Michel and his team ran with that insight and began crafting a solution. But, first, they reached out to even more users to glean info on data transfer. 

Collecting that much knowledge and parsing through all of it is (frankly) a grind. So they launched with one idea, then pivoted to solve pain points they discovered along the way. 

Airbyte’s Discovery: Data Interoperability Issues

During this market research phase, a set of circular issues was brought up consistently: 

1. People hate building connectors, and maintaining them is even more of a burden. 
2. Existing solutions to this were incomplete. Even if teams bought a solution, they usually had to build additional new connectors, which took months internally. 

Even if engineers were willing to grapple with these blockers, the products themselves often weren’t designed to be easily adopted by companies: 

1. Some engineers could not justify the cost of these data solutions to their teams.
2. Others wanted to use these solutions, but internal company rules and processes didn’t permit it.

With this, Airbyte created a product addressing each of these key complaints. 

“Speaking with all of these industry players creates a baseline for the problem you’re solving. From there, you can double-down and execute on those insights.” 

{{demo}}

How Data Sprawl Leads to Short-Sighted Decisions 

Each team uses tools specific to their operational needs and efficiencies within a company. Each tool doesn't necessarily integrate well with other teams' applications. Companies can easily use over 100 SaaS tools across all of their teams. 

As a result, attempting to aggregate data from all these tools gives companies trouble. 

Data warehouses are simple to create and use but lack the insights component. Plus, a lack of interoperability makes it difficult for companies to solve issues that loop in several teams. 

That’s where Airbyte comes into play: to make all data accessible and actionable. When companies leverage data in silos, they make narrow and incremental decisions. For instance, someone might optimize for a factor on the marketing team without considering its broader impact on sales. When every department operates independently, the loss of efficiency is seismic. 

Little information silos everywhere are the opposite of strategic decision-making. Data must exist in a unified, accessible place, so everyone has the same informational baseline. 

“When it comes to data, it needs to be unified to see the whole picture. If it remains siloed, you end up making localized decisions.” 

Integration Pain Points: APIs & Internal Data Platforms

API integration is, at its core, a maintenance problem. 

You need to tinker with it forever to ensure it stays up and running. 

That’s why open source is crucial to Airbyte: They’re building the engine to solve the long-standing issue of scaling data connector maintenance. 

Put simply: Data connectors always fail. Some do so more or less often, but they always fail. So the question is: Do you grow complacent or build a resilient system to avoid these failures? 

On this front, Airbyte is a resilient platform that simplifies the API experience. The more tools a company integrates, the more likely the system is to break.  

Airbyte has a dedicated internal team, an entire community of data engineers prepared to come in and fix connectors, and more. They currently have a catalog of over 200 connectors and are adding more every month. 

If You Build an Internal Data Integration — Be Ready to Maintain It

People don’t initially realize the downstream costs of creating an internal data integration platform. After all, the drain of constant maintenance creeps up on you. 

Building the platform is simple enough, and the first repair isn’t frustrating. 

But then, a colleague asks for more data. Then, the data they receive isn’t correct, so you start monitoring things. Even the script stops working, so you have to monitor that too. 

Bit by bit, that data integration system creeps up to waste time.  

Connector maintenance is a top priority at Airbyte, and they aren’t doing it alone. While Airbyte has a team of software engineers in house to address connector bugs and feature updates, they know they can’t maintain the long tail of connectors alone. So one initiative Airbyte has just launched is the community Maintainer Program, where community members can earn bounties by maintaining connectors (submitting PRs, reviewing PRs, and contributing to Docs).

“The wasted time is small and incremental at first. You don’t immediately realize your entire team is spending time on a failing data integration system.”

{{demo}}