How we made a platform for analysts to build data warehouses and pipelines

At RevInsight, we are building an AI-generation platform that improves revenue operations. Good data lies at the heart of intelligent decisions and execution. Data that is dependable, meaningful, and linked across silos in data warehouses. Analytics and AI need this.

Analysts, database specialists, and data engineers make data warehouses and pipelines. Which takes huge costs and efforts.

Faced with unaffordable costs and timelines, business leaders make do with siloed systems. They trade off the benefits of linked-up data and 360-degree views. They continue to add new systems and data to keep up with the competition. Then they work harder to manage ever more fragmented and inconsistent data.

Can leaders afford 360-degree views that link data across all silos? They need it to transition into the AI-enabled future! For this, we must cut the cost and effort required. We asked: Why can’t analysts make data pipelines and data warehouses themselves, fast and easy?

The problem is scope and complexity. Analysts get to the limits of the system and then turn to engineers to develop the rest. Functionality needs keep growing. Development of the “fast and easy” platform never ends. The data warehousing industry has improved, but it needs a 10x improvement.

What gets us to a 10x improvement?

First: a pattern-language for data warehouses and pipelines. Provide the required scope, specificity, consistency, modularity, and extensibility.

Second: a build and run platform for the pattern language. Make it fast and easy for analysts to build data warehouses and pipelines. Automate deployment and operation. Make it multi-tenant, affordable, robust, scalable, and maintainable.

Third: we limit scope. Focus on delivery of customer commitments.

A well-designed pattern language. Disciplined platform-build. And limited scope. The platform makes it fast and easy for analysts to make data warehouses and pipelines. Each analyst gets to do more, explore further, iterate faster, and learn-by-doing. Our analysts gain productivity, quality, and innovation. They become better at closing the gap between business needs and data availability.

Our engineers focus on extending the platform. Integration with more business systems. Streamlining the entire process.

Our customers are innovative leaders, and they see a momentous change coming. Good data. Data that is dependable, meaningful, and linked across silos. The foundation to unify their revenue operations. To enable AI and analytics.

Why not rent and automate an existing iPaaS? Expert analysts and programmers can encode their knowledge in existing iPaaS platforms. The goal is to get to maximize the use of AI to set up and run the data supply chain (the creation of pipelines, customer onboarding,  maintenance, updates, and operations). The options: (1) integrate automation with an existing iPaaS and face the business hurdle that key functionality of the iPaaS may not be exposed for a third-party software company or (2) harness a curated collection of open-source tooling into a data supply chain platform so that there is no business hurdle for ever-increasing automation.

What are the pros and cons of each approach?

• Approach 1: integrate automation with an iPaaS. This strategy leverages a commercial proprietary iPaaS platform as the foundation for the data supply chain. Build or buy automation tools that sit on top of it.
  • Pros: The iPaaS handles complex, low-level tasks like connection management, security, and runtime orchestration. Focus on building the semantic models and business logic rather than worrying about infrastructure. Connectors, templates, and low-code/no-code interfaces accelerate the initial development of pipelines. Make an automation layer to manage these rented components.
  • Cons: the killer “con” is the business hurdle. No iPaaS vendor exposes their “secret sauce” components with a comprehensive API for third-party automation. APIs are limited to what the vendor deems necessary for their ecosystem. You get locked into the iPaaS’s automation capabilities and development priorities, limiting the ability to innovate with AI tooling. The entire data supply chain becomes dependent on a single vendor. If they raise prices, change their platform, or go out of business, the cost and effort of migrating can be immense. Upgrades and bug-fixes made by the iPaaS can break the automations, leaving you scrambling to rebuild your vital data supply chains. And the enterprise-grade iPaaS solutions are expensive, with pricing often based on data volume, number of connections, or other usage metrics. This can become prohibitive as you scale to hundreds or thousands of customers.
• Approach 2: build a data supply chain platform from a curated collection of open-source tools.
  • Pros: complete control over every component with no business hurdles or dependency on 3rd party APIs. We can deeply integrate AI and automation into the core of the platform, as we have access to the underlying code. We can design it to be cost-effective.
  • Cons: carry the engineering burden of development, maintenance, and operations. This burden feeds back to the “pro” of learning and automating to further drive the functionality and cost-effectiveness of the platform. You feel the pain, you solve the pain. And you incur the pain of slow onboarding of your early customers as we build and evolve the components.

We chose to build this next-gen data supply chain platform. We expect to drive a 10x reduction in data warehouse costs.