Published by Sue Johns-Chapman
Published Date 21.10.2025 
A contribution from dbt Labs, Silver Winner of SaaS Company of the Year 2025, UK Business Tech Awards.
Most businesses recognise that data holds enormous potential, but data alone is not enough. The real advantage comes from turning that data into insights that drive smarter decisions. This requires the ability to bring together organisational data from multiple sources and refine it into reliable, high-quality outputs.
But handling large amounts of data isn’t as simple as running a basic automated script. It requires reliable systems that automatically check and process new data as it comes in. Many organisations employ data teams to build these systems in-house, but they can quickly get overwhelmed with troubleshooting and user support issues. Data teams can end up spending most of their time fixing problems instead of working on new initiatives that drive the business forward.
The result is more than just an inconvenience – it prevents companies from realising the full value of their data investments.
The hidden cost of DIY data infrastructure
When organisations first build their data capabilities, the do-it-yourself approach seems like a natural starting point. The aim is to automate data pipelines, replacing manual steps like collecting, cleaning, and moving data with a system that does it automatically and reliably. Open-source and commercial tools are readily available to handle key steps from ingesting data (e.g., Fivetran, Apache Kafka) and transforming and testing it (e.g. dbt), to orchestrating workflows (e.g., Apache Airflow, Dagster) and monitoring performance (e.g., Prometheus and Grafana). This allows data teams to choose the tools that best fit their needs, often with minimal licensing costs.
However, this approach often creates unexpected headaches.
Gaps in accessibility
One of the most significant, yet often overlooked, challenges is making data tools accessible to everyone who needs them. Modern data work involves diverse stakeholders, such as technical engineers, business analysts, and domain experts who understand the business but may not be software developers.
To contribute to data pipelines, most DIY systems require users to install and configure a long list of technical tools like Git, dbt, and various command-line utilities. For less technical team members, this setup can be difficult and error-prone, especially when software conflicts arise.
This complexity limits who can participate and increases the support burden on engineering teams. The result is that organisations lose the insights that could come from broader participation in data work.
Meanwhile, data engineering teams become gatekeepers by necessity, spending their days answering questions like “Where can I find this data?” or “When was this last updated?” These aren’t complex technical problems. They’re basic discovery issues that shouldn’t require engineering intervention.
The time sink of testing
Another major drain on productivity is the testing cycle. Testing ensures that any mistakes in new code written for the data pipeline are detected before it goes live. Most DIY date pipeline setups don’t come with easy ways to test code changes locally. When data engineers make changes, they often must submit their work, wait for automated systems to process it, review the results, and then repeat the entire cycle if problems emerge.
This back-and-forth consumes enormous amounts of time. What could take minutes with proper local testing tools instead stretches into hours or days. The inability to quickly iterate and troubleshoot locally doesn’t just frustrate engineers, it delays the delivery of critical business capabilities.
Streamlining data pipelines
Forward-thinking organisations are rethinking their approach to data infrastructure. Rather than building everything from scratch, they’re adopting platforms that handle operational complexity while enabling teams to focus on generating value.
For instance, dbt is a central command centre for managing how data is transformed, that teams can use to build, deploy, and monitor data transformation pipelines at a fraction of the time and cost of a DIY solution. dbt also provides tooling that enables all participants in the data lifecycle to find, understand, and utilise analytics code – not just data engineers.
With dbt, data teams can automate how analytics code is reviewed, tested, and deployed. Changes are tracked in Git, a version control tool that helps teams manage and review code safely. Once changes are submitted, they’re automatically tested in a safe environment. If everything passes, the code is deployed without any manual steps required.
This reduces errors, speeds up delivery, and frees engineers to focus on higher-value work.
By reducing operational friction, these approaches transform how data teams spend their time. Instead of serving as help desks, they can work on building sophisticated analytics capabilities that support business stakeholders and strategic initiatives.
Rethinking the costs of DIY data
DIY data infrastructure can create more problems than it solves. When highly skilled data teams are tied up maintaining systems, it slows down progress and limits their ability to focus on strategic work.
The better path is to adopt tools and platforms that follow best practices, reduce manual effort, and make data more accessible across the business. This helps teams spend less time on maintenance and more time using data to drive insight and impact.
In a world where data is everywhere, the real challenge is making it useful – and giving teams the space and tools to do just that.
