Metaplane app adds data observability for Snowflake users

New capabilities

Data observability is a growing need.

In recent years, data volume has grown exponentially. Organizations ingest data from an increasing number of sources, and the amount of data those sources produce is on the rise. In addition, data is becoming more complex, with unstructured data such as text, images, videos and audio files estimated to account for more than 80% of all new data.

The increases in volume and complexity, with nearly half of all organizations now managing at least 500 petabytes of data, make it impossible for data teams to manually observe data for quality.

In response, data observability vendors emerged, providing platforms that automate monitoring data for quality by looking at its accuracy, freshness, schema and lineage, among other characteristics.

Now, surging enterprise interest in developing AI tools, including generative AI, is compounding the need for data observability.

Even when the data used to train AI models and applications is high-quality, AI tools are prone to hallucinations, which are incorrect, misleading and sometimes even offensive outputs. Add in poor data, and the probability of a model or application delivering an incorrect output increases. Models and applications are only as good as the underlying data used to train them.

Given the convergence of rising interest in AI and the need for high-quality data to inform AI tools, the current state of data management presents an opportunity for data observability vendors, according to Catanzano.

“Market conditions, including the surge in data volumes and the growing interest in AI, have made data observability more vital,” he said. “The sheer complexity and scale of modern data ecosystems combined with the demand for trusted data for AI and ML mean that observability tools are essential for maintaining data quality and preventing costly errors or failures.”

Kevin Hu, Metaplane’s co-founder and CEO, similarly said that enterprise interest in developing generative AI tools potentially provides growth opportunities for data observability vendors. However, he noted that only a small minority of technology-forward enterprises have actually put generative AI tools into production and presented data observability vendors with growth potential.

“When they have a GenAI use case, that’s amazing,” Hu said. “Selfishly, it’s great for us, and we’re one of the boats on the rising tide. But for the majority of data teams, they are still building up the maturity curve.”

While current conditions present at least some growth opportunities for data observability vendors, their platforms serve a real need for enterprises.

As a result, Metaplane’s native application for Snowflake stands to benefit Snowflake customers seeking to either start developing AI tools or improve their existing development process, according to Catanzano.

Unlike traditional integrations that serve as conduits between data systems and improve movement between them, Metaplane’s native application for Snowflake eliminates any need to move data out of Snowflake and into Metaplane, which can be costly and risk data to exposure, to observe its quality.

Instead, the application, which is available through the Snowflake Marketplace, enables joint customers to observe their data where it resides. During public preview, the application is free. Once made generally available, users will have to pay for the application. However, Metaplane has not yet determined a pricing structure, according to Hu.

“The addition of a native app for Snowflake is significant as it integrates data observability directly into the Snowflake environment,” Catanzano said. “This allows joint customers to leverage Snowflake’s robust security and governance features seamlessly while ensuring data quality and performance. It provides a streamlined approach for data teams to monitor and secure their data.”

According to the vendor, benefits of using Metaplane’s native application for Snowflake include the following:

• Automated observability from data ingestion through analysis powered by machine learning that frees data teams to focus on other initiatives rather than checking on data as it moves through pipelines.
• Visibility into the health of data throughout its lifecycle, including receiving alerts when there are problems, to ensure that data is reliable for AI and machine learning initiatives as well core business operations.
• Security and compliance given that Metaplane’s tools observe data within the Snowflake environment rather than requiring data to be exported, which is particularly beneficial to enterprises in highly regulated industries that need to meet compliance standards.

Currently, Metaplane does not plan to develop similar native applications for other data platforms such as Databricks, AWS, Google Cloud and Microsoft Azure, according to Hu.

With Snowflake an investor in Metaplane, the two share a close relationship as well as numerous joint customers. But beyond that, Snowflake’s Snowpark Container Services, which allows users to run containerized applications within Snowflake’s infrastructure, makes it conducive to a native application.

“As far as I know, there’s no equivalent in other platforms,” Hu said. “There are a lot of [data governance features] Snowflake has been building out, and they continue to build out more and more. AWS was the original hyperscaler, and they support all the functionality [needed to run a native application], but not in such a containerized way.”

Regarding the impetus for developing a native application for Snowflake, Hu said the idea originated with Metaplane. But once customers were informed about the possibility, they were enthusiastic.

In addition, the native application could serve to attract new customers, he continued.

Next steps

With the application for Snowflake now in public preview, a major focus for Metaplane will be improving the application’s performance to prepare it for general availability, according to Hu.

Beyond the native application, one focal point of the vendor’s product development plans is adding integrations with more databases and other data sources, he said.

Market conditions, including the surge in data volumes and the growing interest in AI, have made data observability more vital. The sheer complexity and scale of modern data ecosystems combined with the demand for trusted data for AI and ML mean that observability tools are essential for maintaining data quality.

A second focal point is to extend data quality beyond data teams to better enable other personas within organizations to understand the quality of their data so they can trust it to inform decisions.

“If I had a critique of our category — data observability — and ourselves included, it’s that we talk a lot about data observability for data teams, and that’s important,” Hu said. “But it doesn’t matter if the data teams trust the data and the business does not. We have to do a better job of making sure that [others within organizations] can monitor what matters to them and understand the state of data.”

Catanzano, meanwhile, suggested that Metaplane’s plan to add more integrations is sound. In addition, he said the vendor would be wise to develop native applications for vendors beyond just Snowflake.

“As Metaplane continues to grow, expanding its integrations and features to cover more diverse data environments beyond Snowflake would be valuable,” Catanzano said.

In addition, adding predictive analytics capabilities would benefit customers, he continued.

“More advanced machine learning algorithms for proactive issue detection could further empower data teams to anticipate and prevent issues before they impact business operations,” Catanzano said.

Eric Avidon is a senior news writer for TechTarget Editorial and a journalist with more than 25 years of experience. He covers analytics and data management.