Latest from Vast Data aims to simplify, speed AI development

Despite enterprises continuing to increase their investments in AI development, AI projects still fail more than 80% of the time, according to some estimates. While Vast Data’s new feature doesn’t eradicate hindrances such as bias and lack of infrastructure resources, it does address problems like poor data quality and lack of relevant data.

Motivation for developing SyncEngine came from observing the trouble customers were having discovering and feeding distributed data into AI pipelines, according to Aaron Chaisson, Vast Data’s vice president of product and solutions marketing.

“The impetus was a direct response to the ‘last mile’ problem hindering customer AI strategies,” he said. “While the Vast AI OS provides automated AI data pipeline services, customers were challenged with finding and then moving all of their distributed data sources into the Vast pipeline.”

Meanwhile, because it directly addresses one of the main barriers to successful AI development, Stephen Catanzano, an analyst at Enterprise Strategy Group, now part of Omdia, like Ni called SyncEngine a compelling addition to Vast Data’s AI OS.

“Vast’s SyncEngine is a significant addition for customers that want to eliminate the ‘last mile’ problem of data fragmentation that has become one of the biggest constraints in AI deployment,” he said. “Allowing organizations to unify scattered data … without additional costs or complex third-party tools is critical.”

SyncEngine includes the following features:

• Data migration capabilities built for the massive files and data sets AI models and applications require to reduce the occurrence of AI hallucinations.
• Metadata indexing, enabling enterprises to catalog hundreds of trillions of files.
• Throughput levels that are limited by only source and target systems.
• Parallel processing for input/output workloads to reduce bottlenecks and improve performance.

Combined, the features enable users to build a catalog that connects data from disparate systems such as object storage and enterprise applications, migrate and synchronize data at scale, and ultimately speed the process of feeding AI pipelines.

That combination, rather than any one or two individual capabilities, is what is most significant about SyncEngine, according to Ni.

“Enterprises continue to struggle to find, contextualize and feed their petabytes of siloed data into [AI] workflows,” he said. “Vast folds those steps into the OS, eliminating the hand-coded, often brittle, tool chains and making [data] searchable and AI-ready.”

By folding the different steps into its AI OS — leading to high performance but also possibly fears regarding vendor lock-in — Vast Data is taking a different approach to feeding AI pipelines than many other data management vendors, Ni continued.

For example, Snowflake and Databricks lay governance and intelligence on top of data while separating compute from storage. Collibra and Informatica, meanwhile, excel at metadata management but don’t specialize in data migration and preparation capabilities.

“Vast is deliberately separating itself from the pack by collapsing categories … into one AI operating system,” Ni said. “This consolidation delivers high performance and low-latency pipelines for real-time and agentic AI, but it also challenges buyers to embrace a single vendor spanning so much ground.”

Catanzano likewise suggested that Vast Data is taking a unique approach to data management and AI pipeline development.

“Vast differentiates by offering a unified platform that combines storage, database and compute capabilities specifically optimized for AI workloads, rather than retrofitting traditional storage systems for modern AI requirements,” he said.

Meanwhile, regarding SyncEngine, Catanzano highlighted the significance of metadata indexing.

“Metadata indexing stands out as particularly valuable because it enables cataloging and searching hundreds of trillions of files within the Vast DataBase,” he said.