Databricks Apps a toolkit that simplifies AI development

New capabilities

Fueled by the potential of generative AI to aid data management and analytics, enterprise interest in AI is surging.

One of its promises is that it can enable true natural language processing (NLP), which lets non-technical workers use analytics to inform decisions. Another part of its potential is that it can be used to generate code and automate processes, which can make data experts more efficient.

However, developing generative AI applications, including chatbots that let users query and analyze data and tools that use machine learning to take on tasks such as monitoring for data quality, is not easy.

Databricks Apps is designed to simplify application development by enabling developers to do all their work in the secure Databricks environment while still providing them with choices as they build data and AI applications.

Before Databricks Apps, Databricks customers had to use platforms from third-party vendors to complete the development of generative AI chatbots, AI-powered analytics applications and other intelligent capabilities.

However, mixing proprietary data, AI systems such as LLMs and third-party development platforms risked accidental data beaches. In addition, it was expensive.

Part of what makes developing data and AI applications difficult, risky and expensive is all the movement they require. Relevant data needs to be discovered and moved out of a data storage platform to train an application. The application needs to be developed in an integrated development environment or other data science platform. And then the application needs to be moved to its host environment for deployment and management.

Databricks Apps eliminates the need for that labor intensive, expensive and risky movement.

Instead, it enables developers to build applications natively in Databricks using development frameworks including Dash, Flask, Gradio, Shiny and Streamlit. In addition, it comes with prebuilt Python templates designed to speed up the development process.

However, if developers prefer to work in an integrated development environment such as Visual Studio Code or PyCharm, Databricks Apps supports that as well.

Following development, Databricks Apps eliminates the need to build infrastructures for deploying and running applications by running the applications on automatically allotted serverless compute storage within Databricks, according to the vendor. Management, meanwhile, includes security measures and governance capabilities such as access control and data lineage accessed through the Unity Catalog.

“There are some features here which are potentially very impactful,” Farmer said. “For example, the support for popular developer frameworks, which enables application developers to work with familiar tools of choice, expands the Databricks ecosystem to a new market of application developers.” 

In addition, eliminating the need to develop infrastructures for managing applications is noteworthy, he continued.

“The automatic provisioning of serverless compute will be significant because it enables developers to focus on building applications rather than the complex process of deploying a data architecture, which was a barrier to those who were not data specialists,” he said.

From a competitive standpoint, Databricks’ aggressive development of an environment for building, deploying and managing AI and machine learning tools over the past couple of years has differentiated it from other data platform vendors, Farmer said.

While AWS, Google Cloud, Microsoft and Snowflake have all similarly made AI a focal point of their product development, their tools for developing and managing AI models and applications are not as integrated as what Databricks has built, he continued. Databricks Apps furthers the separation between Databricks and its peers.

“Snowflake has been catching up or at least playing catch-up, but this continuous development from Databricks is very compelling,” Farmer said. “Microsoft Fabric, of course, is aiming to be a unified platform similar to Databricks, but it’s still an inferior product. Google Cloud Platform and AWS have a wide range of AI and ML services, but they’re not so deeply integrated across the platform.”

Despite the additive capabilities of Databricks Apps, Petrie cautioned that the applications — in particular, generative AI applications — customers will be able to develop will not suddenly enable anyone within an organization to freely work with data.

While Databricks aims to help enterprises broaden the reach of data and AI beyond a small audience of users, training and expertise are still required to use data and AI to inform decisions and take actions based on those decisions.

“Like many vendors, Databricks aims to ‘democratize’ the consumption of data, analytics and AI,” Petrie said. “But I think users of these applications will still require significant expertise in data, AI and the business domain, depending on the use cases involved.”

While Databricks Apps extends what customers could do with Mosaic AI and demonstrates that Databricks is continuing to focus on improving its AI and machine learning development environment, the impetus for the new features came from customer feedback, according to Shanku Niyogi, the vendor’s vice president of product management.

Developing and deploying internal applications has always been complex, he noted. But with enterprise interest in AI rapidly increasing, there is greater need for vendors such as Databricks to simplify developing and deploying AI applications.

“Customers … have shared that building and deploying internal data apps has historically been a complex and time-consuming process,” Niyogi said. “They specifically asked for easier ways to test new features while maintaining a secure environment. With the explosion of AI, this need has only grown.”

Looking ahead

Databricks Apps does not end Databricks’ focus on enabling application development and deployment, according to Niyogi.

The vendor’s goal is to make data and AI available to a broad audience of users, he noted. Toward that end, Databricks plans to invest in developing new Mosaic AI features as well as adding other capabilities through partnerships.

“Databricks will continue to make AI more accessible for organizations,” Niyogi said. “This includes further ways to simplify the app development process, new Mosaic AI capabilities that help teams build, deploy and measure compound AI systems, and a continued investment in a collaborative AI partner ecosystem.”

Farmer, meanwhile, said that Databricks’ focus on improving AI and machine learning workflows is appropriate. In particular, he suggested that the vendor enhance its support for developing applications for non-technical users as well as emerging AI technologies such as multimodal models.

“Multimodal will become critical over the next couple of years,” Farmer said. “I think we should also see more development for non-technical users. This release includes a first attempt at that, and no doubt this is the start of a new direction for Databricks and a very welcome one at that.”

Eric Avidon is a senior news writer for TechTarget Editorial and a journalist with more than 25 years of experience. He covers analytics and data management.