Prophecy, a company that provides a low-code data engineering solution, has created a Databricks integrated system, which allows anyone to quickly and easily build data pipelines on the Apache Spark-based data platform.
The process of producing data pipelines, which provide essential business intelligence and machine learning, is complex. Dozens of data engineers must define them individually and then execute scripts to test, deploy, and manage their entire workflow in production. This is especially due to the increasing volume of internal and external data across organizations.
Prophecy for Databricks
Anyone using Databricks, whether it be a experienced data engineer or a non-programmer data citizen, may leverage a visual, drag-and-drop canvas to develop, deploy, and monitor data pipelines. This transforms the visual data pipeline into 100% open-source Spark code (PySpark or Scala), with interactive development and execution to verify that the pipeline works at every step of the way.
According to Venturebeat, the principal advantage (of this integration) is productivity. Instead of data engineers having to manually code in notebooks, they can use Prophecy to quickly and easily drag-and-drop components to interact with data pipelines.
Join us at the most prominent webinar on applications AI for enterprise business and technology decision makers in person on July 19 and virtually from July 20-28.
The next benefit is that it simplifies working with Apache Spark and Databricks to non-programmers, thus increasing the pool of people who can do data engineering. Overall, these capabilities will enable companies to scale data engineering to keep up with the surge of incoming data.
How to connect?
The Databricks user interface [UI] can connect Prophecy to their existing data stack, enabling them to simplify pipeline execution and management in any cloud. Departure the solution will include several additional tools, such as Delta Lake.
Databricks Partner Connect, a databricks interface, provides an easy on-ramp to Prophecy from the Databricks app, according to Bains.
While data engineering firms like Matillion can integrate with Databricks through Partner Connect, they are limited to SQL transformations. Prophecy, as Bains said, provides two things that no other such product can transform visual data pipelines into 100% open-source Spark code in Scala or PySpark and extensibility.
Prophecys'' integration with Databricks is very deep, and it includes support for Spark Streaming, Delta Lake, and Databricks Jobs, as well as the possibility of scheduling no other product with such a close and extensive integration, according to the author.
According to IDC, worldwide data creation is expected to exceed the expected annual growth rate of 23% in 2025. Besides, Prophecy will come in handy to keep up. The company, which raised $25 million earlier this year, is also looking to expand its capabilities with other data platforms, including Snowflake.