Datarios logo
If something goes awry in a data streaming system, the corrections must be made as quickly as possible. After all, the data will just keeping flowing until the issue is fixed.
To this end, a start-up based in Tel Aviv has released an observability platform for Apache Flink — a data streaming platform gaining adoption thanks to its ability to do both real-time and batch processing.
The Datorios data observability platform for Apache Flink provides a glimpse of data as it is being transformed inside of Flink itself, easing the process of development, troubleshooting and auditing.
Available as a cloud service, the platform can now be tested at no charge.
Apache Flink can process hundreds of thousands of data points in a second but remains a bit of a black box when it comes to observability.
“Flink is an amazing solution for processing data, but it doesn’t have the ability to give the user a good observability of what’s happening under the hood,” said Ronen Korman, CEO and co-founder of Datorios, in an interview with TNS.
This is true both of how Flink processes the data and also how the infrastructure supports the data ingestion.
Flink processes data streams through operators, and Datorios can show how data is being run through the operator, offering the developer insight into the current state of the operator instance. This information is also correlated with the CPU and memory usage of the host.
Datorios can record incoming data, so it can be played back for step-by-step inspection. It also provides a debugging layer to check your data processing code.
According to the Datorios press materials, the service can do all this:
- Event visualization – see the journey of data as it flows through each processing operation.
- Event search – trace specific events; great for auditing and regulatory compliance.
- State monitoring – see how the state changes during each processing step.
- Window analysis – dive into the internal processing of windows including order of processing, lateness, discarded records, state evolution, and window emits.
- Record and replay – share videos of data processing with other developers.
Like many IT start-ups in Israel, The Tel Aviv-based Datorois was born from military experience. Korman and the other co-founders Asaf (Pizzer) Cohen, Idan Shchori, and Avi Hadad came up with the idea for the technology while serving time in the military.
Israel has “the challenge of recruiting 18-year olds into the military. You want them to be productive very fast, Korman said. Observability tools were one way of giving the recruits “the ability to learn the technology work in ways that made that very, very efficient.”
Real-Time Streaming
Real time streaming of data has a bright future, Korman said. For one, interactive generative AI apps require real-time data processing. Not surprisingly, Datorois has support for Python, a favorite language of data scientists writing data pipelines.
Flink is particularly valuable for such workloads given its dual-use nature of supporting both batch programming and real-time streaming, so data scientists can test each approach, looking for ways to move from batch to speedier streaming, Korman said.
Datorois is run as a software service. The user’s copy of Flink is instrumented data collectors that send the required information back to the service. The users can then use the graphical interface as a “second screen” in addition to their primary code editing environment.
For the time being, users can use the service for free, in order for the company to get a better idea of how the service would be most productively used. Over time, commercial pricing will be introduced.
In addition to Tel Aviv, Datarios has an office in Palo Alto, CA. The company is funded by Grove Ventures and Eclipse Ventures.
The post Apache Flink Gets Some Observability With Datorios appeared first on The New Stack.
Apache Flink is rapidly gaining traction as a platform for real-time Generative AI apps, though remains a bit of a black box as far as observability and debugging goes. Datorios wants to change that.