Flink Development Platform: Apache Zeppelin — (Part 3). Streaming

  • Streaming ETL (Real time data cleaning and transformation)
  • Streaming Data Analytics (Build real time dashboard in Zeppelin)

Prerequisites

In this blog, I will use kafka as data source and sink. I will read one kafka topic and do some processing on the data, then write it back to another kafka topic. In order to use kafka connector, you need to configure flink.execution.packages to include the necessary dependencies

  • org.apache.flink:flink-connector-kafka_2.11:1.10.0,org.apache.flink:flink-connector-kafka-base_2.11:1.10.0,org.apache.flink:flink-json:1.10.0
docker-compose up -dcurl -X POST http://localhost:8083/connectors \-H 'Content-Type:application/json' \-H 'Accept:application/json' \-d @connect.source.datagen.json
127.0.0.1        broker

Streaming ETL

Now I will demonstrate how to do streaming ETL via Flink SQL. You can find all the code here in the tutorial note Flink Tutorial/Streaming ETL which is included in Zeppelin.

Streaming Data Analytics

After the Streaming ETL job is started, we can do streaming data analytics on the processed table (sink_kafka) via select statement. The result of sql select statement will be pushed to Zeppelin frontend. That means we can use Zeppelin to create real time dashboard. You can find the examples in tutorial note Flink Tutorial/Streaming Data Analysis which is included in Zeppelin.

  • Single
  • Update
  • Append

Other Features

You can also use other same features in batch sql, such as using udf, connect with hive. You can refer Part 2 for more details.

Summary

This is how you can do streaming in flink. Here’s several highlights of this article.

  • You can do most of the streaming work via SQL + UDF
  • Streaming ETL
  • Streaming Data Analytics
  • You can also use udf and connect with hive in the same way as batch

Flink on Zeppelin Series

References

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Jeff Zhang

Jeff Zhang

Apache Member, Open source veteran, Big Data, Data Science,