Thanks for the article, another option is to use Scala UDF in PySpark directly. Apache Zeppelin support SparkContext/SparkSession sharing between different languages.