Of course, you can play with REST API directly using requests package but in my opinion pylivy will simplify our code a lot. To play with the Livy server, we’ll use a Python library called pylivy. As I mentioned earlier, now we need to create the client script to communicate with the Spark server using REST API.īefore we start coding, I recommend creating a separate project where we put our code. ![]() Now, we will focus on the business logic of our project-the client site. If you’re here, I assume you went through all previous steps successfully and all containers are running. ![]() You can find pylivy examples and documentation here. No, thankfully there’s a dedicated library called pylivy that I’m going to use in the sample project. If we don’t want to play with the command line to reach the cluster directly using SSH then Apache Livy comes into play with its REST API interface.ĭo you have to create an additional layer of logic to manage connections and all REST API functionalities? You might be wondering how to make Apache Spark simpler to use in automated processing.įor example, we can imagine a situation where we submit Spark code written in Python or Scala into a cluster, just like we submit SQL queries into a database engine. share cached RDDs or data frames across multiple jobs and clients,.long-running SparkContext can be reused by many Spark jobs,.managing multiple SparkContexts simultaneously,. ![]()
0 Comments
Leave a Reply. |