Running a Simple Spark Application

This page is intended to guide a new user in running a Spark application on QDS. For more information about composing and running Spark commands from the QDS UI, see Composing Spark Commands in the Analyze Page.

Before You Start

You must have an active QDS account; for instructions on creating one, see Managing Your Accounts. Then sign in to the QDS UI.

Submitting a Spark Scala Application

After signing in, you’ll see the Workbench page.

Proceed as follows to run a Spark command. In this example we’ll be using Scala.

  1. Click + Create New. Select the Spark tab near the top of the page. Scala is selected by default.

  2. Enter your query in the text field (Query Statement is selected by default in the drop-down list near the top right of the page; this is what you want in this case). For example:

    import org.apache.spark._
object FirstProgram {
  def main(args : Array[String]) {
    val sc = new SparkContext(new SparkConf())
    val result = sc.parallelize(1 to 10).collect()
    result.foreach(println)
   }
}
  1. Click Run to execute the query.

The query result appears the Results tab, and the query logs under the Logs tab. Note the clickable Application UI link to the Spark application UI under the Logs tab. See also Downloading Results and Logs.

For information on running other types of Spark command, see:

Using Notebooks

To run Spark applications in a Notebook, follow this quick-start guide, Running Spark Applications in Notebooks.