Using GCP Preemptible Instances in Qubole Clusters

Google Cloud Platform (GCP) offers two types of instances that are suitable for use as Qubole cluster nodes: on-demand instances and preemptible instances.

On-demand instances: These are normal compute instances. An on-demand instance is likely to remain available for the life of the cluster, helping to ensure that the cluster performs its work smoothly and reliably. The disadvantage of on-demand instances is cost: a cluster composed entirely or mostly of on-demand nodes may be many times more expensive than a similarly-sized cluster composed partly of preemptible nodes.

Preemptible instances: Preemptible instances are short-lived, up to 24 hours, and Compute Engine can terminate these instances at any time. But preemptible instances are much cheaper than on-demand instances, so if your applications can withstand possible instance terminations, then preemptible instances can reduce your Compute Engine costs significantly.

To compare prices between on-demand instances and preemptible instances, see Google Compute Engine Pricing in the GCP documentation.

Cluster Composition Choices

You can choose to create a cluster in any of the following configurations:

  • On-demand nodes only
  • Preemptible nodes only
  • A mix of preemptible and on-demand nodes

For most purposes, the third option is the best because it provides a balance between cost and stability.

The remainder of this section focuses on the settings and mechanisms Qubole provides to help safeguard the overall functioning of a cluster that includes preemptible nodes.

How You Configure Preemptible Instances into a Qubole Cluster

The items to configure when creating a new cluster in GCP are these:

  • Coordinator and Minimum Worker Nodes: Choose whether Coordinator and minimum worker nodes should be on-demand or preemptible. These nodes are essential to the functioning of the cluster, so you will normally choose on-demand nodes for them.
  • Auto-scaling Worker Nodes: Choose whether the auto-scaling nodes should be on-demand or preemptible. Auto-scaling worker nodes are not as essential to the functioning of the cluster, so you might consider preemptible nodes for them.
  • Preemptible Nodes Percentage: In a mixed cluster, this specifies the maximum percentage of autoscaling nodes that can be preemptible instances. Autoscaling nodes are those that comprise the difference between the Minimum Worker Nodes and the Maximum Worker Nodes. Qubole adds and removes these nodes according to the cluster workload. In a preemptible-only cluster, this is always set to 100.
  • Fallback to On-demand Nodes: You should normally choose Fallback to On-demand Nodes (check the box). This option causes Qubole to launch on-demand instances if it cannot obtain enough preemptible instances when adding nodes during autoscaling. This means that the cluster could possibly at times consist entirely of on-demand nodes, if no preemptible nodes are available. But unless cost is all-important, this is a sensible option to choose because it allows the cluster to do its work even if preemptible nodes are not available.
  • Use Qubole Placement Policy: If selected, this setting causes Qubole to make a best effort to store one replica of each HDFS block on a stable node (normally an on-demand node, except in the case of a preemptible-only cluster). Qubole recommends you select this option to prevent job failures that could occur if all replicas were lost as a result of Compute Engine reclaiming many preemptible instances at once.
  • Cool-Down Period: The time period in minutes that Qubole allows to elapse before removing nodes designated as not needed, based on the cluster’s current workload.

Configuring a Mixed Cluster (Normal and Preemptible Nodes)

You configure a mixed cluster by doing all of the following:

  • Setting Coordinator and Minimum Work Nodes to On-demand nodes.
  • Setting Auto-scaling Worker Nodes to Preemptible nodes.
  • Setting the Preemptible Nodes Percentage to a number less than 100.

This configures a cluster in which the core nodes (the Coordinator Node and the nodes comprising the Minimum Worker Nodes) are On-Demand instances, and a percentage of the autoscaling nodes are preemptible instances as specified by the Preemptible Nodes Percentage.

For example, if the Minimum Worker Nodes is 2 and the Maximum Worker Nodes is 10, and you set the Preemptible Nodes Percentage to 50, the resulting cluster will have, at any given time:

  • A minimum of 3 nodes: the Coordinator Node plus the Minimum Worker Nodes, all of them on-demand instances (the core nodes).
  • (Usually) a maximum of 11 nodes, of which up to 4 (50% of the difference between 2 and 10) will be preemptible instances, and the remainder on-demand instances. (The cluster size can occasionally rise above the maximum for brief periods while the cluster is autoscaling.)

Qubole also falls back to on-demand nodes when coordinator-and-minimum-number-of-nodes’ cluster composition is preemptible nodes.

How Qubole Manages Preemptible Nodes While the Cluster is Running

Qubole’s primary goal in managing cluster resources is productivity, making sure that the work you need to do gets done as efficiently and reliably as possible, and at the lowest cost that is consistent with that goal.

Qubole uses the following mechanisms to help ensure maximum productivity in running clusters that deploy preemptible instances:

  • The Fallback to On-demand Nodes option described above.
  • The Qubole Placement Policy described above.