18. Heaps and Priority Queues

In the previous lecture, we saw how to leverage the branching structure of a tree to design a data structure, a BST, that can (when actively balanced) support fast add(), remove(), and contains() queries. Today, we’ll introduce another data structure that is built atop a binary tree, a heap. Heaps provide \(O(1)\) access to the “smallest” or “largest” element that they store, making them a useful data structure to implement a new ADT called a priority queue. We’ll also see how we can use a heap to define another efficient sorting algorithm, heapsort.

Priority Queues

In an emergency room, there are often more patients who require treatment at any given time than there are resources (bed spaces, operating rooms, hospital staff, etc.) to treat them. For this reason, a queue of patients will often form in a waiting room, and patients are removed from this queue (i.e., admitted to the ER) once there is space for them. This paradigm, adding patients to a collection (the waiting room) and systematically removing them one at a time, is reminiscent of some earlier data structures that we studied, stacks and queues. However, neither of these offers an appropriate solution in this case. Selecting patients based on their arrival order does not ensure the best overall health outcomes. Instead, patients should be admitted based on their urgency to receive care; a patient experiencing cardiac arrest should take priority over a patient with a sprained ankle. To address this, emergency rooms triage incoming patients, assigning them a priority value that induces an ordering over the patients in the waiting room. As soon as resources are freed, the next patient to be admitted should be the one with the highest priority (ignoring any application-specific complicating factors like the inability to put certain patients in certain rooms, etc.).

Let’s design an ADT that models the behaviors of a hospital waiting room, which we’ll call a PriorityQueue, a queue-like structure that enforces a notion of priority. Similar to stacks and queues, there are three primary operations: add()ing a new element (now associated with a particular priority), remove()ing a particular element (the one with the highest priority), and peek()ing at the highest priority element. Finally, we’ll add an isEmpty() method that allows us to check if the priority queue contains any elements.

List-Based Priority Queues

Because of its similarity to stacks and queues, it may be appealing to implement the PriorityQueue ADT using a List. When we add new elements to the priority queue, they can go at the end of the list. When we want to access the highest-priority element (either to peek() or remove() it), we can search over the list elements. Take some time to determine the complexity of each PriorityQueue operation under this design.

While adding elements to this priority queue implementation is fast, the linear searching in peek() and remove() hurts the overall performance. One way to improve this is to maintain a sorted order invariant on the entries of the list. When we add a new element to this sorted list, we will need to determine its sorted position. Take some time to determine the complexity of each PriorityQueue operation under this alternate design.

These implementations make the peek() and remove() operations fast at the expense of making the add() operation slow. Is there a way that we can get good performance for all the operations? We have seen that (actively balanced) BSTs allow us to both add() and remove() elements while maintaining an order invariant; however, active balancing takes a lot of work (and falls outside of our scope). Next, we’ll introduce another tree-based data structure, a max heap, that maintains a different order invariant that is particularly well-suited to locating its largest element. Moreover, this looser order invariant will allow us to more easily enforce an additional balance constraint. In the end, we’ll find that a max heap-based PriorityQueue can support add()ing and remove()ing elements in \(O(\log N)\) time and peek()ing in \(O(1)\) time.

Max Heaps

A (binary) max heap is a binary tree that maintains two additional invariants, a shape invariant and an order invariant. Its shape invariant requires that the heap’s elements form a nearly complete binary tree.

In the following figure, the first tree is nearly complete. The second tree is not nearly complete because it has a height of 3 but only 3 nodes at depth 2. While the third tree fills all but its last level, it is not nearly complete because its last level is not filled from the left.

The heap order invariant imposes a condition on each parent-child connection in the tree.

In the following figure, the tree shown on the left is a max heap, while the tree shown on the right is not a max heap; the max heap order invariant is violated because node 10 is the child of the lesser node 8.

The heap order invariant implies a stronger condition about the values of its nodes; each node is less than or equal to all of its ancestors and greater than or equal to all of its descendants. In particular, the heap element with the maximum value will be at the root of this tree (explaining the name of the max heap).

Representing a Max Heap

To develop a MaxHeap class, we must first decide on its state representation. As with all of our other data structures, the MaxHeap will be a generic class with a type parameter T that represents the type of elements that it stores. For the heap order invariant to make sense, we must be able to compare elements of type T, which we can enforce with the generic type bound T extends Comparable<T>.

While we can represent the heap using the same recursive linked structure that we developed in our earlier lectures (making MaxHeap a subclass of BinaryTree), a simpler representation is actually possible. Because of the heap’s shape invariant, there is only one possible shape that a heap can have for each size. If a heap contains 10 elements, we know that it must have the shape shown above. It must have a height of at least 3 since only 7 elements can fit in a binary tree with a height of 2. Then, it must fill its top 3 levels, with its root node at depth 0, two nodes at depth 1, and 4 nodes at depth 2. Its final three nodes must all be at depth 3, occupying the three leftmost positions. Any deviation from this would cause it not to be a nearly complete binary tree.

We can, therefore, store the elements in a List, whose indices refer to specific positions in the binary tree. We’ll number the positions from top to bottom in the tree and from left to right in each level, which is often referred to as a level-order traversal of the tree. This translation between the tree and List (which we’ll visualize as a row of boxes, like an array) representations of a heap is visualized below.

We’ll use an ArrayList<T> for this backing storage to practice working with a Java library data structure as a client.

To “navigate” the tree in this ArrayList representation, it will be helpful to have methods that can return the indices of the parent(), leftChild(), and rightChild() of the node at a particular index i in this heap. Take some time to study the indices in the above picture to determine the formulas for computing these parent and children’s indices. Then, use this to complete the definition of these helper methods in the editor below. The rest of the MaxHeap is provided for you, so you can press Run to check your work against a set of tests.

To complete our state representation, we can add an invariantSatisfied() method to enforce the max heap order invariant and a MaxHeap() constructor that initializes an empty heap (and asserts the invariant).

We can immediately give the definitions of some simpler heap methods. For example, we can get the size() of the heap by returning heap.size(), and we can peek() at the largest element in the heap by returning its root, accessible through heap.getFirst().

The two remaining methods, add() and remove(), will require more work to maintain both heap invariants.

The add() Method

Let’s consider the add() method.

If we naively add elem to the end of the heap list, this will not compromise the shape invariant. The fact that our array representation lists the elements in level-traversal order means that the new element will be placed in the leftmost available spot of the bottom level of the tree (or in the leftmost spot of a new level if the lowest level was previously full). However, if elem is sufficiently large, this may violate the max heap order invariant. For example, if we add 12 to the end of the max heap that we’ve been considering, we end up in the following state.

Note that we will continue to draw out the tree representation of a heap (even though its underlying storage is a list), as this makes it easier to reason about its structure and check the order invariant. In this case, we see that the order invariant is violated because 12 is greater than its parent node, 8.

We must find an efficient way to restore the order invariant. We can start by fixing the violation that we just introduced. 12 cannot be a child of the smaller node 8, and we can correct this by swapping 8 and 12. Let’s extract this subroutine into a swap() helper method.

After performing this swap(), our heap is left in the following state:

We have not yet restored the order invariant; 12 is still greater than its parent 10. However, we have moved the problem up higher in the heap. 12 and 8 are now in the correct relative order, as are 12 and its new left child, 5. In fact, such a swap can never introduce a problem lower in the tree. Consider the following picture:

If the max heap order invariant was violated by the addition of node \(c\), this must be because \(a < c\). If the heap order invariant held prior to the addition of \(c\), then \(b \leq a\). Combining these inequalities, we find that \(b < c\), so the order invariant is re-established among these three nodes by swapping \(a\) and \(c\). Similar reasoning applies if the violation was between \(a\) and \(b\) rather than \(a\) and \(c\).

Now, we can repeat our reasoning from above to continue restoring the order invariant. 12 is greater than its parent 10, and we can remove this violation by swapping 12 and 10. This leaves our heap in the following state:

Now, 12 is less than its parent 15, which signals to us that the order invariant has been restored. As we explained above, all the new (parent, child) relationships that we formed below 12 satisfy the order invariant, and now 12’s relationship with its parent satisfies it as well. All the other (parent, child) relationships remain unchanged.

We can extract this subroutine, which restores the heap order invariant by repeatedly swapping the new element upward with its parent, into a private helper method bubbleUp() that takes in the index where the new element currently resides. We can formulate this method recursively. There are two stopping conditions for bubbleUp(): either the new element has bubbled all the way to reside at the root (in which case, it is the maximum element) or the new element is less than its parent. Otherwise, we should swap the new element (at index i) with its parent and then recurse on the parent index to continue bubbling.

Our add() method simply needs to call bubbleUp() after adding elem to the end of the heap list.

To reason about the complexity of add(), let’s start by analyzing its helper methods. swap() only affects two entries of the ArrayList, so it runs in \(O(1)\) time. Notice that our bubbleUp() operation performs at most one swap per level of the tree, so it runs in \(O(\textrm{height})\) time. Since the heap is nearly complete, it is a balanced binary tree, and its height will be \(O(\log N)\), in fact, exactly \( \lfloor \log_2(N) \rfloor \). Thus, bubbleUp() runs in \(O(\log N)\) time. This dominates the runtime of add(), which also runs in \(O(\log N)\) time. As written, the space complexity of add() (not counting the possibility of an array resize) is dominated by the \(O(\log N)\) recursive depth of bubbleUp(). Since this is a tail-recursive method, it is easy to re-implement bubbleUp() in a non-recursive manner using a loop, which results in an \(O(1)\) space complexity (see Exercise 18.7).

The remove() Method

By the max heap order invariant, the return value is the element at index 0 of heap. After storing this return value, we’ll need to remove it from the heap and restore both the shape and order invariants. It is not a good idea to simply remove the first entry from heap and shift down the remaining entries. Since each list index refers to a particular location in the tree representation of the heap, this shifting will scramble all the elements in a way that can introduce multiple violations to the heap order invariant. We’ve highlighted the two violations in the following figure.

Instead, we’d like a way to remove the old root 15 while minimally disrupting the rest of the tree. We can do this by swapping the last list element with the first element (making it the new root) and then removing it from the end of the list. This results in a heap with the correct set of elements in which the only potential order invariant violations involve the new root element. Then, to restore the heap invariant, we can do a similar process to bubbleUp(), swapping this new root element downward in the heap until the order invariant is restored. We’ll call this subroutine a bubbleDown(). The following animation steps through the full remove() procedure.

The code for our remove() and bubbleDown() methods is given below.

Similar to bubbleUp(), our bubbleDown() method does a constant amount of work (up to two element comparisons and one swap) per level of the tree, so it runs in \(O(\textrm{height}) = O(\log N)\) time. This dominates the runtime of remove(), which also runs in \(O(\log N)\) time. The space complexity is dominated by the \(O(\log N)\) recursive depth of bubbleDown(), and we can again use a non-recursive implementation to achieve an \(O(1)\) space complexity.

The following client adds a few elements to a MaxHeap and then removes some, printing the heap’s tree representation after each step. Press Run to watch the heap order invariant being maintained.

Implementing a Priority Queue

We can use our MaxHeap class as the basis for an efficient implementation of a PriorityQueue, which we’ll call a MaxHeapPriorityQueue.

This class will use a MaxHeap to represent its state. Each node in the heap will need to store both an element (of type T) and its priority (which is used to order the elements). We can collect these two components in a (static) nested record class, Entry<T>. To add these Entrys to a heap, they will need to be Comparable, and we’ll define the compareTo() method to compare the priorities of the entries.

Now that we have established a composition relationship between this priority queue class and the MaxHeap class, we can define the PriorityQueue methods to call the respective MaxHeap methods. Take some time to complete these definitions in the editor below before looking at our implementation. The MaxHeap is provided, and pressing Run checks your queue against tests.

The runtimes of these methods directly relate to those of the MaxHeap methods. The isEmpty() and peek() methods both run in \(O(1)\) time. The add() and remove() methods both run in \(O(\log N)\) time. We have achieved our desired complexities.

As a quick demonstration, this client models an emergency room waiting room as a priority queue: patients are treated in order of urgency, not in order of arrival.

Sometimes, we might like the ability to update the priorities of elements while they are in the priority queue. For example, this will be necessary for an efficient implementation of Dijkstra’s shortest path algorithm in graphs, which we will study in a few lectures. We’ll discuss soon how to add support for these priority updates while maintaining the same complexity guarantees.

Heap Sort

To conclude today’s lecture, we’ll explore how we can use a heap to develop a new, efficient sorting algorithm. The idea for this algorithm is very straightforward. First, we’ll form a max heap out of all the elements that we wish to sort. We’ll call this the heapify() step. Then, we can remove the elements from this max heap one by one. The first element to be removed will be the largest, followed by the second-largest, followed by the third-largest, etc. We extract the elements in descending sorted order.

We can perform both of these steps in-place within an array if we modify our heap methods to manipulate an array range rather than an ArrayList. Then, both steps amount to a single pass over the array, which we can reason about with a loop invariant.

Forward Pass: heapify()

First, we’ll build up the heap from the front of the array to the end. Initially, we don’t know about the contents of the array.

After finishing this forward pass, we’d like the array’s elements to form a valid binary max heap.

During the ith iteration, our invariant will be that the range a[..i) forms a valid binary max heap.

We can initialize i = 0 to establish the invariant (since a[..0) is the empty range, which is trivially a valid (empty) binary max heap), and we can guard the loop on the condition i < a.length (since a[..a.length) is the entire array a). To make progress in each iteration, we must add() a[i] to the heap; this causes it to grow to occupy a[..i+1), allowing us to increment i and preserve the loop invariant.

The main structure for heapify() is shown below. The full implementation, which adapts the logic from earlier in the lecture to work on array ranges, is provided with the lecture release code.

An alternate, more efficient version of heapify() is discussed in Exercise 18.8. The overall runtime of heapsort remains the same for this alternate approach, since it is dominated by the backward pass that we will discuss next.

Backward Pass: Iterative Removal

Once we have a max heap, we can iteratively remove the largest element, using these removed elements to fill the array from the back. This will result in a sorted array since the removals happen in descending sorted order. Our precondition for this backward pass is that the entire array is a valid binary max heap.

After finishing the backward pass, our array should be sorted.

During the ith iteration (counting backward), our invariant will be that the range a[..i) forms a valid binary max heap, and the range a[i..] is sorted and greater than or equal to a[0] (the maximum element in the heap).

Using these array diagrams, we should initialize i = a.length and guard the loop on the condition i > 0. To make progress in each iteration, we should swap a[0] and a[i-1], removing the largest element a[0] from the now-smaller heap range a[..i-1), and then bubble down the new root. This allows us to decrement i and preserve the loop invariant.

Both passes rely on array-range versions of bubbleUp() and bubbleDown(). Try writing them yourself below; the rest of the heap sort is provided, so pressing Run checks that your code sorts arrays correctly (and in \(O(N \log N)\) comparisons). Note that bubbleDown() takes an extra argument j: only the array range a[..j) is part of the heap.

Once you have heapsort() working, here is a client that uses it on arrays of Integers and Strings. Since heapsort() is generic in any Comparable subtype, it works on any such array.

Visualizing the Heap Sort Algorithm

The following animation walks through one invocation of the heapsort algorithm to sort an array of 7 elements.

If you focus on the array entries in this animation, they appear to be jumping around erratically, especially during the forward pass. For this reason, heapsort seems like a somewhat “magical” sorting algorithm; the “magic” is just carefully managing and reasoning about different invariants.

Overall, the runtime of heapsort is dominated by the \(O(N)\) calls to bubbleUp() and bubbleDown() during the forward and backward passes (respectively). Since these bubbling operations have an \(O(\log N)\) runtime, the runtime of heapsort is \(O(N \log N)\). Since the heap construction is handled in-place within the array, heapsort has an \(O(1)\) space complexity (when bubbleUp() and bubbleDown() are implemented iteratively, otherwise \(O(\log N)\)). Heap sort is not a stable sorting algorithm, which we ask you to verify in Exercise 18.9.

Exercises