5. Analyzing Complexity

Code that is used in commercial software applications needs to make efficient use of its resources. We are perceptive to code that seems to be running slower than it needs to. Graphical applications can start to lag if their background computation takes longer than the refresh rate of the screen. When we use a search engine, we are used to millions of results populating within a few milliseconds, and we would be frustrated if the search instead took minutes to complete. In time-critical applications such as code that analyzes sensor readings in flight software, a one-second shorter processing time can make the difference in preventing a collision. Beyond the runtime of our code, we should be mindful of its memory usage. While modern computers seem to have an “unbounded” amount of memory when compared to their predecessors, this can be quickly diminished when processing large amounts of data. Further, you may write code for embedded systems where memory space is far more constrained.

As the next step in our journey toward becoming better programmers, we need to start to take these factors, the time and space efficiency of the code that we write, into consideration. Asymptotic analysis is a theoretical tool that allows us to formally reason about time and space complexity. We will introduce these concepts in this lecture and use them throughout the course (and throughout many of the subsequent CS courses that you will take) to analyze the code that we write.

Measuring Runtime

We’ll start by discussing how to analyze the time complexity of a method. Then, we’ll extend these techniques to discuss space complexity. To guide our discussion, we’ll consider the following two methods. The first is the paritySplit() method from last lecture (and its helper swap() method).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

/** 
 * Rearranges the elements of `a` so that all even elements appear before 
 * all odd elements. Returns the index of the first odd element, or returns
 * `a.length` if all elements are even.
 */
static int paritySplit(int[] a) {
  int i = 0;
  int j = a.length;
  /* Loop invariant: `a[..i)` is even, `a[j..]` is odd */
  while (i < j) {
    if (a[i] % 2 == 0) {
      i++;
    } else {
      swap(a,i,j-1);
      j--;
    }
  }
  return j;
}

/** 
 * Swaps the entries `a[x]` and `a[y]`. Requires that `0 <= x < a.length`
 * and `0 <= y < a.length`.
 */
static void swap(int[] a, int x, int y) {
  int temp = a[x];
  a[x] = a[y];
  a[y] = temp;
}

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

/** 
 * Rearranges the elements of `a` so that all even elements appear before 
 * all odd elements. Returns the index of the first odd element, or returns
 * `a.length` if all elements are even.
 */
static int paritySplit(int[] a) {
  int i = 0;
  int j = a.length;
  /* Loop invariant: `a[..i)` is even, `a[j..]` is odd */
  while (i < j) {
    if (a[i] % 2 == 0) {
      i++;
    } else {
      swap(a,i,j-1);
      j--;
    }
  }
  return j;
}

/** 
 * Swaps the entries `a[x]` and `a[y]`. Requires that `0 <= x < a.length`
 * and `0 <= y < a.length`.
 */
static void swap(int[] a, int x, int y) {
  int temp = a[x];
  a[x] = a[y];
  a[y] = temp;
}

The second is a new method that determines whether an array contains duplicate elements. As review from last lecture, try drawing out the array diagrams for both the inner and outer loops in this method and use this to write out their loop invariants.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

/** 
 * Returns whether `a` contains duplicate entries, distinct indices 
 * `i != j` with `a[i] == a[j]`.
 */
static boolean hasDuplicates(int[] a) {
    for (int i = a.length - 1; i >= 0; i--) {
        for (int j = 0; j < i; j++) {
            if (a[i] == a[j]) {
                return true;
            }
        }
    }
    return false;
}

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

/** 
 * Returns whether `a` contains duplicate entries, distinct indices 
 * `i != j` with `a[i] == a[j]`.
 */
static boolean hasDuplicates(int[] a) {
    for (int i = a.length - 1; i >= 0; i--) {
        for (int j = 0; j < i; j++) {
            if (a[i] == a[j]) {
                return true;
            }
        }
    }
    return false;
}

How should we think about reporting these methods’ runtimes? A first suggestion might be to list how much time it takes for them to execute: “paritySplit() executed in 0.026456 milliseconds.” However, there are some issues with this. What input was used? Since more elements must be inspected when a longer array is passed in, we might expect paritySplit() to be faster for smaller inputs. We can refine our statement to “paritySplit() executed in 0.026456 milliseconds on an input with 1000 entries.” This still has issues. Which input was used? Since less work is done for even entries than odd entries (no swap is necessary), we might expect the method to run faster when most of the array is even. If we report the exact input that was used, we’ll have a clear understanding of what was run, but it isn’t clear how we can use this information. What if we want to run the code on a different input, or a different sized input? Perhaps we could run the code for a large collection of inputs (with different sizes) and report all of these runtimes. We do this here for both of the above methods, each input array of length \(N\) consists of the numbers \(0,1,...,N-1\), in order.

These charts provide a better sense of the performance of the code. The time scale for the hasDuplicates() method is multiple orders of magnitude larger. Beyond this, the hasDuplicates() method has a runtime that gets steeper as the input array gets larger, whereas the paritySplit() method’s runtime (after some initial noise) appears to grow at a consistent rate. We’d like to find a more principled way to explain this difference in performance.

Reporting the “wall-clock” time required to execute a method is not a reliable metric for the code’s performance, as this depends on many factors external to the design of the code. First, it will vary based upon the hardware the code is running on. A modern system using advanced hardware will typically execute the same sequence of machine instructions at a faster rate than an older system. The execution time may also vary based what operating system is being used, which versions of a software package are installed, and what other programs are running at the same time. Even factors such as temperature can affect the “wall-clock” time. We can see some of this variance in the values on the left of the paritySplit() graph; for very short times, the effects of the “noise” from these factors overwhelm the “signal” that we get from the true execution time. To avoid these factors, we’d like a metric that will be consistent across executions of the same piece of code. We can achieve this consistency by measuring the time complexity of our code, the number of “basic” operations that the code performs. On a given system, we expect the execution speed to be in a roughly linear relationship with the operation count (meaning these metrics are well-aligned), and across systems, we expect the operation count to remain roughly stable.

Counting Basic Operations

One subtlety in this notion of runtime is what counts as a “basic operation”. For our purposes, every memory access (i.e., reading from or writing to a variable) and every computation (including the evaluation of arithmetic operations, logical comparisons, etc.) counts as one basic operation. This simple metric (which avoids considering different relative speeds of different operations, hardware concerns based on the size and memory location of certain variables or objects, etc.) will be sufficient for our purposes.

As we already noted above, the number of operations will depend on the size of the inputs to a method. Larger inputs will tend to require more operations to process. Therefore, we will express the operation count as a function of the input size, which we will parameterize using one or more variables. For each of our two example methods, which take a single int[] array as input, we can let \(N\) represent the length of this array and then find a function \(T(N)\) that expresses the number of operations in terms of this array length. We also noted earlier that this operation count may depend not only on the input size, but also the particular choice of input. To resolve this ambiguity, we will often focus on the worst-case time complexity; for a given value of \(N\), we’ll let \(T(N)\) be the maximum number of operations performed by the method for any input of size \(N\). We can similarly define the best-case time complexity as the minimum number of operations over all inputs of size \(N\).

Let’s go through some examples of computing a worst-case operation count. This will seem incredibly tedious. Later in the lecture, we’ll see a way that we can “short-cut” this accounting. We’ll start with the paritySplit() method, reproduced below (with documentation omitted for brevity):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

static int paritySplit(int[] a) {
  int i = 0;
  int j = a.length;
  while (i < j) {
    if (a[i] % 2 == 0) {
      i++;
    } else {
      swap(a,i,j-1);
      j--;
    }
  }
  return j;
}

static void swap(int[] a, int x, int y) {
  int temp = a[x];
  a[x] = a[y];
  a[y] = temp;
}

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

static int paritySplit(int[] a) {
  int i = 0;
  int j = a.length;
  while (i < j) {
    if (a[i] % 2 == 0) {
      i++;
    } else {
      swap(a,i,j-1);
      j--;
    }
  }
  return j;
}

static void swap(int[] a, int x, int y) {
  int temp = a[x];
  a[x] = a[y];
  a[y] = temp;
}

First, we can think about the swap() helper method. This method consists of 3 assignment statements that each read from and write to one variable, as well as 4 array index operations, for a total of 10 operations. Now, let’s go through the body of the paritySplit() method line-by-line. For each line, we’ll keep track of two things, the number of operations performed on that line and the number of times (in the worst case) that the line is executed during the method call.

• Line 2 is executed once and assigns an int literal to a variable, requiring 1 operation.
• Line 3 is executed once. It must evaluate a.length (1 operation) and then assign this result to j (1 operation), a total of 2 operations.
• The loop guard on line 4 consists of two variable accesses (i and j) and one comparison, for a total of 3 operations. At the start i and j are a.length = \(N\) units apart, and they get 1 unit closer in each iteration. Thus, the loop guard will evaluate to true \(N\) times and false once for a total of \(N+1\) executions.
• The boolean expression within the condition on line 5 requires 4 operations (access i, access a[i], the modulus computation, and the == comparison). It is executed once per loop iteration for a total of \(N\) times.
• The else branch of this conditional structure performs more operations than the if branch, so a worst-case analysis would assume we always follow the else branch. Within this branch, there are a total of 20 operations performed: we evaluate each argument to swap() (4 operations, including the subtraction), write their values to the parameters in the swap() stack frame (3 operations), run the swap() method (10 operations, from above), and then decrement j (3 operations, read its value, subtract 1, and write its new value). This else body is executed once per loop iteration for a total of \(N\) times.
• The return statement on line 12 is executed once and requires 1 operation (a variable access).

We summarize our findings in the following table. To obtain the total operation count, we multiply the entries in the second and third column of each row (to get the total operation count from each line) and then add the entries in the final column.

In total, we find that \(T(N) = 27N+7\) operations are performed by paritySplit() in the worst case.

Let’s do a similar analysis for the hasDuplicates() method.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

static boolean hasDuplicates(int[] a) {
  for (int i = a.length - 1; i >= 0; i--) {
    for (int j = 0; j < i; j++) {
      if (a[i] == a[j]) {
        return true;
      }
    }
  }
  return false;
}

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

static boolean hasDuplicates(int[] a) {
  for (int i = a.length - 1; i >= 0; i--) {
    for (int j = 0; j < i; j++) {
      if (a[i] == a[j]) {
        return true;
      }
    }
  }
  return false;
}

• The for-loop declaration on Line 2 consists of three statements. The initialization consists of an assignment to a.length-1 (4 operations) and is executed once. The loop guard compares a variable and a literal (2 operations) and is executed \(N+1\) times in the worst case (since the loop runs for a.length\(=N\) iterations). The decrement (3 operations) is executed once per outer-loop iteration for a total of \(N\) times in the worst case.
• The for-loop declaration on Line 3 consists of three statements. The initialization consists of an assignment statement to a literal (1 operation) and is executed once per iteration of the outer loop for a total of \(N\) times. The loop guard compares two variables (3 operations) and is executed \(i+1\) times (corresponding to the \(i\) iterations of the inner loop) during the \(i\)’th iteration of the outer loop, for a total of \( \sum_{i=0}^{N-1} (i+1) \) times. The increment (3 operations) is executed once per inner-loop iteration for a total of \( \sum_{i=0}^{N-1} i \) times.
• The boolean expression within the condition on line 4 requires 5 operations (access i,j,a[i],a[j], and the comparison). It is executed once per inner-loop iteration for a total of \( \sum_{i=0}^{N-1} i \) times.
• The early return statement on line 5 would cause the method’s execution to end after fewer than the worst-case number of operations, so we assume it is never reached in our worst-case analysis.
• The return statement on line 9 is executed once and doesn’t require any computation (we’re returning a boolean literal).

In our tabular form:

Doing some algebra, and using the fact that \(\displaystyle\sum_{i=0}^{N-1} i = \frac{N(N-1)}{2}\), we find that hasDuplicates() has a worst-case runtime of

Plotting these functions \(T(N)\) for both methods displays the same qualitative behavior as the time plots.

The process that we just completed is a lot of work, and we’d like a way to summarize these findings in a simpler, yet still theoretically-sound, way. Asymptotic notation affords us this ability.

Asymptotic Notation

As we noted above, the time complexity of a method is a description of how quickly the number of operations it performs increases as the size of its inputs increase. We can model the worst-case time complexity as a function \(T\) in parameters of the input size (such as the array length \(N\) in our earlier examples). Since we primarily care about the scaling of this function, particularly for very large inputs where performance concerns become important, we can extract some information away.

• We can extract away any constant coefficients: whether a method performs \(27N\) operations or \(30N\) operations is not such a significant distinction, since the exact time for these operations varies between systems (and different ways of counting basic operations will result in different constants).
• We can extract away any “lower-order” terms: in the hasDuplicates() runtime \(T(N) = \tfrac{11}{2} N^2 + \tfrac{7}{2} N + 6\), the \(\tfrac{11}{2} N^2\) term plays a much bigger role than \(\tfrac{7}{2} N + 6\) for large values of \(N\). For example, when \(N = 1,000,000\), this method will require \(T(1,000,000) = 5,500,003,500,006 \) basic operations in the worst case, and almost all (\(5,500,000,000,000\)) of these operations come from the \( \tfrac{11}{2} N^2 \) term.

After these simplifications, our paritySplit() worst-case runtime simplifies to \(T(N) = 27N + 7 \approx N\) and our hasDuplicates() worst-case runtime simplifies to \(T(N) = \tfrac{11}{2} N^2 + \tfrac{7}{2} N + 6 \approx N^2\). The parabolic function \(N^2\) grows faster than the linear function \(N\), so these simplifications provide us enough information to conclude that paritySplit() will have a better asymptotic performance than hasDuplicates().

We can formalize this idea of “simplifying” a function to its primary scaling term(s) using asymptotic or big-O complexity classes.

This definition probably looks a bit scary, and mastering its formalism is beyond what we expect in this course. Rather, we’ll focus on the intuitive meaning of this definition and how we can apply it in practice to reason about complexities. When we apply this definition, our runtime \(T(N)\) will play the role of the \(f\) function, and the simpler function (dropping the coefficients and lower-order terms) plays the role of \(g\). A complexity class like \(O(N)\) includes all functions whose scaling in \(N\) is no-worse-than linear, so the worst-case runtime of paritySplit() is \(O(N)\). The complexity class \(O(N^2)\) includes all functions whose scaling in \(N\) is no-worse-than quadratic, which includes the worst-case runtimes of both hasDuplicates() and paritySplit().

In practice, it is useful to have a “catalog” of common complexity classes at your disposal that you can use to describe method runtimes. The following table summarizes most of the runtimes that we will encounter throughout the course. The complexity classes are listed in order, with the smallest (i.e., best) at the top.

We can use these complexity classes to get a good estimate of how the “wall-clock” execution time of an algorithm will vary with its input size. For example, suppose I report to you that hasDuplicates() took 5 seconds to run on an array with 250,000 elements. If you have an array with 1,000,000 (4 times more) elements, you should expect hasDuplicates() to take around \(5 \cdot 4^2 = 80\) seconds since it has an \(O(N^2)\) worst-case runtime.

Simplified Time Complexity Analysis

Since we ultimately care only about finding a complexity class that describes a method’s runtime rather than the exact function \(T(N)\) we can take short-cuts in our analysis that leverage the following properties of asymptotic notation.

Practically, this says that if we split a method (or loop) body into multiple sequential steps, the runtime of the slowest step will dominate and determine the asymptotic complexity. All other steps can be dropped from the analysis.

Practically, this says that if we repeat a block of code (e.g., in the body of a loop), then we can determine (an upper bound on ) the total contribution of this block to the runtime complexity by multiplying the complexity class of the block by the complexity class of its number of repetitions.

Let’s apply these rules to re-do the complexity analysis of paritySplit():

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

static int paritySplit(int[] a) {
  int i = 0;
  int j = a.length;
  while (i < j) {
    if (a[i] % 2 == 0) {
      i++;
    } else {
      swap(a,i,j-1);
      j--;
    }
  }
  return j;
}

static void swap(int[] a, int x, int y) {
  int temp = a[x];
  a[x] = a[y];
  a[y] = temp;
}

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

static int paritySplit(int[] a) {
  int i = 0;
  int j = a.length;
  while (i < j) {
    if (a[i] % 2 == 0) {
      i++;
    } else {
      swap(a,i,j-1);
      j--;
    }
  }
  return j;
}

static void swap(int[] a, int x, int y) {
  int temp = a[x];
  a[x] = a[y];
  a[y] = temp;
}

First, we note swap() has an \(O(1)\) runtime complexity since nothing in the method body depends on the length of a (there are no loops or additional method calls). In the definition of paritySplit() the loop will dominate the \(O(1)\) runtime of the initialization and return. It runs for \(O(N)\) iterations and does only a constant amount of work (a fixed number of calculations and a call to an \(O(1)\) method) per iteration. Thus, the overall worst-case runtime is \(O(N)\).

We can do a similar, simplified analysis for hasDuplicates():

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

static boolean hasDuplicates(int[] a) {
  for (int i = a.length - 1; i >= 0; i--) {
    for (int j = 0; j < i; j++) {
      if (a[i] == a[j]) {
        return true;
      }
    }
  }
  return false;
}

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

static boolean hasDuplicates(int[] a) {
  for (int i = a.length - 1; i >= 0; i--) {
    for (int j = 0; j < i; j++) {
      if (a[i] == a[j]) {
        return true;
      }
    }
  }
  return false;
}

Here, the outer for-loop runs for \(O(N)\) iterations. For a particular outer loop iteration \(i\), the inner for-loop runs for \(O(i)\), which we can bound by \(O(N)\), iterations. Thus, the code in the inner-loop body runs \(O(N^2)\) times and involves \(O(1)\) operations, so the worst-case runtime of the method is \(O(N^2)\).

Search Algorithms

Let’s combine the concept of loop invariants from last lecture with the concept of time complexity to develop and analyze two search algorithms. A search algorithm takes in a data structure and a value v (of the type stored in the data structure) and returns whether v is present and/or where it can be found in the data structure. For today, we’ll focus on searching for an int in an int[] array and develop two algorithms, linear search and binary search.

Linear Search

The most basic way to search an array is by performing a “linear scan” over its entries, starting at the beginning of the array and looking at the entries until we either find v or reach the end of the array. We call this algorithm linear search.

1
2
3
4
5

/**
 * Returns the smallest index `i` such that `a[i] == v` or returns 
 * `a.length` if `a` does not contain `v`. 
 */
static int linearSearch(int[] a, int v) { ... }

1
2
3
4
5

/**
 * Returns the smallest index `i` such that `a[i] == v` or returns 
 * `a.length` if `a` does not contain `v`. 
 */
static int linearSearch(int[] a, int v) { ... }

We will need a variable i to keep track of the next index of a to inspect. Let’s summarize the behavior of this method using array diagrams. At the start, have no information about the contents of a.

When we exit the loop, i should refer to the first index of a containing v, or we should have confirmed that v is not in a.

When we begin the iteration corresponding to a particular value of i, we are guaranteed that all entries in a[..i) do not contain v (our loop invariant), otherwise the method would have returned.

Using the reasoning from the previous lecture, we should initialize i = 0 and guard our loop on the condition i < a.length. In the loop body, we inspect a[i] and check if it is equal to v. If so, we can return i early, since we will have met the post-condition. Otherwise, we can increment i and maintain the loop invariant.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

/**
 * Returns the smallest index `i` such that `a[i] == v` or returns 
 * `a.length` if `a` does not contain `v`. 
 */
static int linearSearch(int[] a, int v) { 
  int i = 0;
  /* Loop invariant: `a[..i)` does not contain `v`. */
  while (i < a.length) {
    if (a[i] == v) {
      return i;
    }
    i++;
  }
  return a.length;
}

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

/**
 * Returns the smallest index `i` such that `a[i] == v` or returns 
 * `a.length` if `a` does not contain `v`. 
 */
static int linearSearch(int[] a, int v) { 
  int i = 0;
  /* Loop invariant: `a[..i)` does not contain `v`. */
  while (i < a.length) {
    if (a[i] == v) {
      return i;
    }
    i++;
  }
  return a.length;
}

We can streamline the code a bit more by incorporating the inspection of a[i] into the loop guard.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

/**
 * Returns the smallest index `i` such that `a[i] == v` or returns 
 * `a.length` if `a` does not contain `v`. 
 */
static int linearSearch(int[] a, int v) { 
  int i = 0;
  /* Loop invariant: `a[..i)` does not contain `v`. */
  while (i < a.length && a[i] != v) {
    i++;
  }
  return i;
}

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

/**
 * Returns the smallest index `i` such that `a[i] == v` or returns 
 * `a.length` if `a` does not contain `v`. 
 */
static int linearSearch(int[] a, int v) { 
  int i = 0;
  /* Loop invariant: `a[..i)` does not contain `v`. */
  while (i < a.length && a[i] != v) {
    i++;
  }
  return i;
}

The runtime of this method (both variants) is dominated by the while loop, which runs for \(O(N)\) iterations in the worst-case (if all elements are inspected) and performs \(O(1)\) work per iteration. Fittingly, the worst-case runtime of linear search is \(O(N)\), linear. This is the best runtime that we can hope for if we have no additional information about the contents of a, since we may need to check all indices. Next, we’ll see that if we have additional information about a, we can leverage this in our search to avoid checking many entries and improve the worst-case runtime.

Binary Search

If we want to find a value in an array that we know is sorted, we can use this knowledge to cleverly streamline our search. If you’ve ever looked up a word in a (printed) dictionary, you are probably familiar with this approach. If you want to find the definition of “java”, you won’t linearly scan the dictionary, reading all of the entries starting with the first word “a” until your reach “java”. Instead, you’ll flip to an arbitrary page toward the middle. If you happen to land in the “k” section, you’ll know that “java” can’t come after the page you are on (dictionaries list words in sorted order), so you can focus the search on the first half. If you instead land in the “h” section, you’ll know “java” can’t come before the page you are on, so you can focus the search on the second half.

We’ll follow a similar approach in binary search, using two variables l and r to maintain bounds on a window where v could be. Initially this window will cover the entire array, but it will shrink in each iteration. Once the window is one-entry wide, we’ll either have located an index of v or confirmed that the v is not present. The spec for (our implementation of) binarySearch() looks a bit different than linearSearch():

1
2
3
4
5

/**
 * Returns the value `r` with `0 <= r <= a.length` such that `a[..r) < v`
 * and `a[r..) >= v`. Requires that `a` is sorted (in ascending order). 
 */
static int binarySearch(int[] a, int v) { ... }

1
2
3
4
5

/**
 * Returns the value `r` with `0 <= r <= a.length` such that `a[..r) < v`
 * and `a[r..) >= v`. Requires that `a` is sorted (in ascending order). 
 */
static int binarySearch(int[] a, int v) { ... }

We have switched from calling the return value i to r (for reasons that will soon be clear). We have also added the sorted pre-condition on a. Most notably, we handle the absence of a differently. In linearSearch() we used the sentinel value a.length to signal the absence of v. In binarySearch(), we can do better. Since a is sorted, we can report to the client the index where a could be inserted and preserve the sorted order. This will be beneficial in settings where binary search is used as a subroutine in an application that modifies the contents of an array/list but wishes to preserve sorted order. Note that when v is present in a, the return value coincides with that of linearSearch(), the index of the first occurrence of v.

Let’s draw out the array diagrams for this method. In the “Pre” diagram, we can now document the fact that the array is sorted.

We can use the specification about the return value to draw the “Post” diagram.

Finally, in the invariant diagram, l and r will delineate a shrinking “middle” window where the first occurrence of v could be.

Now, let’s define the binarySearch() method using these diagrams. Using the “Pre” and “Inv” diagrams, we see that we should initialize l = 0 and r = a.length. Using the “Inv” and “Post” diagrams, we should continue iterating until l == r, so we can select the loop guard l < r (or l != r). The return value should be r (or, equivalently, l).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

/**
 * Returns the value `r` with `0 <= r <= a.length` such that `a[..r) < v`
 * and `a[r..) >= v`. Requires that `a` is sorted (in ascending order). 
 */
static int binarySearch(int[] a, int v) { 
  int l = 0; // left window boundary (inclusive)
  int r = a.length; // right window boundary (exclusive)
  /* Loop invariant: `a[..l) < v`, `a[r..] >= v` */
  while (l < r) {

  }
  return r;
}

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

/**
 * Returns the value `r` with `0 <= r <= a.length` such that `a[..r) < v`
 * and `a[r..) >= v`. Requires that `a` is sorted (in ascending order). 
 */
static int binarySearch(int[] a, int v) { 
  int l = 0; // left window boundary (inclusive)
  int r = a.length; // right window boundary (exclusive)
  /* Loop invariant: `a[..l) < v`, `a[r..] >= v` */
  while (l < r) {

  }
  return r;
}

It remains to develop the loop body. In each iteration, we’d like to inspect one entry of a and use this to shrink the window of possible return values. The best choice of entry is the midpoint of the window, as this will allow us to either eliminate the entire left half (if this entry is smaller than v) or right half (if this entry is at least v) of the search window. The midpoint index is given by m = (l + r) / 2, but we can use the similar expression m = l + (r - l)/2 to avoid the possibility of overflow.

Let’s carefully consider the possibilities after we inspect a[m].

• If a[m] < v, then every value in a[..m] < v, so this range should be entirely to the left of our window. We can update l = m + 1 and maintain the loop invariant.
• If a[m] >= v, then every value in a[m..] >= v, so this range should be entirely to the right of our window (note that the right endpoint of the window is exclusive). We can update r = m and maintain the loop invariant.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

/**
 * Returns the value `r` with `0 <= r <= a.length` such that `a[..r) < v`
 * and `a[r..) >= v`. Requires that `a` is sorted (in ascending order). 
 */
static int binarySearch(int[] a, int v) { 
  int l = 0; // left window boundary (inclusive)
  int r = a.length; // right window boundary (exclusive)
  /* Loop invariant: `a[..l) < v`, `a[r..] >= v` */
  while (l < r) {
    int m = l + (r - l)/2; // midpoint
    if (a[m] < v) {
      l = m + 1;
    } else {
      r = m;
    }
  }
  return r;
}

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

/**
 * Returns the value `r` with `0 <= r <= a.length` such that `a[..r) < v`
 * and `a[r..) >= v`. Requires that `a` is sorted (in ascending order). 
 */
static int binarySearch(int[] a, int v) { 
  int l = 0; // left window boundary (inclusive)
  int r = a.length; // right window boundary (exclusive)
  /* Loop invariant: `a[..l) < v`, `a[r..] >= v` */
  while (l < r) {
    int m = l + (r - l)/2; // midpoint
    if (a[m] < v) {
      l = m + 1;
    } else {
      r = m;
    }
  }
  return r;
}

Note that our definition of m ensures that \(l <= m < r\) whenever \(l < r\). Thus, in both cases, the width of the search window \(r-l\) decreases by the end of the loop iteration, ensuring termination. The following animation steps through the execution of binarySearch() on an array with length 10.

Finally, let’s consider the runtime of binarySearch(). Each iteration of the while-loop performs \(O(1)\) work, so the final runtime will be the number of iterations. In each iteration, the width of the search window is at most half of its width in the previous iteration. The width is initially \(N\), and the loop terminates when the width is reduced to 0. Thus, the number of loop iterations, which we’ll denote \(i\), is upper bounded by the number of times that we can repeatedly halve \(N\) and remain \(\geq 1\). Mathematically, \(N \cdot \big(\frac{1}{2}\big)^i \geq 1\), which rearranges to \( i \leq \log_2(N) \); the exact formula uses the “floor” function, \( i = \lfloor \log_2(N) \rfloor + 1 = O(\log N)\). Thus, binarySearch() has an \(O(\log N)\) runtime, a significant improvement over the \(O(N)\) runtime of linearSearch() (but with the extra pre-condition of a sorted array).

Space Complexity

Just as we can use asymptotic notation to describe the time complexity of a method, we can also use it to describe the space complexity.

The main subtlety of this definition is the phrase “at one time.” Unlike time, space can be re-used (i.e., memory can be over-written) at a different point in the computation. The notion of space complexity seeks to bound how much physical memory space will be needed to execute a method, which concerns an instantaneous amount of memory that is simultaneously in use.

All of the methods that we have looked at so far today have O(1) space complexity, since they only allocate a constant number of local variables during their execution (remember, the int[] argument that is passed to swap() in paritySplit() is a reference, so the entire array is not copied). Methods that allocate new objects as “scratch space” for their computations can have larger space complexities. Additionally, the space complexity of recursive methods can be more nuanced, as we will discuss in the next lecture.

A Subtle Example

We’ll end today with an example that shows how we need to be careful when analyzing time and space complexity. Consider the following method:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

/** 
 * Returns the string consisting of `n` asterisks (*).
 */
static String starString(int n) {
  String s = "";
  int i = 0;
  /* Loop invariant: `s` is the string of `i` asterisks. */
  while (i < n) {
    s = s + "*";
    i++;
  }
  return s;
}

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

/** 
 * Returns the string consisting of `n` asterisks (*).
 */
static String starString(int n) {
  String s = "";
  int i = 0;
  /* Loop invariant: `s` is the string of `i` asterisks. */
  while (i < n) {
    s = s + "*";
    i++;
  }
  return s;
}

What are the time and space complexities of this method, expressed as functions of \(n\)? Try working this yourself before reading on and checking your answers.

The key to understanding these complexity analyses is the line s = s + "*"; which reassigns s to the concatenation of s and “*”. Recall that in Java, Strings are immutable, meaning the concatenation operation constructs (i.e., allocates in heap memory) a new String object containing the characters of both argument Strings. If the concatenated String has length l, this operation requires both \(O(l)\) time (to copy over the characters to this new String) and \(O(l)\) additional space. In total, \( \sum_{l=1}^{n} l = O(n^2) \) time is required for these repeated concatenations, and \(O(n^2)\) space is utilized in the worst-case to construct the \(n\) String objects used over the course of the method (which we have no guarantee are cleared before the method returns). We can improve the performance of this code by substituting in a StringBuilder object (likely recommended by IntelliJ if you copy this code over), a provided Java class that enables more efficient concatenation. This example demonstrates the importance of carefully considering the runtime of all operations or calls that you use in your methods.

Exercises

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

static double variance(int[] a) {
  double sum = 0;
  double sumSq = 0;
  for (int i = 0; i < a.length; i++) {
    sum += a[i];
    sumSq += a[i] * a[i];
  }
  double mean = sum / a.length;
  return (sumSq / a.length) - (mean * mean); 
}

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

static double variance(int[] a) {
  double sum = 0;
  double sumSq = 0;
  for (int i = 0; i < a.length; i++) {
    sum += a[i];
    sumSq += a[i] * a[i];
  }
  double mean = sum / a.length;
  return (sumSq / a.length) - (mean * mean); 
}

1
2
3
4
5
6
7

static int skip(int[] a) {
    int count = 0;
    for (int i = 0; i < a.length; i *= 3) {
        count += i;
    }
    return count;
}

1
2
3
4
5
6
7

static int skip(int[] a) {
    int count = 0;
    for (int i = 0; i < a.length; i *= 3) {
        count += i;
    }
    return count;
}

1
2
3
4
5
6

static void foo(int[] a) {
  int i = 0;
  while (i < a.length) {
    i = a[i] < 0 ? i + 2 : 2 * i;
  }
}

1
2
3
4
5
6

static void foo(int[] a) {
  int i = 0;
  while (i < a.length) {
    i = a[i] < 0 ? i + 2 : 2 * i;
  }
}

1
2
3
4
5
6
7
8
9

static void bar(int[] a) {
  for (int i = 0; i < a.length; i++) {
    for (int j = i; j < a.length; j++) {
      for (int k = 0; k < 100; k++) {
        // Do a constant amount of work
      }
    }
  }
}

1
2
3
4
5
6
7
8
9

static void bar(int[] a) {
  for (int i = 0; i < a.length; i++) {
    for (int j = i; j < a.length; j++) {
      for (int k = 0; k < 100; k++) {
        // Do a constant amount of work
      }
    }
  }
}

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

/**
 * Requires `a` and `b` are sorted in ascending order.
 */
static int[] numGreaterThan(int[] a, int[] b) {
  int[] r = new int[a.length];
  int j = 0;
  for (int i = 0; i < a.length; i++) {
    while (j < b.length && b[j] < a[i]) {
      j++;
    }
    r[i] = j;
  }
  return r;
}

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

/**
 * Requires `a` and `b` are sorted in ascending order.
 */
static int[] numGreaterThan(int[] a, int[] b) {
  int[] r = new int[a.length];
  int j = 0;
  for (int i = 0; i < a.length; i++) {
    while (j < b.length && b[j] < a[i]) {
      j++;
    }
    r[i] = j;
  }
  return r;
}

1
2
3
4
5
6
7

static int[] method1(int[] nums) {
  int[] half = new int[nums.length / 2];
  for (int i = 0; i < nums.length / 2; i++) {
    half[i] = nums[i];
  }
  return half;
}

1
2
3
4
5
6
7

static int[] method1(int[] nums) {
  int[] half = new int[nums.length / 2];
  for (int i = 0; i < nums.length / 2; i++) {
    half[i] = nums[i];
  }
  return half;
}

1
2
3
4
5
6
7

static int method2(int[] nums) {
  int i = 0;
  while (i < nums.length && linearSearch(nums, i) < nums.length) {
    i++;
  }
  return i;
}

1
2
3
4
5
6
7

static int method2(int[] nums) {
  int i = 0;
  while (i < nums.length && linearSearch(nums, i) < nums.length) {
    i++;
  }
  return i;
}

1
2
3
4
5
6
7
8

/**
 * Requires `nums` is sorted in ascending order.
 */
static void method3(int[] nums) {
  for (int i = 0; i < nums.length; i++) {
    System.out.println(binarySearch(nums, i));
  }
}

1
2
3
4
5
6
7
8

/**
 * Requires `nums` is sorted in ascending order.
 */
static void method3(int[] nums) {
  for (int i = 0; i < nums.length; i++) {
    System.out.println(binarySearch(nums, i));
  }
}

1
2
3
4

/**
 * Returns whether `v` is in `a`.
 */
static boolean binarySearchExists(int[] a, int v) { ... }

1
2
3
4

/**
 * Returns whether `v` is in `a`.
 */
static boolean binarySearchExists(int[] a, int v) { ... }

1
2
3
4
5

/**
 * Returns the value `l` with `0 <= l <= a.length` such that `a[..l) <= v`
 * and `a[l..) > v`. Requires that `a` is sorted (in ascending order). 
 */
static int rightBinarySearch(int[] a, int v) { ... }

1
2
3
4
5

/**
 * Returns the value `l` with `0 <= l <= a.length` such that `a[..l) <= v`
 * and `a[l..) > v`. Requires that `a` is sorted (in ascending order). 
 */
static int rightBinarySearch(int[] a, int v) { ... }

1
2
3
4
5

/**
 * Returns the frequency of `v` in `a`. Requires that `a` is sorted 
 * (in ascending order). 
 */
static int countFrequency(int[] a, int v) { ... }

1
2
3
4
5

/**
 * Returns the frequency of `v` in `a`. Requires that `a` is sorted 
 * (in ascending order). 
 */
static int countFrequency(int[] a, int v) { ... }

1
2
3
4

/**
 * Returns whether `v` is in `a`.
 */
static boolean search2D(int[][] a, int v) { ... }

1
2
3
4

/**
 * Returns whether `v` is in `a`.
 */
static boolean search2D(int[][] a, int v) { ... }

1
2
3
4
5

/**
 * Returns whether `v` is in `a`. Requires the rows and columns of `a`
 * to be sorted in ascending order.
 */
static boolean saddlebackSearch(int[][] a, int v) { ... }

1
2
3
4
5

/**
 * Returns whether `v` is in `a`. Requires the rows and columns of `a`
 * to be sorted in ascending order.
 */
static boolean saddlebackSearch(int[][] a, int v) { ... }

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

/**
 * Returns a mirrored version of the input string by appending its reverse
 * to the end of the original string.
 */
static String mirrorString(String input) { 
  String result = "";
  for (int i = 0; i < input.length; i++) {
    result += input.charAt(i);
  }
  for (int i = input.length - 1; i >= 0; i--) {
    result += input.charAt(i);
  }
  return result;
}

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

/**
 * Returns a mirrored version of the input string by appending its reverse
 * to the end of the original string.
 */
static String mirrorString(String input) { 
  String result = "";
  for (int i = 0; i < input.length; i++) {
    result += input.charAt(i);
  }
  for (int i = input.length - 1; i >= 0; i--) {
    result += input.charAt(i);
  }
  return result;
}

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

/**
 * Returns a string formed by concatenating all prefixes of the input string.
 */
static String progressiveConcat(String input) { 
  String result = "";
  for (int i = 0; i < input.length; i++) {
    result += input.substring(0, i);
  }
  return result;
}

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

/**
 * Returns a string formed by concatenating all prefixes of the input string.
 */
static String progressiveConcat(String input) { 
  String result = "";
  for (int i = 0; i < input.length; i++) {
    result += input.substring(0, i);
  }
  return result;
}

1
2
3
4
5
6

/**
 * Returns an array `ranges` of size `queries.length` such that `ranges[i]` is the sum
 * of the elements in `a[queries[i][0]..queries[i][1]]`. Requires `queries[i].length` 
 * = 2 and 0 <= `queries[i][0]` <= `queries[i][1]` < `a.length`.
 */
static int[] rangeQueries(int[] a, int[][] queries) { ... }

1
2
3
4
5
6

/**
 * Returns an array `ranges` of size `queries.length` such that `ranges[i]` is the sum
 * of the elements in `a[queries[i][0]..queries[i][1]]`. Requires `queries[i].length` 
 * = 2 and 0 <= `queries[i][0]` <= `queries[i][1]` < `a.length`.
 */
static int[] rangeQueries(int[] a, int[][] queries) { ... }