map/reduce, List Comprehensions & other Python Features I Love - Part 1

Python is not functional, but it provides many features found typically in a functional programming language. Vectorized operations, which apply a computation to an entire list using only a single line of code, are one such feature that I enjoy using in Python. Vectorization makes for concise syntax and allows you to apply powerful computations to data in a simple and quick way.

Here I’ll summarize three functions and features, map(), reduce() and list comprehensions, that provide programmers with the ability to create a compact representation for repetitive computations over data structures.

0.1 map()

The best way to understand map() is to illustrate its use with a simple example. Say you have a collection of data in a list and you want to apply the same computation to all the elements. How could map() help you with this? Let’s see:

1
2

ages = [12,14,15,16,2,3,4,20,\
        89,45,34,23,34,54,34,23]

In the list above, we’ve stored ages of driver’s license applicants and we want to determine if they are eligible to apply for a driver’s license based on their age. To do so, we’ll process this data and then create another list which will contain either “true” or “false” values based on whether each individual is old enough to quality for a drivers license.

One way to create our True/False list is to iterate through the list of ages, check a condition and then populate the new True/False list with the corresponding value for each age.

But using vectorization, could we get away without writing a loop? With map() we can apply any function to each element of the list, using just a single line of code. Here’s some syntax:

1
2
3
4
5
6
7
8
9

def check_dl_age(x):
    if(x >= 18 and x <= 85):
        return(True)
    else:
        return(False)

dl_qualify = map(check_dl_age, ages)

bool_vals = list(dl_qualify)

When we run this program, we see the output:

1
2
3

[False, False, False, False, False, \
False, False, True, False, True, True,\
 True, True, True, True, True]

Pay special attention to line 7, where we map each element of the ages list to the computation (or function) check_dl_age. This function is mapped or applied to each list element and a resulting list of return values is created, which we store as the list dl_qualify.

There you go! No loops. Just a single call to map and we’re done.

0.2 reduce()

Now taking this example further, what if we wanted to evaluate the list of driver’s license eligibilities to determine if they were all OK, or whether a particular batch of submissions had some inelegible candidates. Perhaps each one of these batches of applications are being submitted by a clerk and we expect that inelegible applications would already be filtered out. So, by checking if inelegible applications are making it through to us we are really trying to determine if the humans in the loop are doing their job properly. Those pesky humans!

How might we evaluate all the records in this list? The simplest approach would be to use Python’s “in” operation which allows the existence of an element to be verified in a list. But we’re trying to demonstrate the reduce() operation, so we’re going to go about out job a bit differently.

How’s this for a plan? If we could apply a logical AND operation to all the list elements we would get a “True” result only if all the elements in the list were True. So, if no mistakes were made by our human clerk, then we’d get True as an answer. But how do we apply a two-operand operation such as AND to all the elements of a list? That’s where reduce() comes in!

Here’s some code:

1
2
3
4
5

def apply_and(x,y):
    return(x and y)

result = reduce(apply_and,bool_vals)
print(result)

When we run it, we get the answer “False”

Basically, reduce starts off by running our apply_and function with the first two elements of our list (first pair) as arguments. It then computes the output of the function and pairs the return value with the next element in the list. So, if a list is:

1

[True, False, True, False]

reduce will apply apply_and first to the first two elements:

1

True AND False = False

Then, it will take this output False and AND it with the next element of the list, which is True, so:

1

False AND True = False

and then the last element of the list.

1

False AND False = False

You could achieve the same result with a loop, but reduce() cuts all the code down to a single line and may have performance advantages. If you combine it with lambda functions, it’s even more compact! But we’ll save further discussion about lambda functions for some other day.

0.3 List Comprehensions

I use list comprehensions all the time and they are one of my absolute favorites. The idea behind list comprehensions can take some time to really “gel” if you’re new to the concept. Essentially, comprehensions are a way to generate a new list in a vectorized manner with support for conditionals. Let me unpack that a bit with an example:

1

new_list = [x*2 for x in range(10)]

This single line create a new list which multiplies each number from 1 to 9 (i.e. range(10)) by 2 and stores the result in a list called new_list.

Here is the output:

1
2
3

print(new_list)

[0, 2, 4, 6, 8, 10, 12, 14, 16, 18]

But list comprehensions can do much more. For example, what if you wanted to create a list with such numbers but exclude any multiple of 10. You could add a condition to your list comprehension to achieve exactly such an outcome.

1
2
3
4

new_list = [x*2 for x in range(10) if (x*2)%10 != 0]
print(new_list)

[2, 4, 6, 8, 12, 14, 16, 18]

We see that we’ve now excluded 0 and 10, both of which are multiples of 10.

This is just the tip of the iceberg in terms of what’s possible with list comprehensions. You can create multi-dimensional lists, you can filter existing lists, apply operations to multiple lists and combine the results in a new list and much, much more.

0.4 Conclusion of Part I

I hope you found some of these examples helpful. If you program in Python but haven’t used these three delightful features, give them a try! You’ll soon find them indispensible to your programming tasks.