Python : List Comprehension vs Generator expression explained with examples

In this article we will discuss the differences between list comprehensions and Generator expressions.

Need of Generator Expression ?

Sometimes we want to operate on selected elements only and that too one by one. For such scenarios we should avoid keeping unnecessary elements in memory by creating lists or set or tuple etc.

Let’s understand by an example,
Suppose we want to calculate the sum of all the numbers divisible by 3 & 5 in the range 1 to 1000.

Now let’s see how to do that using list comprehension,

# Create a list of numbers which are divisible by 3 & 5 and are in range from 1 to 1000
listOfNums = [n for n in range(1000) if n % 3 == 0 and n %5 == 0]

# get the sum of all numbers in list
total = 0
for num in listOfNums:
    total += num

print('Total = ', total)

Output:

Total =  33165

We created a list of numbers i.e. listOfNums and the iterated over each element in the list to calculate the sum. Here creating a list of so many elements and keeping them in memory was unnecessary and could be avoided using Generators.

Generators in Rescue

Getting the Sum of selected numbers using Generators

As we wanted the sum only, therefore there is no need to create a heavy list of these numbers and then calculate the sum. The efficient way is creating a Generator for these selected numbers and then iterate over the selected numbers one by one using generator object and get the sum. This way we will avoid keeping all the elements unnecessary in memory.

Let’s create a Generator that yields numbers divisible by 3 & 5 one by one i.e.

def selectedNumbers():
    ''' A Generator that yields multiples of 3 & 5 in range 0 to 1000'''
    for num in range(1000):
        if num % 3 == 0 and num % 5 == 0:
            yield num

selectedNumbers() function returns a Generator object. A Generator is like an Iterator using which we can iterate over all the yielded values one by one i.e.

# Get a Generator Object
generatorObj = selectedNumbers()

# Iterate over yielded values one by one and calculate the sum
total = 0
for num in generatorObj:
    total += num

print('Total = ', total)

Output:

Total =  33165

It does not keep any thing in memory, during iteration Generator object calls the associated function again and again for next yielded value.
To know more about Generators check this article –> Python : Yield Keyword and Generators explained with examples

So, we avoided keeping unnecessary numbers in memory using Generator. But, do we always need to create a functions for creating Generator?
The answer is no. Here comes the Generator Expression in picture.

What is a generator expression ?

In python, a generator expression is used to generate Generators. It looks like List comprehension in syntax but (} are used instead of [].
Let’s get the sum of numbers divisible by 3 & 5 in range 1 to 1000 using Generator Expression.

Create a Generator expression that returns a Generator object i.e.

# Get a Generator object using Generator Expression
generatorObj = (n for n in range(1000) if n % 3 == 0 and n %5 == 0)

This generator expression did exactly same thing as function selectedNumbers() i.e. it returned the Generator object. Now let’s use this Generator object to Iterate over all the numbers and get their sum i.e.

# Iterate over yielded values one by one and calculate the sum
total = 0
for num in generatorObj:
    total += num

print('Total = ', total)

Output:

Total =  33165

Instead of Iterating using for loop we could have directly passed the returned Generator object to sum() function i.e.

# Pass the Generator object returned by Generator Expression to sum()
total = sum((n for n in range(1000) if n % 3 == 0 and n %5 == 0))

print('Total = ', total)

Output:

Total =  33165

List Comprehension vs Generator Expression

Generator expression’s syntax is just like List comprehension except the brackets, but the main difference between List Comprehension & Generator Expression is that later returns a Generator object instead of list. We should use Generators when we are only interested in looping over the items one at a time and avoid keeping unnecessary elements in memory, as we explained in above examples.

Complete example is as follows,

from datetime import datetime


def selectedNumbers():
    ''' A Generator that yields multiples of 3 & 5 in range 0 to 1000'''
    for num in range(1000):
        if num % 3 == 0 and num % 5 == 0:
            yield num


def main():

    print('*** Getting the Sum of selected numbers using List Comprehension ***')

    # Create a list of numbers which are divisible by 3 & 5 and are in range from 1 to 1000
    listOfNums = [n for n in range(1000) if n % 3 == 0 and n %5 == 0]

    # get the sum of all numbers in list
    total = 0
    for num in listOfNums:
        total += num

    print('Total = ', total)

    print('*** Getting the Sum of selected numbers using Generators ***')

    # Get a Generator Object
    generatorObj = selectedNumbers()

    # Iterate over yielded values one by one and calculate the sum
    total = 0
    for num in generatorObj:
        total += num

    print('Total = ', total)

    print('*** Getting the Sum of selected numbers using Generator Expression ***')

    # Get a Generator object using Generator Expression
    generatorObj = (n for n in range(1000) if n % 3 == 0 and n %5 == 0)

    # Iterate over yielded values one by one and calculate the sum
    total = 0
    for num in generatorObj:
        total += num

    print('Total = ', total)

    print('*** Getting the Sum of selected numbers using Generator Expression & sum() ***')

    # Pass the Generator object returned by Generator Expression to sum()
    total = sum((n for n in range(1000) if n % 3 == 0 and n %5 == 0))

    print('Total = ', total)

if __name__ == '__main__':
  main()

Output:

*** Getting the Sum of selected numbers using List Comprehension ***
Total =  33165
*** Getting the Sum of selected numbers using Generators ***
Total =  33165
*** Getting the Sum of selected numbers using Generator Expression ***
Total =  33165
*** Getting the Sum of selected numbers using Generator Expression & sum() ***
Total =  33165