Image by Author | Ideogram
Let’s be honest. When you’re learning Python, you’re probably not thinking about performance. You’re just trying to get your code to work! But here’s the thing: making your Python code faster doesn’t require you to become an expert programmer overnight.
With a few simple techniques that I’ll show you today, you can improve your code’s speed and memory usage substantially.
In this article, we’ll walk through five practical beginner-friendly optimization techniques together. For each one, I’ll show you the “before” code (the way many beginners write it), the “after” code (the optimized version), and explain exactly why the improvement works and how much faster it gets.
1. Replace Loops with List Comprehensions
Let’s start with something you probably do all the time: creating new lists by transforming existing ones. Most beginners reach for a for loop, but Python has a much faster way to do this.
Before Optimization
Here’s how most beginners would square a list of numbers:
import time
def square_numbers_loop(numbers):
result = []
for num in numbers:
result.append(num ** 2)
return result
# Let's test this with 1000000 numbers to see the performance
test_numbers = list(range(1000000))
start_time = time.time()
squared_loop = square_numbers_loop(test_numbers)
loop_time = time.time() - start_time
print(f"Loop time: {loop_time:.4f} seconds")
This code creates an empty list called result, then loops through each number in our input list, squares it, and appends it to the result list. Pretty straightforward, right?
After Optimization
Now let’s rewrite this using a list comprehension:
def square_numbers_comprehension(numbers):
return [num ** 2 for num in numbers] # Create the entire list in one line
start_time = time.time()
squared_comprehension = square_numbers_comprehension(test_numbers)
comprehension_time = time.time() - start_time
print(f"Comprehension time: {comprehension_time:.4f} seconds")
print(f"Improvement: {loop_time / comprehension_time:.2f}x faster")
This single line [num ** 2 for num in numbers] does exactly the same thing as our loop, but it’s telling Python “create a list where each element is the square of the corresponding element in numbers.”
Output:
Loop time: 0.0840 seconds
Comprehension time: 0.0736 seconds
Improvement: 1.14x faster
Performance improvement: List comprehensions are typically 30-50% faster than equivalent loops. The improvement is more noticeable when you work with very large iterables.
Why does this work? List comprehensions are implemented in C under the hood, so they avoid a lot of the overhead that comes with Python loops, things like variable lookups and function calls that happen behind the scenes.
2. Choose the Right Data Structure for the Job
This one’s huge, and it’s something that can make your code hundreds of times faster with just a small change. The key is understanding when to use lists versus sets versus dictionaries.
Before Optimization
Let’s say you want to find common elements between two lists. Here’s the intuitive approach:
def find_common_elements_list(list1, list2):
common = []
for item in list1: # Go through each item in the first list
if item in list2: # Check if it exists in the second list
common.append(item) # If yes, add it to our common list
return common
# Test with reasonably large lists
large_list1 = list(range(10000))
large_list2 = list(range(5000, 15000))
start_time = time.time()
common_list = find_common_elements_list(large_list1, large_list2)
list_time = time.time() - start_time
print(f"List approach time: {list_time:.4f} seconds")
This code loops through the first list, and for each item, it checks if that item exists in the second list using if item in list2. The problem? When you do item in list2, Python has to search through the entire second list until it finds the item. That’s slow!
After Optimization
Here’s the same logic, but using a set for faster lookups:
def find_common_elements_set(list1, list2):
set2 = set(list2) # Convert list to a set (one-time cost)
return [item for item in list1 if item in set2] # Check membership in set
start_time = time.time()
common_set = find_common_elements_set(large_list1, large_list2)
set_time = time.time() - start_time
print(f"Set approach time: {set_time:.4f} seconds")
print(f"Improvement: {list_time / set_time:.2f}x faster")
First, we convert the list to a set. Then, instead of checking if item in list2, we check if item in set2. This tiny change makes membership testing nearly instantaneous.
Output:
List approach time: 0.8478 seconds
Set approach time: 0.0010 seconds
Improvement: 863.53x faster
Performance improvement: This can be of the order of 100x faster for large datasets.
Why does this work? Sets use hash tables under the hood. When you check if an item is in a set, Python doesn’t search through every element; it uses the hash to jump directly to where the item should be. It’s like having a book’s index instead of reading every page to find what you want.
3. Use Python’s Built-in Functions Whenever Possible
Python comes with tons of built-in functions that are heavily optimized. Before you write your own loop or custom function to do something, check if Python already has a function for it.
Before Optimization
Here’s how you might calculate the sum and maximum of a list if you didn’t know about built-ins:
def calculate_sum_manual(numbers):
total = 0
for num in numbers:
total += num
return total
def find_max_manual(numbers):
max_val = numbers[0]
for num in numbers[1:]:
if num > max_val:
max_val = num
return max_val
test_numbers = list(range(1000000))
start_time = time.time()
manual_sum = calculate_sum_manual(test_numbers)
manual_max = find_max_manual(test_numbers)
manual_time = time.time() - start_time
print(f"Manual approach time: {manual_time:.4f} seconds")
The sum function starts with a total of 0, then adds each number to that total. The max function starts by assuming the first number is the maximum, then compares every other number to see if it’s bigger.
After Optimization
Here’s the same thing using Python’s built-in functions:
start_time = time.time()
builtin_sum = sum(test_numbers)
builtin_max = max(test_numbers)
builtin_time = time.time() - start_time
print(f"Built-in approach time: {builtin_time:.4f} seconds")
print(f"Improvement: {manual_time / builtin_time:.2f}x faster")
That’s it! sum() gives the total of all numbers in the list, and max() returns the largest number. Same result, much faster.
Output:
Manual approach time: 0.0805 seconds
Built-in approach time: 0.0413 seconds
Improvement: 1.95x faster
Performance improvement: Built-in functions are typically faster than manual implementations.
Why does this work? Python’s built-in functions are written in C and heavily optimized.
4. Perform Efficient String Operations with Join
String concatenation is something every programmer does, but most beginners do it in a way that gets exponentially slower as strings get longer.
Before Optimization
Here’s how you might build a CSV string by concatenating with the + operator:
def create_csv_plus(data):
result = "" # Start with an empty string
for row in data: # Go through each row of data
for i, item in enumerate(row): # Go through each item in the row
result += str(item) # Add the item to our result string
if i < len(row) - 1: # If it's not the last item
result += "," # Add a comma
result += "n" # Add a newline after each row
return result
# Test data: 1000 rows with 10 columns each
test_data = [[f"item_{i}_{j}" for j in range(10)] for i in range(1000)]
start_time = time.time()
csv_plus = create_csv_plus(test_data)
plus_time = time.time() - start_time
print(f"String concatenation time: {plus_time:.4f} seconds")
This code builds our CSV string piece by piece. For each row, it goes through each item, converts it to a string, and adds it to our result. It adds commas between items and newlines between rows.
After Optimization
Here’s the same code using the join method:
def create_csv_join(data):
# For each row, join the items with commas, then join all rows with newlines
return "n".join(",".join(str(item) for item in row) for row in data)
start_time = time.time()
csv_join = create_csv_join(test_data)
join_time = time.time() - start_time
print(f"Join method time: {join_time:.4f} seconds")
print(f"Improvement: {plus_time / join_time:.2f}x faster")
This single line does a lot! The inner part ",".join(str(item) for item in row) takes each row and joins all items with commas. The outer part "n".join(...) takes all those comma-separated rows and joins them with newlines.
Output:
String concatenation time: 0.0043 seconds
Join method time: 0.0022 seconds
Improvement: 1.94x faster
Performance improvement: String joining is much faster than concatenation for large strings.
Why does this work? When you use += to concatenate strings, Python creates a new string object each time because strings are immutable. With large strings, this becomes incredibly wasteful. The join method figures out exactly how much memory it needs upfront and builds the string once.
5. Use Generators for Memory-Efficient Processing
Sometimes you don’t need to store all your data in memory at once. Generators let you create data on-demand, which can save massive amounts of memory.
Before Optimization
Here’s how you might process a large dataset by storing everything in a list:
import sys
def process_large_dataset_list(n):
processed_data = []
for i in range(n):
# Simulate some data processing
processed_value = i ** 2 + i * 3 + 42
processed_data.append(processed_value) # Store each processed value
return processed_data
# Test with 100,000 items
n = 100000
list_result = process_large_dataset_list(n)
list_memory = sys.getsizeof(list_result)
print(f"List memory usage: {list_memory:,} bytes")
This function processes numbers from 0 to n-1, applies some calculation to each one (squaring it, multiplying by 3, and adding 42), and stores all results in a list. The problem is that we’re keeping all 100,000 processed values in memory at once.
After Optimization
Here’s the same processing using a generator:
def process_large_dataset_generator(n):
for i in range(n):
# Simulate some data processing
processed_value = i ** 2 + i * 3 + 42
yield processed_value # Yield each value instead of storing it
# Create the generator (this doesn't process anything yet!)
gen_result = process_large_dataset_generator(n)
gen_memory = sys.getsizeof(gen_result)
print(f"Generator memory usage: {gen_memory:,} bytes")
print(f"Memory improvement: {list_memory / gen_memory:.0f}x less memory")
# Now we can process items one at a time
total = 0
for value in process_large_dataset_generator(n):
total += value
# Each value is processed on-demand and can be garbage collected
The key difference is yield instead of append. The yield keyword makes this a generator function – it produces values one at a time instead of creating them all at once.
Output:
List memory usage: 800,984 bytes
Generator memory usage: 224 bytes
Memory improvement: 3576x less memory
Performance improvement: Generators can use “much” less memory for large datasets.
Why does this work? Generators use lazy evaluation, they only compute values when you ask for them. The generator object itself is tiny; it just remembers where it is in the computation.
Conclusion
Optimizing Python code doesn’t have to be intimidating. As we’ve seen, small changes in how you approach common programming tasks can yield dramatic improvements in both speed and memory usage. The key is developing an intuition for choosing the right tool for each job.
Remember these core principles: use built-in functions when they exist, choose appropriate data structures for your use case, avoid unnecessary repeated work, and be mindful of how Python handles memory. List comprehensions, sets for membership testing, string joining, generators for large datasets are all tools that should be in every beginner Python programmer’s toolkit. Keep learning, keep coding!
Bala Priya C is a developer and technical writer from India. She likes working at the intersection of math, programming, data science, and content creation. Her areas of interest and expertise include DevOps, data science, and natural language processing. She enjoys reading, writing, coding, and coffee! Currently, she’s working on learning and sharing her knowledge with the developer community by authoring tutorials, how-to guides, opinion pieces, and more. Bala also creates engaging resource overviews and coding tutorials.
