Python Profiling: A Complete Guide¶
Is Python slow? Maybe. But first measure where.
cProfile¶
python -m cProfile -s cumulative app.py
snakeviz¶
python -m cProfile -o profile.prof app.py snakeviz profile.prof
line_profiler¶
@profile def slow_function(): …
kernprof -l -v script.py¶
memory_profiler¶
python -m memory_profiler script.py
py-spy — Sampling Without Overhead¶
py-spy record -o profile.svg – python app.py
Optimization¶
- Generators instead of lists
- dict/set for lookups
- functools.lru_cache
- numpy for numerical work
- multiprocessing for CPU-bound tasks
- asyncio for I/O-bound tasks
When to Use Which Tool¶
cProfile is the default choice for an overall overview — it shows how much time is spent in each function. snakeviz visualizes cProfile data as an interactive sunburst chart. line_profiler is essential for identifying slow lines within a function — decorate the suspect function with @profile and run kernprof.
py-spy is a sampling profiler that attaches to a running process without restarting and without overhead. It is ideal for production debugging. memory_profiler reveals memory leaks and unnecessary allocations — each line of a function shows the memory increment. For NumPy/Pandas code, use scalene, which distinguishes between Python and C code. Key optimization rules: generators instead of list comprehensions for large datasets, dict/set for O(1) lookup instead of linear search in lists, and asyncio for I/O-bound operations.
Workflow¶
cProfile -> snakeviz -> line_profiler -> optimize -> measure again.