Pandas Performance: Vectorization, dtypes, PyArrow, Parquet & Polars

Q: Why is pandas apply slow?

With axis=1 (row-wise) or custom functions, apply calls Python code for each row or group, losing the benefit of vectorized C operations. Replace it with column arithmetic, NumPy functions, string and datetime accessors, or built-in aggregations wherever possible.

Q: How can I reduce pandas memory usage?

Convert repeated strings to category, use PyArrow-backed strings, downcast numeric types, use nullable dtypes instead of object columns, load only needed columns, and process large files in chunks or with out-of-core tools like DuckDB.

Pandas Performance & Memory

Pandas is fast when used as intended, meaning vectorized operations over whole columns executed in optimized C, and painfully slow when used like plain Python, looping row by row. The same transformation can take 20 minutes with iterrows() and under a second with a vectorized expression. Memory is the other constraint: pandas holds data in RAM, often using several times the on-disk size, so a 2 GB CSV can need 10 GB or more of memory with poor dtypes.

Most performance problems have simple fixes: vectorize, choose efficient dtypes (categories, nullable and PyArrow-backed types), read only the data you need from Parquet, and avoid apply. When data outgrows a single machine's memory, or you need multi-core speed, Polars and DuckDB are natural next steps that interoperate with pandas.

TL;DR

Vectorize: use column operations, np.where/np.select, .str/.dt accessors, and built-in aggregations. Avoid iterrows, itertuples loops, and row-wise apply.
Right-size dtypes: category for repeated strings, downcast numerics, nullable ints, and PyArrow strings (the default string dtype in pandas 3).
Measure memory with df.memory_usage(deep=True), and time with %timeit and profilers.
Use Parquet (columnar, compressed, typed) and read only needed columns and filtered rows.
Process in chunks, or push work to a database, when data exceeds memory.
Reach for Polars (multithreaded, lazy) or DuckDB (SQL on DataFrames and files) for large or complex workloads.

Quick Example

Core Concepts

Vectorization

Pandas and NumPy operations apply to entire arrays in compiled code. Vectorized equivalents exist for most row-wise logic:

df.eval() and df.query() can speed up large arithmetic expressions, and cut memory for intermediates, via numexpr.

apply Is Usually a Loop

df.apply(func, axis=1) calls a Python function once per row. It's convenient, but runs at Python speed. It's acceptable for small data or genuinely complex logic. Otherwise, express the logic vectorized, or use Numba, Cython, or Polars expressions for heavy custom computation.

Dtypes and Memory

df.info(memory_usage="deep") and memory_usage(deep=True) show the real footprint, because object columns hold Python strings far larger than their text.

PyArrow Backend

Pandas can store columns in Apache Arrow format (dtype_backend="pyarrow" in readers, or ArrowDtype). Benefits include compact strings, true missing values for all types, faster I/O with Parquet, and zero-copy interchange with Polars, DuckDB, and Spark. pandas 3.0 uses Arrow-backed strings by default when PyArrow is installed.

File Formats and I/O

CSV: slow to parse, untyped, large. Specify dtype=, usecols=, and parse_dates=, or use engine="pyarrow" for faster reading.
Parquet: columnar, compressed, typed, and splittable. Read only needed columns, and push row filters down to skip data. It's the default choice for analytical data. See data lakehouse.
Partitioned datasets (for example year=2026/month=09/) let readers skip entire directories.

Out-of-Memory Data

Options when data doesn't fit in RAM:

Chunking: pd.read_csv(..., chunksize=500_000), processing and aggregating each chunk incrementally.
Push computation down: filter and aggregate in the database or warehouse with SQL, and pull only results.
DuckDB: query Parquet or CSV files larger than memory with SQL, then return a small DataFrame (duckdb.sql(...).df()). See DuckDB.
Polars lazy and streaming mode, Dask, or Spark for distributed or multi-core processing.

Profiling

Measure before optimizing: %timeit and %prun in notebooks, line_profiler for line-level timing, memray or memory_profiler for memory, and df.memory_usage. Often one step (a row-wise apply, or an object-dtype join) dominates runtime.

pandas vs Polars vs DuckDB

They interoperate via Arrow: use DuckDB or Polars for heavy lifting, and pandas for ecosystem tools (plotting, scikit-learn, statsmodels) on the results.

Best Practices

Load Less Data

The fastest computation is the one you skip: select columns, filter rows at read time, and sample during development.

Set dtypes at Load Time

Declare dtypes in read_csv (and use dtype_backend="pyarrow") instead of converting after loading everything as objects. You save time and peak memory.

Avoid Repeated Concatenation and Copies

Build lists of frames and concat once. Chain methods rather than creating many intermediate copies, which Copy-on-Write helps with. See pandas indexing.

Cache Intermediate Results

Save cleaned or expensive intermediate datasets as Parquet, instead of recomputing them in every notebook run.

Common Mistakes

iterrows for Transformations

iterrows creates a Series per row, and loses dtypes. It's one of the slowest ways to process data. Vectorize, or at minimum use itertuples for unavoidable loops.

Object Columns Everywhere

Loading CSVs without dtypes leaves strings, and even numbers with stray characters, as object, which inflates memory and slows every operation. Clean and cast early.

Growing DataFrames Row by Row

df.loc[len(df)] = row in a loop reallocates repeatedly. Accumulate Python dicts or lists, and build the DataFrame once.

FAQ

Why is pandas apply slow?

With axis=1 (row-wise) or custom functions, apply calls Python code for each row or group, losing the benefit of vectorized C operations. Replace it with column arithmetic, NumPy functions, string and datetime accessors, or built-in aggregations wherever possible.

How can I reduce pandas memory usage?

Convert repeated strings to category, use PyArrow-backed strings, downcast numeric types, use nullable dtypes instead of object columns, load only needed columns, and process large files in chunks or with out-of-core tools like DuckDB.

Should I switch from pandas to Polars?

For large datasets and performance-critical pipelines, Polars is often several times faster, thanks to multithreading and lazy optimization. Pandas remains valuable for its ecosystem and familiarity. Many teams use Polars or DuckDB for heavy processing, and pandas for final analysis, and they interoperate via Arrow.

What file format is fastest for pandas?

Parquet, for most analytical workloads: it's columnar, compressed, typed, and supports column selection and predicate pushdown. Feather/Arrow IPC is excellent for fast temporary interchange. CSV is the slowest, and loses type information.