SciPy: Optimization, Integration, Sparse & Stats

Q: How do I know if an optimization converged?

Check the returned result's success flag and message, look at the objective value and iteration counts, and confirm the solution is stable when you change starting points and tolerances.

Q: Which ODE solver should I use?

RK45 for most non-stiff problems; Radau, BDF, or LSODA when the problem is stiff — typically when components evolve on very different time scales and explicit methods take tiny steps.

SciPy

SciPy builds on NumPy to provide the numerical algorithms scientists and engineers rely on: optimization, root finding, integration, differential equations, interpolation, linear algebra, sparse matrices, statistics, signal processing, and spatial algorithms. Most of it wraps decades-old, heavily tested libraries written in Fortran and C, exposed through consistent Python interfaces. scikit-learn, statsmodels, and much of the scientific Python ecosystem depend on it.

SciPy will almost always return an answer. The skill lies in knowing whether that answer is right: whether an optimizer converged, whether an integral's error estimate is acceptable, whether an ODE solver suits a stiff problem, and whether a statistical test's assumptions hold. This page covers the main subpackages and the checks that make results trustworthy.

TL;DR

scipy.optimize: minimization, curve fitting, least squares, root finding, and linear and mixed-integer programming.
scipy.integrate: integrals (quad) with error estimates and ODE solvers (solve_ivp) for stiff and non-stiff problems.
scipy.linalg and scipy.sparse: dense decompositions and memory-efficient sparse matrices with solvers.
scipy.stats: distributions, hypothesis tests, bootstrap and permutation methods.
Also signal, spatial, interpolate, fft, and ndimage.
Always inspect result objects (success, message, error estimates), set tolerances deliberately, and validate on known cases.

Quick Example

Fit a saturation curve to load-test measurements and report parameter uncertainty:

Compare two latency samples without assuming normality:

Reporting uncertainty and residuals — not just the fitted numbers — is what turns a fit into evidence.

Core Concepts

Optimization

Results come back as an OptimizeResult with x, fun, success, message, and evaluation counts. Check success and message every time, scale variables to similar magnitudes, provide gradients when possible, and try several starting points for non-convex problems.

Integration and Differential Equations

quad integrates functions adaptively and returns an error estimate alongside the value. solve_ivp solves initial value problems with methods for non-stiff problems (RK45, DOP853) and stiff problems (Radau, BDF, LSODA); a non-stiff method on a stiff problem crawls or fails. Set rtol and atol to match the precision you need, and use dense_output or t_eval for smooth results.

Linear Algebra and Sparse Matrices

scipy.linalg extends NumPy with decompositions (LU, QR, Cholesky, Schur), matrix functions such as expm, and specialized solvers. scipy.sparse stores mostly-zero matrices efficiently: build in COO or LIL format, convert to CSR or CSC for arithmetic, and solve with sparse.linalg.spsolve or iterative solvers (cg, gmres) for very large systems; eigsh finds a few eigenvalues without dense decomposition. The newer sparse array classes (csr_array) are recommended over the older matrix classes. See Probability and Linear Algebra for Computing.

Statistics

scipy.stats provides probability distributions (pdf, cdf, sampling, fitting), descriptive statistics, and hypothesis tests — t-tests, Mann-Whitney U, chi-square, Kolmogorov-Smirnov, correlation tests — plus resampling methods (bootstrap, permutation_test) that make fewer assumptions. Choose tests whose assumptions match the data, report effect sizes and confidence intervals alongside p-values, and correct for multiple comparisons. See A/B Testing.

Other Subpackages

Checking Numerical Results

Validate on known answers: test functions and integrals with analytic solutions before trusting new ones.
Inspect diagnostics: optimizer status, iteration counts, integration error estimates, solver messages.
Vary tolerances and starting points: answers that change significantly are not converged.
Check residuals and conditioning: large residuals or ill-conditioned matrices signal unreliable solutions.
Watch units and scaling: mixing milliseconds and hours, or parameters differing by many orders of magnitude, hurts convergence.
Prefer stable formulations: use log-likelihoods, solve systems instead of inverting matrices, and avoid subtracting nearly equal numbers.

Best Practices

Read the Result Object

Never use result.x without checking result.success and result.message; failed optimizations still return numbers.

Choose Methods for the Problem

Use bounded methods for constrained parameters, stiff solvers for stiff ODEs, and sparse solvers for large sparse systems. The default method is not always appropriate.

Quantify Uncertainty

Report parameter uncertainties from covariance matrices or bootstrapping, and confidence intervals for statistical estimates.

Vectorize Objective Functions

Write objective and model functions with NumPy array operations; Python loops inside functions called thousands of times are slow.

Common Mistakes

Ignoring Convergence Status

Dense Matrices for Sparse Problems

Tests Without Checking Assumptions

FAQ

What is SciPy used for?

Scientific and technical computing in Python: optimization, curve fitting, integration, differential equations, interpolation, linear algebra, sparse matrices, statistics, signal processing, and spatial algorithms.

What is the difference between NumPy and SciPy?

NumPy provides the array type and core numerical operations. SciPy builds on NumPy with higher-level algorithms for optimization, integration, statistics, sparse matrices, signal processing, and more.

How do I know if an optimization converged?

Check the returned result's success flag and message, look at the objective value and iteration counts, and confirm the solution is stable when you change starting points and tolerances.

Which ODE solver should I use?

RK45 for most non-stiff problems; Radau, BDF, or LSODA when the problem is stiff — typically when components evolve on very different time scales and explicit methods take tiny steps.

When should I use sparse matrices?

When most entries are zero and the matrix is large, such as graphs, finite element models, and text features. Sparse formats save memory and enable solvers that scale far beyond dense methods.