Property-Based Testing
Example-based tests check the cases a developer thought of. Property-based tests check a rule that should hold for all inputs — "decoding an encoded value returns the original", "sorting twice is the same as sorting once", "the balance never goes negative" — and let a library generate hundreds of inputs, including the empty strings, huge numbers, Unicode oddities, and boundary values developers rarely write by hand.
When a property fails, the library shrinks the failing input to the smallest example that still fails, so a bug triggered by a 300-element list with a negative number at position 217 is reported as [-1]. The result is a technique that finds edge-case bugs cheaply and produces unusually clear bug reports.
TL;DR
- Instead of fixed examples, describe what kind of input to generate and what must always be true.
- Good properties include round trips, invariants, idempotence, comparison with a reference implementation, and "never crashes".
- Shrinking reduces a failing input to a minimal counterexample, which makes debugging fast.
- Failing examples are saved and replayed, so a found bug becomes a permanent regression test.
- Stateful (model-based) testing generates sequences of operations and compares the system against a simple model.
- Property tests complement example tests; they do not replace a few readable examples of expected behavior.
Quick Example
Run-length encoding with two properties, written with Hypothesis for Python.
Two short properties cover behavior that would otherwise need dozens of hand-written examples.
Core Concepts
Generators and Strategies
A generator (Hypothesis calls them strategies) describes the space of inputs: integers in a range, lists of emails, dictionaries shaped like an API payload, or values built from other generators. Generators are deliberately biased towards troublesome values — zero, negative numbers, empty collections, maximum lengths, unusual characters.
Properties
A property is a statement that must hold for every generated input. The hardest part of property-based testing is finding properties that are true, meaningful, and not just a restatement of the implementation. Common patterns:
Shrinking
When a property fails, the library searches for a smaller input that still fails — shorter lists, smaller numbers, simpler strings — and reports the minimal counterexample. Consider a chunking helper tested with chunk sizes starting at zero:
Hypothesis reports the failure as xs=[], size=0 with ValueError: range() arg 3 must not be zero — the simplest possible demonstration that the function does not validate its size argument.
Regression Examples
Libraries store failing examples (Hypothesis keeps a local example database; fast-check prints a seed and path to replay). Pin important ones explicitly — @example(xs=[], size=0) in Hypothesis — so they run on every build, even on machines without the database.
Stateful and Model-Based Testing
Some bugs only appear after a particular sequence of operations: add, remove, add again, then resize. Stateful property testing generates random sequences of commands against the real system and a simple model of how it should behave — often just a dictionary or list — and checks they agree after every step. It is very effective for caches, data structures, state machines, and APIs with create/update/delete operations.
Tools
The same idea in TypeScript with fast-check and Vitest:
Best Practices
Start With Round Trips and Invariants
Serialization, parsing, encoding, and data transformations almost always have a round-trip or invariant property. They are the easiest place to start and frequently find real bugs.
Generate Realistic Inputs
Build generators that match your domain — valid emails, plausible money amounts, well-formed API payloads — and separately test invalid inputs. Generators that are too broad waste effort on inputs rejected at the boundary; generators that are too narrow miss bugs.
Keep Properties Independent of the Implementation
A property that recomputes the answer the same way the code does will pass even when both are wrong. Prefer properties that describe what must be true, or compare against a simpler, obviously correct implementation.
Control Runtime
Tune the number of examples per test (often 100 by default) and run larger budgets in nightly builds. Slow properties are usually a sign that each example does too much setup.
Keep a Few Readable Examples
Example-based tests document intended behavior for readers; property tests explore the edges. Use both.
Common Mistakes
Reimplementing the Code in the Test
Filtering Away Most Inputs
Ignoring a "Flaky" Failure
FAQ
What is property-based testing?
A testing technique where you state properties that should hold for all valid inputs, and a library generates many inputs automatically to try to break them. When it finds a failure, it shrinks the input to a minimal counterexample.
How is it different from example-based testing?
Example-based tests check specific inputs and outputs you chose. Property-based tests check general rules against inputs the library chooses, including edge cases you did not think of. Example tests document behavior; property tests explore it.
How do I find good properties?
Look for round trips (encode/decode, serialize/parse), invariants (totals, element counts, ordering), idempotent operations, a simpler reference implementation to compare against, and the rule that valid input never causes a crash. Many domain rules — "a refund never exceeds the payment" — are properties too.
Does property-based testing replace unit tests?
No. It complements them. Keep readable example tests for documented behavior and important cases, and add property tests where the input space is large or edge cases are likely: parsers, encoders, calculations, data structures, and state machines.
Are property-based tests flaky?
They are random but reproducible. Libraries report the seed or store the failing example, so any failure can be replayed exactly. A failure that appears only occasionally is almost always a real bug in a rare input.
Related Topics
- Testing & Quality Assurance — Property tests within a wider test strategy
- Unit Testing — Example-based tests that properties complement
- pytest — Where Hypothesis tests usually run in Python
- Vitest — Running fast-check properties in JavaScript projects
- Test-Driven Development — Writing tests first, including properties
- Security Regression Test Suites — Keeping discovered failures fixed
- Hypothesis — Property-based testing in Python
- Fuzz Testing — Coverage-guided fuzzing
- Formal Methods and Program Verification — Specifications, model checking, and proofs beyond testing