Data Classification and Discovery
You cannot protect, retain, or delete data you do not know you have. Personal data spreads quietly: a customer email copied into an analytics table, phone numbers in support ticket text, card numbers in debug logs, passport scans in an S3 bucket created for a one-off project. Data classification defines categories of sensitivity and the rules that apply to each. Discovery finds where sensitive data actually lives — including the places nobody intended — so the rules can be applied.
Together they turn privacy and security policy into something enforceable: encryption, access control, masking, retention, and deletion all depend on knowing which data is sensitive and who owns it.
TL;DR
- Define a small set of classification levels (for example Public, Internal, Confidential, Restricted) with concrete handling rules.
- Assign owners for datasets; classification without accountability decays.
- Discover sensitive data with scanners for structured stores, object storage, documents, and logs — combining patterns, validation, context, and ML.
- Record results as tags on columns, tables, and buckets in a catalog and in code.
- Enforce rules from tags: masking policies, column and row access, encryption, retention, and DLP controls.
- Keep inventories current with continuous scanning, lineage-based propagation, and review of new data sources.
Quick Example
Detecting personal data in free text with Microsoft Presidio:
Recording classification next to the model definition (dbt) so it travels with the code:
Enforcing the rule in the warehouse with a masking policy (Snowflake):
Core Concepts
Classification Levels and Handling Rules
Keep the scheme small enough that people apply it correctly; map regulatory categories (personal data under GDPR, cardholder data under PCI DSS, health information) onto it. See PII Handling and PCI DSS.
Ownership
Each dataset needs a data owner accountable for classification and access decisions and, often, a steward who maintains metadata. Owners approve access, review classifications when schemas change, and respond to discovery findings.
Discovery Techniques
Managed services — Amazon Macie for S3, Google Sensitive Data Protection, Microsoft Purview — and open-source tools such as Presidio cover many of these. Combine automated results with human review for high-impact decisions.
Where to Look
- Databases and warehouses: tables, views, and especially free-text columns such as notes and comments.
- Object storage: exports, backups, uploads, and forgotten buckets.
- Logs and traces: request bodies, headers, and error messages. See Audit Logging.
- Collaboration tools: tickets, chat exports, shared drives.
- Non-production environments: copies of production data in staging and developer databases.
Tags, Catalogs, and Lineage
Record classifications as tags on columns, tables, and buckets in a data catalog, schema metadata, or infrastructure code. Lineage lets tags propagate: a column derived from a confidential column is confidential unless deliberately transformed (for example aggregated or tokenized). Warehouses and catalogs increasingly support tag-based policies — attach a masking or access policy to a tag, and every tagged column inherits it.
Enforcing Handling Rules
Best Practices
Classify at Creation
Require classification metadata when new tables, buckets, and event schemas are created — in code review, schema registries, and infrastructure templates — so discovery is a safety net rather than the primary mechanism.
Scan Continuously
Schedule discovery scans for new and changed data stores and alert owners about unexpected sensitive data, such as card numbers appearing in a logging bucket.
Tune for Precision
Excessive false positives cause alert fatigue. Validate matches (checksums, context words), suppress known-safe columns, and review findings with owners.
Prefer Removing Data to Protecting It
When discovery finds sensitive data where it is not needed, delete, aggregate, or tokenize it rather than adding controls around it. See Privacy by Design.
Common Mistakes
Classification Only in a Policy Document
Scanning Production Only
Trusting Column Names
FAQ
What is data classification?
Assigning sensitivity levels to data — such as public, internal, confidential, and restricted — with defined handling rules for each level covering access, encryption, retention, sharing, and logging.
What is sensitive data discovery?
The process of scanning data stores, files, and logs to find where sensitive information such as personal data, payment data, or credentials actually resides, often using pattern matching, validation, and machine learning.
How many classification levels should we have?
Usually three to five. Fewer levels are easier to apply consistently; each must map to clear, enforceable handling rules.
How do I find personal data in unstructured text?
Use recognizers combining patterns, checksums, context words, and NLP models — tools such as Presidio or cloud data protection services — and tune them for your languages and data, with human review of uncertain findings.
How does classification support GDPR compliance?
It identifies where personal data lives, supports records of processing, drives access and retention controls, and makes it possible to locate and delete a person's data when they exercise their rights.
Related Topics
- PII Handling — Protecting personal data
- GDPR Engineering — Privacy regulation in code
- Privacy by Design — Minimizing data from the start
- Data Retention — Keeping data only as long as needed
- PCI DSS — Handling payment card data
- Data Quality Management — Catalogs, metadata, and ownership