Binary Data and Character Encoding

Everything a program stores or sends is a sequence of bytes. A byte has no meaning on its own: the same four bytes can be a 32-bit integer, a fraction of a floating-point number, two UTF-16 characters, or part of a JPEG. Meaning comes from an encoding that the writer and the reader agree on — and most data corruption bugs are disagreements about that encoding.

The symptoms are familiar: 0.1 + 0.2 printing 0.30000000000000004, an emoji counted as two characters, é appearing where é should be, an integer that turns negative after adding one, or a binary protocol that works on one machine and returns nonsense on another. This guide explains the representations behind each.

TL;DR

Quick Example

One emoji, several different lengths — and a number packed in two byte orders.

A user sees seven characters: n, a, ï, v, e, a space, and one thumbs-up. No single built-in length function reports that number.

Core Concepts

Bits, Bytes, and Hexadecimal

A byte is 8 bits, giving 256 values (0–255). Hexadecimal writes each byte as two digits (0x00–0xFF), which is why memory dumps, hashes, and color codes use it: one hex digit is exactly four bits.

Integers and Two's Complement

Unsigned integers map bit patterns directly to non-negative numbers: a 32-bit unsigned integer holds 0 to 4,294,967,295. Signed integers almost universally use two's complement, where the top bit carries a negative weight; a 32-bit signed integer holds −2,147,483,648 to 2,147,483,647.

Arithmetic that exceeds the range overflows. Languages differ in what happens next:

Floating Point (IEEE 754)

A double-precision float stores a sign, an 11-bit exponent, and a 52-bit fraction — about 15–17 significant decimal digits. Values are binary fractions, so 0.1 cannot be represented exactly, in the same way that 1/3 cannot be written exactly in decimal. Consequences:

For money, use integer minor units (cents) or a decimal type (decimal.Decimal, BigDecimal, SQL NUMERIC).

Byte Order (Endianness)

A multi-byte number can be stored most-significant byte first (big-endian) or least-significant byte first (little-endian). x86-64 and almost all ARM systems run little-endian; network protocols such as IP and TCP define network byte order as big-endian. Byte order only matters when bytes cross a boundary — files, sockets, memory-mapped structures — and every binary format must specify it.

Text Encodings: Unicode and UTF-8

Unicode assigns each character a code point, written U+0041 for A or U+1F44D for 👍. It is a catalog, not a byte format. Encodings turn code points into bytes:

UTF-8 is the right default for anything stored or transmitted. Its design means ASCII text is already valid UTF-8, and a stray byte cannot be mistaken for the start of a different character.

Grapheme Clusters and Normalization

What a user perceives as one character can be several code points: é may be one code point (U+00E9) or two (e + combining accent U+0301); a family emoji joins several people with zero-width joiners; flags pair two regional-indicator letters. The user-visible unit is the grapheme cluster (defined in Unicode UAX #29).

Because the same text can be spelled several ways, compare strings after normalization: NFC (composed) for storage and comparison in most systems, NFKC when you also want compatibility characters folded (such as full-width letters in usernames).

Binary-to-Text and Serialization Formats

Hex and Base64

Binary data that must travel through text channels — JSON fields, URLs, email — is encoded as text. Hex doubles the size; Base64 adds about 33% and has a URL-safe variant (- and _ instead of + and /). Neither is encryption: anyone can decode them.

Choosing a Serialization Format

Text formats are self-describing and easy to inspect; binary formats are smaller and faster to parse but need tooling and schemas. Whatever you choose, version the schema — see Event Schema Evolution.

Best Practices

Use UTF-8 End to End

Declare UTF-8 explicitly at every boundary: HTTP Content-Type: application/json; charset=utf-8, HTML <meta charset="utf-8">, database column collations (utf8mb4 in MySQL, not the 3-byte utf8), and file reads (open(path, encoding="utf-8")). Decode bytes to text once at the input boundary and encode once at the output boundary.

Normalize Before Comparing or Indexing

Normalize user-entered identifiers (usernames, email local parts, search keys) to NFC or NFKC before storing and comparing them, so visually identical strings match.

Count What the User Sees

Truncate and validate lengths by grapheme cluster (Intl.Segmenter in JavaScript, the regex module's \X in Python) and enforce storage limits in bytes. Never cut a UTF-8 string at an arbitrary byte offset.

Specify Width and Byte Order in Binary Formats

Use fixed-width integer types (uint32_t, i64) and state the byte order explicitly when reading or writing binary data (struct.pack("<I", …), DataView with an explicit littleEndian argument).

Keep Money Out of Floats

Store currency as integer minor units or decimal types, and round only at defined points such as invoice totals.

Common Mistakes

Decoding With the Wrong Charset

Using .length for User-Facing Limits

Floats for Money

Treating Base64 as Security

FAQ

Why does 0.1 + 0.2 not equal 0.3?

Floating-point numbers are binary fractions, and 0.1 and 0.2 have no exact binary representation, just as 1/3 has none in decimal. Each is stored as the nearest representable value, and the tiny errors show up in the sum. Compare floats with a tolerance and use decimal types where exact decimal arithmetic matters.

What is the difference between Unicode and UTF-8?

Unicode is a catalog that gives every character a number (a code point). UTF-8 is one way to turn those numbers into bytes, using one to four bytes per code point. UTF-16 and UTF-32 are other encodings of the same catalog.

Why does my emoji count as two characters in JavaScript?

JavaScript strings are sequences of UTF-16 code units, and characters outside the Basic Multilingual Plane — including most emoji — take two units (a surrogate pair). Modifiers and joiners add more. Use Intl.Segmenter to count user-perceived characters, or [...str].length to count code points.

When does endianness matter?

Only when raw bytes cross a boundary: binary file formats, network protocols, memory-mapped hardware registers, and data shared between systems with different conventions. Text formats such as JSON avoid it entirely, and well-designed binary formats state their byte order.

Should I use JSON or a binary format?

Use JSON for public APIs, configuration, and anything humans debug, because it is universal and self-describing. Use a schema-based binary format such as Protocol Buffers for high-volume internal traffic where size and parse time matter, and columnar formats such as Parquet for analytics. See gRPC for a common binary RPC stack.

Related Topics

References