encodingbase64unicode

Encoding and Decoding: The Complete Developer Guide

April 30, 2026 · Cosyslabs

Encoding converts data from one representation to another so it can be safely transmitted, stored, or processed by systems that expect a specific format. Decoding reverses the transformation. Understanding encoding is essential for working with APIs, file uploads, authentication, and web security — nearly every developer encounters encoding issues daily.

Why Encoding Matters

Many systems were designed for specific character sets or byte ranges. When data crosses these boundaries, certain bytes get misinterpreted as control characters, structure markers, or illegal sequences. Encoding solves this by transforming arbitrary data into a format the target system accepts.

Common encoding challenges:

Binary data in text-only protocols (SMTP, HTTP headers)
Special characters in URLs (&, ?, #, spaces)
HTML metacharacters in user-generated content (<, >, ")
Non-ASCII characters in legacy systems

Base64 Encoding

Base64 encodes binary data as printable ASCII text by regrouping 8-bit bytes into 6-bit groups, then mapping each group to one of 64 printable characters.

// Browser / Node.js
const encoded = btoa("Hello, World!");        // "SGVsbG8sIFdvcmxkIQ=="
const decoded = atob("SGVsbG8sIFdvcmxkIQ=="); // "Hello, World!"

// Node.js Buffer (handles binary data)
const buf = Buffer.from([0xff, 0xfe, 0x00, 0x01]);
const b64 = buf.toString("base64");            // "/v4AAQ=="
const back = Buffer.from(b64, "base64");

Base64 increases size by ~33%. The URL-safe variant replaces + with - and / with _, used in JWTs and OAuth tokens.

URL Encoding (Percent Encoding)

URLs may only contain a limited set of characters. Percent encoding represents unsafe characters as %XX where XX is the hexadecimal byte value.

// Full URL — preserve structure characters (: / ? & =)
encodeURI("https://example.com/search?q=hello world&lang=en");
// "https://example.com/search?q=hello%20world&lang=en"

// Query parameter value — encode everything including & = ?
encodeURIComponent("price >= $5 & color=red");
// "price%20%3E%3D%20%245%20%26%20color%3Dred"

// Python
from urllib.parse import quote, urlencode
quote("hello world")           # "hello%20world"
urlencode({"q": "hello&world"}) # "q=hello%26world"

HTML Entity Encoding

HTML entities prevent browsers from interpreting user content as markup. This is a critical defense against Cross-Site Scripting (XSS).

// Manual escaping (or use DOMPurify for full sanitization)
function escapeHtml(str) {
  return str
    .replace(/&/g, "&amp;")
    .replace(/</g, "&lt;")
    .replace(/>/g, "&gt;")
    .replace(/"/g, "&quot;")
    .replace(/'/g, "&#039;");
}

escapeHtml('<script>alert("xss")</script>');
// "&lt;script&gt;alert(&quot;xss&quot;)&lt;/script&gt;"

Named entities for common characters:

Character	Entity	Numeric
`<`	`<`	`<`
`>`	`>`	`>`
`&`	`&`	`&`
`"`	`"`	`"`
`©`	`©`	`©`
`→`	`→`	`→`

Unicode and Character Encodings

Unicode assigns a unique code point to every character. UTF-8 is the dominant encoding that represents Unicode code points as 1–4 bytes.

# Python — unicode by default
text = "Héllo Wörld 🌍"
utf8_bytes = text.encode("utf-8")    # b'H\xc3\xa9llo W\xc3\xb6rld \xf0\x9f\x8c\x8d'
back = utf8_bytes.decode("utf-8")   # "Héllo Wörld 🌍"

# UTF-16 for Windows APIs and Java
utf16_bytes = text.encode("utf-16")

// JavaScript — TextEncoder/TextDecoder for explicit control
const encoder = new TextEncoder(); // always UTF-8
const bytes = encoder.encode("Héllo 🌍");

const decoder = new TextDecoder("utf-8");
const text = decoder.decode(bytes);

Binary Encoding: Hex and Octal

Hexadecimal provides a compact, human-readable representation of binary data commonly used in debugging, cryptography, and network protocols.

// Number to hex
(255).toString(16);   // "ff"
(255).toString(2);    // "11111111" (binary)
(255).toString(8);    // "377" (octal)

// Hex string to buffer
const bytes = Buffer.from("deadbeef", "hex"); // <Buffer de ad be ef>

// Go
import "encoding/hex"
encoded := hex.EncodeToString([]byte{0xde, 0xad, 0xbe, 0xef}) // "deadbeef"
decoded, _ := hex.DecodeString("deadbeef")

Choosing the Right Encoding

Use Case	Encoding
Binary data in JSON/HTTP	Base64
URL query parameters	encodeURIComponent
Full URL with structure	encodeURI
HTML user content	HTML entity encoding
File storage of text	UTF-8
Cryptographic output	Hex
JWT tokens	Base64URL (no padding)

Tools

Base64 Encoder/Decoder — encode/decode text and files
URL Encoder/Decoder — percent-encode URL components
HTML Entity Encoder — escape HTML special characters

← Back to all topics