bastianplsfix

โ† JavaScript + TypeScript

Text, and the three things called a character

Three different things get called a character, and .length counts the least useful one.

A JavaScript string is a sequence of UTF-16 code units, and that is what .length, indexing, and slice work with; they are the same code units whose ordering the ordering and sorting page measured. A code point is one Unicode character as the standard defines it, and it takes one or two code units. A grapheme cluster is what a person reading the screen would call one character, and it takes one or more code points.

For English text all three agree, which is why this stays hidden until the day it does not.

Create programs/text-and-characters.test.ts for this reference and keep it open. Start it with the assertion helpers used on this page:

import { assertEquals, assertFalse } from "@std/assert";

Follow the page as you add and revise the runnable examples below that import.

three things get called a character

The file opens with a helper that measures all three levels at once: counts returns [code units, code points, grapheme clusters], using a tool for each level that the next steps introduce properly. Start with the prediction everyone makes: one smiley, length one. Save:

const graphemes = new Intl.Segmenter("en", { granularity: "grapheme" });

function counts(text: string): [number, number, number] {
  return [
    text.length,
    Array.from(text).length,
    Array.from(graphemes.segment(text)).length,
  ];
}

Deno.test("three things get called a character", () => {
  assertEquals("๐Ÿ™‚".length, 1);
});
Check programs/text-and-characters.test.ts
running 1 test from ./programs/text-and-characters.test.ts
three things get called a character ... FAILED (8ms)

 ERRORS

three things get called a character => ./programs/text-and-characters.test.ts:14:11
error: AssertionError: Values are not equal.

    [Diff] Actual / Expected

-   2
+   1

 FAILURES

three things get called a character => ./programs/text-and-characters.test.ts:14:11

FAILED | 0 passed | 1 failed (9ms)

error: Test failed

One visible character, length two. .length counts code units, and this emoji takes two of them. Replace the prediction with the three-level survey:

Deno.test("three things get called a character", () => {
  assertEquals(counts("oak"), [3, 3, 3]);
  assertEquals(counts("๐Ÿ™‚"), [2, 1, 1]);
  assertEquals(counts("๐Ÿ‘จโ€๐Ÿ‘ฉโ€๐Ÿ‘ง"), [8, 5, 1]);
});
three things get called a character ... ok (0ms)

Read the three rows.

  1. "oak" is [3, 3, 3]: on plain English text the levels agree, which is exactly why the distinction hides.
  2. "๐Ÿ™‚" is [2, 1, 1]: one code point, encoded as two code units.
  3. The family emoji is [8, 5, 1]: five code points wearing eight code units, and one character to anybody looking at it.

Each level has its own tool, and the next three steps take them in order.

indexing gives you code units

Deno.test("indexing gives you code units", () => {
  const smiley = "๐Ÿ™‚";
  assertEquals(smiley.length, 2);
  assertEquals(smiley[0].length, 1);
  assertEquals(smiley.split(""), ["\uD83D", "\uDE42"]);
});
indexing gives you code units ... ok (0ms)

smiley[0] is not a character. It is half of one: a one-unit string that renders as nothing useful, which split("") makes explicit by cutting the emoji into its two halves, written here as \u escapes because there is no better way to show them. Everything positional on a string, indexing, slice, .length, works at this level.

iteration gives you code points

Every iteration protocol on strings splits by code point instead:

Deno.test("iteration gives you code points", () => {
  assertEquals(Array.from("A๐Ÿ™‚"), ["A", "๐Ÿ™‚"]);
  assertEquals([..."A๐Ÿ™‚"], ["A", "๐Ÿ™‚"]);

  const seen: string[] = [];
  for (const codePoint of "A๐Ÿ™‚") seen.push(codePoint);
  assertEquals(seen, ["A", "๐Ÿ™‚"]);
});
iteration gives you code points ... ok (0ms)

Array.from, spread, and for...of all walk the string through its Symbol.iterator, the hook from the symbols page, and the string's iterator deals in code points. So Array.from(text).length counts code points while text.length counts code units, and that single difference fixes most naive character counting. This is where counts gets its middle number.

Intl.Segmenter gives you what a reader sees

Nothing in the language itself splits text into grapheme clusters, because doing that correctly requires Unicode tables that change every year. Intl.Segmenter, the same Intl family as the numbers page's formatting is a separate job, carries the tables:

Deno.test("Intl.Segmenter gives you what a reader sees", () => {
  const segments = Array.from(graphemes.segment("A๐Ÿ™‚๐Ÿ‘จโ€๐Ÿ‘ฉโ€๐Ÿ‘ง"));

  assertEquals(segments.map((s) => s.segment), ["A", "๐Ÿ™‚", "๐Ÿ‘จโ€๐Ÿ‘ฉโ€๐Ÿ‘ง"]);
  assertEquals(segments[2].index, 3);
});
Intl.Segmenter gives you what a reader sees ... ok (0ms)

Three segments for what a reader would call three characters. Each segment also carries its index, where it starts in code units, and the family emoji begins at 3: after one unit of A and two of the smiley. That index is how you slice safely, by asking the segmenter where the boundaries are rather than guessing, and it is where counts gets its third number.

why a character can take two code units

Unicode has more characters than 16 bits can address. UTF-16 solves that by reserving two ranges of code units that never appear alone in valid text, and pairing one from each range to encode everything above the first 65,536 characters. The halves are called the leading and trailing surrogate, and every tool on this step names which level it works at:

Deno.test("why a character can take two code units", () => {
  const smiley = "๐Ÿ™‚";
  assertEquals(smiley.codePointAt(0), 0x1f642);
  assertEquals(smiley.charCodeAt(0), 0xd83d);
  assertEquals(smiley.charCodeAt(1), 0xde42);

  assertEquals("\u{1F642}", smiley);
  assertEquals(String.fromCodePoint(0x1f642), smiley);
});
why a character can take two code units ... ok (0ms)

The pairs to keep straight.

  1. codePointAt(0) answers 0x1f642, the smiley's one code point. charCodeAt(0) and charCodeAt(1) answer the two surrogates it is stored as.
  2. In string literals, \uXXXX spells one code unit, and \u{...} spells one code point, so "\u{1F642}" is the whole smiley while step two's "\uD83D" was half of it.
  3. String.fromCodePoint builds from code points. Prefer the code point versions, codePointAt, fromCodePoint, \u{...}, unless you have a reason not to.

truncating by index cuts characters in half

Here is the bug this entry exists to explain:

Deno.test("truncating by index cuts characters in half", () => {
  const smiley = "๐Ÿ™‚";
  const half = smiley.slice(0, 1);

  assertEquals(half.length, 1);
  assertFalse(half.isWellFormed());
  assertEquals(JSON.stringify(half), '"\\ud83d"');

  const atZero = smiley.at(0);
  assertEquals(atZero !== undefined && atZero.isWellFormed(), false);

  assertEquals(half.toWellFormed(), "๏ฟฝ");
});
truncating by index cuts characters in half ... ok (0ms)

Walk the wreckage.

  1. slice(0, 1) works in code units, so it cut the surrogate pair down the middle. The result is a lone surrogate: a perfectly valid string value that is not valid text.
  2. It survives in memory and rides through JSON as "\ud83d", so nothing stops it from reaching a database or another program. It renders as the replacement character or as nothing, depending on who draws it, which is the whole mechanism behind "why does my truncated preview end in a black diamond".
  3. .at(0) has the same problem, because it is the same code-unit indexing with politer syntax.
  4. Two ES2024 methods help at boundaries: isWellFormed reports lone surrogates, and toWellFormed replaces each with ๏ฟฝ. Repaired, not recovered: the other half of the character is gone. Use them to validate arriving text, and fix the truncation rather than leaning on the repair.

one character, many code points

Surrogate pairs are only the first layer. Several Unicode mechanisms build one visible character out of several code points:

Deno.test("one character, many code points", () => {
  assertEquals(counts("๐Ÿ‡ฏ๐Ÿ‡ต"), [4, 2, 1]);
  assertEquals(counts("๐Ÿ‘๐Ÿฝ"), [4, 2, 1]);
  assertEquals(counts("๐Ÿ‘จโ€๐Ÿ‘ฉโ€๐Ÿ‘ง"), [8, 5, 1]);

  assertEquals(counts("๐Ÿ‡ฏ๐Ÿ‡ต๐Ÿ‘๐Ÿฝ๐Ÿ‘จโ€๐Ÿ‘ฉโ€๐Ÿ‘ง")[2], 3);
});
one character, many code points ... ok (0ms)

Three mechanisms, one lesson.

  1. A flag is two regional indicator letters, J and P, which a renderer that recognizes the pair draws as one flag.
  2. A skin tone is a modifier code point following a base emoji.
  3. A family is three person emoji stitched together with zero-width joiners.

Iterating by code point splits every one of these into pieces, which is why code point counting beats code unit counting and is still not enough. The last line is the summary: three characters to a reader, and only the segmenter agrees.

the same letter, spelled two ways

The last layer catches people comparing text rather than rendering it. Type these two literals exactly as shown, one รฉ produced as a single character and one typed as e followed by a combining accent, and save:

Deno.test("the same letter, spelled two ways", () => {
  const precomposed = "รฉ";
  const decomposed = "eฬ";
  assertEquals(precomposed === decomposed, false);
});
Check programs/text-and-characters.test.ts
TS2367 [ERROR]: This comparison appears to be unintentional because the types '"รฉ"' and '"รฉ"' have no overlap.
    assertEquals(precomposed === decomposed, false);
    at file:///programs/text-and-characters.test.ts:81:18

error: Type checking failed.

  info: The program failed type-checking, but it still might work correctly.
  hint: Re-run with --no-check to skip type-checking.

Stop and read that diagnostic twice: two types, printed identically, declared to have no overlap. The checker reads code points, not pixels, so it can see what no amount of staring at the source will show you. "รฉ" here is the single code point U+00E9; "รฉ" is e followed by the combining accent U+0301; both are correct Unicode and both render the same.

Widen the literals with : string annotations so JavaScript can be asked, and predict what deep equality says about them:

Deno.test("the same letter, spelled two ways", () => {
  const precomposed: string = "รฉ";
  const decomposed: string = "eฬ";
  assertEquals(precomposed, decomposed);
});
Check programs/text-and-characters.test.ts
running 8 tests from ./programs/text-and-characters.test.ts
...
the same letter, spelled two ways ... FAILED (9ms)

 ERRORS

the same letter, spelled two ways => ./programs/text-and-characters.test.ts:75:11
error: AssertionError: Values are not equal.

    [Diff] Actual / Expected

-   รฉ
+   รฉ

 FAILURES

the same letter, spelled two ways => ./programs/text-and-characters.test.ts:75:11

FAILED | 7 passed | 1 failed (10ms)

error: Test failed

A diff showing two identical lines, one removed and one added. This is the failure as it arrives in real life: two strings that print the same and are not the same, from the equality page's rules working exactly as specified on values that merely differ invisibly. Text from a macOS filesystem tends to be decomposed; text typed into a browser tends to be precomposed. Record the truth and the fix:

Deno.test("the same letter, spelled two ways", () => {
  const precomposed: string = "รฉ";
  const decomposed: string = "eฬ";

  assertEquals(precomposed.length, 1);
  assertEquals(decomposed.length, 2);
  assertEquals(precomposed === decomposed, false);

  assertEquals(precomposed.normalize("NFC"), decomposed.normalize("NFC"));
  assertEquals(counts(decomposed)[2], 1);
});
the same letter, spelled two ways ... ok (1ms)

normalize("NFC") composes where it can, and after it the two spellings agree. NFC is the sensible default for storage and comparison; NFD decomposes, and is occasionally what a search index wants. Normalize once, at the boundary where text enters your program, and compare afterwards. And note the last line: the grapheme count is 1 for the decomposed spelling too, because a combining mark joins the cluster it modifies, which is one more argument for asking the segmenter instead of counting anything yourself.

regular expressions can be told to see code points

By default, . in a regular expression matches one code unit, which means it can match half an emoji:

Deno.test("regular expressions can be told to see code points", () => {
  assertEquals(("๐Ÿ™‚".match(/./g) ?? []).length, 2);
  assertEquals("๐Ÿ™‚".match(/./gv), ["๐Ÿ™‚"]);

  assertEquals("A๐Ÿ™‚๐Ÿ‘จโ€๐Ÿ‘ฉโ€๐Ÿ‘ง".match(/\p{RGI_Emoji}/gv), ["๐Ÿ™‚", "๐Ÿ‘จโ€๐Ÿ‘ฉโ€๐Ÿ‘ง"]);
});
regular expressions can be told to see code points ... ok (4ms)

Line by line.

  1. /./g finds two matches in one smiley: one per code unit, each a lone surrogate.
  2. The v flag moves the pattern to the code point level, so /./gv finds the smiley whole.
  3. v also enables Unicode property escapes, and \p{RGI_Emoji} matches whole emoji including the multi-code-point family, which is as close as a regular expression gets to grapheme clusters.

Put v on any pattern that touches text you did not write.

the segmenter does more than characters

Deno.test("the segmenter does more than characters", () => {
  const words = new Intl.Segmenter("en", { granularity: "word" });

  assertEquals(
    Array.from(words.segment("Hi there!")).map((s) => s.segment),
    ["Hi", " ", "there", "!"],
  );
});
the segmenter does more than characters ... ok (0ms)

Word granularity splits "Hi there!" the way a reader would, keeping the space and the punctuation as their own segments. Sentence granularity exists too, and both are locale-aware in the same way as the grapheme tables, which matters for languages that do not put spaces between words at all. Splitting on /\s+/ is a guess that happens to work for English.

In practice