bastianplsfix

← JavaScript + TypeScript

A regular expression has to decide what a character is, and the flag decides for it. With no Unicode flag, a character is a UTF-16 code unit, the level the text and characters page called out in indexing gives you code units. With v, a character is a code point, and a few multi-code-point sequences can be matched as single units too. v also brings three features the older flag lacks: property escapes for whole categories of character, literal sequences inside a class, and set operations between classes.

That is why the regular expressions page recommended v on every pattern without qualification. This page is the argument for that recommendation, in full.

Create programs/unicode-in-patterns.test.ts for this reference and keep it open. Start it with the assertion helpers used on this page:

import {
  assert,
  assertEquals,
  assertFalse,
  assertThrows,
} from "@std/assert";

Below the import, add the fixture several steps ask about: a face with spiral eyes, one emoji written as its three code points.

const faceWithSpiralEyes = "\u{1F635}\u{200D}\u{1F4AB}";

Follow the page as you add and revise the runnable examples below that import.

the flag decides what a character is

Deno.test("the flag decides what a character is", () => {
  const smile = "🙂";
  assertEquals(smile.length, 2);

  assertEquals(smile.match(/./g)?.length, 2);
  assertEquals(smile.match(/./gv)?.length, 1);
});

Check programs/unicode-in-patterns.test.ts
running 1 test from ./programs/unicode-in-patterns.test.ts
the flag decides what a character is ... ok (0ms)

ok | 1 passed | 0 failed (1ms)

smile.length is 2: two UTF-16 code units, from the surrogate pair the text page's why a character can take two code units explained. Without v, /./g matches one code unit at a time and finds two matches inside a single emoji. With v, /./gv matches one code point at a time and finds one. That is the entire argument for the flag, stated as evidence rather than assertion.

without a flag, everything works on code units

The dot was one instance. Here is the worse one: a character class holding a single emoji.

Deno.test("without a flag, everything works on code units", () => {
  const smile = "🙂";

  assertEquals(/^[🙂]$/.test("🙂"), false);
  assertEquals(/^[🙂]$/.test("\uD83D"), true);

  assertEquals(smile.match(/\D/g)?.length, 2);

  assertEquals(/^[🙂]$/v.test("🙂"), true);
  assertEquals(smile.match(/\D/gv)?.length, 1);
  assertEquals(/^🙂{2}$/v.test("🙂🙂"), true);
});

without a flag, everything works on code units ... ok (0ms)

Read the first two assertions as one fact seen from both sides.

1. /^[🙂]$/.test("🙂") is false. A class containing the emoji does not match the emoji.
2. /^[🙂]$/.test("\uD83D") is true. It matches the lone leading surrogate instead, half a character, because without a flag the class holds the two code units of the emoji as two separate alternatives, and the anchored pattern only has room for one.

That is worth sitting with: /^[🙂]$/ is a pattern that tests for a lone surrogate. It does not error, it does not warn, and it passes every test written with ASCII input, the same silent-until-foreign-text failure mode the text page measured in truncating by index cuts characters in half. The complement \D shows the same thing from a third angle: it finds two matches inside one emoji, because it too counts code units. v repairs all three: the class matches the whole emoji, the complement finds one match, and a quantifier like {2} counts two whole emoji rather than four surrogate halves.

the ASCII class escapes stay ASCII

Deno.test("the ASCII class escapes stay ASCII", () => {
  assertEquals("a7x4".match(/\d/gv), ["7", "4"]);
  assertEquals("high - low".match(/\w+/gv), ["high", "low"]);

  assertFalse(/\w/v.test("ü"));
  assertFalse(/\w/v.test("é"));
  assert(/\p{Letter}/v.test("ü"));
});

the ASCII class escapes stay ASCII ... ok (0ms)

\d is [0-9], \w is [A-Za-z0-9_], and \s is the whitespace characters. v does not change any of them, and never will, because too much existing code depends on the narrow definition. So \w is not "a word character"; it is "an ASCII word character", and it says no to ü and é the same way it would to any accented letter in any European language. \p{Letter} says yes, and that is the next section's whole subject.

character property escapes

Every code point in Unicode carries metadata, and \p{...} matches by it; \P{...} is the negation.

Deno.test("character property escapes", () => {
  assert(/^\p{White_Space}+$/v.test("\t \n\r"));
  assertEquals("a\tb c".match(/\P{White_Space}/gv), ["a", "b", "c"]);

  assert(/^\p{Script=Greek}+$/v.test("μετά"));
  assertEquals("1π2ü3é4".replaceAll(/\p{Letter}/gv, ""), "1234");
  assertEquals("AbCdEf".replaceAll(/\p{Lowercase_Letter}/gv, ""), "ACE");
});

character property escapes ... ok (0ms)

Two forms. \p{Name=Value} names a property with values, such as Script=Greek. \p{Value} is shorthand that works for General_Category values like Letter, and for boolean properties like White_Space. Stripping \p{Letter} from "1π2ü3é4" leaves only the digits, "1234", regardless of script; stripping \p{Lowercase_Letter} from mixed-case ASCII leaves only the uppercase letters. The ones worth remembering: Letter, Number, Uppercase_Letter, Lowercase_Letter, White_Space, Punctuation, Script=..., and ASCII. There are hundreds more, in the Unicode character database annex.

This is the feature most people do not know exists, and the one that makes a pattern work outside English. \p{Letter} instead of [A-Za-z] is usually a one-word fix for a bug you have not hit yet.

string property escapes match listed sequences only

v adds a second kind of property, one whose values are sequences of code points rather than single ones. Predict what happens when the three code points of an emoji sequence arrive in reverse order:

Deno.test("string property escapes match listed sequences only", () => {
  const reversed = "\u{1F4AB}\u{200D}\u{1F635}";
  assertEquals(/^\p{RGI_Emoji}$/v.test(reversed), true);
});

Check programs/unicode-in-patterns.test.ts
running 5 tests from ./programs/unicode-in-patterns.test.ts
...
string property escapes match listed sequences only ... FAILED (11ms)

 ERRORS

string property escapes match listed sequences only => ./programs/unicode-in-patterns.test.ts:45:11
error: AssertionError: Values are not equal.

    [Diff] Actual / Expected

-   false
+   true

 FAILURES

string property escapes match listed sequences only => ./programs/unicode-in-patterns.test.ts:45:11

FAILED | 4 passed | 1 failed (10ms)

error: Test failed

The reversed sequence does not match, even though it is built from the identical three code points as one that does. That is not a bug. Correct the prediction and put both orders side by side:

Deno.test("string property escapes match listed sequences only", () => {
  const reversed = "\u{1F4AB}\u{200D}\u{1F635}";

  assertEquals([...faceWithSpiralEyes].length, 3);

  assertEquals(/^\p{RGI_Emoji}$/v.test(faceWithSpiralEyes), true);
  assertEquals(/^\p{RGI_Emoji}$/v.test(reversed), false);
});

string property escapes match listed sequences only ... ok (5ms)

faceWithSpiralEyes is U+1F635 U+200D U+1F4AB, three code points joined by a zero-width joiner, and RGI_Emoji matches it as one unit. reversed is the same three code points in the other order, and it does not match. RGI means recommended for general interchange, and the property is an enumerated list of specific sequences vendors have agreed to render, not a rule that accepts anything joined by a joiner. Swap the order and you have a sequence nobody publishes a glyph for, so it is not on the list. The natural assumption, that this is a general facility for multi-code-point characters, is the wrong one to carry forward: it is a lookup table.

a character property only ever matches one code point

The other half of the distinction, between a character property and a string property:

Deno.test("a character property only ever matches one code point", () => {
  assert(/^\p{Emoji}$/v.test("🙂"));
  assertFalse(/^\p{Emoji}$/v.test(faceWithSpiralEyes));

  assert(/^\p{Basic_Emoji}$/v.test("🙂"));
  assert(/^\p{Emoji_Keycap_Sequence}$/v.test("1️⃣"));
  assert(/^\p{RGI_Emoji_Flag_Sequence}$/v.test("\u{1F1E9}\u{1F1EA}"));
});

a character property only ever matches one code point ... ok (1ms)

Emoji is a character property, so it matches one code point and refuses the three-code-point sequence that RGI_Emoji accepted in the previous step. RGI_Emoji is a string property, and it is the union of narrower ones: Basic_Emoji, Emoji_Keycap_Sequence, RGI_Emoji_Flag_Sequence, and the modifier and tag sequences alongside them. One more asymmetry worth knowing: a string property cannot be negated. There is no \P{RGI_Emoji}.

a class can hold a sequence, with \q

A character class is ordinarily a set of alternatives, so three code points inside one mean "any of these three". \q{...} says "this exact sequence", which is how a multi-code-point value gets into a set at all:

Deno.test("a class can hold a sequence, with \\q", () => {
  assert(
    /^[\q{\u{1F635}\u{200D}\u{1F4AB}}]$/v.test(faceWithSpiralEyes),
  );
  assertFalse(
    /^[\u{1F635}\u{200D}\u{1F4AB}]$/v.test(faceWithSpiralEyes),
  );

  assert(/^[\q{oak|elm}]$/v.test("oak"));
  assert(/^[\q{oak|elm}]$/v.test("elm"));
  assertFalse(/^[\q{oak|elm}]$/v.test("oat"));
});

a class can hold a sequence, with \q ... ok (0ms)

The second assertion shows why \q is needed at all: the same three code points written without it inside [...] are three alternatives, "any of these three code points", and the sequence does not match. Wrapped in \q{...}, they are one alternative, the whole sequence. Several sequences go in one \q, separated by |, which reads oddly inside square brackets and is correct: oak and elm both match, oat matches neither.

set operations on classes

v lets classes nest, and once they nest you can combine them: -- subtracts and && intersects. Union has no operator, because writing two things next to each other in one class was always union.

Deno.test("set operations on classes", () => {
  assertFalse(/^[\w--[a-g]]$/v.test("a"));
  assert(/^[\w--[a-g]]$/v.test("h"));
  assert(/^[\p{Number}--[0-9]]$/v.test("٣"));
  assertFalse(/^[\p{Number}--[0-9]]$/v.test("3"));

  assert(/^[\p{ASCII}&&\p{Letter}]$/v.test("D"));
  assertFalse(/^[\p{ASCII}&&\p{Letter}]$/v.test("Δ"));
  assert(/^[\p{Script=Arabic}&&\p{Number}]$/v.test("٣"));

  assert(/^[\p{Emoji_Keycap_Sequence}[a-z]]+$/v.test("a1️⃣c"));
});

set operations on classes ... ok (0ms)

Walk one of each.

1. \w--[a-g] subtracts a range from the ASCII word characters: a through g are gone, h remains.
2. \p{Number}--[0-9] reads as "every number that is not an ASCII digit", and an Arabic-Indic digit qualifies while an ASCII 3 does not.
3. \p{ASCII}&&\p{Letter} intersects to "letters, but only the ASCII ones": D passes, the Greek Δ does not. \p{Script=Arabic}&&\p{Number} intersects a script with a category directly.
4. The last line unions a string property with a plain range by writing them side by side in one class.

This is the feature that makes property escapes usable rather than merely available. Before it, both queries meant writing the set out by hand or combining lookaheads. One caution: -- and && are only valid inside a class, and a malformed one is a construction-time SyntaxError rather than a silent misreading, which is the right trade and the subject of a later step.

two ways of negating agree under /v and not under /u

This is the reason v is a replacement rather than an extension, and it is worth reading all eight assertions slowly.

Deno.test("two ways of negating agree under /v and not under /u", () => {
  assert(/^\P{Lowercase_Letter}$/iu.test("A"));
  assert(/^\P{Lowercase_Letter}$/iu.test("a"));
  assertFalse(/^[^\p{Lowercase_Letter}]$/iu.test("A"));
  assertFalse(/^[^\p{Lowercase_Letter}]$/iu.test("a"));

  assertFalse(/^\P{Lowercase_Letter}$/iv.test("A"));
  assertFalse(/^\P{Lowercase_Letter}$/iv.test("a"));
  assertFalse(/^[^\p{Lowercase_Letter}]$/iv.test("A"));
  assertFalse(/^[^\p{Lowercase_Letter}]$/iv.test("a"));
});

two ways of negating agree under /v and not under /u ... ok (1ms)

Under /u with /i, "not a lowercase letter" written as \P{...} matches both "A" and "a", and written as [^\p{...}] matches neither. Two spellings of the same idea, opposite answers, and neither is what anyone wants. Under /v they agree, and they agree on the answer that actually follows from case-insensitivity: with /i there is no way to be "not a lowercase letter" and still be a letter at all, because the comparison folds case first.

The general shape of the bug is worth keeping regardless of this specific property: adding /i to a pattern should make it match more strings, never fewer. Under /u and a negated property, it made it match fewer.

/v is stricter, and says so at construction

Deno.test("/v is stricter, and says so at construction", () => {
  assertThrows(
    () => new RegExp("-", "uv"),
    SyntaxError,
    "Invalid flags supplied to RegExp constructor 'uv'",
  );

  assertThrows(
    () => new RegExp(String.raw`\a`, "v"),
    SyntaxError,
    "Invalid escape",
  );

  assert(/\a/.test("a"));
});

/v is stricter, and says so at construction ... ok (0ms)

Three facts, three lines.

1. u and v cannot be combined, and the error says so plainly. Pick v.
2. Under either Unicode flag, escaping a character that has no meaning as an escape is a SyntaxError. Without a flag, \a silently means the letter a, which is how a typo in a pattern becomes a pattern that works by accident, exactly the plausible-failure pattern the strings page's substring is not slice warned about in a different API. The strictness under v exists so that \p, \q, and future escapes can be added later without breaking anything already written.
3. Note when the error arrives. A pattern written as a literal fails at parse time, so the file will not load at all; the constructor form used here fails when the line runs, which is one more reason to prefer a literal, from the regular-expressions page's two ways to build one, whenever the pattern is fixed rather than built from data.

do not reach for regular expressions for real characters

v handles code points and an enumerated list of emoji sequences, and that is where it stops. A family emoji, a flag with a modifier, an accented letter written as a base plus a combining mark: all of those are one thing to a reader and several code points to a pattern.

Deno.test("do not reach for regular expressions for real characters", () => {
  function graphemes(text: string): string[] {
    const segmenter = new Intl.Segmenter("en", { granularity: "grapheme" });
    return [...segmenter.segment(text)].map((segment) => segment.segment);
  }

  assertEquals(graphemes("a🙂b"), ["a", "🙂", "b"]);

  assertEquals(graphemes(faceWithSpiralEyes), [faceWithSpiralEyes]);
  assertEquals(faceWithSpiralEyes.match(/./gv)?.length, 3);
});

do not reach for regular expressions for real characters ... ok (5ms)

The last two lines draw the boundary exactly. Intl.Segmenter, from the text page's Intl.Segmenter gives you what a reader sees, counts faceWithSpiralEyes as one grapheme: one character to a reader. /./gv counts it as three: three code points, one of them the joiner that has no visible form of its own. Neither tool is wrong; they answer different questions, and this is the page where that distinction has consequences for a pattern you write. When the unit of work is "what a person calls one character", reach for Intl.Segmenter. v fixes code points; it was never going to fix graphemes too.

In practice

• Put /v on every pattern you write.
• Use \p{...} instead of hand-written ranges for letters, digits, scripts, and case.
• Use Intl.Segmenter, not a regular expression, when the unit is a reader-perceived character rather than a code point.
• Rely on v rejecting invalid escapes and malformed set operations instead of silently matching something unintended.