bastianplsfix

← JavaScript + TypeScript

Matching and replacing

Six methods run a pattern against a string, and you pick one by what you want back: test for a boolean, search for a position, match for one match or all the matched text, matchAll for every match with its captures, replace and replaceAll for a new string, and split for pieces.

Then there is the thing that makes this harder than it should be. A regular expression with g or y is stateful: it carries a lastIndex property that some methods read, some write, and some ignore. Half this page is about getting caught by that, with three wrong predictions along the way.

The short version of the advice: use matchAll, and never call test on a pattern with g. The regular expressions page is the pattern language; this one is the API that runs it.

Create programs/matching-and-replacing.test.ts for this reference and keep it open. Start it with the assertion helpers used on this page:

import { assert, assertEquals, assertThrows } from "@std/assert";

Follow the page as you add and revise the runnable examples below that import.

six methods, by what they hand back

Deno.test("six methods, by what they hand back", () => {
  assert(/\d/v.test("a1"));
  assertEquals("a1".search(/\d/v), 1);

  assertEquals("ab aab".match(/(a+)b/v)?.[1], "a");
  assertEquals([..."ab aab".matchAll(/(a+)b/gv)].map((m) => m[1]), ["a", "aa"]);

  assertEquals("a1b2".replaceAll(/\d/gv, "#"), "a#b#");
  assertEquals("a1b2".split(/\d/v), ["a", "b", ""]);
});
Check programs/matching-and-replacing.test.ts
running 1 test from ./programs/matching-and-replacing.test.ts
six methods, by what they hand back ... ok (1ms)

ok | 1 passed | 0 failed (2ms)

Two of the six carry warnings from earlier pages.

  1. search answers -1 when nothing matches, an in-band marker with all the problems the sentinels page priced: -1 is truthy, so if (text.search(re)) is wrong in both directions.
  2. split on "a1b2" gives three pieces, and the third is empty, because there is nothing after the 2. split never drops an empty piece at either end, the same behavior the strings page's splitting an empty string is not an empty list measured with a string separator.

the match object is an array with extras

Deno.test("the match object is an array with extras", () => {
  const match = "ab aab".match(/(?<as>a+)b/v);

  assertEquals(match?.[0], "ab");
  assertEquals(match?.[1], "a");
  assertEquals(match?.groups?.as, "a");
  assertEquals(match?.index, 0);
  assertEquals(match?.input, "ab aab");
});
the match object is an array with extras ... ok (0ms)

A match object is an array with extra properties bolted on. Element 0 is the whole match and the rest are the numbered captures, so a named group is reachable twice: as a number and inside groups. index is where the match started, and input is the whole string you searched.

groups is a bare object

Deno.test("groups is a bare object", () => {
  const groups = "ab".match(/(?<first>a)/v)?.groups;

  assertEquals(Object.getPrototypeOf(groups!), null);
  assertEquals((groups as Record<string, unknown>).toString, undefined);
});
groups is a bare object ... ok (0ms)

groups has a null prototype, so it inherits nothing: no toString, no hasOwnProperty, none of it. The conversion-and-coercion page's Object wraps, and a primitive is an instance of nothing met the same kind of object from the other direction, where the missing toString made every primitive conversion throw.

Here it is deliberate and correct. A capture group could be named constructor or toString, and on an ordinary object that capture would collide with an inherited member. Stripping the prototype means a group name can be anything without ambiguity. The practical consequence is that spreading it into {...groups} gives you an ordinary object when you want one.

match returns three different shapes

One method, one argument, three possible return shapes, and the flag decides which. Predict what /g does to the captures:

Deno.test("match returns three different shapes", () => {
  assertEquals("ab aab".match(/(a+)b/gv), ["ab", "a", "aab", "aa"]);
});
Check programs/matching-and-replacing.test.ts
running 4 tests from ./programs/matching-and-replacing.test.ts
match returns three different shapes ... FAILED (8ms)

 ERRORS

match returns three different shapes => ./programs/matching-and-replacing.test.ts:4:11
error: AssertionError: Values are not equal.

    [Diff] Actual / Expected

    [
      "ab",
+     "a",
      "aab",
+     "aa",
    ]

 FAILURES

match returns three different shapes => ./programs/matching-and-replacing.test.ts:4:11

FAILED | 3 passed | 1 failed (10ms)

error: Test failed

The two captures are missing from the actual result. With /g, match hands back a plain array of the matched text and throws the captures away, along with every index. Correct the prediction and put all three shapes side by side:

Deno.test("match returns three different shapes", () => {
  const one = "ab aab".match(/(a+)b/v);
  assertEquals(one?.slice(0), ["ab", "a"]);
  assertEquals(one?.index, 0);

  assertEquals("ab aab".match(/(a+)b/gv), ["ab", "aab"]);

  assertEquals("xyz".match(/(a+)b/gv), null);
  assertEquals("xyz".match(/(a+)b/v), null);
});
match returns three different shapes ... ok (0ms)

Three shapes from one method.

  1. Without /g: a match object with captures and an index.
  2. With /g: a plain array of matched text, captures and indices discarded.
  3. Either way, no match is null rather than an empty array, so text.match(re).length is a crash waiting for the wrong input. ?? [] handles it, and the better answer is the next step.

matchAll wants /g and gives you everything

Deno.test("matchAll wants /g and gives you everything", () => {
  const matches = [..."ab aab".matchAll(/(?<as>a+)b/gv)];

  assertEquals(matches.map((m) => [m[0], m.groups?.as, m.index]), [
    ["ab", "a", 0],
    ["aab", "aa", 3],
  ]);

  assertThrows(
    () => "ab".matchAll(/a/v),
    TypeError,
    "String.prototype.matchAll called with a non-global RegExp argument",
  );
});
matchAll wants /g and gives you everything ... ok (0ms)

This is the method to reach for. It returns an iterator of full match objects, so every match keeps its captures and its index, and it throws without /g rather than quietly doing something else.

Compare the three failure modes now visible: match with /g silently discards captures, exec in a loop silently depends on state, and matchAll refuses to run. Refusing is better, the same loud-over-plausible preference the strings page drew out in replace does one, replaceAll does all.

lastIndex, and what each method does with it

lastIndex is the only writable property a regular expression has, and it holds where the next match starts. Four methods, four different behaviors:

Deno.test("lastIndex, and what each method does with it", () => {
  const plain = /#/v;
  plain.lastIndex = 1;
  assertEquals("##-#".match(plain)?.index, 0);
  assertEquals(plain.lastIndex, 1);

  const sticky = /#/vy;
  sticky.lastIndex = 1;
  assertEquals("##-#".match(sticky)?.index, 1);
  assertEquals(sticky.lastIndex, 2);

  const global = /#/gv;
  global.lastIndex = 1;
  assertEquals("##-#".match(global), ["#", "#", "#"]);
  assertEquals(global.lastIndex, 0);

  const all = /#/gv;
  all.lastIndex = 1;
  assertEquals([..."##-#".matchAll(all)].map((m) => m.index), [1, 3]);
  assertEquals(all.lastIndex, 1);
});
lastIndex, and what each method does with it ... ok (1ms)

Read the four blocks as four rows of a table.

  1. Neither flag: lastIndex is ignored entirely, the match starts at 0, and the property is left untouched at 1.
  2. /y (sticky): the match must begin exactly at lastIndex, so it starts at 1, and the property advances to 2.
  3. /g with match: lastIndex is ignored, so the match at index 0 is included, and then it is reset to 0. That is a write you did not ask for.
  4. matchAll: lastIndex is honored, so the match at index 0 is skipped and only 1 and 3 are found, and the property is left exactly as you set it.

There is no principle to derive here; it is a table, and the specification is the table. The way to be safe is to never share a /g pattern between calls.

test with /g alternates between true and false

Same pattern, same input, called twice. Predict the second answer:

Deno.test("test with /g alternates between true and false", () => {
  const stateful = /^X/gv;

  assertEquals(stateful.test("Xa"), true);
  assertEquals(stateful.test("Xa"), true);
});
Check programs/matching-and-replacing.test.ts
running 7 tests from ./programs/matching-and-replacing.test.ts
test with /g alternates between true and false ... FAILED (8ms)

 ERRORS

test with /g alternates between true and false => ./programs/matching-and-replacing.test.ts:4:11
error: AssertionError: Values are not equal.

    [Diff] Actual / Expected

-   false
+   true

 FAILURES

test with /g alternates between true and false => ./programs/matching-and-replacing.test.ts:4:11

FAILED | 6 passed | 1 failed (9ms)

error: Test failed

The same call on the same string answers differently the second time. Correct the prediction and add a third call:

Deno.test("test with /g alternates between true and false", () => {
  const stateful = /^X/gv;

  assertEquals(stateful.test("Xa"), true);
  assertEquals(stateful.test("Xa"), false);
  assertEquals(stateful.test("Xa"), true);
});
test with /g alternates between true and false ... ok (0ms)

Trace the cycle.

  1. The first call matches at index 0 and advances lastIndex past the match.
  2. The second call starts from there, and ^X cannot match away from the start, so it fails, and a failed match resets lastIndex to 0.
  3. The third call therefore starts fresh and succeeds again.

A pure-looking predicate whose answer depends on how many times you have called it. test almost never wants /g, and this happens when a pattern written for replaceAll gets reused for a check. Never call test on a pattern with /g.

a shared regular expression carries its position

The previous step was one function misbehaving. Here is the same statefulness crossing a function boundary, with a pattern somebody has already used:

Deno.test("a shared regular expression carries its position", () => {
  function countMatchesBadly(pattern: RegExp, text: string): number {
    let count = 0;
    while (pattern.test(text)) count++;
    return count;
  }

  function countMatches(pattern: RegExp, text: string): number {
    if (!pattern.global) throw new TypeError("pattern needs /g");
    return [...text.matchAll(pattern)].length;
  }

  const shared = /a/gv;
  shared.lastIndex = 4;

  assertEquals(countMatchesBadly(shared, "babaa"), 1);
  assertEquals(shared.lastIndex, 0);
  assertEquals(countMatches(shared, "babaa"), 3);
});
a shared regular expression carries its position ... ok (0ms)

"babaa" contains three as, and the first function reports one.

  1. shared.lastIndex starts at 4, exactly as a previous call would have left it. countMatchesBadly therefore begins looking at index 4, finds the single a there, counts it, and the next test fails.
  2. That failure resets lastIndex to 0, which the middle assertion records. The bad function left the pattern in a clean state, purely by accident.
  3. countMatches runs on the same object and answers 3, correctly, because matchAll starts from a lastIndex that happens to be zero now.

Read those together and the danger is clearer than either alone: the wrong answer and the right answer came from the same pattern object, and which one you get depends on what ran before you. The while (pattern.test(text)) loop is the one people write, and it works exactly once, on a fresh pattern.

Four ways this bites, all real: a pattern inlined in a loop condition is rebuilt each time so lastIndex is always zero and the loop never ends; removing /g from a pattern a test loop relies on makes that loop never end; adding /g to a pattern used with test makes the answer alternate; and a non-zero lastIndex from a previous call makes a fresh call skip the beginning.

Three ways out. Throw if the pattern is not what your function needs, as countMatches does. Clone it with new RegExp(pattern, flags), from the regular-expressions page's flags are immutable, so changing them means cloning, so you cannot affect the caller's copy. Or use a method that does not depend on state, which means matchAll.

exec in a loop is the old way

Deno.test("exec in a loop is the old way", () => {
  const pattern = /(a+)b/gv;
  const found: string[] = [];

  let match: RegExpExecArray | null;
  while ((match = pattern.exec("ab aab")) !== null) {
    found.push(match[1]);
  }

  assertEquals(found, ["a", "aa"]);
  assertEquals(pattern.lastIndex, 0);
});
exec in a loop is the old way ... ok (0ms)

You will read this in existing code, so it is worth being able to see what it does. exec with /g returns the next match each time and null when there are none left, and that final null resets lastIndex, which the last assertion confirms.

Three things have to be right: the pattern must live outside the loop, it must have /g, and lastIndex must start at zero. matchAll needs none of the three, which is why it exists.

replace and replaceAll disagree about what they accept

Deno.test("replace and replaceAll disagree about what they accept", () => {
  assertEquals("aaa".replace("a", "x"), "xaa");
  assertEquals("aaa".replaceAll("a", "x"), "xxx");

  assertEquals("aaa".replace(/a/v, "x"), "xaa");
  assertEquals("aaa".replace(/a/gv, "x"), "xxx");

  assertThrows(
    () => "aaa".replaceAll(/a/v, "x"),
    TypeError,
    "String.prototype.replaceAll called with a non-global RegExp argument",
  );
});
replace and replaceAll disagree about what they accept ... ok (0ms)

Five combinations and one exception. A string search means literal text: once for replace, all for replaceAll. A pattern without /g means once for replace, and a TypeError for replaceAll. A pattern with /g means all, for both.

The asymmetry is history. replace predates replaceAll by twenty years and had to handle /g itself, so that behavior survives as a duplicate of the newer method, and replaceAll throwing is the newer design being stricter on purpose. Reach for replaceAll with a plain string whenever the search text is literal: no flag, no escaping, no pattern.

the dollar language in a replacement string

A replacement string is not plain text; $ introduces six things. Predict what $0 inserts:

Deno.test("the dollar language in a replacement string", () => {
  assertEquals("a".replaceAll(/a/gv, "$0"), "a");
});
Check programs/matching-and-replacing.test.ts
running 11 tests from ./programs/matching-and-replacing.test.ts
the dollar language in a replacement string ... FAILED (8ms)

 ERRORS

the dollar language in a replacement string => ./programs/matching-and-replacing.test.ts:4:11
error: AssertionError: Values are not equal.

    [Diff] Actual / Expected

-   $0
+   a

 FAILURES

the dollar language in a replacement string => ./programs/matching-and-replacing.test.ts:4:11

FAILED | 10 passed | 1 failed (9ms)

error: Test failed

$0 is not the whole match; it is the literal text $0, because capture numbering starts at one and there is no group zero. Correct that and collect the rest of the syntax:

Deno.test("the dollar language in a replacement string", () => {
  assertEquals(
    "a1 a2".replaceAll(/a/gv, "($`|$&|$')"),
    "(|a|1 a2)1 (a1 |a|2)2",
  );

  assertEquals(
    "first: Ada".replaceAll(/^([A-Za-z]+): (.*)$/gv, "KEY: $1, VALUE: $2"),
    "KEY: first, VALUE: Ada",
  );
  assertEquals(
    "first: Ada".replaceAll(/^(?<key>[A-Za-z]+): (?<value>.*)$/gv, "$<key>!"),
    "first!",
  );

  assertEquals("a".replaceAll(/a/gv, "$$"), "$");
  assertEquals("a".replaceAll(/a/gv, "$0"), "$0");
});
the dollar language in a replacement string ... ok (0ms)

The six: $1 is a numbered capture, $<name> a named one, $& the whole match, $` everything before it, $' everything after it, and $$ a literal dollar. The first assertion shows the three positional ones at once, which is why its expected value reads like line noise and is worth tracing character by character once.

Two traps. $0 is the one just measured. And a replacement string containing a dollar sign that came from data will be interpreted, a small injection with the same shape the strings page's the replacement string has its own syntax demonstrated; the fix there is the fix here, which is the next step. Prefer $<name>: it is the only one of the six that says what it means.

a replacement function, and where the groups arrive

Deno.test("a replacement function, and where the groups arrive", () => {
  const seen: unknown[] = [];
  "x2y".replaceAll(/(?<digit>\d)/gv, (...args) => {
    seen.push(args.slice(0, -1), args.at(-1));
    return "#";
  });

  assertEquals(seen, [
    ["2", "2", 1, "x2y"],
    { digit: "2" },
  ]);

  assertEquals(
    "3 cats and 4 dogs".replaceAll(
      /[0-9]+/gv,
      (all) => String(2 * Number(all)),
    ),
    "6 cats and 8 dogs",
  );
});
a replacement function, and where the groups arrive ... ok (0ms)

Pass a function and it is called once per match, with its return value used as the replacement, which is how you compute a replacement rather than describe one: the second assertion doubles every number in a sentence.

The parameter list is where this gets ugly, and the recorded seen array shows the exact order: the whole match, then one argument per numbered capture, then the offset, then the input string, and only then, and only if the pattern has named groups, the groups object. Because the position of that last argument depends on how many capture groups you have, the reliable way to reach it is from the end, with args.at(-1). An unpleasant signature, worth seeing once so you recognize the idiom.

the /d flag adds positions

Deno.test("the /d flag adds positions", () => {
  const match = /(?<as>a+)(?<bs>b+)/dv.exec("aaaabb");

  assertEquals(match?.indices?.[1], [0, 4]);
  assertEquals(match?.indices?.groups?.as, [0, 4]);
  assertEquals(match?.indices?.groups?.bs, [4, 6]);
});
the /d flag adds positions ... ok (0ms)

With /d, a match object gains an indices property mirroring the captures: a [start, end] pair per numbered group, and the same under indices.groups for named ones. The use case is anything reporting a location back to a person: a parser pointing at where a syntax error begins, an editor underlining part of a line, a linter's column numbers. Without /d you would compute those from the match text and its index, which is possible and easy to get wrong.

split keeps what a group captured

Deno.test("split keeps what a group captured", () => {
  assertEquals("a x:yyy b".split(/x+:y+/v), ["a ", " b"]);
  assertEquals("a x:yyy b".split(/(x+):(y+)/v), ["a ", "x", "yyy", " b"]);

  assertEquals("a: b: c".split(/(?<=:) */v), ["a:", "b:", "c"]);

  assertEquals("a,b,c".split(",", 2), ["a", "b"]);
  assertEquals("a🙂".split(""), ["a", "\uD83D", "\uDE42"]);
});
split keeps what a group captured ... ok (0ms)

Four things worth knowing.

  1. Captures become elements. Adding groups to a separator pattern interleaves the captured text with the pieces, which is either exactly what a tokenizer wants or a surprise.
  2. A lookaround keeps the separator in the piece, because it matches a position without consuming anything, from the regular-expressions page's four kinds of lookaround, and none of them consume. That is how you split after a delimiter rather than around it.
  3. A limit truncates rather than combining the rest, so the tail is discarded entirely.
  4. split("") splits into code units, not characters, so it breaks the emoji into the surrogate halves the text and characters page named. [...text] gives code points, and Intl.Segmenter gives what a person calls a character.

split and search are the only two methods that ignore /g and /y completely, which makes them the only two safe to call on a shared pattern.

treat groups as possibly missing

The last step is about the one place in this API where the types actively mislead:

Deno.test("treat groups as possibly missing", () => {
  const ENTRY = /^(?<key>[a-z]+)=(?<value>.*)$/v;

  type Entry = { key: string; value: string };

  function parseEntry(line: string): Entry | undefined {
    const groups = ENTRY.exec(line)?.groups as
      | Record<string, string | undefined>
      | undefined;
    if (!groups?.key || groups.value === undefined) return undefined;
    return { key: groups.key, value: groups.value };
  }

  assertEquals(parseEntry("size=10"), { key: "size", value: "10" });
  assertEquals(parseEntry("nonsense"), undefined);

  const typo = ENTRY.exec("size=10")?.groups;
  assertEquals(typo?.mispelled, undefined);

  const alternative = /(?<first>a)|(?<second>b)/v.exec("b")?.groups;
  assertEquals(alternative?.first, undefined);
});
treat groups as possibly missing ... ok (0ms)

groups is typed as {[key: string]: string} | undefined, and the last two assertions show that type is wrong twice over.

  1. typo?.mispelled type-checks and compiles, because an index signature accepts every key, so a misspelled group name is a silent undefined rather than an error. That is the same index-signature looseness the what-a-type-is page described, here costing you a compile-time check you would expect to have.
  2. alternative?.first is undefined at run time, because the group sits in an alternative that did not match, while the type promises a string.

So the cast inside parseEntry is not pedantry; it is the type the value actually has. Narrow the result of exec into a shape you declared, at the boundary, and the rest of your code gets to trust it, which is the same edge-narrowing discipline the any, unknown, and never page argued for with unknown.

In practice