bastianplsfix

← JavaScript + TypeScript

Regular expressions

A regular expression is a small program for matching text. It has two parts: a pattern, which describes what to match, and flags, which change how the pattern is interpreted.

Always use the v flag. It is the recommendation of everyone who works on this feature, it fixes real bugs, and it costs one character. What it buys you is the unicode in patterns page's whole subject.

This entry is the pattern language: the pieces you write inside /.../, and the ones that read like trivia until they cost you an afternoon. It is not a full reference; MDN has one. For the methods that run a pattern against a string, and the statefulness /g adds, see matching and replacing.

Create programs/regular-expressions.test.ts for this reference and keep it open. Start it with the assertion helpers used on this page:

import { assert, assertEquals, assertFalse } from "@std/assert";

Follow the page as you add and revise the runnable examples below that import.

two ways to build one

Deno.test("two ways to build one", () => {
  const literal = /^a+b$/v;
  const constructed = new RegExp(String.raw`^a+b$`, "v");

  assert(literal.test("aab"));
  assert(constructed.test("aab"));
  assertFalse(literal.test("abb"));

  assertEquals(literal.source, "^a+b$");
  assertEquals(literal.flags, "v");
});
Check programs/regular-expressions.test.ts
running 1 test from ./programs/regular-expressions.test.ts
two ways to build one ... ok (0ms)

ok | 1 passed | 0 failed (2ms)

/^a+b$/v is compiled when the file loads. new RegExp(String.raw\^a+b$`, "v")` is compiled when the line runs. Use the literal when the pattern is known as you write it, which is nearly always. Use the constructor when part of the pattern comes from data, or when you build it out of pieces, which a long pattern is code, so write it like code does at the end of this page.

String.raw matters in the constructor form, and RegExp.escape is deliberately unreadable explains why: a pattern is full of backslashes, and without String.raw JavaScript's own string escaping would resolve them before the regex engine ever saw them, the same tagged-templates distinction from cooked text and raw text. Two properties to know: .source is the pattern as text, and .flags is the flags as text.

the pieces, one example each

Deno.test("the pieces, one example each", () => {
  assert(/^[A-Z][a-z]+$/v.test("Ada"));
  assert(/^\d{4}-\d{2}$/v.test("2077-01"));
  assert(/^(?:ab)+$/v.test("abab"));
  assertEquals(/(\d+)-(\d+)/v.exec("10-20")?.slice(1), ["10", "20"]);
  assertEquals(
    /(?<from>\d+)-(?<to>\d+)/v.exec("10-20")?.groups,
    { from: "10", to: "20" },
  );
  assert(/\bcat\b/v.test("a cat sat"));
  assertFalse(/\bcat\b/v.test("concatenate"));
});
the pieces, one example each ... ok (0ms)

Seven lines, and they cover most of what you will write.

  1. A character class, [A-Z], matches one character from a set. [a-z]+ repeats that with a quantifier: + for one or more. {4} is a quantifier too, counted repetition, exactly four.
  2. (?:ab)+ is a group that repeats without capturing: it groups ab for the +, and produces no capture of its own.
  3. (\d+)-(\d+) captures with plain parentheses, and .exec(...)?.slice(1) reads the two captures off the match array. (?<from>...) captures under a name instead, and .groups reads named captures back by that name.
  4. \bcat\b is a word boundary, an anchor: it matches a position, not a character, so cat matches inside "a cat sat" and not inside "concatenate", where the letters are there but no boundary surrounds them.

The rest of the language is . for any character, \d \w \s and their uppercase complements for digits, word characters, and whitespace, | for alternatives, and the lookarounds further down.

flags report themselves in alphabetical order

Deno.test("flags report themselves in alphabetical order", () => {
  assertEquals(/x/gymdivs.flags, "dgimsvy");
  assertEquals(/a+b/gi.source, "a+b");
  assert(/a/i.ignoreCase);
  assertFalse(/a/.ignoreCase);
});
flags report themselves in alphabetical order ... ok (0ms)

Written as gymdivs, reported back as dgimsvy: whatever order you type them in, .flags reports them alphabetically, and matching that order in your own source keeps it lined up with what you will see in a debugger. Eight flags exist, each with a long-named property:

flags are immutable, so changing them means cloning

Deno.test("flags are immutable, so changing them means cloning", () => {
  const original = /oak/i;
  const withGlobal = new RegExp(original, "gi");

  assertEquals(original.flags, "i");
  assertEquals(withGlobal.flags, "gi");
  assertEquals(withGlobal.source, original.source);
});
flags are immutable, so changing them means cloning ... ok (0ms)

original is untouched after withGlobal is built. Every property on a regular expression except lastIndex is read-only, so there is no way to add a flag to an existing one. The second form of the constructor takes a regular expression and a new set of flags, which is how you copy one with a change: useful when a caller hands you a pattern and your algorithm needs /g, a technique a shared regular expression carries its position builds on.

escaping, and the characters that need it

Deno.test("escaping, and the characters that need it", () => {
  assert(/\*/v.test("*"));
  assert(/\//v.test("/"));
  assert(new RegExp("/", "v").test("/"));
  assert(/\$\d/v.test("$5"));
});
escaping, and the characters that need it ... ok (0ms)

Thirteen characters are special at the top level of a pattern and need a backslash to match literally:

^ $ \ . * + ? ( ) [ ] { } |

Plus / inside a literal, since that is what ends the literal, which is why the constructor form on line three needs no escape for the same character. Inside a character class the rules are different, and under v more characters need escaping than before. The honest advice is not to learn either table.

If the text comes from data, escape it with a function, which is the next step. If you are writing the pattern by hand, the linter catches an invalid one: no-invalid-regexp is on by default. So is a rule for the mistake you cannot see:

error[no-regex-spaces]: more than one consecutive spaces in RegExp is not allowed
 --> lintdemo.ts:1:26
  |
1 | export const twoSpaces = /Hello,   world/v;
  |                          ^^^^^^^^^^^^^^^^^

  docs: https://docs.deno.com/lint/rules/no-regex-spaces

Three spaces or two, in a pattern, look identical on the page. no-regex-spaces wants \x20{3} instead, which says what it means.

RegExp.escape is deliberately unreadable

Deno.test("RegExp.escape is deliberately unreadable", () => {
  assertEquals(RegExp.escape("(*)"), String.raw`\(\*\)`);
  assertEquals(RegExp.escape("_oak"), "_oak");

  assertEquals(RegExp.escape("oak"), String.raw`\x6fak`);
  assertEquals(RegExp.escape("1oak"), String.raw`\x31oak`);

  assertEquals(RegExp.escape("a-b"), String.raw`\x61\x2db`);
  assertEquals(RegExp.escape("oak elm"), String.raw`\x6fak\x20elm`);

  assert(new RegExp(RegExp.escape("a.b"), "v").test("a.b"));
  assertFalse(new RegExp(RegExp.escape("a.b"), "v").test("axb"));
});
RegExp.escape is deliberately unreadable ... ok (0ms)

RegExp.escape(text) returns a pattern that matches text literally, and the output will surprise you the first time. (*) becomes \(\*\), which is what you expected. But oak becomes \x6fak, with the leading o turned into a hex escape, and a-b becomes \x61\x2db.

Both are deliberate. The guarantee is that the result is safe in any context, including immediately after another pattern and inside a character class. A leading letter or digit is escaped so it cannot merge with whatever precedes it; a hyphen is escaped because inside a class it would mean a range; a space is escaped because a pattern might be written with insignificant whitespace, the same whitespace a long pattern is code, so write it like code relies on being able to strip.

So the output is not for reading, and that is the feature. The last two lines prove it does what it promises regardless of how strange the middle looks. Reach for it when part of a pattern must match text you did not write, but check first whether you need a pattern at all, which check whether you need a pattern at all comes back to.

greedy by default, reluctant with a ?

Every quantifier takes as much as it can and gives characters back only if the rest of the pattern fails. Predict what /<.+>/ matches against text with two tags:

Deno.test("greedy by default, reluctant with a ?", () => {
  assertEquals(/<.+>/v.exec("<a> and <b>")?.[0], "<a>");
});
Check programs/regular-expressions.test.ts
running 7 tests from ./programs/regular-expressions.test.ts
...
greedy by default, reluctant with a ? ... FAILED (28ms)

 ERRORS

greedy by default, reluctant with a ? => ./programs/regular-expressions.test.ts:66:11
error: AssertionError: Values are not equal.

    [Diff] Actual / Expected

-   <a> and <b>
+   <a>

 FAILURES

greedy by default, reluctant with a ? => ./programs/regular-expressions.test.ts:66:11

FAILED | 6 passed | 1 failed (10ms)

error: Test failed

It matched from the first < all the way to the last >, swallowing "a> and <b" in between. .+ took as much as it could, and the rest of the pattern, one closing >, was still satisfied by the very last one in the string. Correct the prediction, and add the fix alongside the simpler pair that motivates it:

Deno.test("greedy by default, reluctant with a ?", () => {
  assertEquals(/X.*X/v.exec("XoakXelmX")?.[0], "XoakXelmX");
  assertEquals(/X.*?X/v.exec("XoakXelmX")?.[0], "XoakX");

  assertEquals(/<.+>/v.exec("<a> and <b>")?.[0], "<a> and <b>");
  assertEquals(/<.+?>/v.exec("<a> and <b>")?.[0], "<a>");
});
greedy by default, reluctant with a ? ... ok (0ms)

Adding ? after a quantifier reverses it: take as little as possible, and add more only if needed. /<.+>/ looks like it works, right up until the input has two tags, which is why noticing you need .+? is the skill rather than the syntax.

| binds more loosely than anything else

^aa|zz$ reads like "the whole string is aa or zz". Predict what it actually tests:

Deno.test("| binds more loosely than anything else", () => {
  assertEquals(/^aa|zz$/v.test("aa!!"), false);
});
Check programs/regular-expressions.test.ts
running 8 tests from ./programs/regular-expressions.test.ts
...
| binds more loosely than anything else ... FAILED (8ms)

 ERRORS

| binds more loosely than anything else => ./programs/regular-expressions.test.ts:74:11
error: AssertionError: Values are not equal.

    [Diff] Actual / Expected

-   true
+   false

 FAILURES

| binds more loosely than anything else => ./programs/regular-expressions.test.ts:74:11

FAILED | 7 passed | 1 failed (10ms)

error: Test failed

"aa!!" matches. The alternation | is the outermost operator in this pattern, so it splits the whole thing including the anchors: "starts with aa", or "ends with zz", not "the whole string is one of the two". Correct the prediction, and group the pattern to say what you meant:

Deno.test("| binds more loosely than anything else", () => {
  assert(/^aa|zz$/v.test("aa!!"));
  assertFalse(/^(?:aa|zz)$/v.test("aa!!"));
  assert(/^(?:aa|zz)$/v.test("zz"));
});
| binds more loosely than anything else ... ok (0ms)

^(?:aa|zz)$ is what you meant, and the non-capturing group from the pieces, one example each says you only wanted the grouping, no capture attached.

four kinds of lookaround, and none of them consume

A lookaround asserts something about the text around a match without becoming part of the match. Predict the negative case: which runs of lowercase letters are not followed by X?

Deno.test("four kinds of lookaround, and none of them consume", () => {
  assertEquals("oakX def".match(/[a-z]+(?!X)/gv), ["def"]);
});
Check programs/regular-expressions.test.ts
running 9 tests from ./programs/regular-expressions.test.ts
...
four kinds of lookaround, and none of them consume ... FAILED (9ms)

 ERRORS

four kinds of lookaround, and none of them consume => ./programs/regular-expressions.test.ts:80:11
error: AssertionError: Values are not equal.

    [Diff] Actual / Expected

    [
-     "oa",
      "def",
    ]

 FAILURES

four kinds of lookaround, and none of them consume => ./programs/regular-expressions.test.ts:80:11

FAILED | 8 passed | 1 failed (11ms)

error: Test failed

"oa" is a match too, and it is the one worth staring at. [a-z]+ is greedy, so it first tries to take all of oak, then checks the lookahead: the next character is X, so (?!X) fails and the quantifier gives one character back. oa is not followed by X either, so the assertion is satisfied there and the match stops. Correct the prediction, and see all four kinds together:

Deno.test("four kinds of lookaround, and none of them consume", () => {
  assertEquals("oakX def".match(/[a-z]+(?=X)/gv), ["oak"]);
  assertEquals("oakX def".match(/[a-z]+(?!X)/gv), ["oa", "def"]);
  assertEquals("Xoak def".match(/(?<=X)[a-z]+/gv), ["oak"]);
  assertEquals("Xoak def".match(/(?<!X)[a-z]+/gv), ["ak", "def"]);

  assertEquals(
    "Node.js: index.js and main.js".replaceAll(/(?<!Node)\.js/gv, ".html"),
    "Node.js: index.html and main.html",
  );
});
four kinds of lookaround, and none of them consume ... ok (0ms)

Note the first result: "oak", not "oakX", even though the X had to be there for the match to succeed. None of the four consume what they check: a lookahead or lookbehind reads text without including it in the match, positive or negative, ahead or behind. Negative lookaround plus a greedy quantifier is exactly the place to check your assumptions, as the wrong prediction just showed.

Lookbehind is what makes the last line possible: replace .js except after Node, in one pass, with no way to write it otherwise.

a pattern modifier applies a flag to part of a pattern

Deno.test("a pattern modifier applies a flag to part of a pattern", () => {
  const shouted = /^x(?i:HELLO)x$/v;

  assert(shouted.test("xHELLOx"));
  assert(shouted.test("xhellox"));
  assertFalse(shouted.test("XhelloX"));

  const exact = /^x(?-i:HELLO)x$/iv;
  assert(exact.test("XHELLOX"));
  assertFalse(exact.test("XhelloX"));
});
a pattern modifier applies a flag to part of a pattern ... ok (0ms)

(?i:...) turns a flag on for one group; (?-i:...) turns it off. Only i, m, and s are supported, because those are the only flags that make sense applied to part of a pattern rather than the whole thing.

  1. shouted matches "xHELLOx" and "xhellox", the middle case-insensitive, but not "XhelloX": the outer xs stay case-sensitive because they sit outside the group.
  2. exact runs the other direction, /iv makes everything insensitive except the group where -i turns it back off.

Two real uses: matching one part of an otherwise exact pattern case-insensitively, as above, and keeping flags inside the pattern text itself, which matters when a pattern is stored in a configuration file or composed from fragments that each want different flags.

matching everything, and matching nothing

Deno.test("matching everything, and matching nothing", () => {
  assertEquals(String(new RegExp("")), "/(?:)/");

  assert(/(?:)/v.test(""));
  assert(/(?:)/v.test("oak"));

  assertFalse(/.^/v.test(""));
  assertFalse(/.^/v.test("oak"));
});
matching everything, and matching nothing ... ok (1ms)

Two limits, occasionally useful as defaults. /(?:)/ matches everything, including the empty string, and it is what new RegExp("") becomes on its own, because // in source would start a comment rather than an empty pattern. /.^/ matches nothing: . consumes one character, and ^ only matches at the very start, a position the dot has already moved past.

Worth recognizing on sight: seeing /(?:)/ in output means somebody constructed a regular expression from an empty string, deliberately or not.

a long pattern is code, so write it like code

A pattern with no whitespace and no comments is unreadable for the same reason a program would be, and String.raw plus one replaceAll gets you both without a library:

Deno.test("a long pattern is code, so write it like code", () => {
  const API_SIGNATURE = new RegExp(
    String.raw`
      ^
      (?<prefix> new \x20 | get \x20 )?
      (?<name> [A-Za-z0-9_.\[\]]+ )
    `.replaceAll(/\s+/gv, ""),
    "v",
  );

  assertEquals(
    String(API_SIGNATURE),
    String.raw`/^(?<prefix>new\x20|get\x20)?(?<name>[A-Za-z0-9_.\[\]]+)/v`,
  );

  assertEquals(
    API_SIGNATURE.exec("get Map.prototype.size")?.groups,
    { prefix: "get ", name: "Map.prototype.size" },
  );
});
a long pattern is code, so write it like code ... ok (0ms)

Three things make it work.

  1. String.raw, from two ways to build one, means the backslashes reach the constructor literally, so the pattern reads as it would inside a /.../ literal, and the template can span multiple lines.
  2. .replaceAll(/\s+/gv, "") strips every run of whitespace before the constructor ever sees it, which is why every intentional space inside the pattern is written \x20 instead: a real space would vanish along with the formatting ones.
  3. The result, proven by the first assertion, is exactly the compact pattern you would have written by hand. The second assertion runs it, so the readable version is not just decoration; it is the pattern.

Libraries exist that do this more comfortably, with real comments and interpolated fragments. Two lines of standard JavaScript get most of the benefit, and a dependency for readability is a decision to make deliberately.

Write a test per pattern as a habit, not only for this one: three inputs and their expected captures document a pattern better than a comment can, and they fail when somebody edits it.

check whether you need a pattern at all

Deno.test("check whether you need a pattern at all", () => {
  assertEquals("(a) and (a)".replaceAll("(a)", "@"), "@ and @");

  assertEquals(
    "(a) and (a)".replaceAll(new RegExp(RegExp.escape("(a)"), "gv"), "@"),
    "@ and @",
  );
});
check whether you need a pattern at all ... ok (0ms)

Both lines produce the same result. replaceAll with a string searches for literal text, needing no escaping, no flag, and no regular expression at all; the strings page's replace does one, replaceAll does all already showed this method without the regex detour. The same goes for includes, startsWith, endsWith, and split with a string separator: a surprising amount of pattern-shaped code is asking a question a string method already answers, and RegExp.escape from RegExp.escape is deliberately unreadable exists for the cases where you truly do need the pattern.

In practice