bastianplsfix

← JavaScript + TypeScript

A generator is a function that can pause. yield hands a value out and suspends the body where it stands, and the next .next() resumes from exactly there, with every local variable as it was. That one ability is the whole feature, and everything below is a consequence of it, including the two you will use daily: a generator is the shortest way to write an iterable, and the values it produces are computed one at a time, when asked for.

The iterator helpers page showed that laziness from the consuming side, where a pipeline pulls one element at a time through a chain. This page is the same picture from the producing side, and the two features were built to meet in the middle.

Create programs/generators.test.ts for this reference and keep it open. Start it with the assertion helpers used on this page:

import { assertEquals, assertStrictEquals, assertThrows } from "@std/assert";

Follow the page as you add and revise the runnable examples below that import.

a star after function

Deno.test("a star after function", () => {
  function* declared(): Generator<string> {
    yield "declaration";
  }

  const expressed = function* (): Generator<string> {
    yield "expression";
  };

  const holder = {
    *method(): Generator<string> {
      yield "object literal method";
    },
  };

  class Container {
    *method(): Generator<string> {
      yield "class method";
    }
  }

  assertEquals([...declared()], ["declaration"]);
  assertEquals([...expressed()], ["expression"]);
  assertEquals([...holder.method()], ["object literal method"]);
  assertEquals([...new Container().method()], ["class method"]);

  assertThrows(
    () => new Function("const gen = *() => 1;"),
    SyntaxError,
    "Unexpected token '*'",
  );
});

Check programs/generators.test.ts
running 1 test from ./programs/generators.test.ts
a star after function ... ok (628µs)

ok | 1 passed | 0 failed (1ms)

Four places the star can go: a declaration, a function expression, an object literal method, and a class method. That is all of them, and in particular there is no generator arrow function: the syntax does not exist, and trying it is a parse error rather than a subtle failure, observed the usual way, by compiling a string, from the functions page's a function built at run time. The return type is Generator<T>, where T is the type of the values it yields; two more type parameters exist, and later steps get to them.

calling one runs nothing

declared sets started on its first line. Call it, and predict the flag:

Deno.test("calling one runs nothing", () => {
  let started = false;

  function* declared(): Generator<string> {
    started = true;
    yield "declaration";
  }

  const walker = declared();
  assertStrictEquals(started, true);
  assertEquals([...walker], ["declaration"]);
});

Check programs/generators.test.ts
running 2 tests from ./programs/generators.test.ts
...
calling one runs nothing ... FAILED (8ms)

 ERRORS

calling one runs nothing => ./programs/generators.test.ts:37:6
error: AssertionError: Values are not strictly equal.

    [Diff] Actual / Expected

-   false
+   true

 FAILURES

calling one runs nothing => ./programs/generators.test.ts:37:6

FAILED | 1 passed | 1 failed (10ms)

error: Test failed

The body never ran. Calling a generator does not execute anything: it builds an iterator and hands it back, and the body runs only when something asks for a value, which is what the spread eventually does. Correct the prediction and pin what came back:

Deno.test("calling one runs nothing", () => {
  let started = false;

  function* declared(): Generator<string> {
    started = true;
    yield "declaration";
  }

  const walker = declared();
  assertStrictEquals(started, false);

  assertEquals([...walker], ["declaration"]);
  assertStrictEquals(started, true);

  const another = declared();
  assertStrictEquals(another[Symbol.iterator](), another);
  assertEquals(another.map((s) => s.toUpperCase()).toArray(), ["DECLARATION"]);
});

calling one runs nothing ... ok (97µs)

What comes back is an iterable iterator: it has next, and it returns itself from Symbol.iterator, which is why for-of, spread, Array.from, and destructuring all work on it directly, the protocol from the iterables and iterators page. It also inherits every helper method, so .map and .toArray are available on it without an adapter, which the last line uses.

pausing, traced

Deno.test("pausing, traced", () => {
  let location = 0;

  function* traced(): Generator<string> {
    location = 1;
    yield "first";
    location = 2;
    yield "second";
    location = 3;
  }

  const walker = traced();
  assertStrictEquals(location, 0);

  assertEquals(walker.next(), { value: "first", done: false });
  assertStrictEquals(location, 1);

  assertEquals(walker.next(), { value: "second", done: false });
  assertStrictEquals(location, 2);

  assertEquals(walker.next(), { value: undefined, done: true });
  assertStrictEquals(location, 3);
});

pausing, traced ... ok (43µs)

Read the four assertions on location in order.

1. After traced() the body has not started, so location is still 0, which is the previous step again.
2. The first .next() runs the body as far as the first yield and stops there, leaving location at 1 and handing out "first".
3. The second .next() resumes after that yield, runs to the next one, and leaves location at 2.
4. The last .next() resumes again, runs off the end of the body, sets location to 3, and reports done: true with no value.

yield is like return in that it leaves the function with a value. It is unlike return in that the function is still there afterwards, waiting.

yield* delegates, and forgetting it is silent

forgot calls inner() where outer writes yield* inner(). Predict both spreads:

Deno.test("yield* delegates, and forgetting it is silent", () => {
  function* inner(): Generator<string> {
    yield "b";
    yield "c";
  }

  function* outer(): Generator<string> {
    yield "a";
    yield* inner();
    yield "d";
  }

  function* forgot(): Generator<string> {
    yield "a";
    inner();
    yield "d";
  }

  assertEquals([...outer()], ["a", "b", "c", "d"]);
  assertEquals([...forgot()], ["a", "b", "c", "d"]);
});

Check programs/generators.test.ts
running 4 tests from ./programs/generators.test.ts
...
yield* delegates, and forgetting it is silent ... FAILED (8ms)

 ERRORS

yield* delegates, and forgetting it is silent => ./programs/generators.test.ts:82:6
error: AssertionError: Values are not equal.

    [Diff] Actual / Expected

    [
      "a",
+     "b",
+     "c",
      "d",
    ]

 FAILURES

yield* delegates, and forgetting it is silent => ./programs/generators.test.ts:82:6

FAILED | 3 passed | 1 failed (10ms)

error: Test failed

Two values are missing and nothing said so. yield* inner() yields every value inner produces, one at a time, as though they had been written out in place. Without the star you have called a generator and thrown away the iterator it returned, which, by the previous step, ran nothing and yielded nothing: no error, no warning, two values gone. The code in forgot looks reasonable, which is what makes it worth staring at. Correct the prediction, and note that the right side of yield* can be any iterable:

Deno.test("yield* delegates, and forgetting it is silent", () => {
  function* inner(): Generator<string> {
    yield "b";
    yield "c";
  }

  function* outer(): Generator<string> {
    yield "a";
    yield* inner();
    yield "d";
  }

  function* forgot(): Generator<string> {
    yield "a";
    inner();
    yield "d";
  }

  assertEquals([...outer()], ["a", "b", "c", "d"]);
  assertEquals([...forgot()], ["a", "d"]);

  function* mixed(): Generator<string> {
    yield* ["a", "b"];
    yield* new Set(["c"]);
    yield* "de";
  }

  assertEquals([...mixed()], ["a", "b", "c", "d", "e"]);
});

yield* delegates, and forgetting it is silent ... ok (61µs)

mixed delegates to an array, a Set, and a string, which arrives by code point, from the text and characters page.

recursion is the payoff

Deno.test("recursion is the payoff", () => {
  class Tree {
    constructor(readonly value: string, readonly children: Tree[] = []) {}

    *[Symbol.iterator](): Generator<string> {
      yield this.value;
      for (const child of this.children) yield* child;
    }
  }

  const tree = new Tree("a", [
    new Tree("b", [new Tree("c"), new Tree("d")]),
    new Tree("e"),
  ]);

  assertEquals([...tree], ["a", "b", "c", "d", "e"]);
  assertEquals(Iterator.from(tree).take(2).toArray(), ["a", "b"]);
});

recursion is the payoff ... ok (75µs)

Two lines of body give a complete external iterator over a recursive structure, in the order a reader would guess. yield* child works because a Tree is iterable, so delegation and the iteration protocol compose without an adapter between them. Then look at the second assertion: the traversal stops after two nodes, and the rest of the tree is never visited. Writing the same thing by hand means maintaining an explicit stack, and writing it with a callback means giving up the ability to stop, a trade the pull-against-push step at the end returns to.

composition stays lazy

Deno.test("composition stays lazy", () => {
  const work: string[] = [];

  function* lines(): Generator<string> {
    for (const line of ["alpha", "beta"]) {
      work.push(`read ${line}`);
      yield line;
    }
  }

  function* numbered(source: Iterable<string>): Generator<string> {
    let n = 1;
    for (const line of source) {
      work.push(`number ${n}`);
      yield `${n++}: ${line}`;
    }
  }

  const walker = numbered(lines());
  assertEquals(work, []);

  assertStrictEquals(walker.next().value, "1: alpha");
  assertEquals(work, ["read alpha", "number 1"]);

  assertStrictEquals(walker.next().value, "2: beta");
  assertEquals(work, ["read alpha", "number 1", "read beta", "number 2"]);
});

composition stays lazy ... ok (54µs)

work records every step both generators take, and the log is the argument. Building the pair ran nothing, by the calling-runs-nothing rule. The first result arrived after one line had been read, not after all of them, and the two generators interleaved rather than one finishing before the other began. That is why pausing matters rather than being a curiosity: a generator that reads and a generator that transforms compose into a pipeline that costs one element of work per element of output, and each stage is an ordinary loop you can read. Generators compose with each other exactly as they compose with the helpers, the laziness the iterator helpers page measured in nothing happens until something pulls.

a return value is not an iteration value

counting both yields and returns. Predict the spread:

Deno.test("a return value is not an iteration value", () => {
  function* counting(): Generator<string, string> {
    yield "x";
    return "total: 1";
  }

  const walker = counting();
  assertEquals(walker.next(), { value: "x", done: false });
  assertEquals(walker.next(), { value: "total: 1", done: true });

  assertEquals([...counting()], ["x", "total: 1"]);
});

Check programs/generators.test.ts
running 7 tests from ./programs/generators.test.ts
...
a return value is not an iteration value ... FAILED (7ms)

 ERRORS

a return value is not an iteration value => ./programs/generators.test.ts:162:6
error: AssertionError: Values are not equal.

    [Diff] Actual / Expected

    [
      "x",
+     "total: 1",
    ]

 FAILURES

a return value is not an iteration value => ./programs/generators.test.ts:162:6

FAILED | 6 passed | 1 failed (10ms)

error: Test failed

The returned value is not in the sequence. It arrives on the result that says done: true, as the two .next() calls show, and every consumer that iterates for values throws that result away: spread gets ["x"], toArray gets ["x"], and for-of never sees the total either. This is the second type parameter, since Generator<string, string> yields strings and returns a string, and it is the mechanism behind the sentinels page's iteration already made this decision, where a value rides along on done: true and is easy to miss for exactly this reason. There is one convenient way to read it:

Deno.test("a return value is not an iteration value", () => {
  function* counting(): Generator<string, string> {
    yield "x";
    return "total: 1";
  }

  const walker = counting();
  assertEquals(walker.next(), { value: "x", done: false });
  assertEquals(walker.next(), { value: "total: 1", done: true });

  assertEquals([...counting()], ["x"]);
  assertEquals(counting().toArray(), ["x"]);

  function* wrapping(): Generator<string> {
    const summary = yield* counting();
    yield `saw ${summary}`;
  }

  assertEquals([...wrapping()], ["x", "saw total: 1"]);
});

a return value is not an iteration value ... ok (36µs)

yield* yields everything the inner generator yields, and then evaluates to what it returned. So a return in a generator is best understood as a message to a delegating caller rather than as part of the sequence. If you want a total, either yield it as the last value or accept that only yield* will collect it.

next(value) sends a value in

Deno.test("next(value) sends a value in", () => {
  function* asking(): Generator<string, void, string> {
    const name = yield "name?";
    const colour = yield `hello ${name}, colour?`;
    yield `${name} likes ${colour}`;
  }

  const walker = asking();

  assertStrictEquals(walker.next().value, "name?");
  assertStrictEquals(walker.next("Ada").value, "hello Ada, colour?");
  assertStrictEquals(walker.next("green").value, "Ada likes green");
});

next(value) sends a value in ... ok (21µs)

yield is an expression, and the argument to the next .next() becomes its value, so data flows both ways through the same keyword: out on the yield, back in on the .next() that resumes it. That is the third type parameter, since Generator<string, void, string> yields strings, returns nothing, and is sent strings. Be warned that this reads confusingly, and not because you are missing something: the argument and the value it becomes are in different statements, and the first .next() argument is discarded because no yield is suspended yet to receive it. Treat two-way generators as a specialist tool, and reach for an ordinary parameter or a callback first.

return and throw are how a generator is stopped

Deno.test("return and throw are how a generator is stopped", () => {
  const events: string[] = [];

  function* guarded(): Generator<number> {
    try {
      yield 1;
      yield 2;
    } finally {
      events.push("cleaned");
    }
  }

  const stopped = guarded();
  assertStrictEquals(stopped.next().value, 1);
  assertEquals(stopped.return(99), { value: 99, done: true });
  assertEquals(events, ["cleaned"]);
  assertEquals(stopped.next(), { value: undefined, done: true });

  events.length = 0;
  const thrown = guarded();
  thrown.next();
  assertThrows(() => thrown.throw(new Error("boom")), Error, "boom");
  assertEquals(events, ["cleaned"]);

  events.length = 0;
  for (const _ of guarded()) break;
  assertEquals(events, ["cleaned"]);
});

return and throw are how a generator is stopped ... ok (70µs)

An iterator can have two more methods beside next, and a generator implements both. .return(value) finishes the generator at the point where it is suspended, as though a return statement stood there, and hands back that value with done: true; .throw(error) behaves as though a throw stood there instead. The important part is the finally in both cases: a generator suspended inside a try is not abandoned when it is stopped, because the finally runs first, so cleanup written where the resource was acquired still happens, the rule from the errors and exceptions page's finally runs on every way out holding even for a function that is paused. And the last three lines are the mechanism behind a fact the iterables and iterators page demonstrated in leaving early closes the iterator: a for-of that leaves early calls .return() for you, which is why try/finally around a yield is the right place for cleanup, and why it works with break, an early return, and an exception alike.

yield is only a keyword inside a generator body

You cannot yield from inside a callback, and the refusal starts before the code can even run. Put the tidy-looking version in a scratch file programs/yield-in-callback.ts:

function* g(): Generator<number> {
  [1].forEach((n) => {
    yield n;
  });
}

error: SyntaxError: Expected ';', '}' or <eof>
  |
3 |     yield n;
  |           ~
    at file:///programs/yield-in-callback.ts:3:11

The message is strange until you see what the parser did. The arrow passed to forEach is a separate function and not a generator, so yield is not a keyword there at all: the parser read it as an ordinary identifier and then had no idea what n was doing after it, which is exactly what the squiggle under n says. It is not refusing permission; the word does not exist where you wrote it. That rules out forEach, map, and every other callback-taking method, and it is a real difference between forEach and for-of on top of the one the loops page gives in .forEach() cannot stop, because yield inside a for-of loop in the generator's own body is ordinary and correct: the loop is part of the body. Delete the scratch file, and pin how the same mistake reports at run time:

Deno.test("yield is only a keyword inside a generator body", () => {
  assertThrows(
    () =>
      new Function(
        "'use strict'; function* g() { [1].forEach((n) => { yield n; }); }",
      ),
    SyntaxError,
    "Unexpected strict mode reserved word",
  );

  assertEquals(
    typeof new Function("var yield = 1; return yield;"),
    "function",
  );
});

yield is only a keyword inside a generator body ... ok (45µs)

Module code is strict, and in strict mode yield is at least a reserved word, so the compiled string names that as the problem. In sloppy code it is not even reserved, as the second probe shows by using it as a variable name without complaint, which nobody writes on purpose and which explains the first message: the parser was looking for an identifier or a reserved word, not a keyword, and that is exactly what it says it found.

pull against push

Deno.test("pull against push", () => {
  class Rows {
    #rows = ["a", "b", "c"];

    forEachRow(callback: (row: string) => void): void {
      this.#rows.forEach(callback);
    }

    *rows(): Generator<string> {
      yield* this.#rows;
    }
  }

  const source = new Rows();

  const pushed: string[] = [];
  source.forEachRow((row) => pushed.push(row));
  assertEquals(pushed, ["a", "b", "c"]);

  const pulled: string[] = [];
  for (const row of source.rows()) {
    pulled.push(row);
    break;
  }
  assertEquals(pulled, ["a"]);

  assertEquals(
    source.rows().map((row) => row.toUpperCase()).take(2).toArray(),
    ["A", "B"],
  );
});

pull against push ... ok (55µs)

The same traversal offered two ways, and the difference is who is in control. forEachRow pushes: it owns the loop, every caller gets every row, stopping early means throwing an exception through somebody else's loop, and combining two such traversals means nesting callbacks. rows lets the caller pull: stopping early is break, transforming is a helper, and holding the traversal for later is just holding the iterator. This is the design lesson of the whole feature. A callback-taking method commits every future caller to your loop, while the same code written as a generator commits nobody, costs no more to write, and reads the same. When you are exposing a traversal, expose a generator.

batches of anything iterable

Deno.test("batches of anything iterable", () => {
  function* batches<T>(values: Iterable<T>, size: number): Generator<T[]> {
    let batch: T[] = [];
    for (const value of values) {
      batch.push(value);
      if (batch.length === size) {
        yield batch;
        batch = [];
      }
    }
    if (batch.length > 0) yield batch;
  }

  assertEquals([...batches(["a", "b", "c", "d", "e"], 2)], [
    ["a", "b"],
    ["c", "d"],
    ["e"],
  ]);
  assertEquals([...batches(new Set([1, 2, 3]), 3)], [[1, 2, 3]]);

  function* naturals(): Generator<number> {
    let n = 1;
    while (true) yield n++;
  }

  assertEquals(batches(naturals(), 3).take(2).toArray(), [
    [1, 2, 3],
    [4, 5, 6],
  ]);
});

batches of anything iterable ... ok (87µs)

Ten lines, and it batches anything iterable: an array, a Set, a file's lines, another generator, endlessly many values without holding them, as the naturals pull shows, where two batches cost six pulls and the endless rest was never asked for. The short last batch is the detail to keep. That ratio of work to result is why the feature is worth the syntax.

In practice

• Use a generator when the alternative is duplicating a traversal, exposing callback control flow, or maintaining a manual stack or index.
• Produce values with a generator and transform them with iterator helpers.
• Use yield* for recursion and delegation; a missing star fails silently.
• Put cleanup in a finally around yield so consumers that leave early still release resources.
• Give a reusable container a generator method that creates a fresh sequence for every iteration.
• Leave direct calls to next(value), .return(), and .throw() for the rare cases that need the underlying machinery.

Related

• Iterator helpers covers lazy transformations.
• Iterables and iterators explains why a generator method is a factory rather than a one-shot sequence.