name : api-documentation.md # Node.js API Documentation Tooling The Node.js API documentation is generated by an in-house tooling that resides within the [tools/doc](https://github.com/nodejs/node/tree/main/tools/doc) directory. The build process (using `make doc`) uses this tooling to parse the markdown files in [doc/api](https://github.com/nodejs/node/tree/main/doc/api) and generate the following: 1. Human-readable HTML in `out/doc/api/*.html` 2. A JSON representation in `out/doc/api/*.json` These artifacts are published to nodejs.org for multiple versions of Node.js. As an example the latest version of the human-readable HTML is published to [nodejs.org/en/doc](https://nodejs.org/en/docs/), and the latest version of the json documentation is published to [nodejs.org/api/all.json](https://nodejs.org/api/all.json) The artifacts are built as part of release builds by running the [doc-upload](https://github.com/nodejs/node/blob/1a83ad6a693f851199608ae957ac5d4f76871485/Makefile#L1218-L1224) Makefile target as part of the release-sources part of the iojs+release job. This target runs the `doc` target to build the docs and then uses `scp` to copy them onto the staging/www server into a directory of the form `/home/staging/nodejs/<type>/<full_version>/docs` where <type> is e.g. release, nightly, etc. The promotion step (either automatic for nightlies or manual for releases) then moves the docs to `/home/dist/nodejs/docs/\<full\_version>` where they are served by node.org. **The key things to know about the tooling include:** 1. The entry-point is `tools/doc/generate.js`. 2. The tooling supports the CLI arguments listed in the table below. 3. The tooling processes one file at a time. 4. The tooling uses a set of dependencies as described in the dependencies section. 5. The tooling parses the input files and does several transformations to the AST (Abstract Syntax Tree). 6. The tooling generates a JSON output that contains the metadata and content of the Markdown file. 7. The tooling generates a HTML output that contains a human-readable and ready to-view version of the file. This documentation serves the purpose of explaining the existing tooling processes, to allow easier maintenance and evolution of the tooling. It is not meant to be a guide on how to write documentation for Node.js. #### Vocabulary & Good to Know's * AST means "Abstract Syntax Tree" and it is a data structure that represents the structure of a certain data format. In our case, the AST is a "graph" representation of the contents of the Markdown file. * MDN means [Mozilla Developer Network](https://developer.mozilla.org/en-US/) and it is a website that contains documentation for web technologies. We use it as a reference for the structure of the documentation. * The [Stability Index](https://nodejs.org/dist/latest/docs/api/documentation.html#stability-index) is used to community the Stability of a given Node.js module. The Stability levels include: * Stability 0: Deprecated. (This module is Deprecated) * Stability 1: Experimental. (This module is Experimental) * Stability 2: Stable. (This module is Stable) * Stability 3: Legacy. (This module is Legacy) * Within Remark YAML snippets `<!-- something -->` are considered HTML nodes, that's because YAML isn't valid Markdown content. (Doesn't abide by the Markdown spec) * "New Tooling" references to the (written from-scratch) API build tooling introduced in `nodejs/nodejs.dev` that might replace the current one from `nodejs/node` ## CLI Arguments The tooling requires a `filename` argument and supports extra arguments (some also required) as shown below: | Argument | Description | Required | Example | | --------------------- | -------------------------------------------------------------------------------------------------------------------------------------- | -------- | ---------------------------------- | | `--node-version=` | The version of Node.js that is being documented. It defaults to `process.version` which is supplied by Node.js itself | No | v19.0.0 | | `--output-directory=` | The directory where the output files will be generated. | Yes | `./out/api/` | | `--apilinks=` | This file is used as an index to specify the source file for each module | No | `./out/doc/api/apilinks.json` | | `--versions-file=` | This file is used to specify an index of all previous versions of Node.js. It is used for the Version Navigation on the API docs page. | No | `./out/previous-doc-versions.json` | **Note:** both of the `apilinks` and `versions-file` parameters are generated by the Node.js build process (Makefile). And they're files containing a JSON object. ### Basic Usage ```bash # cd tools/doc npm run node-doc-generator ${filename} ``` **OR** ```bash # nodejs/node root directory make doc ``` ## Dependencies and how the Tooling works internally The API tooling uses an-AST-alike library called [unified](https://github.com/unifiedjs/unified) for processing the Input file as a Graph that supports easy modification and update of its nodes. In addition to `unified` we also use [Remark](https://github.com/remarkjs/remark) for manipulating the Markdown part, and [Rehype](https://github.com/rehypejs/rehype)to help convert to and from Markdown. ### What are the steps of the internal tooling? The tooling uses `unified` pipe-alike engine to pipe each part of the process. (The description below is a simplified version) * Starting from reading the Frontmatter section of the Markdown file with [remark-frontmatter](https://www.npmjs.com/package/remark-frontmatter). * Then the tooling goes to parse the Markdown by using `remark-parse` and adds support to [GitHub Flavoured Markdown](https://github.github.com/gfm/). * The tooling proceeds by parsing some of the Markdown nodes and transforming them to HTML. * The tooling proceeds to generate the JSON output of the file. * Finally it does its final node transformations and generates a stringified HTML. * It then stores the output to a JSON file and adds extra styling to the HTML and then stores the HTML file. ### What each file is responsible for? The files listed below are the ones referenced and actually used during the build process of the API docs as we see on <https://nodejs.org/api>. The remaining files from the directory might be used by other steps of the Node.js Makefile or might even be deprecated/remnant of old processes and might need to be revisited/removed. * **`html.mjs`**: Responsible for transforming nodes by decorating them with visual artifacts for the HTML pages; * For example, transforming man or JS doc references to links correctly referring to respective External documentation. * **`json.mjs`**: Responsible for generating the JSON output of the file; * It is mostly responsible for going through the whole Markdown file and generating a JSON object that represent the Metadata of a specific Module. * For example, for the FS module, it will generate an object with all its methods, events, classes and use several regular expressions (ReGeX) for extracting the information needed. * **`generate.mjs`**: Main entry-point of doc generation for a specific file. It does e2e processing of a documentation file; * **`allhtml.mjs`**: A script executed after all files are generated to create a single "all" page containing all the HTML documentation; * **`alljson.mjs`**: A script executed after all files are generated to create a single "all" page containing all the JSON entries; * **`markdown.mjs`**: Contains utility to replace Markdown links to work with the <https://nodejs.org/api/> website. * **`common.mjs`**: Contains a few utility functions that are used by the other files. * **`type-parser.mjs`**: Used to replace "type references" (e.g. "String", or "Buffer") to the correct Internal/External documentation pages (i.e. MDN or other Node.js documentation pages). **Note:** It is important to mention that other files not mentioned here might be used during the process but are not relevant to the generation of the API docs themselves. You will notice that a lot of the logic within the build process is **specific** to the current <https://nodejs.org/api/> infrastructure. Just as adding some JavaScript snippets, styles, transforming certain Markdown elements into HTML, and adding certain HTML classes or such things. **Note:** Regarding the previous **Note** it is important to mention that we're currently working on an API tooling that is generic and independent of the current Nodejs.org Infrastructure. [The new tooling that is functional is available at the nodejs.dev repository](https://github.com/nodejs/nodejs.dev/blob/main/scripts/syncApiDocs.js) and uses plain ReGeX (No AST) and [MDX](https://mdxjs.com/). ## The Build Process The build process that happens on `generate.mjs` follows the steps below: * Links within the Markdown are replaced directly within the source Markdown (AST) (`markdown.replaceLinks`) * This happens within `markdown.mjs` and basically it adds suffixes or modifies link references within the Markdown * This is necessary for the `https://nodejs.org` infrastructure as all pages are suffixed with `.html` * Text (and some YAML) Nodes are transformed/modified through `html.preprocessText` * JSON output is generated through `json.jsonAPI` * The title of the page is inferred through `html.firstHeader` * Nodes are transformed into HTML Elements through `html.preprocessElements` * The HTML Table of Contents (ToC) is generated through `html.buildToc` ### `html.mjs` This file is responsible for doing node AST transformations that either update Markdown nodes to decorate them with more data or transform them into HTML Nodes that attain a certain visual responsibility; For example, to generate the "Added at" label, or the Source Links or the Stability Index, or the History table. **Note:** Methods not listed below are either not relevant or utility methods for string/array/object manipulation (e.g.: are used by the other methods mentioned below). #### `preprocessText` **New Tooling:** Most of the features within this method are available within the new tooling. This method does two things: * Replaces the Source Link YAML entry `<-- source_link= -->` into a "Source Link" HTML anchor element. * Replaces type references within the Markdown (text) (i.e.: "String", "Buffer") into the correct HTML anchor element that links to the correct documentation page. * The original node then gets mutated from text to HTML. * It also updates references to Linux "MAN" pages to Web versions of them. #### `firstHeader` **New Tooling:** All features within this method are available within the new Tooling. Is used to attempt to extract the first heading of the page (recursively) to define the "title" of the page. **Note:** As all API Markdown files start with a Heading, this could possibly be improved to a reduced complexity. #### `preprocessElements` **New Tooling:** All features within this method are available within the new tooling. This method is responsible for doing multiple transformations within the AST Nodes, in majority, transforming the source node in respective HTML elements with diverse responsibilities, such as: * Updating Markdown `code` blocks by adding Language highlighting * It also adds the "CJS"/"MJS" switch to Nodes that are followed by their CJS/ESM equivalents. * Increasing the Heading level of each Heading * Parses YAML blocks and transforms them into HTML elements (See more at the `parseYAML` method) * Updates BlockQuotes that are prefixed by the "Stability" word into a Stability Index HTML element. #### `parseYAML` **New Tooling:** Most of the features within this method are available within the new tooling. This method is responsible for parsing the `<--YAML snippets -->` and transforming them into HTML elements. It follows a certain kind of "schema" that basically constitutes in the following options: | YAML Key | Description | Example | Example Result | Available on new tooling | | ------------- | ------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------- | --------------------------- | ------------------------ | | `added` | It's used to reference when a certain "module", "class" or "method" was added on Node.js | `added: v0.1.90` | `Added in: v0.1.90` | Yes | | `deprecated` | It's used to reference when a certain "module", "class" or "method" was deprecated on Node.js | `deprecated: v0.1.90` | `Deprecated since: v0.1.90` | Yes | | `removed` | It's used to reference when a certain "module", "class" or "method" was removed on Node.js | `removed: v0.1.90` | `Removed in: v0.1.90` | No | | `changes` | It's used to describe all the changes (historical ones) that happened within a certain "module", "class" or "method" in Node.js | `[{ version: v0.1.90, pr-url: '', description: '' }]` | -- | Yes | | `napiVersion` | It's used to describe in which version of the N-API this "module", "class" or "method" is available within Node.js | `napiVersion: 1` | `N-API version: 1` | Yes | **Note:** The `changes` field gets prepended with the `added`, `deprecated` and `removed` fields if they exist. The table only gets generated if a `changes` field exists. In the new tooling only "added" is prepended for now. #### `buildToc` **New Tooling:** This feature is natively available within the new tooling through MDX. This method generates the Table of Contents based on all the Headings of the Markdown file. #### `altDocs` **New Tooling:** All features within this method are available within the new tooling. This method generates a version picker for the current page to be shown in older versions of the API docs. ### `json.mjs` This file is responsible for generating a JSON object that (supposedly) is used for IDE-Intellisense or for indexing of all the "methods", "classes", "modules", "events", "constants" and "globals" available within a certain Markdown file. It attempts a best effort extraction of the data by using several regular expression patterns (ReGeX). **Note:** JSON output generation is currently not supported by the new tooling, but it is in the pipeline for development. #### `jsonAPI` This method traverses all the AST Nodes by iterating through each one of them and infers the kind of information each node contains through ReGeX. Then it mutate the data and appends it to the final JSON object. For a more in-depth information we recommend to refer to the `json.mjs` file as it contains a lot of comments.