Using the CLI#

dscat reads a prebuilt catalogue of the dataset dictionaries. Run every command from the repository root, where data/ and .catalogue/ live, through uv:

uv run dscat <command> [options]

Every read command prints an aligned table by default and can serialise its results to JSON, CSV, TSV, or a Markdown table with --format. See Output formats.

Build the catalogue first#

The read commands need the index. Build or refresh it from data/:

uv run dscat ingest                # all datasets
uv run dscat ingest -d spark       # one dataset
uv run dscat ingest --convert-docs # also pre-convert every document to markdown

ingest is safe to re-run: it rebuilds the named datasets and leaves the rest intact. A single progress bar reports its progress, since scanning the data CSVs for row counts takes a moment. By default documents are converted to markdown lazily, on first dscat doc; pass --convert-docs to convert them all up front instead (slower, using the same per-format engines), which is handy before working offline.

The search-then-read workflow#

The catalogue holds metadata, not data values. The usual path is:

  1. dscat search "<intent>" finds candidate features by meaning or keyword.

  2. dscat feature <table.name> shows one feature’s full card, including its measurement scale, value codings, and the source CSV path.

  3. With the column name and file path from step 2, read just that column from the CSV with a streaming tool. Reading the whole file is rarely necessary.

Commands#

Command

What it does

ingest

Build or refresh the catalogue from data/.

datasets

List indexed datasets with version and feature counts.

versions

List versions per dataset, newest first.

tables

List tables: name, role(s), rows, columns, title.

describe <table>

Show a table’s stats and a page of its features.

search <query>

Full-text feature search with synonym expansion.

feature <key>

Show one feature’s full metadata card.

diff

Compare a dataset’s dictionary between two versions.

docs

List a version’s non-dictionary documentation files.

doc <name>

Convert a documentation file to markdown and show a section.

Run uv run dscat <command> --help for the full options of any command.

Scope flags#

Read commands default to the latest version of each dataset. Three flags change that scope:

Flag

Meaning

-d, --dataset NAME

Limit to one dataset (spark, ssc).

-v, --version VER

Pin a version; requires --dataset.

--all-versions

Consider every version, not only the latest.

Other common filters are -t/--table, -s/--scale, -r/--role (an SSC family role: proband, mother, father, sibling, mz_twin), -g/--grep, and -n/--limit.

Output formats#

Read commands print an aligned text table by default. Pick another format with -f/--format:

Format

Use

table

Aligned columns for reading in a terminal (the default).

json

A JSON array of records, for scripts and pipelines.

csv

Comma-separated values with a header row.

tsv

Tab-separated values with a header row.

markdown

A GitHub-flavoured Markdown table, for pasting into docs or an issue.

--json is kept as a shortcut for --format json. The table and Markdown views truncate long free-text cells for readability; the csv, tsv, and json formats carry the full values.

uv run dscat search "sleep problems" -d spark -f csv > sleep.csv
uv run dscat tables -d ssc -r proband -f markdown
uv run dscat datasets --json

The feature key#

feature accepts a bare variable name (q01_phrases), a table.name (scq.q01_phrases), or a qualified id. When a key matches more than one feature, the command lists the matches so you can narrow with --dataset or --table.

Search expressions#

search expands each query term through a synonym table, so sleep also matches insomnia and naps, and iq matches cognitive and Mullen. It then runs an FTS5 BM25 match and orders results best first. Pass --raw to send an FTS5 MATCH expression through unchanged, which lets you use OR, NEAR, prefix *, and column filters directly. See Synonyms for how the expansion works and how to extend it.

Converting documents#

docs lists the non-dictionary files shipped with a version (welcome packets, protocols, release notes). doc <name> converts one of them to Markdown, caches it under .catalogue/docs/, and prints either a preview or the windows of text matching a regex passed to -s/--section.

By default the engine is chosen by file type:

Format

Engine

Why

PDF

marker

Layout-aware extraction; higher quality, but slower (loads ML models on first use).

.docx, .txt, and the rest

markitdown

Fast, with broad format coverage.

.doc, .rtf

textutil

The macOS tool; the only one of the three that reads these.

Pass -e/--engine to force marker or markitdown for a given file (legacy .doc and .rtf always use textutil). markitdown and marker ship as dependencies, so uv sync installs them; textutil is part of macOS. Each engine caches to its own file, so you can convert the same document with two engines and compare.

uv run dscat doc "Welcome Packet" -d spark                 # PDF, so marker
uv run dscat doc "Welcome Packet" -d spark -e markitdown   # force the faster engine

Examples#

uv run dscat search "adaptive behaviour" -d ssc -s standard
uv run dscat feature vineland-3.composite_standard_score -d spark
uv run dscat tables -d ssc -r proband -g ados
uv run dscat diff -d spark --from 2025-03-31 --to 2026-03-23 --features -n 100
uv run dscat doc "Welcome Packet" -d spark -s "data access"