Adapt Specification: Core Engine

Status: This document describes the current implementation. The running code on main wins if it differs. See the documentation contract and user manual.

1. File discovery

Adapt recursively scans the document root. It ignores hidden paths, .adapt, virtual environments, __pycache__, and node_modules. Discovery selects a candidate plugin from AdaptConfig.plugin_registry by file extension. It then calls Plugin.detect(). When this method returns True, Discovery loads the file.

flowchart TD A[Scan document root recursively] B{Path excluded?<br/>hidden, .adapt, venv,<br/>__pycache__, node_modules} C[Skip path] D{Extension in<br/>plugin registry?} E[Ignore file] F[Select candidate plugin] G["Run Plugin.detect(path)"] H{Detect returned true?} I[Do not load resource] J[Load resource with plugin] K[Create or refresh companion files] A --> B B -->|Yes| C B -->|No| D D -->|No| E D -->|Yes| F F --> G G --> H H -->|No| I H -->|Yes| J J --> K

The default registry contains these mappings:

Extensions Plugin
.csv CSV
.xlsx, .xls Excel
.parquet Parquet
.py Python handler
.html HTML
.md Markdown
.txt, .pdf, .json, .xml, .svg Generic file
.png, .jpg, .jpeg, .gif, .webp Generic file
.mp4, .mp3, .avi, .mkv, .webm, .ogg, .wav Media

The Excel plugin reads .xlsx and legacy .xls workbooks. Legacy .xls workbooks are read-only. Discovery ignores extensions that are not in the registry.

2. Dataset engine

The dataset engine handles CSV files, Excel sheets, and Parquet files. It provides row reads, query controls, and mutation envelopes for create, update, and delete actions. Legacy .xls resources reject mutations with 405. Row identifiers are one-based positions named _row_id.

Schema inference uses string, integer, number, and boolean labels for CSV and Excel samples. These labels control response conversion and UI columns. They also make sure that supplied create and update values are correct before a file changes.

Each Excel sheet has a sub_namespace. For example, the People sheet in staff.xlsx has these extensionless routes:

  • /api/staff/People/
  • /schema/staff/People/
  • /ui/staff/People/

Adapt also mounts the extension-qualified staff.xlsx/People namespace. A sheet uses these companion paths:

  • .adapt/staff.People.schema.json
  • .adapt/staff.People.index.html
  • .adapt/staff.People.options.json

The header_row option selects the one-based header row for CSV and Excel resources. An absent or invalid value uses row 1.

3. Schema and companion files

Dataset plugins derive column metadata from the file. During discovery, Adapt creates a missing schema file and DataTables UI file. Adapt reads an options file but does not create it.

Generated schema files contain a generated_by marker. Adapt can refresh a marked schema after the derived shape changes. When its content differs from the derived schema, Adapt preserves a hand-maintained schema without this marker.

Generated and hand-maintained schemas make sure that supplied create and update fields are correct. Adapt rejects unknown columns and values incompatible with the common string, integer, number, and boolean types with 422. It accepts and normalizes numeric and boolean strings for compatibility with the generated HTML form. The schema format permits blank strings and null because it does not specify required columns or nullability. Unknown custom types remain as metadata only. Adapt does not make sure that their values are correct.

{
  "type": "object",
  "name": "people",
  "primary_key": "_row_id",
  "columns": {
    "name": {"type": "string"},
    "age": {"type": "integer"}
  }
}

Dataset UI files use the *.index.html suffix. The media plugin is different. It writes JSON metadata to its assigned ui_path.

4. Writes and locks

The lock manager stores one unique lock record for each resource. Built-in dataset writes retry a held lock for up to 30 seconds. The retry delay starts at 0.1 seconds, doubles, and stops at 1 second.

flowchart TD A[Start write operation] --> B[Attempt lock acquisition] B --> C{Lock available?} C -->|Yes| D[Write through temp file path] C -->|No| E[Wait current delay] E --> F[Increase delay: 0.1s, 0.2s, 0.4s, 0.8s, then 1.0s cap] F --> G{Elapsed time > 30s?} G -->|No| B G -->|Yes| H[Raise lock timeout] H --> I[Shared write path returns 409]

Locks expire after five minutes by default. Startup deletes locks older than five minutes. The Admin UI can list, delete, and clean lock records.

CSV, XLSX, and Parquet writes use a temporary file. They replace the target with os.replace() where supported. An EXDEV error uses a copy fallback. These measures reduce conflict and partial-write risk. They do not remove all races or make a writer uninterruptible.

Legacy .xls workbooks do not use this write path. The Excel plugin marks each legacy sheet as read-only to prevent data loss from an incompatible writer.

The lock manager raises lock-specific exceptions for immediate conflicts and exhausted retries. The shared write method converts both exceptions to 409.

5. Cache

adapt/cache.py stores plugin cache entries in the SQLite cache table. The table contains key, value, expires_at, resource, and user columns.

Plugins cache selected values. These values include parsed dataset rows, schemas, rendered HTML or Markdown, and media metadata. Supported writes invalidate resource cache entries.

Adapt does not cache generic file bodies or streamed media bodies. Adapt does not apply one automatic cache wrapper to every GET response.