# Secret Sauce

> Technical architecture and design blueprint for the Hyperclay hosted platform, from the DOM-as-database save pipeline to routing, storage, and sync.

Source: https://hyperclay.com/docs/secret-sauce/

This is a technical blueprint for the Hyperclay hosted platform. Every architectural decision, every code pattern, and every trade-off is documented here. A developer could read this and rebuild the system from scratch.

The stack is intentionally boring: Node.js, Express, PostgreSQL with Sequelize, Edge.js templates, PM2 for process management, Stripe for payments, Postmark for email, and Server-Sent Events for real-time sync. The interesting part isn't the technology: it's how the pieces fit together.

## The core innovation: HTML files as a database

Traditional web applications shuttle data between a frontend, an API, and a database. Hyperclay collapses all three into one artifact: the malleable document. The DOM is the data model. In [ClayJS](https://clayjs.com/docs/), [saving](https://clayjs.com/docs/guide/saving/) means cloning the root element, synchronizing `[persist]` form state, running snapshot hooks (`onbeforesnapshot` and `clay.onSnapshot`), stripping [regions](https://clayjs.com/docs/regions/) marked `clay="editor-ui"` or `no-snapshot` along with browser extension debris, running save hooks (`onbeforesave` and `clay.addDocumentTransform`), converting mode attributes (`editmode:contenteditable`, `viewmode:disabled`) to their inert saved form, stripping `no-save` content and runtime root attributes (`editmode`, `pageowner`, `canedit`, `draftmode`, `unsaved`, `savestatus`, `savetoken`, `documentetag`), then writing `<!DOCTYPE html>` plus the prepared root to disk via `POST /_/save`. The next page load serves the modified file, so changes are permanent.

### The save pipeline

When a user saves, the client sends the full HTML string via `POST /_/save`. On the server, `saveHTML` runs through these steps:

1. **Format**: Formatting is opt-in. If the `<html>` element carries `formathtml="true"`, the raw HTML is passed through js-beautify with force-expanded multiline attributes and 2-space indentation (SVG, script, and style tags are left unformatted). Otherwise the bytes are stored exactly as sent.

2. **Admit**: The current file is read from disk. A stale `If-Match` answers `412`. Then the write is admitted: a save that declares a lower block editor format than the document's floor (`X-Clay-Blockeditor-Format` against `<meta name="clay-blockeditor-format">`), or that replaces a block editor document whose live session it does not descend from, answers `409`. Nothing is backed up or written for a refused save.

3. **Deduplicate**: The formatted HTML is compared to the current file. If the content hasn't changed, the save is skipped entirely, with no backup, no broadcast, and no wasted I/O.

4. **First-save safety net**: If this document has zero backups in the database (for example, it was created before the backup system existed), the *current* content is backed up before the new content overwrites it.

5. **Backup**: The formatted HTML is passed to `BackupService.createBackup`, which stores either a full snapshot or a unified diff inside a PostgreSQL transaction.

6. **Write**: The formatted HTML is written to `/sites/{username}/{folderpath}/{sitename}.html` via `dx('sites', ownerUsername, sitePath).createFileOverwrite()`.

7. **Tailwind compile**: If the HTML contains any Tailwind `<link>`, the Tailwind JIT compiler extracts classes and writes the compiled CSS to `/public-assets/tailwindcss/{username}/{folderpath}/{sitename}.css`.

8. **Broadcast**: The new content and a SHA-256 checksum are broadcast over SSE to any connected live-sync clients and the owner's disk-sync engine.

```js
// The actual save handler (simplified)
const formattedHtml = formatHtml(html);
const currentHtml = await dx('sites', ownerUsername, sitePath, node.name).getContents();

if (currentHtml === formattedHtml) return next(); // Skip if unchanged

await BackupService.createBackup(node.id, formattedHtml, userId);
await dx('sites', ownerUsername, sitePath).createFileOverwrite(node.name, formattedHtml);

if (hasAnyTailwindLink(formattedHtml)) {
  const css = await compileTailwind(formattedHtml);
  const baseName = node.name.replace(/\.(html|htmlclay)$/, '');
  await dx('public-assets/tailwindcss', ownerUsername, sitePath).createFileOverwrite(`${baseName}.css`, css);
}

liveSync.broadcastNodeSaved(ownerUsername, buildNodeSavedPayload(node, {
  content: formattedHtml
}));
```

## The three-dimensional state machine router

Every request to Hyperclay is classified along three axes: **who** is making the request, **what** they're accessing, and **what action** they're performing. These three dimensions form a routing key like `dev:regular_site:edit`, and the routing table maps keys to handler pipelines.

### User types

| Type | Description |
|------|-------------|
| `no_auth` | Not logged in |
| `app_user` | Logged-in user on a multi-tenant app (`account_type = 'app_user'`) |
| `dev` | Developer account (`account_type = 'dev'`), can create documents and has billing |
| `admin` | Superadmin (`isSuperadmin = true`), falls through to `dev` routes if no admin-specific route exists |

### Resource types

| Type | Description |
|------|-------------|
| `main_app` | The hyperclay.com application itself (dashboard, auth pages, billing) |
| `regular_site` | A standard user-created document |
| `multi_tenant` | A document with `enableSignups = true` |
| `instance` | A copy of a multi-tenant app owned by an `app_user` |
| `folder` | Dashboard folder navigation |
| `upload` | User-uploaded files |

### The eight-step state detection pipeline

Before any route handler executes, eight middleware functions build a complete picture of the request:

```bash
state__init         -> Initialize req.state with safe defaults
state__meta         -> Parse domain, method, path, body, query, subdomain
state__user         -> Load Person from auth_token cookie, detect user type
state__resource     -> Detect resource type, load Node from database, check ownership
state__shareAccess  -> If valid share token, elevate no_auth -> app_user with ownership
state__action       -> Map URL path to action name (e.g. /edit -> 'edit', /login GET -> 'login-page')
state__key          -> Build final key: '{userType}:{resourceType}:{action}'
state__authcookies  -> Set ownership and resource cookies for the client
```

After detection, `req.state` contains everything needed for routing:

```js
{
  meta: { fullDomain, rootDomain, method, path, subdomain, isCustomDomain, body, query },
  user: { person, isLoggedIn, isOwner, username, hasSubscription, siteLimit, canCreateSites },
  resource: { node, siteName, customDomain, sourceNode, instanceOwner, currentFolderId },
  userType: 'dev',
  resourceType: 'regular_site',
  action: 'edit',
  key: 'dev:regular_site:edit'
}
```

### Action detection

The `state__action` middleware maps URL paths to action names. The rules:

- **Root path** (`/`): `view`
- **First segment** is the action name: `/edit` maps to `edit`, `/save` maps to `save`, `/dashboard` maps to `dashboard`
- **GET on auth pages** gets a `-page` suffix: `GET /login` maps to `login-page`, `POST /login` maps to `login`. This lets the routing table serve the form on GET and process the submission on POST with the same URL.
- **Compound routes** have special mappings: `/edit/uploads/...` maps to `edit-upload`, `/save/uploads/...` maps to `save-upload`, `/live-sync/stream` maps to `live-sync-stream`. Upload saves via `POST /_/save` with a `Document-URL` header containing `/edit/uploads/...` are also detected as `save-upload`.
- **Versioning routes** remap: `/version/123` maps to `view-version`, `/restore/123` maps to `restore-version`, `/versions` maps to `backups`
- **Unknown paths on documents** fall through to `view`. If the resource is a document and the action isn't in the known actions list (`edit`, `save`, `login`, `signup`, etc.), it's treated as a `view`. This enables client-side routing: any path on a document renders the root HTML, and the client code can read `window.location.pathname`.

The remaining path segments are stored in `req.state.actionParams`, available to handlers. For `/version/123`, the action is `view-version` and `actionParams` is `['123']`.

### The expansion syntax

The `+` operator lets one route definition cover multiple combinations. This:

```js
'dev+app_user:regular_site+instance:save': [requireOwnership, saveHTML, respond]
```

Expands at startup into four separate routes:

```bash
dev:regular_site:save
dev:instance:save
app_user:regular_site:save
app_user:instance:save
```

### Wildcard matching

When the exact key has no match, the router tries fallbacks in order:

1. Exact match: `dev:regular_site:edit`
2. Admin fallback to dev: `dev:regular_site:edit` (if user is admin)
3. Wildcard user: `*:regular_site:edit`
4. Wildcard resource: `dev:*:edit`
5. Full wildcard: `*:*:edit`

If nothing matches, the request gets a 403.

### The state factory

Route handlers use a `state()` factory for inline condition checks:

```js
// Require ownership - returns 400 if false
'dev:regular_site:edit': [
  requireOwnership,
  serveCodeEditor
]

// Require condition with custom message
state(s => s.user.person?.hasActiveSubscription).require('Active subscription required')

// Redirect if condition fails
state(s => false).redirect('/_/dashboard', 'You already have an account.')
```

### Key routes

| Key | What it does |
|-----|-------------|
| `*:main_app:view` | Serve the homepage |
| `dev:main_app:dashboard` | Serve the dashboard (folder browser, document list) |
| `dev:main_app:new` | Create a new document |
| `*:regular_site+multi_tenant+instance:view` | Serve a document's HTML |
| `dev:regular_site+multi_tenant+instance:edit` | Serve the code editor (ownership required) |
| `dev+app_user:regular_site+multi_tenant+instance:save` | Save HTML (ownership required) |
| `*:multi_tenant:signup` | Sign up for a multi-tenant app, create instance |
| `dev+app_user:*:live-sync-stream` | SSE stream for real-time sync |
| `*:regular_site+multi_tenant+instance:data` | Data extraction API |

### The response layer

The routing table maps keys to handler pipelines, and each pipeline produces output through the `respond` module, which handles the browser-vs-AJAX split automatically.

| Function | What it does |
|----------|-------------|
| `respond.success(res, { msg, html })` | Sends JSON with `msgType: 'success'`. Used by save, create, and update operations. |
| `respond.error(res, error)` | Browser requests get an Edge.js error page. AJAX requests get JSON. Sequelize errors are auto-classified: `SequelizeUniqueConstraintError` maps to 409, `SequelizeValidationError` maps to 400. |
| `respond.redirect(res, url, params)` | Builds a URL with query parameters (`msg`, `msgType`, `return`) and issues a 302. |
| `respond.html(res, html)` | Sends raw HTML, prepending `<!DOCTYPE html>` if missing. |
| `respond.nodes({ res, person, folderPath, msg })` | Reloads the person via `getPersonWithNodes`, renders the `components/nodes` Edge.js partial, and returns JSON with the rendered HTML fragment. |

`respond.nodes` is how the legacy dashboard stays in sync: after creating, deleting, or moving a document, the client gets back the updated node list as a pre-rendered HTML string, with no second request needed.

The routing table also uses `serve('home')` (renders a named Edge.js template for browser requests) and `asyncHandler(fn)` (wraps async handlers with `try/catch`, routing uncaught errors to `respond.error`).

The request type is detected via `isBrowserRequest()`, which checks the `Accept` header. This means the same route can serve both a full HTML page (direct navigation) and a JSON response (AJAX call from the client).

### The db middleware factory

Route handlers often need to check database conditions before proceeding, such as whether an email already exists or a document name conflicts. The `db` object provides a Proxy-based API for expressing these checks declaratively inside the routing table:

```js
// Ensure no Person exists with this email (for signup)
db.person({ email: req => req.body.email }).exists(false)

// Ensure the node exists (404 if not)
db.node({ name: req => req.body.name }).exists(true)

// Update the current user's record
db.person().update({ username: req.body.username })

// Delete matching records
db.customDomain({ id: req => req.body.domainId }).destroy()
```

Each call returns a standard Express middleware function. The query parameter accepts static values or functions that receive `req`: values are resolved at execution time, not at startup.

`.exists(false)` returns 409 if a record is found. `.exists(true)` returns 404 if not found. `.update(data)` finds the record and updates it. `.destroy()` deletes matching records. `.create(data)` inserts a new record and stores it on `req.createdRecord` for later middleware.

Special handling for `Person` and `Node`: calling `db.person().update(...)` with no query defaults to `req.state.user.person`, and `db.node().update(...)` defaults to `req.state.resource.node`. These records were already loaded by the state pipeline, so no redundant lookup is needed.

The Proxy intercepts any property access, so `db.customDomain(...)`, `db.message(...)`, `db.siteBackups(...)`, and others all work for any registered Sequelize model.

## The hybrid storage system

Hyperclay uses PostgreSQL for metadata and the filesystem for content.

### Database models

| Model | Purpose |
|-------|---------|
| `Person` | User accounts: email, username, password (bcrypt), `stripeCustomerId`, `account_type` (`dev`/`app_user`), `isSuperadmin` |
| `Node` | Unified entity for documents, folders, and uploads: `name`, `type`, `parentId`, `path`, `enableSignups`, `shareToken`, `sourceNodeId` |
| `PersonNode` | Ownership junction table (many-to-many between `Person` and `Node`) |
| `SiteBackups` | Version history: `html`/`diffContent`, `isDiff`, `diffFromId`, `snapshotId`, `contentHash`, `backupNumber` |
| `CustomDomain` | Domain mappings: `domain`, `status` (`pending`/`verifying`/`active`/`failed`), `sslStatus`, `saasCustomDomainsId` |
| `LoggedInToken` | Auth tokens: `token` (secure random), `personId`, `expires` (30 days from creation) |
| `EmailConfirmToken` | Email verification tokens |
| `Message` | Contact form submissions |
| `Event` | Activity tracking (logins, saves, signups) |
| `ApiKey` | Sync API keys: `keyHash` (SHA-256), `keyPrefix` (`hcsk_`), `expiresAt`, `isActive` |

The `Node` model is the unifying abstraction. Documents, folders, and uploads are all Nodes with a `type` field (`'site'`, `'folder'`, or `'upload'`). Folders form a tree via `parentId`. Instances link to their source via `sourceNodeId`. This lets the workspace and dashboard load documents, folders, and uploads in a single query via the `ownerPersonId`-keyed `OwnedNodes` association.

### Filesystem layout

```bash
/sites/{username}/{folderpath}/{sitename}.html                  # Live document files
/sites-versions/{username}/{folderpath}/{base}/{timestamp}.html # Timestamped backup files (e.g. on instance creation)
/sites-deleted/{sitename}.html                                  # Soft-deleted documents
/uploads/{username}/{folderpath}/{filename}                     # User-uploaded files
/public-assets/tailwindcss/{username}/{folderpath}/{name}.css   # Compiled Tailwind CSS per document
```

### The dx.js file utility

All filesystem operations go through `dx`, a chainable async API that absorbs errors instead of throwing:

```js
// Read - returns null if file doesn't exist
const html = await dx('sites', 'mysite.html').getContents();

// Write - creates directories automatically
await dx('sites').createFileOverwrite('mysite.html', html);

// Check existence
const exists = await dx('sites', 'mysite.html').exists();

// Copy with automatic directory creation
await dx('sites-deleted', 'mysite.html').copyFileFrom('sites', 'mysite.html');

// JSON operations
await dx('config.json').appendJSON({ username: 'alice' });
const value = await dx('config.json').getKey('username');

// Chainable - copy then rename
await dx('backups', 'site.html')
  .copyFileFrom('sites', 'site.html')
  .renameTo('site-backup.html');

// Null safety - returns null instead of crashing
await dx('users', undefined).append('data'); // returns null
```

The chainable mechanism uses a Proxy that wraps each async operation, passing the resolved path from one step to the next. Native array methods like `.length`, `.reverse()`, and `[0]` work on the resolved values.

## The diff-based backup system

Every save creates a backup in [version history](https://hyperclay.com/docs/version-history/). Every 20th backup is a full HTML snapshot stored in the `SiteBackups` table. The 19 saves between snapshots are stored as unified diffs (via the `diff` npm package) against the previous backup.

### How it works

```js
// Inside BackupService.createBackup, within a transaction:
const backupNumber = lastBackupNumber + 1;
const isSnapshot = backupNumber % 20 === 0 || backupNumber === 1;

if (isSnapshot) {
  // Store full HTML
  await SiteBackups.create({
    nodeId, html, contentHash, isDiff: false, backupNumber
  });
} else {
  // Get previous backup and create diff
  const previousHtml = await lastBackup.getFullHtml();
  const diff = createPatch(String(lastBackup.id), previousHtml, html);

  await SiteBackups.create({
    nodeId, diffContent: diff, diffFromId: lastBackup.id,
    contentHash, isDiff: true, snapshotId, backupNumber
  });
}
```

### Reconstruction

To reconstruct any backup's full HTML, `getFullHtml()` does:

1. If the backup is a snapshot (`isDiff: false`), return `html` directly
2. Otherwise, load the nearest snapshot (tracked by `snapshotId`)
3. Load all diffs between the snapshot and this backup, ordered by ID
4. Validate chain continuity: each diff's `diffFromId` must equal the previous backup's ID
5. Apply diffs sequentially using `applyPatch`
6. Verify the final SHA-256 hash matches `contentHash`

### Concurrency safety

The entire `createBackup` call runs inside a PostgreSQL transaction that takes a row-level lock on the Node:

```js
await Node.findByPk(nodeId, {
  transaction,
  lock: transaction.LOCK.UPDATE
});
```

This serializes concurrent saves for the same document, preventing two backups from both reading the same "last backup" and creating conflicting diffs.

## Multi-tenant architecture

Any document becomes a multi-tenant [sign-up app](https://hyperclay.com/docs/sign-up-apps/) by setting `enableSignups = true` on its Node record. The state machine detects this and classifies the resource as `multi_tenant` instead of `regular_site`.

### Instance creation flow

When a user signs up on a multi-tenant app:

1. **Create account**: A `Person` record with `account_type: 'app_user'` and a bcrypt-hashed password
2. **Generate name**: The instance gets named `{appname}.html`. If taken, it becomes `{appname}-1.html`, `{appname}-2.html`, and so on.
3. **Create Node**: A new `Node` with `type: 'site'`, `ownerPersonId` set to the new `app_user`, and `sourceNodeId` pointing to the source app
4. **Copy HTML**: The source document's HTML file is copied to `/sites/{username}/{instancename}`
5. **Create backup**: An initial timestamped backup is created
6. **Send confirmation**: An email with a confirmation link is sent

```js
const instanceName = await generateInstanceName(baseName, person);

const instanceNode = await Node.create({
  name: instanceName,
  type: 'site',
  sourceNodeId: sourceNode.id,
  parentId: 'root',
  ownerPersonId: person.id
});

const sourceHtml = await dx('sites', sourceOwner.username, sourcePath, sourceNode.name).getContents();
await dx('sites', person.username).createFileOverwrite(instanceNode.name, sourceHtml || '');
```

### Instance URLs

The subdomain is the owner's username, and the instance filename is a path segment:

- Hyperclay subdomain: `https://{username}.hyperclay.com/{instance.name}` (for example, `https://alice.hyperclay.com/myapp.html`)
- Custom domain: `{username}.myapp.com/{instance.name}` (requires wildcard domain `*.myapp.com`)

Each instance evolves independently from the source. Editing the source app doesn't change existing instances.

### Wildcard domain requirements

If a multi-tenant app uses custom domains, enabling signups requires a wildcard domain (`*.myapp.com`) so that instance subdomains can be routed. Without custom domains, instances use Hyperclay subdomains by default.

## Authentication and access control

### Token-based auth

Login creates a `LoggedInToken` record with a cryptographically secure random ID and a 30-day expiry:

```js
// LoggedInToken model
{
  token: idSecure(),        // Secure random string
  personId: person.id,
  expires: Date.now() + (30 * 24 * 60 * 60 * 1000) // 30 days
}
```

The token is stored as an `httpOnly` cookie. For Hyperclay subdomains, it's scoped to `.hyperclay.com` (the leading dot covers all subdomains). For custom domains, `SameSite: none` and `secure: true` enable cross-domain cookie sharing.

On every request, `state__user` looks up the token, loads the `Person` with their `Nodes` (via eager loading), calculates document limits, and sets the user type.

### Eager loading with getPersonWithNodes

The `getPersonWithNodes` function is used everywhere a fully-populated user is needed: during login, on every authenticated request, and after account creation. It loads the `Person` with all their `Nodes` in a single query via Sequelize eager loading:

```js
const person = await Person.findOne({
  where: { id: personId },
  include: [{
    model: Node,
    as: 'OwnedNodes',
    required: false,
    include: [{
      model: Node,
      as: 'SourceNode',
      required: false
    }]
  }],
  order: [sequelize.literal('LOWER("OwnedNodes"."name") ASC')]
});
```

This returns a `Person` with `person.Nodes` (aliased to `OwnedNodes`), an array of all documents, folders, and uploads they own, each with its `SourceNode` if it's an instance. The dashboard, document limit calculation, ownership checks, and folder navigation all read from this pre-loaded array instead of making additional queries. The `LOWER(name) ASC` ordering means the dashboard lists documents alphabetically, case-insensitively.

### Domain-aware cookie management

The `cookieManager` handles the complexity of setting cookies that work across both Hyperclay subdomains and custom domains:

```js
// For Hyperclay domains - scope to apex so all subdomains share auth
setCookieOnApexAndAllSubdomains(res, 'auth_token', token, {
  maxAge: 30 * 24 * 60 * 60 * 1000,
  httpOnly: true
});

// For custom domains - SameSite=none enables cross-domain sharing
res.cookie('auth_token', token, {
  maxAge, httpOnly: true, secure: true,
  sameSite: 'none',
  domain: req.state.meta.rootDomain
});
```

The same split applies to ownership cookies. When a user visits a document, `cookieManager.setOwnershipCookie` sets `currentResource` (the document name) and `isAdminOfCurrentResource` (whether they own it) on the appropriate domain. These cookies are readable by client JavaScript: [ClayJS](https://clayjs.com/docs/edit-mode/) reads `isAdminOfCurrentResource` at boot to set `clay.canEdit` and `clay.isOwner`. The document still opens in view, so `clay.isEditMode` stays `false` until the owner switches to editing. ClayJS writes `editmode="true|false"`, `pageowner="true|false"` and, while the person may edit, `canedit="true"` onto `<html>` to control editing UI visibility (`[show-when:editmode="true"]`) and mode attributes.

On logout, `clearAuthCookies` clears all auth and ownership cookies across every possible domain variation to prevent stale sessions.

### Two account types

| Type | How they're created | What they can do |
|------|-------------------|------------------|
| `dev` | Sign up via Stripe checkout (or free signup when `FREE_SIGNUP=on`) and set password | Create documents, manage uploads, and on paid plans edit code, manage billing, and attach custom domains |
| `app_user` | Sign up on a multi-tenant app | Edit their own instance, manage their uploads |

### Share token system

Any document can use [share links](https://hyperclay.com/docs/share-links/). In the token-based share flow, a random string is stored on the Node. When a visitor arrives with `?token=...` in the URL, the `state__shareAccess` middleware:

1. Validates the token against the Node's `shareToken` field (must have `shareEnabled: true`)
2. Sets `req.state.hasShareAccess = true` and `req.state.user.isOwner = true`
3. Elevates `no_auth` users to `app_user` (devs keep their `dev` type)
4. Stores the token in a document-specific cookie (`share_{sitename}`) for future visits

This lets edit share link recipients save changes without creating an account. Free accounts can create both view and edit share links.

### Password security

- Bcrypt hashing with salt rounds of 10
- 8-character minimum
- Checked against a top-100k common passwords list loaded at server startup
- Timing-safe comparison on failed lookups (dummy bcrypt compare to prevent enumeration)

## Real-time collaboration via SSE

The `livesync-hyperclay` library handles real-time [live sync](https://clayjs.com/docs/plugins/live-sync/) between browser editors and the disk sync engine over the [ClayJS wire protocol](https://clayjs.com/docs/wire-protocol/).

### SSE stream

Clients connect via GET. The `document-url` query parameter names the document because `EventSource` cannot set headers:

```bash
GET /_/sync?document-url=<full page URL>&lane=live
GET /_/sync?document-url=<full page URL>&lane=saved
```

The `live` lane carries pre-strip snapshots to editors. The `saved` lane carries durable documents to viewers after an accepted save.

The server sets SSE headers, registers the client, sends a `: connected` comment, and starts 30-second keep-alive pings. On disconnect, the client is unregistered.

```js
const lane = req.query.lane === 'saved' ? 'saved' : 'live';

res.setHeader('Content-Type', 'text/event-stream');
res.setHeader('Cache-Control', 'no-cache');
res.setHeader('Connection', 'keep-alive');
res.setHeader('X-Accel-Buffering', 'no');

liveSync.subscribe(file, res, { lane });

const keepAlive = setInterval(() => {
  res.write(': ping\n\n');
}, 30000);
```

### Save broadcast

When a save happens (from the code editor, the visual editor, or the Sync API), two broadcasts fire:

1. **Browser broadcast**: `liveSync.broadcast(file, { html, sender }, { lane: 'saved' })` sends the durable document to viewers on the `saved` lane. The `sender` ID lets clients ignore their own saves.
2. **Sync engine broadcast**: `liveSync.broadcastNodeSaved(username, buildNodeSavedPayload(node, { content }))` notifies the owner's [Hyperclay Local](https://hyperclaylocal.com/docs/) sync client. `buildNodeSavedPayload` builds the per-type wire shape (for a document, it carries `content` plus an auto-computed SHA-256 `checksum`).

Editor snapshots travel separately through `POST /_/sync` and are broadcast on the `live` lane without writing to disk.

### Access control

The two lanes use different authorization. The `live` lane is gated like a write and requires ownership. The `saved` lane is gated like a read, so public documents are open while private documents require owner, share-token, or admin access.

```js
app.get('/_/sync', gateLiveSyncStream, handleLiveSyncStream);
```

Edit share guests get live-lane access because `state__shareAccess` elevates them and sets `isOwner = true`.

## The Sync API

The [Sync API](https://hyperclay.com/docs/sync-api/) powers [Hyperclay Local](https://hyperclaylocal.com/docs/), the desktop app that syncs a local folder of documents with hyperclay.com, as well as external agents and scripts. It uses a separate authentication system from the browser-based state machine: API keys instead of cookies.

### API key authentication

Keys use a `hcsk_` prefix followed by 32 random bytes (hex-encoded). The raw key is shown once during generation and stored as a SHA-256 hash. Keys expire after 1 year. Each request validates the `X-API-Key` header by hashing the provided key and looking up the hash in the `ApiKey` table.

```js
const keyHash = crypto.createHash('sha256').update(apiKeyHeader).digest('hex');
const apiKey = await ApiKey.findValidKey(keyHash);
```

The `authenticateApiKey` middleware runs before all sync endpoints, separate from the state machine pipeline. On success, it sets `req.state.user.person` to the key's owner, so downstream handlers can use the same person reference.

### Endpoints

| Endpoint | Method | Purpose |
|----------|--------|---------|
| `/_/sync/status` | GET | Server time for clock synchronization |
| `/_/sync/stream` | GET | SSE stream for real-time file change notifications |
| `/_/sync/nodes` | GET | List all nodes (documents, uploads, and folders) |
| `/_/sync/nodes` | POST | Create a node |
| `/_/sync/nodes/:id/content` | GET | Download node content |
| `/_/sync/nodes/:id/content` | PUT | Write node content |
| `/_/sync/nodes/:id/rename` | PATCH | Rename a node |
| `/_/sync/nodes/:id/move` | PATCH | Move a node to a different parent |
| `/_/sync/nodes/:id` | DELETE | Delete a node |

### Sync upload flow

A node write (`POST` or `PUT /_/sync/nodes...`) does more than write a file:

1. **Parse path**: Splits `folder/subfolder/sitename` into folder components and document name
2. **Create folders**: If the path includes folders, creates `Node` records for each level that doesn't exist yet, with parent-child relationships
3. **Check name collisions**: If the document name exists globally and is owned by someone else, returns 409. If owned by the same user in a different folder, reuses the existing `Node`.
4. **Backup**: If the file already exists on disk, creates a backup via `BackupService.createBackup` before overwriting
5. **Write**: Writes to `/sites/{username}/{folderpath}/{sitename}.html`, scoped by owner username and folder path on disk
6. **Tailwind**: Compiles Tailwind CSS if the HTML references a Tailwind stylesheet
7. **Broadcast**: Sends the update to both the live-sync system (for connected browsers) and the sync engine (for other devices running Hyperclay Local)

### SSE stream per user

Unlike browser live-sync (which subscribes to individual files), the sync stream subscribes to an entire user's file changes via `liveSync.subscribeUser`. When any of the user's files is saved, whether from the browser code editor, another device's Hyperclay Local, or in-page editing, the stream pushes the update.

### Dual broadcast

When a sync upload includes a `snapshotHtml` field and `senderId`, two additional broadcasts fire:

1. `liveSync.broadcast(siteName, { html, sender })` pushes to browser editors viewing the file
2. `liveSync.broadcastToUser(username, siteName, { html, sender })` pushes to other Hyperclay Local instances

This enables three-way sync across the local filesystem, the hosted platform, and the browser editor. The `sender` ID prevents echo loops, as each client ignores broadcasts from itself.

## Tailwind CSS on-save compilation

The `tailwind-hyperclay` library powers [Tailwind on save](https://hyperclay.com/docs/tailwind/) with four functions used across the codebase:

| Function | Purpose |
|----------|---------|
| `hasTailwindLink(html, siteName)` | Checks if HTML contains `<link href="/tailwindcss/{siteName}.css">` |
| `hasAnyTailwindLink(html)` | Checks if HTML contains any Tailwind link (for copies and renames) |
| `compileTailwind(html)` | Extracts classes from HTML, runs Tailwind v4 JIT, and returns a CSS string |
| `replaceTailwindLink(html, oldName, newName)` | Updates the link `href` when a document is renamed or copied |

Compilation hooks into every operation that changes HTML content:

- **Save** (`saveHTML`): compile if `hasAnyTailwindLink`
- **Sync upload** (`uploadSyncFile`): compile if `hasAnyTailwindLink`
- **Backup restore** (`restoreBackup`): compile if `hasAnyTailwindLink`
- **Document copy** (`copySiteComplete`): detect with `hasAnyTailwindLink`, replace link with `replaceTailwindLink`, then compile

Output is written to `/public-assets/tailwindcss/{username}/{folderpath}/{sitename}.css`. The Express server serves this path with a fallback: if the file doesn't exist yet (first load before any save), it returns empty CSS instead of a 404.

## Domain routing

Hyperclay routes both [addresses and subdomains](https://hyperclay.com/docs/addresses/) on `hyperclay.com` and [custom domains](https://hyperclay.com/docs/custom-domains/).

### Domain classification

The `classifyDomain` function in `state__meta` determines the request type:

| Domain | Classification |
|--------|---------------|
| `hyperclay.com` or `www.hyperclay.com` | `main_app` |
| `*.hyperclay.com` | `hyperclay_subdomain` |
| Dev tunnel domain | `dev_tunnel` (treated like `main_app`) |
| Everything else | `custom_domain` |

### Subdomain resolution

For Hyperclay subdomains, `state__resource` resolves the subdomain to the owner's username, then the document by filename in the path:

1. Look up a `Person` whose `username` matches the subdomain
2. Resolve the `.html` or `.htmlclay` filename from the path and find the owner's `Node` by name and folder path
3. If that `Node` has `sourceNodeId` set, load its source node and owner, setting `ResourceType.INSTANCE`
4. Otherwise, if the document has `enableSignups = true`, set `ResourceType.MULTI_TENANT`; else set `ResourceType.REGULAR_SITE`
5. If no document matches, fall back to resolving an uploaded asset at that path

### Custom domain resolution

For custom domains:

1. Look up the exact hostname in the `CustomDomain` table (status must be `active`)
2. If not found and there's a subdomain, try the root domain (for instance subdomains on custom domains)
3. Load the associated `Node`
4. If the `Node` has `enableSignups` and there's a subdomain, look for an instance (`ResourceType.INSTANCE`)
5. Otherwise serve the document directly

### Custom domain lifecycle

Custom domains are managed through the SaasCustomDomains.com API:

1. User adds a domain: `POST` to the SaasCustomDomains API creates a `CustomDomain` record with status `pending`
2. SaasCustomDomains provides DNS instructions (CNAME or TXT records)
3. User updates DNS
4. Webhook fires on DNS verification: status moves to `active`
5. SSL is provisioned automatically
6. Webhook fires on SSL issuance: `sslStatus` moves to `issued`

Wildcard domains (`*.myapp.com`) use the DNS-01 challenge type for SSL. Per-user rate limiting caps API calls at 10 per minute.

## The Data Extraction API

Any document's HTML can be queried as structured JSON through [page data endpoints](https://hyperclay.com/docs/page-data/) by appending `?data={...}` to its URL, or from inside the page with the [ClayJS data plugin](https://clayjs.com/docs/plugins/data/) (`clay.extractData` and `clay.readData`). The extraction rules use a CSS-selector-based syntax.

### Rule types

**String rules** extract text or attributes:

```js
".title"           // text content of first .title element
".logo@src"        // src attribute of first .logo element
"@data-user-id"    // attribute from root element
".tag[]"           // array of text from all .tag elements
```

**Array rules** iterate over elements:

```js
// [selector, shape] - returns array of shaped objects
[".product", { name: ".name", price: ".price" }]

// Result: [{ name: "Widget A", price: "$99" }, { name: "Widget B", price: "$149" }]
```

**Object rules** compose nested structures:

```js
{
  user: { name: ".user-name", role: ".user-role" },
  metrics: { revenue: ".revenue", orders: ".orders" },
  products: [".product", { name: ".name", price: ".price" }],
  tags: ".tag[]"
}
```

### DOM property access

The `@` prefix also supports DOM properties: `@textContent`, `@innerHTML`, `@outerHTML`, `@value`, `@checked`, and others.

### Relaxed JSON parser

Since rules are passed as URL query parameters, the extraction endpoint includes a custom tokenizer that handles unquoted keys and CSS selectors (including pseudo-selectors with colons and attribute selectors with brackets) without requiring strict JSON escaping.

### Caching

Results are cached in memory for 5 minutes per document and query combination. When the map exceeds 100 entries, expired entries are swept; past 200 entries, the oldest entry is evicted.

## File uploads

### Storage

[Uploads](https://hyperclay.com/docs/uploads/) are stored directly on the filesystem at `/uploads/{username}/{folder_path}/{filename}`. The `Node` model tracks them in the database for [folder organization](https://hyperclay.com/docs/files/), with `parentId` linking to a folder `Node`.

When a file is uploaded from inside a document using the [ClayJS upload plugin](https://clayjs.com/docs/plugins/upload/) (`clay.upload` posting to `/_/upload`), it is stored in that document's own folder at `uploads/assets-document-<document id>/` (created on first upload) and linked as `/_/uploads/assets-document-<document id>/<file>`. The folder is named after the document's id, so renaming the document never moves its pictures, and once the account's uploads are protected the file answers only to people who can open the document. The top-level `uploads` folder is locked and cannot be renamed, moved, or deleted.

### Processing

Upload handling uses Formidable for multipart parsing. Files pass through validation:

- Filename sanitization via `sanitize-filename`
- Extension-based type detection via `mime-types` and `shared/file-kinds.js`, plus byte sniffing on free accounts to verify image contents
- Refused extensions: `.shtml` and `.xht` are blocked on all upload lanes, and `.html`, `.htm`, `.xhtml`, `.htmlclay`, `.js`, `.mjs`, and `.xml` are blocked on the in-document upload lane so HTML files go through the save lane
- Tier-based upload and storage caps: Free accounts are limited to images only (PNG, JPG, GIF, WebP, AVIF, or SVG) at 10 MB per file and 25 MB total; Personal allows 20 MB per file and 500 MB total; Maker allows 100 MB per file and 2 GB total. This is separate from the Express body parser limits (5 MB JSON, 16 MB text for whole-document saves, 2 MB URL-encoded forms), with special parsers raising the JSON cap to 150 MB on `/sync/nodes` and `/upload`, lowering it to 1 MB on `/collection`, and setting 10 MB on `/live-sync`.

### Editable files

Clicking an upload in Files opens `/dashboard/site/<id>`, which renders images in place, PDFs in the browser's viewer, text files as plain text, and other file types as a card with **Download** and **Open in new tab**. Text-based uploads (`txt`, `md`, `markdown`, `json`, `html`, `htm`, `htmlclay`, `jsx`, `css`, `js`, `mjs`, `csv`, `tsv`, `xml`, `yml`, `yaml`, `log`) plus `svg` can be edited in that page's **Code** tab with **Save** (`⌘S`) or at `/_/edit/uploads/{username}/{path}/{filename}`, which routes saves to `save-upload`. Editing an upload's code requires the paid `canEditCode` capability.

### Folder management

Folders are `Node` records with `type: 'folder'`. They support:

- 5 levels of nesting (enforced by a `beforeValidate` hook)
- Breadcrumb navigation via the `path` field (stores the ancestor path)
- Move operations between folders (with path recalculation for all descendants)

## Payment integration

### Stripe checkout flow

1. Unauthenticated user submits their email on the pricing page
2. Server creates a Stripe checkout session with the email
3. User completes payment on Stripe's hosted page
4. Stripe webhook (`/stripe-webhook`) fires with the raw body (parsed before Express JSON middleware)
5. Server verifies the webhook signature
6. On `checkout.session.completed`, the server creates a `Person` with `hasActiveSubscription: true` and an `EmailConfirmToken`
7. User follows the confirmation link, sets a username and password, and the account is ready

### Pricing tiers

[Plans and billing](https://hyperclay.com/docs/billing/) offer two paid tiers, billed monthly or annually (saving 33%), plus a Free tier ($0 forever, offered when `FREE_SIGNUP` is enabled; off by default in production):

| Tier | Monthly | Annual | Included highlights |
|------|---------|--------|---------------------|
| Free | $0 | $0 | 10 documents, image-only uploads (10 MB per file, 25 MB total), view and edit share links |
| Personal | $6/month | $48/year | Unlimited documents, unlimited custom domains, 1 to 3 multi-tenant apps, 3 to 10 subdomains, 20 MB / 500 MB uploads, 1,000 monthly AI credits |
| Maker | $15/month | $120/year | Everything in Personal plus [teams](https://hyperclay.com/docs/teams/) (4 people included: owner and 3 editors; $3/month or $2/month on annual per extra editor; viewers free), 5 to 25 multi-tenant apps, 6 to 25 subdomains, 100 MB / 2 GB uploads, 2,000 monthly AI credits |

### Document limits and feature accrual

Limits are tier-based, driven by `TIER_CONFIG` rather than a single formula. Free accounts can create 10 documents. Paid tiers (Personal and Maker) are effectively unlimited: the config sets the cap at 2000, which `calculateSiteLimit` reports as `isUnlimited`. Superadmins are unlimited.

```js
// From TIER_CONFIG
const TIER_CONFIG = {
  free:     { sites: { limit: 10 } },
  personal: { sites: { limit: 2000 } },  // treated as unlimited
  maker:    { sites: { limit: 2000 } },  // treated as unlimited
};

// calculateSiteLimit reports config.limit >= 2000 as isUnlimited
```

Custom domains are unlimited (`domains.maximum: null`) on paid tiers. The `base + monthsActive * perMonth` accrual (capped at `max`) applies to subdomain and multi-tenant app allowances:

- **Personal**: 3 subdomains (+1/month up to 10) and 1 multi-tenant app (+1/month up to 3)
- **Maker**: 6 subdomains (+2/month up to 25) and 5 multi-tenant apps (+2/month up to 25)

### Billing portal and lapsed subscriptions

Subscription management (**Manage billing**) is a redirect via `/_/manage-billing` to Stripe's hosted billing portal. Subscriptions have a 30-day full refund policy and auto-cancel after 6 months of inactivity. When a subscription lapses, `isLapsedFor` in `server-lib/capabilities.js` drops the account to free capabilities: existing documents stay accessible for viewing, content editing, uploads, and copying, while creating new documents, opening the code editor, adding domains or subdomains, and enabling sign-ups require an active paid plan.

## The code editor

The `/_/edit` route and the dashboard **Code** tab (`/_/dashboard/site/:nodeId/code`) serve the [code editor](https://hyperclay.com/docs/code-editor/) built on CodeMirror 6. The editor loads the document's current HTML, provides syntax highlighting, and integrates with the live-sync system.

### Access control

Only `dev` users who own the document and hold the paid `canEditCode` capability can access the code editor. Share guests are elevated to `app_user`, which is explicitly forbidden from the code editor (`app_user:*:edit` returns 403). This is intentional: edit share access grants save permission for in-page editing, not raw code editing.

```js
'dev:regular_site+multi_tenant+instance:edit': [requireOwnership, serveCodeEditor],
'app_user:regular_site+multi_tenant+instance:edit': [forbidden],
```

### Upload editing

Text-based and SVG uploads can be edited in the dashboard **Code** tab or at `/_/edit/uploads/{username}/{path}`. Ownership and `canEditCode` are verified before saving.

## Security model

### Declarative access control

Security is structural. The routing table is the access control policy:

```js
// Only devs who own the document can edit code
'dev:regular_site:edit': [requireOwnership, serveCodeEditor]

// Nobody without auth can edit anything
'no_auth:*:edit': [redirect('/_/login?...')]

// App users can save (their instances), but not edit code
'app_user:regular_site+instance:save': [requireOwnership, requireEmailConfirmForAppUser, saveHTML, respond]
'app_user:regular_site+instance:edit': [forbidden]
```

Every request that doesn't match a routing key gets a 403. There is no default-allow behavior.

### Rate limiting

The `createRateLimiter` factory produces configurable middleware:

- **Login**: 20 attempts per 15 minutes, keyed on `sha256(IP + email)`, resets on success
- **API**: 100 requests per 15 minutes, keyed on IP only
- **Strict** (sensitive operations): 5 attempts per hour

Rate limiters use in-memory `Map` instances with periodic cleanup.

### Password security

- Bcrypt with 10 salt rounds
- 8-character minimum enforced server-side
- Checked against a top-100k common passwords list (loaded from `PwnedPasswordsTop100k.json` at startup)
- Timing-safe comparison: on failed lookup, a dummy bcrypt compare runs to prevent email enumeration via response timing

### Cookie security

| Cookie | Flags |
|--------|-------|
| `auth_token` | `httpOnly`, `secure`, `SameSite=lax`, `domain=.hyperclay.com`, 30-day expiry |
| `auth_token` (custom domain) | `httpOnly`, `secure`, `SameSite=none`, scoped to root domain |
| `isAdminOfCurrentResource` | `secure` (readable by [ClayJS](https://clayjs.com/docs/edit-mode/) for UI state) |
| `share_{sitename}` | `httpOnly`, `secure` (production only), `SameSite=lax`, 1-year expiry |

### Additional protections

- Sequelize parameterized queries prevent SQL injection
- Stripe webhook signature verification (raw body, not parsed JSON)
- Node name validation blocks reserved words and special characters
- File upload sanitization via `sanitize-filename` and byte sniffing for image-only free accounts
- Express body parser limits (5 MB JSON, 16 MB text for whole-document saves, 2 MB URL-encoded), with tier-based per-file upload caps (Free 10 MB, Personal 20 MB, Maker 100 MB) and account storage caps (Free 25 MB, Personal 500 MB, Maker 2 GB) layered on top

## Deployment

### Production

```js
// ecosystem.config.js
{
  name: 'hyperclay',
  script: './hey.js',
  instances: 1,
  autorestart: true,
  max_memory_restart: '1G'
}
```

Production runs a single PM2 instance without cluster mode: the filesystem-based architecture means a single process avoids file locking complexity. The 1 GB memory limit triggers an automatic restart if the process leaks.

### Development

Dev mode runs four PM2 processes:

| Process | What it does |
|---------|-------------|
| `hyperclay` | Express server with file watching on `hey.js` and `server-lib/` |
| `tailwind-dev` | PostCSS watcher compiling the admin UI's Tailwind CSS |
| `backup-dev` | Periodic local backup script |
| `malleabledocs-dev` | Cloudflare tunnel for external access during development |

### Database

PostgreSQL with Sequelize ORM. Connection pool: min 2, max 10, 30-second acquire timeout, 10-second idle timeout. There is no `sequelize.sync()` or model auto-sync at startup. The schema is managed by an explicit numbered migration runner (`node migrate.js --all`), run on both dev startup (the `predev` script) and deploy (the `deploy` script); the numbered migrations live in `migrations/`.

### Graceful shutdown

The server listens for `SIGTERM` and closes the HTTP server cleanly, letting in-flight requests complete before exiting:

```js
process.on('SIGTERM', () => {
  server.close(() => process.exit(0));
});
```

## The philosophy

Hyperclay works because it embraces constraints. A single HTML file per document limits complexity. DOM-as-database eliminates impedance mismatch. Direct manipulation means what you see is what gets saved.

The trade-offs are real:

- **No complex queries**: You can't JOIN across documents or SELECT with SQL from the HTML. The [Data Extraction API](https://hyperclay.com/docs/page-data/) gives you CSS-selector-based reads, not relational queries.
- **Entire HTML in memory**: Every save reads and writes the full file (up to 8 MB on hyperclay.com). Documents with hundreds of thousands of DOM nodes will hit performance limits.
- **Whole-document conflict handling**: Saves send the complete HTML file rather than field-level patches. With [conditional saves and receipts](https://clayjs.com/docs/conditional-saves/) (`If-Match` and HTTP 412) and [live sync](https://clayjs.com/docs/guide/live-sync/) (`?plugins=sync`), open tabs perform a save-gated three-way DOM merge and record colliding edits in `clay.conflicts` rather than silently overwriting, while un-stamped saves fall back to last-write-wins.

What it gains:

- **Radical simplicity**: The entire backend is one Express application with one routing table. There is no separate content API layer, no ORM for document content, and no frontend build step.
- **Instant persistence**: Save and it's live. There is no deploy step, no cache invalidation, and no eventual consistency.
- **True ownership**: Users own their HTML files. They can download, modify, and re-upload them. The platform adds features on top; it doesn't lock content away.
