feat: Documentation of db indexes and ODE Desktop developer mode

docs/getting-started/architecture-overview.md

@@ -98,6 +98,7 @@ Custom App Structure:
 **Key Features:**
 - **Configuration-driven**: `app.config.json` controls app behavior
 - **Form Integration**: Seamless integration with Formulus form system
+- **Local observation indexes**: Device-only index table for fast queries (never synced to Synkronus)
 - **Theme System**: Material Design 3 based theming
 - **Build Process**: Optimized bundle creation for deployment
 

docs/guides/custom-applications.md

@@ -72,10 +72,16 @@ If your template uses **`app.config.json`** (common in React-based examples), it
       "onBackground": "#ffffff",
       "onSurface": "#e0e0e0"
     }
-  }
+  },
+  "observationIndexes": [
+    { "key": "patient_id", "path": "$.patient_id" },
+    { "key": "site_code", "path": "$.site_code", "formTypes": ["visit_*"] }
+  ]
 }
 ```
 
+`observationIndexes` declares **local-only** SQLite indexes for fast `getObservationsByQuery` filters on `data.*` fields. They are maintained on the device and are **not synced**. See [Observation queries](./observation-queries.md).
+
 ### Theme Integration
 
 Themes are automatically generated from your configuration:

docs/guides/observation-queries.md

@@ -0,0 +1,100 @@
+# Observation queries and local indexes
+
+Custom apps query local observations through **`getObservationsByQuery`** using a **structured filter AST** (Abstract Syntax Tree). Formulus and ODE Desktop compile that AST to SQLite (`json_extract` plus a local **observation index** table). Indexes are declared in **`app.config.json`** and are **never synced** to Synkronus.
+
+## API
+
+```javascript
+const people = await formulus.getObservationsByQuery({
+  formType: 'register_person',
+  includeDeleted: false,
+  filter: {
+    op: 'and',
+    conditions: [
+      { field: 'data.village', op: 'eq', value: 'London' },
+      { field: 'data.sex', op: 'eq', value: 'female' },
+    ],
+  },
+});
+```
+
+### Filter shape
+
+| Node | Fields | Notes |
+|------|--------|-------|
+| Condition | `field`, `op`, `value` | `field` is `data.*` or a top-level column (`observation_id`, `deleted`, …) |
+| Logical | `op: 'and' \| 'or'`, `conditions[]` | Parentheses via nesting |
+| Quantifier | `op: 'any'`, `path`, `as`, `where` | Array members via `json_each` |
+
+Supported comparison ops: `eq`, `neq`, `gt`, `gte`, `lt`, `lte`, `in`.
+
+**Age from date of birth** (`age_from_dob(...)`) is **not** compiled to SQL. Formplayer evaluates age in JavaScript after fetching (see [Dynamic choice lists](./dynamic-choice-lists.md)).
+
+Invalid filters **fail closed** (structured error; no unfiltered fallback).
+
+## Declaring indexes (`app.config.json`)
+
+```json
+{
+  "observationIndexes": [
+    { "key": "p_id", "path": "$.p_id", "formTypes": ["hh_person", "p_*"] },
+    { "key": "hh_id", "path": "$.hh_id", "formTypes": ["household", "hh_*"] },
+    { "key": "age", "path": "$.age", "valueType": "number" }
+  ]
+}
+```
+
+| Property | Purpose |
+|----------|---------|
+| `key` | Stable index name; matches `data.<key>` in filters |
+| `path` | JSON path under observation `data` (e.g. `$.p_id`) |
+| `formTypes` | Optional patterns (`*` suffix = prefix match) |
+| `valueType` | `"number"` stores `value_num` for numeric comparisons |
+
+Indexes rebuild automatically when the app bundle updates. On **ODE Desktop**, use **Sync → Re-create index** after changing `observationIndexes` or bulk imports.
+
+## Examples
+
+**Person visits (OR on several fields):**
+
+```javascript
+filter: {
+  op: 'or',
+  conditions: [
+    { field: 'data.p_id', op: 'eq', value: personId },
+    { field: 'data.names', op: 'eq', value: personName },
+  ],
+}
+```
+
+**Household list:**
+
+```javascript
+filter: {
+  op: 'or',
+  conditions: [
+    { field: 'data.hh_id', op: 'eq', value: hhId },
+    { field: 'data.hh_number', op: 'eq', value: hhNumber },
+  ],
+}
+```
+
+## Performance notes
+
+- One **`formType`** per call; use `Promise.all` for multiple types.
+- Declared `data.*` paths use the **index table**; undeclared paths use **`json_extract`** with a dev warning.
+- Expression indexes on `observations(data)` may be created for declared paths to speed fallback queries.
+
+## Platform mapping
+
+| Concept | Formulus | ODE Desktop |
+|---------|----------|-------------|
+| JSON payload | `observations.data` | `observations.payload` |
+| Soft delete | `deleted` column | `observation_extras.deleted` |
+| Observation id | `observation_id` | `id` |
+
+The AST always uses **`data.*`** for JSON fields; each platform compiler maps to the correct column.
+
+## ODE Desktop developer mode
+
+When [ODE Desktop developer mode](./ode-desktop-developer-mode) is on, `app.config.json` and observation indexes are read from the **mirrored** app under `bundles/dev-local/app/`. After you change index declarations in your local project, mirror again with **Refresh app** so the desktop workspace picks up the new config.

docs/guides/ode-desktop-developer-mode.md

@@ -0,0 +1,92 @@
+# ODE Desktop developer mode
+
+**Developer mode** in ODE Desktop lets you iterate on a **local custom app build** (for example `dist/` after `npm run build`) against a profile’s real observations and workspace, without replacing the bundle you downloaded from Synkronus.
+
+## What it is
+
+When developer mode is on, ODE Desktop **mirrors** your selected folder into the profile workspace under `bundles/dev-local/`. The workbench then loads:
+
+- **Custom app** from `bundles/dev-local/app/`
+- **Form preview** from `bundles/dev-local/forms/` (when your folder includes a `forms/` directory)
+
+Observations, attachments, and sync still use the profile database. The Synk-downloaded bundle in `bundles/active/` is **not** overwritten.
+
+## Prerequisites
+
+- ODE Desktop installed (see project README for build-from-source).
+- A **profile** with a configured workspace.
+- A local folder whose root contains **`index.html`** (typical: your custom app `dist/` output).
+- Optional: download an app bundle from Synkronus on the **Bundles** page so `bundles/active/` has forms and `app.config.json` when developer mode is off.
+
+## Enable developer mode
+
+1. Open **Workbench** → **Bundles**.
+2. Turn **Developer mode** **On**.
+3. Click **Browse…** and select the folder that contains `index.html`.
+4. Wait for the first mirror to finish (or click **Refresh app**).
+
+While developer mode is on, an orange **banner** appears on all Workbench pages with the folder path and a **Refresh app** shortcut.
+
+## Folder layout
+
+Your selected folder is the **app root** (same idea as the `app/` directory inside a published bundle):
+
+```
+my-custom-app/dist/
+  index.html          ← required at root
+  app.js
+  assets/
+  forms/              ← optional; mirrored for Form preview
+    my_form/
+      schema.json
+      ui.json
+```
+
+- **Custom app entry:** `index.html` at the selected path (not a parent repo root unless that root is your built app).
+- **Forms:** standard bundle layout under `forms/{formType}/schema.json` and `ui.json`, plus optional `forms/ext.json` and custom question types (same as Formulus bundles).
+
+## Refresh app
+
+After you rebuild the custom app or edit forms locally:
+
+1. Click **Refresh app** on the Bundles page or in the Workbench banner.
+
+This re-runs the mirror (`source` → `bundles/dev-local/app/` and `source/forms/` → `bundles/dev-local/forms/`). **Custom app** and **Form preview** pick up changes on the next load (the embed remounts automatically after a successful mirror).
+
+**Refresh from server** on the Bundles page is separate: it updates `bundles/active/` from Synkronus only.
+
+## What changes when developer mode is on
+
+| Area | Behavior |
+|------|----------|
+| Custom app (Workbench) | Loads mirrored app under `bundles/dev-local/app/` |
+| Form preview | Lists and loads specs from `bundles/dev-local/forms/` when mirrored |
+| Observations / sync | Unchanged — profile SQLite |
+| Downloaded bundle | Stays in `bundles/active/` for server reload and non-dev use |
+| `getCustomAppUri` / `getFormSpecsUri` (in preview bridge) | Point at dev-local roots when mode is on |
+
+## Workbench banner
+
+When developer mode is on, every Workbench route shows a compact status strip **above** sync/activity messages: active folder path and **Refresh app**. Configure the folder and toggle on the **Bundles** page only.
+
+## Limitations
+
+- **Device APIs** (camera, GPS, QR, etc.) are stubbed in ODE Desktop form preview, same as before developer mode.
+- The mirror is a **copy** — edit files in your source folder, then **Refresh app**; ODE Desktop does not watch the filesystem.
+- Invalid or missing folder (no `index.html`) shows a **blocking error**; the custom app does not silently fall back to `bundles/active/`.
+- Observation query indexes use `app.config.json` from the **mirrored** app when developer mode is on; rebuild indexes if you change index declarations (see [Observation queries](./observation-queries)).
+
+## Platform comparison
+
+| | Formulus (device) | ODE Desktop (developer mode) |
+|--|-------------------|----------------------------|
+| Custom app source | Downloaded bundle | Local folder → `bundles/dev-local/app/` |
+| Forms | Bundle on device | Mirrored `forms/` or `bundles/active/forms/` when off |
+| Observations | On-device DB | Profile workspace SQLite |
+| Typical use | Field collection | Local iteration before publish |
+
+## See also
+
+- [Understanding app bundles](../using/app-bundles) — Synk download to `bundles/active/`
+- [Custom applications](../using/custom-applications) — custom app role in bundles
+- [Observation queries](./observation-queries) — `getObservationsByQuery` and indexes

docs/implementer/implementer-index.md

@@ -61,6 +61,18 @@ Get your first form running in 30 minutes:
     </div>
   </div>
 
+  <div className="col col--6 col--12-mobile margin-bottom--md">
+    <div className="card card--compact">
+      <div className="card__header">
+        <h4>ODE Desktop developer mode</h4>
+      </div>
+      <div className="card__body">
+        <p>Test a local custom app build in the Workbench against real profile data.</p>
+        <a className="button button--primary button--sm button--block" href="/docs/guides/ode-desktop-developer-mode">Learn More →</a>
+      </div>
+    </div>
+  </div>
+
   <div className="col col--6 col--12-mobile margin-bottom--md">
     <div className="card card--compact">
       <div className="card__header">

docs/reference/formulus.md

@@ -149,22 +149,37 @@ await api.deleteObservation('survey', 'obs-123');
 
 **Returns:** Promise that resolves when deletion is complete
 
-#### getObservations(formType, filters)
+#### getObservations(formType, isDraft?, includeDeleted?)
 
-Query observations from local database.
+List observations for a form type (no structured filter).
 
 ```javascript
-const observations = await api.getObservations('survey', {
-  status: 'synced',
-  limit: 10
+const observations = await api.getObservations('survey', false, false);
+```
+
+#### getObservationsByQuery(options)
+
+Query observations with a **structured filter AST** (preferred for custom apps). Declared `data.*` paths use a local **observation index**; other paths use `json_extract`. See [Observation queries](../guides/observation-queries.md).
+
+```javascript
+const observations = await api.getObservationsByQuery({
+  formType: 'hh_person',
+  includeDeleted: false,
+  filter: {
+    op: 'and',
+    conditions: [
+      { field: 'data.village', op: 'eq', value: 'kopria' },
+    ],
+  },
 });
 ```
 
 **Parameters:**
-- `formType` (string): The form type identifier
-- `filters` (object): Optional query filters
+- `formType` (string): Form type identifier
+- `includeDeleted` (boolean, optional): Include soft-deleted rows
+- `filter` (ObservationFilter, optional): Structured filter AST
 
-**Returns:** Promise resolving to array of observations
+**Returns:** Promise resolving to an array of observations
 
 #### sync()
 

docs/using/app-bundles.md

@@ -167,6 +167,10 @@ For technical information about app bundle structure, format, and development, s
 - [Custom Applications Guide](/guides/custom-applications) - Building custom applications
 - [Form Design Guide](/guides/form-design) - Creating form specifications
 
+## ODE Desktop Workbench
+
+On **ODE Desktop**, Synkronus bundles download into **`bundles/active/`** in the profile workspace. To iterate on a **local** custom app build without replacing that download, use [ODE Desktop developer mode](/docs/guides/ode-desktop-developer-mode) (mirror to `bundles/dev-local/`).
+
 ## Related Documentation
 
 - [Your First Form](/using/your-first-form) - Get started with data collection

docs/using/custom-applications.md

@@ -107,6 +107,10 @@ Custom applications are suitable for:
 - Integration with external systems
 - Complex navigation requirements
 
+## Testing locally with ODE Desktop
+
+Use [ODE Desktop developer mode](/docs/guides/ode-desktop-developer-mode) to load a local build (folder with `index.html`) in the Workbench **Custom app** page against a profile’s observations, then refresh after each build.
+
 ## Next Steps
 
 - Read the [Custom Applications guide](/guides/custom-apps/overview) for detailed information

sidebars.ts

@@ -93,6 +93,8 @@ const sidebars: SidebarsConfig = {
             'guides/building-custom-apps-v2',
             'guides/form-design',
             'guides/dynamic-choice-lists',
+            'guides/observation-queries',
+            'guides/ode-desktop-developer-mode',
             'guides/custom-extensions',
             'guides/deployment',
             'guides/configuration',