# ICI Orchestrator

> **Experimental.** This feature is under active development. Behavior may change between releases.

> For the standard indexing workflow, see [Indexing & Integration](7a_indexing_integration.md).

> This stage requires center finding and refinement to have run first (`center_x`, `center_y`, `det_shift_x_mm`, `det_shift_y_mm` must exist in the HDF5).

ICI (Iterative Center-shift Indexing) refines the detector shift by repeatedly indexing at a proposed shift, evaluating quality via wRMSD, and proposing the next shift — converging per-frame toward the optimal detector center.

## Inputs

- COSEDA working file (`<dataset>.h5`) with:
  - `entry/data/images`
  - `entry/data/center_x`, `center_y` (from center finding / refinement)
  - `entry/data/det_shift_x_mm`, `det_shift_y_mm` (used for geometry generation)
  - optional `entry/data/index` (if frames were stripped)
- A **cell file** (`.cell`) describing the expected unit cell. CrystFEL accepts `rhombohedral` or `hexagonal` lattice types for trigonal cells; do not use `trigonal` as the `lattice_type`.
- A **geometry file** (`.geom`) describing the detector layout

## How it works

Each iteration of the loop:

1. **Propose shifts.** For each event (frame), select the next detector shift using a two-stage policy:
   - *Step 1 (global exploration):* sample candidate shifts within `radius_mm` using a hillmap weighted by wRMSD — successful prior shifts attract sampling, failed ones repel.
   - *Step 2 (local refinement):* when the last run indexed successfully, apply a damped correction from the per-frame dx/dy residuals: `next = last − λ × dxdy`.
2. **Index.** Run indexamajig at each proposed shift in parallel.
3. **Evaluate.** Ingest per-run metrics (wRMSD, indexed flag) into the run log.
4. **Converge.** An event is marked done when any of the convergence criteria are satisfied (see parameters). The loop ends when all events are done or `max_iters` is reached.

## Parameters

### Convergence and search

| Parameter | Default | Description |
|---|---|---|
| Search radius (mm) | 0.05 | Radius of the Step-1 shift search disk. |
| Minimum shift spacing (mm) | 0.0005 | Minimum distance between candidate shifts; prevents retrying near-identical positions. |
| Events for convergence | 3 | Minimum events needed to evaluate convergence. |
| Recurring shift tolerance | 0.1 | Relative tolerance for detecting recurring shifts (10%). |
| Median change tolerance | 0.1 | Relative change threshold for median-based convergence (10%). |
| No-improvement iterations | 2 | Iterations with no improvement before declaring convergence. |
| Minimum improvement | 0.02 | Relative improvement threshold; below this counts as no improvement (2%). |
| Stability window | 3 | Number of iterations over which stability is evaluated. |
| Stability spread limit | 0.05 | Relative std threshold for stability convergence (5%). |
| Successes in streak | 2 | Minimum successes within an unindexed streak to mark the event done. |
| Streak length | 5 | Length of unindexed streak evaluated against successes-in-streak. |
| Damping factor | 0.8 | λ for Step-2 refined shift updates (0 = full damping, 1 = no damping). |

### Indexamajig settings

These differ from the Simple Indexamajig defaults; they are tuned for ICI's iterative, prediction-focused use.

| Parameter | Default | Description |
|---|---|---|
| Min peaks | 15 | Minimum peaks for a frame to be attempted. |
| Max indexer threads | 1 | Thread limit per indexing backend. |
| Tolerance | 5,5,5,5 | Lattice parameter tolerances (a, b, c in %, angles in °). |
| Push resolution | 0 | Integrate beyond the apparent resolution of each pattern (1/nm). |
| Integration radii | 4,5,8 | Radii for the three integration rings: inner, middle, outer (px). |
| XGandalf tolerance | 0.02 | Relative tolerance of lattice vectors (2%). |
| XGandalf sampling pitch | 5 | Sampling density of reciprocal space (0–7). |
| Min lattice vector length | 40 Å | Shortest allowed lattice vector. |
| Max lattice vector length | 110 Å | Longest allowed lattice vector. |
| Gradient descent iterations | 1 | Refinement iterations (0–5). |

### Optional flags

| Flag | Default | Effect |
|---|---|---|
| Disable retry | on | Do not retry failed frames with weakest reflections removed. |
| Disable half-pixel shift | on | Disable half-pixel offset when reading images. |
| Omit non-hits from stream | on | Exclude non-indexed patterns from the output stream. |
| Skip revalidation | on | Skip filtering peaks too close to detector edge, double peaks, or saturated pixels. |
| Skip prediction refinement | on | Disable post-indexing unit cell refinement. |

An **Additional flags** free-text box accepts any extra indexamajig flags not covered by the structured controls.

## Using the GUI

Open the **Indexing & Integration** window and select **ICI Orchestrator (experimental)** from the method drop-down.

The window has a **Settings** tab and a **Run** tab.

### Settings tab

Configure convergence parameters and indexamajig settings. The top form holds the convergence and search knobs; the **Indexamajig settings** group below it holds the per-iteration indexing parameters and flags.

Click **Save Settings** to persist the current settings into the `.ini` and `ici_settings.json`.

### Run tab

| Field | Description |
|---|---|
| Run folder | Resolved automatically from the selected dataset; shows the `ici_<timestamp>/` path that will be created on start. Read-only. |
| Geom (.geom) | Generated automatically in the ICI run folder when the run starts. |
| Cell (.cell) | Generated automatically in the ICI run folder from the Cell tab / workspace cell. |
| HDF5 files | Resolved automatically from the selected dataset. |
| Max iterations | Upper limit on orchestration loop iterations (default 20). |
| Jobs | Parallel indexing jobs (default: all available CPUs). |

1. **Select a dataset** from the workspace tree. The Settings tab populates with any saved settings.
2. **Prepare cell & geometry files** in the Cell File and Geom File tabs (shared with Simple Indexamajig).
3. **Configure convergence and indexamajig parameters** in the Settings tab and save them.
4. **Start a run** from the Run tab:
   - **Run orchestration** — starts the ICI loop for the selected dataset.
   - **Run all** — runs the ICI loop sequentially for all workspace datasets.
5. **Monitor progress.** A progress bar and live log update as indexing events complete each iteration.
6. **Stop** at any time. The current iteration finishes before the loop exits; partial results are preserved.

## CLI

```python
python3 -m coseda.ici.orchestrator \
    --run-root /path/to/ici_run \
    --geom /path/to/detector.geom \
    --cell /path/to/crystal.cell \
    --h5 /path/to/dataset.h5 \
    --max-iters 20 \
    --jobs 8 \
    --no-session-subdir
```

Key arguments:

| Argument | Description |
|---|---|
| `--run-root` | Output folder where run iterations and logs are written. |
| `--geom` | Geometry file for indexamajig. |
| `--cell` | Cell file for indexamajig. |
| `--h5` | One or more HDF5 source files (space-separated). |
| `--max-iters` | Maximum loop iterations (default 20). |
| `--jobs` | Parallel jobs (default: `os.cpu_count()`). |
| `--no-session-subdir` | Write runs directly into `--run-root` without a timestamped subfolder. |
| `--flags` | Space-separated indexamajig flags; replaces the built-in defaults when provided. |

All convergence and search parameters (`--radius-mm`, `--min-spacing-mm`, `--N-conv`, etc.) accept the same defaults shown in the parameters table above.

## Outputs

Each run creates a timestamped folder `ici_<timestamp>/` under the dataset's run root:

```
ici_<timestamp>/
  run_000/
    event_000000/
      idx.stdout, idx.stderr
      stream_000000.stream
      per_frame_dx_dy.csv
    event_000001/
    ...
    stream_000.stream          (concatenated run-level stream)
  run_001/
  ...
  image_run_log.csv            (living ledger: trials, proposals, convergence)
  image_run_state.json         (per-event state cache)
  ici_settings.json            (parameter snapshot)
  ici_run_manifest.json        (run provenance)
```

- **`image_run_log.csv`:** one section per event recording each trial's shift, indexed flag, wRMSD, and next proposed shift.
- **`image_run_state.json`:** per-event trial and proposal history; used for convergence detection and diagnostics.
- **`ici_settings.json`:** full parameter dump for reproducibility.
- **NXprocess metadata:** written into the source HDF5 files documenting the ICI step.