ICI Orchestrator

Experimental. This feature is under active development. Behavior may change between releases.

For the standard indexing workflow, see Indexing & Integration.

This stage requires center finding and refinement to have run first (center_x, center_y, det_shift_x_mm, det_shift_y_mm must exist in the HDF5).

ICI (Iterative Center-shift Indexing) refines the detector shift by repeatedly indexing at a proposed shift, evaluating quality via wRMSD, and proposing the next shift — converging per-frame toward the optimal detector center.

Inputs

• COSEDA working file (<dataset>.h5) with:

  • entry/data/images

  • entry/data/center_x, center_y (from center finding / refinement)

  • entry/data/det_shift_x_mm, det_shift_y_mm (used for geometry generation)

  • optional entry/data/index (if frames were stripped)

• A cell file (.cell) describing the expected unit cell. CrystFEL accepts rhombohedral or hexagonal lattice types for trigonal cells; do not use trigonal as the lattice_type.

• A geometry file (.geom) describing the detector layout

How it works

Each iteration of the loop:

1. Propose shifts. For each event (frame), select the next detector shift using a two-stage policy:

   • Step 1 (global exploration): sample candidate shifts within radius_mm using a hillmap weighted by wRMSD — successful prior shifts attract sampling, failed ones repel.

   • Step 2 (local refinement): when the last run indexed successfully, apply a damped correction from the per-frame dx/dy residuals: next = last − λ × dxdy.

2. Index. Run indexamajig at each proposed shift in parallel.

3. Evaluate. Ingest per-run metrics (wRMSD, indexed flag) into the run log.

4. Converge. An event is marked done when any of the convergence criteria are satisfied (see parameters). The loop ends when all events are done or max_iters is reached.

Parameters

Convergence and search

Parameter	Default	Description
Search radius (mm)	0.05	Radius of the Step-1 shift search disk.
Minimum shift spacing (mm)	0.0005	Minimum distance between candidate shifts; prevents retrying near-identical positions.
Events for convergence	3	Minimum events needed to evaluate convergence.
Recurring shift tolerance	0.1	Relative tolerance for detecting recurring shifts (10%).
Median change tolerance	0.1	Relative change threshold for median-based convergence (10%).
No-improvement iterations	2	Iterations with no improvement before declaring convergence.
Minimum improvement	0.02	Relative improvement threshold; below this counts as no improvement (2%).
Stability window	3	Number of iterations over which stability is evaluated.
Stability spread limit	0.05	Relative std threshold for stability convergence (5%).
Successes in streak	2	Minimum successes within an unindexed streak to mark the event done.
Streak length	5	Length of unindexed streak evaluated against successes-in-streak.
Damping factor	0.8	λ for Step-2 refined shift updates (0 = full damping, 1 = no damping).

Indexamajig settings

These differ from the Simple Indexamajig defaults; they are tuned for ICI’s iterative, prediction-focused use.

Parameter	Default	Description
Min peaks	15	Minimum peaks for a frame to be attempted.
Max indexer threads	1	Thread limit per indexing backend.
Tolerance	5,5,5,5	Lattice parameter tolerances (a, b, c in %, angles in °).
Push resolution	0	Integrate beyond the apparent resolution of each pattern (1/nm).
Integration radii	4,5,8	Radii for the three integration rings: inner, middle, outer (px).
XGandalf tolerance	0.02	Relative tolerance of lattice vectors (2%).
XGandalf sampling pitch	5	Sampling density of reciprocal space (0–7).
Min lattice vector length	40 Å	Shortest allowed lattice vector.
Max lattice vector length	110 Å	Longest allowed lattice vector.
Gradient descent iterations	1	Refinement iterations (0–5).

Optional flags

Flag	Default	Effect
Disable retry	on	Do not retry failed frames with weakest reflections removed.
Disable half-pixel shift	on	Disable half-pixel offset when reading images.
Omit non-hits from stream	on	Exclude non-indexed patterns from the output stream.
Skip revalidation	on	Skip filtering peaks too close to detector edge, double peaks, or saturated pixels.
Skip prediction refinement	on	Disable post-indexing unit cell refinement.

An Additional flags free-text box accepts any extra indexamajig flags not covered by the structured controls.

Using the GUI

Open the Indexing & Integration window and select ICI Orchestrator (experimental) from the method drop-down.

The window has a Settings tab and a Run tab.

Settings tab

Configure convergence parameters and indexamajig settings. The top form holds the convergence and search knobs; the Indexamajig settings group below it holds the per-iteration indexing parameters and flags.

Click Save Settings to persist the current settings into the .ini and ici_settings.json.

Run tab

Field	Description
Run folder	Resolved automatically from the selected dataset; shows the ici_<timestamp>/ path that will be created on start. Read-only.
Geom (.geom)	Generated automatically in the ICI run folder when the run starts.
Cell (.cell)	Generated automatically in the ICI run folder from the Cell tab / workspace cell.
HDF5 files	Resolved automatically from the selected dataset.
Max iterations	Upper limit on orchestration loop iterations (default 20).
Jobs	Parallel indexing jobs (default: all available CPUs).

1. Select a dataset from the workspace tree. The Settings tab populates with any saved settings.

2. Prepare cell & geometry files in the Cell File and Geom File tabs (shared with Simple Indexamajig).

3. Configure convergence and indexamajig parameters in the Settings tab and save them.

4. Start a run from the Run tab:

   • Run orchestration — starts the ICI loop for the selected dataset.

   • Run all — runs the ICI loop sequentially for all workspace datasets.

5. Monitor progress. A progress bar and live log update as indexing events complete each iteration.

6. Stop at any time. The current iteration finishes before the loop exits; partial results are preserved.

CLI

python3 -m coseda.ici.orchestrator \
    --run-root /path/to/ici_run \
    --geom /path/to/detector.geom \
    --cell /path/to/crystal.cell \
    --h5 /path/to/dataset.h5 \
    --max-iters 20 \
    --jobs 8 \
    --no-session-subdir

Key arguments:

Argument	Description
--run-root	Output folder where run iterations and logs are written.
--geom	Geometry file for indexamajig.
--cell	Cell file for indexamajig.
--h5	One or more HDF5 source files (space-separated).
--max-iters	Maximum loop iterations (default 20).
--jobs	Parallel jobs (default: os.cpu_count()).
--no-session-subdir	Write runs directly into --run-root without a timestamped subfolder.
--flags	Space-separated indexamajig flags; replaces the built-in defaults when provided.

All convergence and search parameters (--radius-mm, --min-spacing-mm, --N-conv, etc.) accept the same defaults shown in the parameters table above.

Outputs

Each run creates a timestamped folder ici_<timestamp>/ under the dataset’s run root:

ici_<timestamp>/
  run_000/
    event_000000/
      idx.stdout, idx.stderr
      stream_000000.stream
      per_frame_dx_dy.csv
    event_000001/
    ...
    stream_000.stream          (concatenated run-level stream)
  run_001/
  ...
  image_run_log.csv            (living ledger: trials, proposals, convergence)
  image_run_state.json         (per-event state cache)
  ici_settings.json            (parameter snapshot)
  ici_run_manifest.json        (run provenance)

• image_run_log.csv: one section per event recording each trial’s shift, indexed flag, wRMSD, and next proposed shift.

• image_run_state.json: per-event trial and proposal history; used for convergence detection and diagnostics.

• ici_settings.json: full parameter dump for reproducibility.

• NXprocess metadata: written into the source HDF5 files documenting the ICI step.