> ## Documentation Index
> Fetch the complete documentation index at: https://docs.reilabs.org/llms.txt
> Use this file to discover all available pages before exploring further.
# Sequential learning, advanced
> Configure policies, rewards, delayed credit, exploration, training, and frozen evaluation.
Configure a Domain for actions that change later states and receive delayed outcomes. Begin with the starting declaration, run the query–execute–feedback loop, then adjust only the settings your task requires.
Sequential learning is Adapt-1's episode-aware policy path for state-changing actions and delayed outcomes. This advanced guide explains how to configure and operate it through the Domain API. When Adapt-1 forms the supported state-dependent action-value and delayed-credit structure from evidence, the Discovery documentation refers to that capability as Sequential Discovery.
The objective is not to find one universal configuration. A good learner is a correctly scoped state-action-reward contract, followed by settings that match the task's horizon, feedback density, nonstationarity, risk tolerance, and data budget.
Sequential learning does not require the whole Domain ontology to be discovered. The public action space, approved state paths, reward semantics, and episode boundaries can be authored, discovered where supported, or mixed. The acquisition schedule is separate again: sequential state can form inside a zero-start run, form in a separate trajectory acquisition phase, or continue from a separately acquired checkpoint.
## 1. When to use sequential learning
Enable sequential learning when all of the following are true:
* the system repeatedly chooses among two or more policies or actions;
* actions influence later observations;
* action quality depends on the current state;
* reward may be delayed, sparse, or distributed over an episode;
* events can be grouped into episodes and ordered by step;
* every action can eventually be linked to a measured outcome.
Examples include control, trading, game play, operations scheduling, treatment sequencing, multi-step remediation, and adaptive workflows.
Use ordinary contextual policy learning instead when each decision is independent and receives an immediate outcome. Use structured transition learning when the main output should be a prediction of the next value rather than a selected policy. Use both when the application needs next-state forecasts and policy selection, but keep their output contracts distinct.
Sequential learning does not register a separate `sequential` entry in `learning_state.active_subsystems`. It is an episode-aware model path inside `feedback_policy`. A Domain can therefore report `feedback_policy` and `structured_transition` as its active subsystems while sequential learning is enabled. When the sequential candidate trains and passes validation, inspect `learning_state.subsystems.feedback_policy.model.report.model_type` for `sequential_q_mlp`.
### Choose zero-start learning, separate acquisition, or both
| Setup | Use it when | Main constraint |
| -------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
| One zero-start live task | Online adaptation inside that task is the goal, and feedback arrives early enough to affect later choices | One long episode is still one episode and cannot supply episode-held-out validation |
| Zero-start repeated episodes | Safe exploration and continuing improvement are part of the declared run | Keep one learner scope and use a new episode ID at each real reset; later episodes still use state earned inside the same zero-start run |
| Separate trajectory acquisition, then freeze | The first held-out action should use learned sequential value, or test-time exploration is unsafe or unavailable | Split acquisition and evaluation episodes before ingestion, preserve action coverage, and stop writes during evaluation |
| Separate acquisition checkpoint plus live episodes | Prior trajectories form state outside the new run, while the deployed process can drift | Preserve a fixed starting checkpoint so current-run adaptation can be measured separately |
The complete declared run determines the acquisition label. A later episode is not warm-started merely because earlier episodes in the same zero-start run produced retained state.
Under the current defaults, the sequential candidate needs at least eight distinct episodes and reserves at least two episodes for validation. A Domain can declare a higher `minimum_episodes`. A single test episode may update ordinary contextual evidence when policy admission succeeds, but it does not establish validated `sequential_q_mlp` behavior.
A separate trajectory acquisition phase helps only when the state fields, actions, reward, terminal meaning, and useful action-value relationship transfer into the new run. Use a zero-start task-local scope when each independently redrawn task has a different hidden mapping and the current context does not identify that difference.
Historical trajectories use the same event and feedback contract as live trajectories. Replay approved records into fresh Domain state, preserve the true episode and step order, wait for asynchronous fitting, and inspect the model report before freezing. See [Choose a learning setup](/docs/neuroadapt/choose-a-learning-setup#when-ordered-interaction-needs-sequential-learning) for the schedule and scope decision.
## 2. The learning contract
Each sequential transition must provide this logical tuple:
```text theme={null}
(episode_id, step, current_context, relation, chosen_policy,
next_context, step_reward, terminal)
```
The normal online loop uses two calls:
1. `POST /domains/{domain_id}/query` supplies the current context and returns a selected policy plus a `decision_id`.
2. `POST /domains/{domain_id}/feedback` returns the observed transition and reward.
A `decision_id` gives the strongest sealed binding to the earlier query, but policy admission can recover decision-time context through other supported routes. Current runtime context sources are a valid `decision_id` (`decision`), a valid `target_memory_id` (`memory`), or explicit structured `context` (`request`). Sequential readiness is downstream of this policy-admission step.
The default field mapping is:
| Logical value | Default path in feedback |
| --------------- | ------------------------ |
| Episode ID | `metadata.episode_id` |
| Ordered step | `metadata.step` |
| Successor state | `values.next_state` |
| Per-step reward | `values.step_reward` |
| Terminal flag | `values.terminal` |
All five values must be present and correctly typed for a record to be eligible for sequential training:
* `episode_id`: string or other stable identifier;
* `step`: finite number, normally a zero-based integer;
* `next_state`: JSON object;
* `step_reward`: finite number;
* `terminal`: JSON Boolean, not the strings `"true"` or `"false"`.
### Scope rule
Keep the same authenticated owner and `domain_id` when learning should carry across episodes. Use `metadata.episode_id` to separate environment episodes.
On the hosted API, the bearer token determines the effective session; changing the body `session_id` does not create a new tenant or learner. Use a new Domain for an independent task-state history, or a separate credential for a separate identity. The examples use `session_id: ignored` for compatibility.
## 3. Recommended starting declaration
The following declaration is a general starting point for a delayed-reward policy task. Replace the state fields, policies, policy features, and reward limits with values from the real application.
```json theme={null}
{
"domain_id": "sequential-controller-v1",
"session_id": "ignored",
"description": "Select a control policy from state and delayed outcomes.",
"schema": {
"entities": ["controller", "workload"],
"relations": ["controls"],
"signals": ["queue_depth", "error_rate", "capacity", "step_reward"],
"event_types": ["state", "feedback"]
},
"hypotheses": [
{
"name": "conservative controller",
"relation": "controls",
"policy": "conservative",
"policy_features": {
"intensity": 0.25,
"risk_class": "low"
},
"when": [],
"predicts": ["stable service with limited intervention"],
"falsified_by": [],
"weight": 1.0
},
{
"name": "balanced controller",
"relation": "controls",
"policy": "balanced",
"policy_features": {
"intensity": 0.55,
"risk_class": "medium"
},
"when": [],
"predicts": ["balanced recovery and intervention cost"],
"falsified_by": [],
"weight": 1.0
},
{
"name": "aggressive controller",
"relation": "controls",
"policy": "aggressive",
"policy_features": {
"intensity": 0.9,
"risk_class": "high"
},
"when": [],
"predicts": ["rapid recovery with higher intervention cost"],
"falsified_by": [],
"weight": 1.0
}
],
"learning": {
"enabled": true,
"context": {
"feature_paths": [
"values.queue_depth",
"values.error_rate",
"values.capacity",
"values.operating_mode"
],
"event_types": ["state"],
"max_samples": 4096
},
"reward": {
"aggregation": "weighted_mean",
"components": [
{
"field": "values.step_reward",
"goal": "maximize",
"min": 0.0,
"max": 1.0,
"weight": 1.0,
"required": true
}
]
},
"policy": {
"decay_half_life_seconds": 0.0,
"transfer_strength": 0.0,
"action_transfer_strength": 0.1,
"min_context_observations": 4,
"model_max_weight": 0.45,
"model_ood_threshold": 0.75,
"ood_confidence_penalty": 0.5,
"exploration_mode": "auto",
"exploration_strength": 1.0,
"minimum_expected_reward": 0.0,
"minimum_confidence": 0.0,
"require_credible_dominance": false,
"dominance_margin": 0.0,
"abstain_on_ood": false
},
"credit_assignment": {
"mode": "eligibility_trace",
"discount": 0.95,
"neutral_reward": 0.5,
"minimum_weight": 0.01,
"delay_path": "values.delay_steps"
},
"sequential": {
"enabled": true,
"episode_path": "metadata.episode_id",
"step_path": "metadata.step",
"next_context_path": "values.next_state",
"reward_path": "values.step_reward",
"terminal_path": "values.terminal",
"discount": 0.95,
"n_step": 6,
"minimum_episodes": 10,
"minimum_validation_skill": 0.1,
"target_sync_interval": 10,
"conservative_weight": 0.05
},
"latent_belief": {
"enabled": true,
"max_regimes": 8,
"novelty_threshold": 0.55,
"temperature": 0.25,
"change_point_threshold": 0.55,
"change_point_patience": 3,
"historical_regime_weight": 0.05
},
"training": {
"enabled": true,
"min_samples": 64,
"retrain_interval": 16,
"dimensions": 96,
"epochs": 100,
"learning_rate": 0.04
}
}
}
```
Create it with:
```bash theme={null}
export ADAPT1_BASE_URL="https://rei-neuroadapt-api.reilabs.org/api/v1"
export ADAPT1_API_KEY="YOUR_API_KEY"
curl -sS -X POST "$ADAPT1_BASE_URL/domains" \
-H "Authorization: Bearer $ADAPT1_API_KEY" \
-H "Content-Type: application/json" \
--data-binary @domain.json
```
## Configure the learner
Each selectable hypothesis needs:
* a `name` for humans and traces;
* the same `relation` for policies competing in one decision;
* a unique, non-null `policy` identifier;
* optional `policy_features` describing the action itself;
* optional `when`, `predicts`, and `falsified_by` clauses for interpretable conditions.
`policy` is the action identity. `policy_features` are observable action attributes that let the learner generalize between related actions. They must describe the candidate action, not its result.
Good policy features:
```json theme={null}
{
"dose": 0.4,
"duration_seconds": 30,
"mode": "pulsed"
}
```
Poor policy features:
```json theme={null}
{
"will_succeed": true,
"future_reward": 0.9
}
```
Do not make the learned outcome or target action itself an autonomous structure target. A policy-less induced hypothesis is predictive evidence, not an executable action. Action selection must resolve to a declared policy.
`learning.context.feature_paths` determines which state fields are visible to contextual and sequential model training and inference.
Include fields that are:
* available before the decision;
* causally or predictively relevant to action value;
* represented consistently at query and feedback time;
* stable in name, unit, and type.
Exclude fields that are:
* generated after the action;
* direct encodings of the answer or reward;
* identifiers with no transferable meaning;
* timestamps when elapsed time or phase would be more useful;
* high-cardinality provenance that encourages memorization.
An empty `feature_paths` list uses all structured leaf fields. This is convenient for exploration but risky in production because IDs, logging fields, and accidental outcome leakage may enter the model. Explicit paths are recommended.
`context.event_types` filters stored events when a query asks Adapt-1 to infer the latest context. It does not replace explicit query context. Supplying `context` directly is the clearest and least ambiguous path.
`context.max_samples` bounds retained policy examples. For sequential tasks, size it to retain several representative episodes:
```text theme={null}
max_samples >= retained_episodes x representative_steps_per_episode
```
For 200-step episodes and 10 useful episodes, start at `2048` or `4096`, not the default `512`. Replay eviction balances episode groups and preserves failure evidence, but it cannot recover transitions that were never retained.
Policy learning only updates from measured reward. A successful HTTP response does not imply that the feedback changed the learner.
**Declarative reward components**
Each component supports:
| Field | Meaning |
| ------------ | ---------------------------------------------------------------- |
| `field` | Dot path inside the feedback envelope, such as `values.profit` |
| `goal` | `maximize`, `minimize`, or `target` |
| `min`, `max` | Expected numeric range used to normalize utility to `[0,1]` |
| `target` | Desired value for a `target` component |
| `tolerance` | Distance from `target` at which utility reaches zero |
| `weight` | Relative contribution to weighted or geometric aggregation |
| `hard` | Force total reward to zero when this component's utility is zero |
| `required` | Treat a missing or non-numeric value as a measured hard failure |
Aggregation modes:
| Mode | Use it when |
| ---------------- | ------------------------------------------------------- |
| `weighted_mean` | Objectives can compensate for one another |
| `geometric_mean` | A very weak component should strongly reduce the total |
| `minimum` | The weakest resolved objective should define the reward |
Example with performance and safety:
```json theme={null}
{
"aggregation": "geometric_mean",
"components": [
{
"field": "values.throughput",
"goal": "maximize",
"min": 0,
"max": 100,
"weight": 2
},
{
"field": "values.error_rate",
"goal": "minimize",
"min": 0,
"max": 0.1,
"weight": 1,
"hard": true,
"required": true
}
]
}
```
Without components, Adapt-1 recognizes numeric `values.reward`, `values.score`, or `values.utility`; Boolean `correct`, `success`, or `accepted`; and recognized binary outcome labels. Fields such as `error_distance` have no implicit reward meaning. Either declare them as components or compute a normalized reward externally.
**Reward scale and neutral reward**
Keep rewards in `[0,1]`. The default neutral point is `0.5`:
* below `0.5` is disadvantageous;
* `0.5` is neutral or unresolved;
* above `0.5` is advantageous.
For a naturally signed reward, define normalization explicitly. For example, map profit from `[-100,100]` with `min: -100`, `max: 100`, and `goal: maximize`.
Do not invent a success or failure label for an unknown outcome. Use an outcome value accepted by the current feedback schema and provide the numeric field required by the reward declaration. Do not rely on undocumented outcome labels; the live validator is authoritative for accepted enum values.
**Sequential parameters**
| Parameter | Default | Effect | Tuning guidance |
| -------------------------- | --------------------: | --------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- |
| `enabled` | `false` | Enables episode-aware return learning | Must be `true` for sequential Q learning |
| `episode_path` | `metadata.episode_id` | Groups transitions into episodes | Keep stable across one episode and distinct across episodes |
| `step_path` | `metadata.step` | Orders transitions within an episode | Use monotonic, unique step values |
| `next_context_path` | `values.next_state` | Supplies the observed successor state | Must resolve to an object on every eligible feedback row |
| `reward_path` | `values.step_reward` | Supplies the sequential reward | Must resolve to a finite number |
| `terminal_path` | `values.terminal` | Marks episode completion | Send a real Boolean on every row |
| `discount` | `0.95` | Controls how much future value affects earlier actions | Lower for short horizons; higher for long delayed effects |
| `n_step` | `3` | Number of observed rewards used before bootstrapping | Increase for delayed rewards; reduce for dense/noisy rewards |
| `minimum_episodes` | `8` | Minimum distinct episodes before sequential fitting | The runtime accepts lower positive values, but fewer episodes weaken episode-held-out validation; increase when episodes vary substantially |
| `minimum_validation_skill` | `0.25` | Required held-out improvement over a constant-return baseline | Lower cautiously for noisy data; never use it to force a failing model into selection |
| `target_sync_interval` | `10` | Epoch interval for copying online weights to the target network | Increase for a slower, more stable target; reduce for faster tracking |
| `conservative_weight` | `0.05` | Penalizes unsupported action values | Increase when action coverage is sparse or overestimation is costly |
The learner uses episode-held-out validation. The last portion of ordered episode IDs is reserved for validation, with at least two validation episodes. A model is installed only when its validation skill meets the declared threshold. This is why one long episode is not a substitute for multiple episodes.
**Picking `discount`**
Use the effective horizon as a guide:
```text theme={null}
effective horizon is approximately 1 / (1 - discount)
```
Approximate examples:
| `discount` | Effective horizon |
| ---------: | ----------------: |
| `0.80` | 5 steps |
| `0.90` | 10 steps |
| `0.95` | 20 steps |
| `0.98` | 50 steps |
| `0.99` | 100 steps |
Very high discount values can propagate noise and make policies hard to distinguish. Start from the shortest horizon that can still reach the delayed consequence of interest.
**Picking `n_step`**
* Dense reward every step: start at `1` to `3`.
* Reward after a short action chain: start at `4` to `8`.
* Sparse terminal reward: start at `8` to `20`, with eligibility traces enabled.
* Highly stochastic outcomes: use a smaller value and more episodes.
`n_step` does not need to equal the episode length. It controls the direct return window before the model bootstraps from the successor state.
`credit_assignment.mode` controls synchronous reward attribution in the online policy state. Sequential Q learning separately consumes the episode transitions and returns.
| Parameter | Default | Effect |
| ---------------- | -------------------: | ----------------------------------------------------------------------------- |
| `mode` | `none` | Use `eligibility_trace` to propagate delayed non-neutral outcomes |
| `discount` | `0.9` | Exponential decay applied per delay step |
| `neutral_reward` | `0.5` | Baseline around which positive and negative credit moves |
| `minimum_weight` | `0.0` | Stops backward propagation after the discounted weight falls below this value |
| `delay_path` | `values.delay_steps` | Optional explicit delay used to adjust the current feedback reward |
With eligibility traces enabled, a later non-neutral `step_reward` can update earlier neutral transitions in the same episode. Propagation stops at a prior non-neutral signal, a terminal boundary, an unchanged transition, or the minimum weight cutoff.
Use `mode: none` when rewards are immediate and accurately assigned. Use `eligibility_trace` when the effect arrives later than the responsible action.
Do not use a large `minimum_weight` for long sparse-reward episodes. With `discount: 0.95`, a cutoff of `0.05` reaches roughly 58 steps; a cutoff of `0.01` reaches roughly 90 steps.
**Exploration modes**
| Mode | Behavior | Best use |
| ---------- | ------------------------------------------------------------------------------------ | -------------------------------------------------- |
| `exploit` | Select the highest current expected return | Frozen evaluation and stable deployment |
| `ucb` | Adds uncertainty-scaled optimism | Broad, controlled exploration |
| `thompson` | Samples from each policy posterior | Stochastic exploration proportional to uncertainty |
| `auto` | Chooses among exploit, UCB, and Thompson from confidence and bound-transition status | General online operation |
The Domain configuration sets the default, but exploration only occurs when the query sends `allow_exploration: true`. A query can temporarily override the mode with `selection_mode`.
`exploration_strength` scales UCB uncertainty. `0` removes the UCB bonus. Values near `0.5` are cautious, `1.0` is a reasonable start, and values above `1.0` deliberately favor underexplored actions.
**Selection gates**
| Parameter | Default | Effect |
| ---------------------------- | ------: | -------------------------------------------------------------- |
| `minimum_expected_reward` | `0.0` | Abstain if the best candidate is below this expected reward |
| `minimum_confidence` | `0.0` | Abstain if contextual confidence is too low |
| `require_credible_dominance` | `false` | Require the winner's credible interval to dominate competitors |
| `dominance_margin` | `0.0` | Required separation when credible dominance is enabled |
| `abstain_on_ood` | `false` | Abstain when the context is outside learned support |
These gates are deployment controls, not training accelerators. Keep them loose while collecting initial coverage, then tighten them using held-out operational data. The client must always handle `status: "abstained"`; abstention is a valid safety output.
**Policy transfer**
| Parameter | Default | Meaning |
| -------------------------- | ------: | ------------------------------------------------------------------------- |
| `transfer_strength` | `0.15` | Weight of same-policy evidence learned under another relation |
| `action_transfer_strength` | `0.05` | Weight of evidence from different policies with similar `policy_features` |
| `min_context_observations` | `4` | Local support target used by confidence and calibration machinery |
| `model_max_weight` | `0.45` | Maximum contribution of the asynchronously trained model |
| `model_ood_threshold` | `0.75` | Novelty distance at which model contribution falls to zero |
| `ood_confidence_penalty` | `0.5` | Confidence reduction under out-of-distribution context |
Set both transfer strengths to `0` when actions or relations have unrelated semantics. Increase `action_transfer_strength` only when policy features have stable physical or operational meaning. Transfer never crosses owner, Domain, or session scope.
**Decay**
`decay_half_life_seconds` exponentially reduces the influence of older feedback. `0` disables time decay.
Use decay for regime changes, market drift, changing users, or equipment aging. Pick a half-life in application time, not request count. It should be longer than ordinary noise but shorter than the expected persistence of obsolete behavior.
Examples:
* stable physical process: `0` or several months;
* weekly operational drift: several days to weeks;
* intraday regime changes: tens of minutes to hours;
* controlled reversal test: long enough to retain evidence, short enough for new evidence to dominate.
Latent belief separates recurring or changing context regimes without requiring the client to provide a regime label.
| Parameter | Default | Effect |
| -------------------------- | ------: | ------------------------------------------------------ |
| `enabled` | `true` | Enables latent regime inference |
| `max_regimes` | `8` | Maximum regimes retained per relation |
| `novelty_threshold` | `0.55` | Context distance needed to create a new regime |
| `temperature` | `0.25` | Softness of regime probability assignment |
| `change_point_threshold` | `0.55` | Residual threshold for suspected behavior change |
| `change_point_patience` | `3` | Consecutive high residuals required for a change point |
| `historical_regime_weight` | `0.05` | Residual influence of evidence from non-active regimes |
Tuning effects:
* Lower `novelty_threshold`: creates regimes more readily; useful for distinct modes, risky with noisy features.
* Higher `novelty_threshold`: merges more contexts into one regime.
* Lower `temperature`: sharper regime assignments.
* Higher `temperature`: blends evidence across nearby regimes.
* Lower `change_point_patience`: reacts faster and risks false changes.
* Higher `historical_regime_weight`: improves recurrence reuse but increases interference from old regimes.
Start with the defaults. Tune latent belief only after verifying that feature scaling, reward, and episode linkage are correct.
Feedback updates the online posterior synchronously. Neural model fitting is asynchronous and feedback-triggered. The API remains available while training runs.
| Parameter | Default | Effect | Guidance |
| ------------------ | ------: | -------------------------------------------------------- | -------------------------------------------------------------------------------------- |
| `enabled` | `true` | Enables asynchronous model fitting | Disable only when posterior-only behavior is required |
| `min_samples` | `24` | First sample count at which training may start | Use `64` to `256` for noisier sequential tasks |
| `retrain_interval` | `8` | New policy samples required after the last completed fit | Increase to reduce training churn; decrease for fast adaptation |
| `dimensions` | `96` | Capacity control for the contextual model | Sequential Q input capacity is derived from the declared feature schema |
| `epochs` | `100` | Optimization epochs per fit | Increase only if validation is underfit, not merely because training loss is high |
| `learning_rate` | `0.04` | Optimizer step size | Reduce if validation is unstable; increase cautiously if learning is consistently slow |
`retrain_interval: 20` is not inherently too early for a 2,000-sample task. It means at most one new fit for every 20 accepted feedback samples after the minimum. Choose the interval from the acceptable compute load and adaptation delay:
* rapid drift, cheap fitting: `8` to `20`;
* stable task, moderate load: `32` to `100`;
* expensive fitting or very large streams: `100` or more.
Training readiness also depends on `minimum_episodes`. A fit triggered before enough episodes exist may train the ordinary contextual candidate; once enough valid episodes accumulate, the sequential candidate can be trained and validated.
## 13. Execute the online loop
### Step 1: query before acting
```bash theme={null}
curl -sS -X POST \
"$ADAPT1_BASE_URL/domains/sequential-controller-v1/query" \
-H "Authorization: Bearer $ADAPT1_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"session_id": "ignored",
"question": "Select the controller for the current state.",
"relation": "controls",
"context": {
"event_type": "state",
"values": {
"queue_depth": 71,
"error_rate": 0.04,
"capacity": 0.62,
"operating_mode": "degraded"
}
},
"allow_exploration": true,
"selection_mode": "auto"
}'
```
Persist these response fields:
* `decision_id`;
* `selection.status`;
* `selection.selected_policy`;
* the exact action actually executed;
* the pre-action context.
If the application overrides Adapt-1's selection, send feedback for the policy that was actually executed. Do not credit an action that was only proposed.
### Step 2: observe the transition
After executing the selected policy, collect:
* the successor state;
* the reward components;
* whether the episode ended;
* the episode ID and step;
* any measured delay.
### Step 3: write decision-linked feedback
```bash theme={null}
curl -sS -X POST \
"$ADAPT1_BASE_URL/domains/sequential-controller-v1/feedback" \
-H "Authorization: Bearer $ADAPT1_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"session_id": "ignored",
"outcome": "success",
"feedback_kind": "execution",
"decision_id": "DECISION_ID_FROM_QUERY",
"relation": "controls",
"policy": "SELECTED_POLICY_FROM_QUERY",
"values": {
"step_reward": 0.65,
"next_state": {
"queue_depth": 54,
"error_rate": 0.025,
"capacity": 0.61,
"operating_mode": "recovering"
},
"terminal": false,
"delay_steps": 1
},
"metadata": {
"episode_id": "episode-0001",
"step": 0
}
}'
```
Repeat query, action, and feedback until the terminal transition. The terminal row still needs a valid `next_state` object and must send `"terminal": true`.
`decision_id` is the preferred binding for a query → action → outcome loop because it uses the sealed pre-action context. It is not the only context route. Current runtime policy admission can also recover context from `target_memory_id` or from explicit structured `context`. In every case, send the executed `relation` and `policy` explicitly and verify that `credit_assignment.contextual_learning_applied` is true.
Current admission checks require the `relation` field to be present and do not validate its value against the bound decision. Send the real relation from the Domain. A successful feedback response without `relation` can store the record while leaving `feedback_policy.sample_count` unchanged.
## 14. Batch feedback without losing sequence semantics
`POST /domains/{domain_id}/batch` accepts ordered event and feedback operations. Every feedback operation retains its own `decision_id`, `metadata.episode_id`, `metadata.step`, successor state, reward, and terminal flag.
Batching reduces HTTP overhead; it does not remove the need for per-transition identity. It is safe when decisions have already been made and their results are being uploaded together. It cannot replace the online query loop when the next action depends on the previous query result.
Minimal shape:
```json theme={null}
{
"session_id": "ignored",
"operations": [
{
"operation": "feedback",
"ref": "feedback-episode-1-step-0",
"outcome": "success",
"feedback_kind": "execution",
"decision_id": "decision-step-0",
"relation": "controls",
"policy": "balanced",
"values": {
"step_reward": 0.5,
"next_state": {"queue_depth": 62, "error_rate": 0.03, "capacity": 0.62, "operating_mode": "degraded"},
"terminal": false,
"delay_steps": 0
},
"metadata": {"episode_id": "episode-0001", "step": 0}
},
{
"operation": "feedback",
"ref": "feedback-episode-1-step-1",
"outcome": "success",
"feedback_kind": "execution",
"decision_id": "decision-step-1",
"relation": "controls",
"policy": "balanced",
"values": {
"step_reward": 0.9,
"next_state": {"queue_depth": 30, "error_rate": 0.01, "capacity": 0.65, "operating_mode": "normal"},
"terminal": true,
"delay_steps": 0
},
"metadata": {"episode_id": "episode-0001", "step": 1}
}
]
}
```
## 15. Read the diagnostics
Request a normal Domain query and inspect:
| Output | What it tells you |
| ------------------------------------------------------------------------- | ------------------------------------------------------------------------- |
| `selection.status` | `selected` or `abstained` |
| `selection.reason` | Why no policy was selected or which gate applied |
| `selection.selected_policy` | Executable action identity |
| `ranked_hypotheses[].contextual_policy.expected_reward` | Fused expected immediate/contextual reward |
| `ranked_hypotheses[].contextual_policy.selection_expected_reward` | Value actually preserved for policy ordering, including sequential return |
| `ranked_hypotheses[].contextual_policy.sequential_expected_reward` | Sequential model estimate when installed |
| `ranked_hypotheses[].contextual_policy.model_weight` | Effective trained-model contribution |
| `ranked_hypotheses[].contextual_policy.out_of_distribution` | Whether context novelty exceeded support |
| `ranked_hypotheses[].contextual_policy.effective_observations` | Weighted evidence supporting the estimate |
| `latent_belief` | Current regime probabilities and novelty |
| `learning_state.subsystems.feedback_policy.sample_count` | Accepted policy-learning observations |
| `learning_state.subsystems.feedback_policy.model.status` | `waiting`, `running`, `installed`, `failed`, or unavailable state |
| `learning_state.subsystems.feedback_policy.model.report.model_type` | Look for `sequential_q_mlp` after successful sequential fitting |
| `learning_state.subsystems.feedback_policy.model.report.validation_skill` | Held-out improvement over the constant-return baseline |
The top-level `learning_state.sample_count` is an aggregate across active learning subsystems. Use the feedback-policy subsystem count when diagnosing sequential policy learning.
Do not wait for a separate `sequential` subsystem to appear. Sequential return learning is represented by the model lifecycle inside `learning_state.subsystems.feedback_policy`. Before `feedback_policy.sample_count` increases, the episode, sample, training, and validation thresholds for `sequential_q_mlp` are not yet relevant. Samples admitted by the separate `structured_transition` subsystem do not satisfy the policy learner's sample or episode gates.
Evidence that sequential learning is active includes all of the following:
1. feedback-policy `sample_count` increases after accepted feedback;
2. multiple valid episode IDs are retained;
3. model status advances from `waiting` to `running` and then `installed`;
4. the installed model type is `sequential_q_mlp`;
5. validation skill meets the configured threshold;
6. sequential expected rewards differ across policies or contexts;
7. frozen exploit decisions improve on held-out episodes without feedback writes.
## 16. Freeze evaluation correctly
To measure learned behavior rather than continued adaptation:
1. train only on the training partition;
2. wait for asynchronous training to complete;
3. stop all `/feedback` writes;
4. query held-out episodes with `allow_exploration: false`;
5. use `selection_mode: "exploit"`;
6. keep the same authenticated owner and `domain_id` so the trained state remains available;
7. use unseen episode IDs, and preferably unseen scenarios or entities;
8. record abstentions as abstentions rather than silently replacing them in the Adapt-1 score.
Frozen query example:
```json theme={null}
{
"session_id": "ignored",
"question": "Select the controller for this held-out state.",
"relation": "controls",
"context": {
"event_type": "state",
"values": {
"queue_depth": 83,
"error_rate": 0.052,
"capacity": 0.58,
"operating_mode": "degraded"
}
},
"allow_exploration": false,
"selection_mode": "exploit"
}
```
Report at least return, policy accuracy if ground truth exists, abstention coverage, selective performance, episode variance, and learning curves by episode. Use multiple seeds or independent streams for stochastic environments.
## Advanced configuration
These are starting profiles, not guaranteed optima.
**Sparse delayed reward**
```json theme={null}
{
"context": {"max_samples": 4096},
"credit_assignment": {
"mode": "eligibility_trace",
"discount": 0.97,
"neutral_reward": 0.5,
"minimum_weight": 0.005
},
"sequential": {
"discount": 0.98,
"n_step": 12,
"minimum_episodes": 12,
"conservative_weight": 0.08
},
"training": {"min_samples": 128, "retrain_interval": 32}
}
```
Use when most steps are neutral and terminal outcomes carry the useful signal.
**Dense short-horizon control**
```json theme={null}
{
"credit_assignment": {"mode": "none"},
"sequential": {
"discount": 0.9,
"n_step": 3,
"minimum_episodes": 8,
"conservative_weight": 0.03
},
"training": {"min_samples": 64, "retrain_interval": 16}
}
```
**Nonstationary or reversal-prone task**
```json theme={null}
{
"policy": {
"decay_half_life_seconds": 86400,
"transfer_strength": 0.0,
"action_transfer_strength": 0.05
},
"latent_belief": {
"enabled": true,
"max_regimes": 12,
"novelty_threshold": 0.45,
"change_point_threshold": 0.4,
"change_point_patience": 2,
"historical_regime_weight": 0.1
},
"training": {"retrain_interval": 8}
}
```
Set the decay half-life in real application time. `86400` is one day and is only an example.
**High-stakes conservative deployment**
```json theme={null}
{
"policy": {
"exploration_mode": "exploit",
"minimum_expected_reward": 0.6,
"minimum_confidence": 0.4,
"require_credible_dominance": true,
"dominance_margin": 0.05,
"abstain_on_ood": true
},
"sequential": {
"minimum_validation_skill": 0.25,
"conservative_weight": 0.12
}
}
```
Collect exploration data in a safe environment before enabling these deployment gates.
**Many structurally similar actions**
Use informative `policy_features`, set `action_transfer_strength` around `0.1` to `0.25`, and increase `conservative_weight` when many actions remain unsupported. Verify with an ablation against `action_transfer_strength: 0` to confirm transfer helps held-out actions rather than blending distinct ones.
`learning.sequential.bound_transition` is an optional non-neural projection for tasks with explicitly bound state objects and action roles. It can estimate how a declared transition action changes a bounded objective and blend that estimate into policy scoring. Most sequential tasks should leave it disabled.
It is appropriate when:
* one context contains multiple indexed entities or objects;
* a policy selects which entity is the current state and optionally which is the action source;
* action roles can be declared from policy features;
* the objective has known numeric bounds;
* state transformations can transfer between structurally similar bindings.
| Parameter | Meaning |
| -------------------------------- | ----------------------------------------------------------------------------------- |
| `enabled` | Enables bound-transition projection |
| `state_selector_feature` | Policy-feature path whose value selects the current state object |
| `state_features` | Map from logical feature name to context path template; `{selector}` is substituted |
| `state_bounds` | Optional `[min,max]` validation bounds by logical state feature |
| `action_selector_feature` | Optional policy-feature path selecting an action/source object |
| `action_features` | Map from logical action feature to templated context path |
| `signature_features` | Policy-feature paths that must match for direct transition evidence |
| `objective_feature` | Logical state feature to maximize after normalization |
| `episode_start_path` | Optional context Boolean that resets episode-local binding |
| `objective_min`, `objective_max` | Numeric normalization limits |
| `transition_values` | Policy-feature path to allowed values identifying transition actions |
| `terminal_values` | Policy-feature path to allowed values identifying terminal actions |
| `minimum_support` | Required matching transitions before prediction |
| `model_weight` | Blend weight of the bound projection in policy scoring |
| `novelty_bonus` | Exploration value assigned to unsupported transitions |
| `transition_cost` | Utility subtracted from predicted transition actions |
This surface is for explicit, reusable structural bindings. Do not enable it merely to increase a benchmark score. If the application cannot state what the selectors, state features, action roles, and objective mean independently of its evaluation labels, use the ordinary sequential learner.
## 19. Common failure modes
### `sample_count` stays at zero
First distinguish storage from learner admission. A successful feedback response can store the record while leaving `feedback_policy.sample_count` unchanged.
Check:
* `learning.enabled` is `true`;
* feedback includes a real measured reward;
* `relation` is present in the feedback request;
* the executed `policy` is present and is one of the declared hypotheses;
* decision-time context is recoverable from a valid `decision_id`, a valid `target_memory_id`, or explicit structured `context`.
Then inspect `credit_assignment.contextual_learning_applied`. If it is `false`, the record did not enter contextual policy learning even if the HTTP request succeeded. Current responses report the context route under `credit_assignment.context_source` as `decision`, `memory`, or `request`.
An unrecognized outcome label does not update policy learning unless a numeric, Boolean, or declarative reward is also present.
### Feedback count rises but no sequential model appears
Check every row for all five configured sequential paths and types. Then check:
* at least `minimum_episodes` distinct episode IDs exist;
* replay capacity retains enough rows from those episodes;
* training has reached `min_samples`;
* Torch or the configured training service is available;
* asynchronous fitting has completed;
* validation skill meets `minimum_validation_skill`.
### Every decision abstains at the start
Use `allow_exploration: true` during data collection. With no policy evidence and exploration disabled, abstention is expected. Keep strict confidence, dominance, and OOD gates disabled until there is enough coverage.
### Decisions return `selected_policy: null`
Only declared hypotheses with non-null policies are executable. Predictive or induced hypotheses without a policy cannot be selected as actions. Verify that all competing policy hypotheses share the requested relation and have unique policies.
### Learning becomes worse across episodes
Check for:
* reward direction or normalization errors;
* incorrect decision-to-feedback linkage;
* outcome leakage in context features;
* reused episode IDs or non-monotonic steps;
* too-small replay capacity;
* excessive transfer between unrelated actions;
* too-high discount or `n_step` for noisy rewards;
* validation episodes drawn from a different regime due to lexicographic episode naming;
* exploration still enabled during evaluation;
* client fallbacks being scored as if Adapt-1 selected them.
### The model trains but policy ordering stays flat
Inspect `validation_skill`, `sequential_expected_reward`, model weight, and OOD status. Common causes are constant rewards, action-independent features, identical policy features, low action coverage, a feature path mismatch between current and successor state, or OOD downweighting.
Use this sequence:
1. **Define the decision.** Write one sentence: "Given state X, choose one policy from Y to maximize measured outcome Z over horizon H."
2. **Declare policies.** Give every executable action a unique policy and observable policy features.
3. **Declare pre-action state.** Select only fields known before acting.
4. **Define reward mathematically.** Normalize every objective to `[0,1]`; specify tradeoffs and hard constraints.
5. **Define episode boundaries.** Establish stable episode IDs, ordered steps, successor state, and terminal semantics.
6. **Estimate the horizon.** Pick the shortest discount and `n_step` that reach the relevant consequences.
7. **Size replay.** Retain enough transitions from enough complete and diverse episodes.
8. **Choose exploration.** Explore in a safe environment; exploit or abstain in high-risk deployment.
9. **Set validation gates.** Require positive held-out skill and keep conservative penalties when action coverage is incomplete.
10. **Instrument the loop.** Persist decisions and verify that every executed action receives correctly linked feedback.
11. **Run ablations.** Compare sequential enabled/disabled, transfer enabled/disabled, and latent belief enabled/disabled.
12. **Freeze and evaluate.** Use unseen episodes with no feedback writes and no exploration.
* [ ] The authenticated owner and Domain stay stable across training episodes.
* [ ] Every policy is unique, executable, and attached to the queried relation.
* [ ] Policy features describe actions, not outcomes.
* [ ] Context features contain no post-action or answer information.
* [ ] Reward is measurable, normalized, and directionally correct.
* [ ] Every feedback row has episode, step, next state, step reward, and Boolean terminal.
* [ ] Every executed action has one valid decision-time context binding: sealed `decision_id`, valid `target_memory_id`, or explicit structured `context`; use `decision_id` for sealed query attribution when available.
* [ ] Replay capacity covers several representative episodes.
* [ ] Exploration is enabled only where exploration is acceptable.
* [ ] The client handles abstention explicitly.
* [ ] A `sequential_q_mlp` is installed only after positive held-out validation skill.
* [ ] OOD and confidence behavior is tested before high-stakes deployment.
* [ ] Evaluation is frozen, held out, and free of feedback writes.
* [ ] Results include variance, coverage, abstention, and learning curves rather than only the best episode.
The strongest sequential learner is produced by a clean data contract and a defensible evaluation protocol. Parameter tuning cannot compensate for missing successor states, ambiguous reward, cross-session fragmentation, or incorrect action-feedback linkage.
Start with the Discovery behavior, public boundary, and minimal contract.
Review Transition, Structure, and Sequential Discovery together.